I see some value on restricting IPs but I wanted to stress this point because it's special. Using this feature could make you think you are safe from Internet attacks which is not true. Would you worry differently if the attack comes from "Internet" or e.g. disposable AWS/GCP VMs? They are not a different threat.