If the Bluetooth connection between the leash and your Apple device is lost, the alert will go off. In this case, re-connect the leash to the Apple device.
Another reason is that the location information in the Apple device is not yet updated. This happens when you are just stepping into the Safe Range. Depending on the network service and environment, it takes 30 seconds to 5 minutes to update service location information. When the service location is updated, the alert would stop if you are located within the Safe Range.