10/02/2019 11:30 am EDTOver the past 12-24 hours we SmartSupport received reports of users having issues in various areas across the US and Canada. Initial analysis and review could not determine the underlying cause, as the issues reported (inability to fix / long times to fix or inability to connect / maintain connection) were not represented in a number of key logs used for troubleshooting during normal operations. Additionally, these issues affected only a small and inconsistent subset of the user base which further hampered root cause analysis.
A deeper analysis was initiated and a GLONASS ephemeris issue was found to be causing the issue; starting in Western Canada and propagating across all of SmartNet Application Infrastructure. Corrective actions have been taken to address the issue, address the underlying cause of the issue and to remove remnants of the offending ephemeris from SmartNet servers. Customers should now see fixing and connection performance return to expected levels.
Normal operations have resumed for all of SmartNet North America.
Resolved at 10/02/2019 11:30 pm EDT