Sync systems are failing to process inbound data
Incident Report for Solutionreach
Resolved
At this time, our sync systems are now operating within normal expectations. All syncs are being processed automatically within tolerances of when they are received. As this is a critical system of our service, we will continue to monitor performance through Friday for any further anomalies and address as needed.
Posted Oct 20, 2022 - 10:44 MDT
Update
Our Engineering team has been monitoring the sync system throughout the morning as a result of the issues experienced yesterday and have identified that we are again falling behind in processing syncs in near-real-time. We are continuing to work toward a long-term solution to address the issues at hand.
Posted Oct 19, 2022 - 12:28 MDT
Monitoring
Our systems have been able to process the backlog of sync files that were queued up during the outage and we are now processing new files in a near-real-time manner. We will continue to monitor the system for the next 24 hours to for any additional anomalies that may arise as a result of this incident.
Posted Oct 18, 2022 - 18:30 MDT
Identified
Our Engineering has implemented some mitigations to help reduce the sync backlog at this time. Our systems have begun to process backlogged syncs that were submitted during the outage however we still have a significant backlog to process through before our systems are caught up.
Posted Oct 18, 2022 - 17:21 MDT
Update
Our engineering team is still investigating the issue at hand. As more information is available, we will provide updates as appropriate.
Posted Oct 18, 2022 - 14:05 MDT
Investigating
Our engineers are investigating an issue where our systems are failing to process new sync files. Currently we do not have a cause of the issue and are investigating the situation. All customers are potentially to be effected based on their individual sync schedules. Bear in mind that this only effects new data (such as patients and appointments) that may have changed or been added since the last time that your system had a successful sync complete. Messages, reminders, and other patient communications are unaffected at this time.
Posted Oct 18, 2022 - 11:27 MDT
This incident affected: Sync - General stability of SR sync software and connection to client PM/EMR/EHR.