Critical Service Disruption Alert
Incident Report for WorkForce Software US8: Phoenix AZ (OCI)
Postmortem

Following the recent service disruption on 17-June-2024, we have conducted a thorough investigation to understand the root cause. As part of our commitment to transparency and continuous improvement, we have implemented measures to prevent future occurrences.

We appreciate your understanding and support during this time. For a detailed Incident Report, please visit the following link: https://workforcesoftware.my.site.com/customers/s/article/WORKFORCE-INCIDENT-REPORT-RCA-1474

Posted Jun 26, 2024 - 10:14 UTC
Resolved
Services have been operational since 4:25 EST, and emergency patching is also now completed. We thank you for your patience during this time and sincerely regret any disruptions to your operations. A detailed analysis and steps we're taking to prevent future incidents will be shared in our upcoming post-mortem report.
Posted Jun 17, 2024 - 23:43 UTC
Update
Emergency patching continues to be rolled out and we are currently 50% completed. Another update will be provided in 30 minutes or sooner.
Posted Jun 17, 2024 - 23:10 UTC
Update
Emergency patching is proceeding as rolling to avoid a disruption of service during the process. This is now estimated to take 1-2 hours for patching to fully complete. The next update will be provided in 30 minutes or sooner.
Posted Jun 17, 2024 - 22:43 UTC
Update
Emergency patching is in progress and is estimated to take approximately 30 minutes to complete. The next update will be provided in 30 minutes or sooner.
Posted Jun 17, 2024 - 22:07 UTC
Update
Emergency patching is now expected to begin by 6:00 EST and is estimated to take 30 minutes to complete. Systems have been stable since 4:25 EST. The next update will be provided in 30 minutes or sooner.
Posted Jun 17, 2024 - 21:41 UTC
Update
Preparation continues for emergency patching. ETA to begin patch application is 5:30 EST and is estimated to complete at 6:00 EST. The next update will be provided in 30 minutes or sooner.
Posted Jun 17, 2024 - 21:08 UTC
Update
We are still working through preparation for the emergency patch. User experience will continue to vary at this time. The next update will be provided in 30 minutes or sooner.
Posted Jun 17, 2024 - 20:42 UTC
Update
Our vendor has identified a bug that is potentially impacting performance. Additional resources from our vendor are actively engaged and working with our teams to apply a patch within the next hour. The next update will be provided in 30 minutes or sooner.
Posted Jun 17, 2024 - 20:12 UTC
Update
While service has been partially restored, teams continue to remediate intermittent performance issues. User experience will vary at this time. Estimated time for full resolution remains unknown. The next update will be provided in 30 minutes or sooner.
Posted Jun 17, 2024 - 19:45 UTC
Update
Teams continue to work with our vendor to restore service. Estimated time for resolution remains unknown. The next update will be provided in 30 minutes or sooner.
Posted Jun 17, 2024 - 19:09 UTC
Update
There has been a recurrence of the performance degradation starting at 2:15 EST. Teams continue to work with our vendor to review logs and restore service. The next update will be provided in 30 minutes or sooner.
Posted Jun 17, 2024 - 18:40 UTC
Update
While progress has been made, teams continue to restore connections to the database. The next update will be provided in 30 minutes or sooner.
Posted Jun 17, 2024 - 18:11 UTC
Update
Systems continue to come online, and the team is restarting app servers as needed to validate proper connections to the database. The next update will be provided in 30 minutes or sooner.
Posted Jun 17, 2024 - 17:39 UTC
Update
The database restart is complete. The team is currently working to re-establish connections. Another update will be provided in 30 minutes or sooner.
Posted Jun 17, 2024 - 17:15 UTC
Update
The issue causing the restart to take longer than expected has been addressed, and the restart is now expected to complete within 15 minutes. The next update will be shared in 30 minutes or sooner.
Posted Jun 17, 2024 - 16:40 UTC
Update
The restart is still in progress and taking longer than expected. The next update will be shared in 30 minutes or sooner.
Posted Jun 17, 2024 - 16:10 UTC
Update
As part of our corrective actions with our vendor a database memory flush has been completed. A rolling restart is now in progress, estimated completion time is 40 minutes. The next update will be shared in 30 minutes or sooner.
Posted Jun 17, 2024 - 15:41 UTC
Update
Current Status:
Our incident response team has been actively engaged in diagnosing the root cause and implementing corrective actions. Here are the key points of our current efforts:

Diagnosis:
We are attempting to take the enhanced log with the suggestion of Oracle.

Actions Taken:
We are getting ready to restart the Database Node to allow the system to become available. This is estimated to complete within 30 minutes.

Ongoing Efforts:
Review the enhanced log to identify a long-term solution

Next update will be provided in 15 minutes.
Posted Jun 17, 2024 - 15:25 UTC
Update
The team continues to work with our vendor on remediation. We appreciate your continued patience and will provide another update in 15 minutes or sooner.
Posted Jun 17, 2024 - 15:09 UTC
Update
The team continues to collect diagnostic information while this is escalated with our vendor Oracle. Estimated time to resolution remains unknown. We appreciate your continued patience and will provide another update in 15 minutes.
Posted Jun 17, 2024 - 14:52 UTC
Update
With Oracle’s assistance the team is collecting diagnostic information. Estimated time to resolution is currently unknown. We appreciate your continued patience and will provide another update in 15 minutes.
Posted Jun 17, 2024 - 14:35 UTC
Identified
We're currently experiencing a service disruption affecting WorkForce Time & Attendance that started approximately at 9:15 EST. Our team is actively investigating the cause and working diligently to restore services as swiftly as possible. We recognize how vital our services are to your operations and are grateful for your patience during this time. To keep you informed and updated, we will be providing updates in Status Hub every hour until the service is fully restored.
Posted Jun 17, 2024 - 13:38 UTC
This incident affected: US8: Phoenix AZ (OCI).
← Current Status Powered by Atlassian Statuspage