RCA now available - US8 SFTP Performance Degradation

Incident Report for WorkForce Software US8

Postmortem

Following the recent service disruption on 15-Oct-2024, we have conducted a thorough investigation to understand the root cause. As part of our commitment to transparency and continuous improvement, we have implemented several measures to prevent future occurrences.

We appreciate your understanding and support during this time. For a detailed Report, please visit the following link: https://workforcesoftware.my.site.com/customers/s/article/WORKFORCE-INCIDENT-REPORT-RCA-1511

Posted Oct 24, 2024 - 13:11 UTC

Resolved

This incident has been resolved.

Posted Oct 16, 2024 - 19:07 UTC

Update

Final Update: We are pleased to report that the SFTP Performance Degradation been resolved. Our services are now fully operational. We thank you for your patience during this time and sincerely regret any disruptions to your operations. A detailed analysis and steps we are taking to prevent future incidents will be shared in our upcoming post incident report. Our teams will continue to monitor the environment.
• RCA Number: RCA-1511
• Impact Start Date/Time: 10/15 10:26 ET
• Impact End Date/Time: 10/16 11:45 ET
• Impacted Application(s): SFTP
• Business Impact: Customers could have experienced intermittent connection issues.
• Issue Type: Intermittent

Posted Oct 16, 2024 - 19:06 UTC

Update

Update #10: WorkForce Software is currently experiencing SFTP Performance Degradation. Our teams are actively investigating the cause and working diligently to restore services as swiftly as possible. We recognize how vital our services are to your operations and are grateful for your patience during this time. To keep you informed and updated, we will be providing updates in Status Hub soon as updated information is available.
• RCA Number: RCA-1511
• Impact Start Date/Time: 10/15 10:26 ET
• Impacted Application(s): SFTP
• Issue Type: Intermittent
• Business Impact: Customers could experience intermittent connection issues.
• Actions Underway: While performance seems to have improved our teams are continuing to monitor.
• ETA: Unknown

Posted Oct 16, 2024 - 17:57 UTC

Monitoring

Update #9: WorkForce Software is currently experiencing SFTP Performance Degradation. Our teams are actively investigating the cause and working diligently to restore services as swiftly as possible. We recognize how vital our services are to your operations and are grateful for your patience during this time. To keep you informed and updated, we will be providing updates in Status Hub soon as updated information is available.
• RCA Number: RCA-1511
• Impact Start Date/Time: 10/15 10:26 ET
• Impacted Application(s): SFTP
• Issue Type: Intermittent
• Business Impact: Customers could experience intermittent connection issues.
• Actions Underway: Our technical teams have identified a possible cause and implemented a workaround. Teams are currently monitoring performance of the environment.
• ETA: Unknown

Posted Oct 16, 2024 - 16:57 UTC

Identified

Update #8: WorkForce Software is currently experiencing SFTP Performance Degradation. Our teams are actively investigating the cause and working diligently to restore services as swiftly as possible. We recognize how vital our services are to your operations and are grateful for your patience during this time. To keep you informed and updated, we will be providing updates in Status Hub soon as updated information is available.
• RCA Number: RCA-1511
• Impact Start Date/Time: 10/15 10:26 ET
• Impacted Application(s): SFTP
• Issue Type: Intermittent
• Business Impact: Customers could experience intermittent connection issues.
• Actions Underway: Our technical teams have identified a possible cause and working to remediate the problem. At the same time, they are continuing to be analyzing log data and high CPU utilizing processes for possible causes.
• ETA: Unknown

Posted Oct 16, 2024 - 15:51 UTC

Investigating

Update #7: WorkForce Software is currently experiencing SFTP Performance Degradation. Our teams are actively investigating the cause and working diligently to restore services as swiftly as possible. We recognize how vital our services are to your operations and are grateful for your patience during this time. To keep you informed and updated, we will be providing updates in Status Hub soon as updated information is available.
• RCA Number: RCA-1511
• Impact Start Date/Time: 10/15 10:26 ET
• Impacted Application(s): SFTP
• Issue Type: Intermittent
• Business Impact: Customers could experience intermittent connection issues.
• Actions Underway: Our technical teams are currently analyzing logs for possible causes.
• ETA: Unknown

Posted Oct 16, 2024 - 14:53 UTC

Monitoring

Update #6: Performance is back to normal. Our teams continue to monitor.
• RCA Number: RCA-1511
• Impact Start Date/Time: 10/15 10:26 ET
• Impacted Application(s): SFTP
• Issue Type: Intermittent
• Business Impact: Customers could experience intermittent connection issues.
• Actions Underway: Our technical teams continue to monitor.

Posted Oct 15, 2024 - 21:02 UTC

Investigating

Update #5: Our teams continue to actively investigate the cause of SFTP Performance Degradation and working diligently to restore services as swiftly as possible. We recognize how vital our services are to your operations and are grateful for your patience during this time. To keep you informed and updated, we will be providing updates in Status Hub soon as updated information is available.
• RCA Number: RCA-1511
• Impact Start Date/Time: 10/15 10:26 ET
• Impacted Application(s): SFTP
• Issue Type: Intermittent
• Business Impact: Customers could experience intermittent connection issues.
• Actions Underway: Our technical teams implemented a possible workaround, but the problem is still on-going. We are continuing to investigate to identify possible causes.
• ETA: Unknown

Posted Oct 15, 2024 - 20:38 UTC

Update

Update #4: Our teams continue to actively investigate the cause of SFTP Performance Degradation and working diligently to restore services as swiftly as possible. We recognize how vital our services are to your operations and are grateful for your patience during this time. To keep you informed and updated, we will be providing updates in Status Hub soon as updated information is available.
• RCA Number: RCA-1511
• Impact Start Date/Time: 10/15 10:26 ET
• Impacted Application(s): SFTP
• Issue Type: Intermittent
• Business Impact: Customers could experience intermittent connection issues.
• Actions Underway: Our technical teams have identified a possible cause and working to remediate the problem.
• ETA: Unknown

Posted Oct 15, 2024 - 19:37 UTC

Identified

Update #3: Our teams continue to actively investigate the cause of SFTP Performance Degradation and working diligently to restore services as swiftly as possible. We recognize how vital our services are to your operations and are grateful for your patience during this time. To keep you informed and updated, we will be providing updates in Status Hub soon as updated information is available.
• RCA Number: RCA-1511
• Impact Start Date/Time: 10/15 10:26 ET
• Impacted Application(s): SFTP
• Issue Type: Intermittent
• Business Impact: Customers could experience intermittent connection issues.
• Actions Underway: Our technical teams have identified a possible cause and working to remediate the problem.
• ETA: Unknown

Posted Oct 15, 2024 - 18:37 UTC

Update

Update #2: Our teams continue to actively investigate the cause of SFTP Performance Degradation and working diligently to restore services as swiftly as possible. We recognize how vital our services are to your operations and are grateful for your patience during this time. To keep you informed and updated, we will be providing updates in Status Hub soon as updated information is available.
• RCA Number: RCA-1511
• Impact Start Date/Time: 10/15 10:26 ET
• Impacted Application(s): SFTP
• Issue Type: Intermittent
• Actions Underway: Our technical teams have identified a possible cause and continue to look into workarounds to improve performance.
• ETA: Unknown

Posted Oct 15, 2024 - 17:33 UTC

Update

Update #1: Our teams continue to actively investigate the cause of SFTP Performance Degradation and working diligently to restore services as swiftly as possible. We recognize how vital our services are to your operations and are grateful for your patience during this time. To keep you informed and updated, we will be providing updates in Status Hub soon as updated information is available.
• RCA Number: RCA-1511
• Impact Start Date/Time: 10/15 10:26 ET
• Impacted Application(s): SFTP
• Issue Type: Intermittent
• Actions Underway: Our technical teams have identified a possible cause and looking into workarounds to improve performance.
• ETA: Unknown

Posted Oct 15, 2024 - 16:35 UTC

Investigating

Initial: WorkForce Software is currently experiencing a SFTP Performance Degradation. Our teams are actively investigating the cause and working diligently to restore services as swiftly as possible. We recognize how vital our services are to your operations and are grateful for your patience during this time. To keep you informed and updated, we will be providing updates in Status Hub soon as updated information is available.
• RCA Number: RCA-1511
• Impact Start Date/Time: 10/15 10:26 ET
• Impacted Application(s): SFTP
• Issue Type: Intermittent
• Actions Underway: Our technical teams are currently analyzing logs for possible causes.
• ETA: Unknown

Posted Oct 15, 2024 - 15:38 UTC

This incident affected: US8: Phoenix AZ (OCI).