Impact Dashboard and Impact Messages are unavailable.

Incident Report for Instructure

Postmortem

Incident Report: Impact Dashboard and Impact Messages are unavailable.

Timeline:

March 6, 2025, 3:38 PM UTC: Incident started.
March 6, 2025, 8:22 PM UTC: Issue mitigated.
March 6, 2025, 10:52 PM UTC: Issue resolved.

Summary:

Between 3:38 PM and 10:52 PM UTC on March 6, 2025, a significant portion of users experienced slowness and unresponsiveness across the Impact platform. This was caused by a code change designed to enhance sub-account functionality which resulted in a much greater database load than anticipated, overloading the US-East database clusters. The incident resulted in widespread service disruption, preventing users from accessing and utilizing the platform.

Remediation:

Initial mitigation efforts involved scaling down the deployment platform to limit requests to the RDS clusters. However, this proved insufficient, and ultimately traffic had to be diverted from the affected cluster to allow the deployment platform to launch new instances with the rolled-back version.

‌We sincerely apologize for the inconvenience this incident caused. We value your understanding and are dedicated to continuous improvement of our platform's reliability and performance.

Posted Mar 13, 2025 - 08:34 MDT

Resolved

This incident has been resolved.

Posted Mar 06, 2025 - 14:30 MST

Update

We are continuing to monitor for any further issues.

Posted Mar 06, 2025 - 13:28 MST

Monitoring

A fix has been implemented and we are monitoring the results.

Posted Mar 06, 2025 - 13:25 MST

Update

We are continuing to work on a fix for this issue.

Posted Mar 06, 2025 - 12:48 MST

Update

We are continuing to investigate this issue but don’t have any new information to share at this time. Thank you for your patience as we work towards a resolution. We will provide an update as soon as we have more details.

Posted Mar 06, 2025 - 10:25 MST

Identified

Users may experience slowness, blank pages, or high latency in the Impact Dashboard and delays with Impact messages. Users may also be unable to access Campaign Templates. We have identified the root cause as a recent software update and are actively working to resolve the issue with the highest priority.

Posted Mar 06, 2025 - 08:41 MST

This incident affected: Impact.