Amazon reports that they've resolved the Amazon Web Services (AWS) API issues that caused this morning's Canvas problems. Our DevOps team is working through the process of transitioning back from the manual workaround measure they implemented to our normal, automated scaling mechanisms that rely on the AWS APIs that were affected. Until we're sure that the automated toolset is working as intended, we'll err on the side of extra capacity to keep Canvas performance solid.
We will gather information from AWS about the root cause of their problems this morning and will prepare an incident report that we'll email to Canvas admins. We will provide this in the next few days; we'll prepare it as soon as we can, depending on when Amazon provides us with the information we'll need.
Posted Sep 20, 2015 - 13:06 MDT
Update
Our DevOps team has made manual changes that have restored Canvas performance for almost all users. They will continue to closely monitor the situation and manually respond to unexpected traffic spikes until the AWS API issues are fully resolved.
Posted Sep 20, 2015 - 10:39 MDT
Monitoring
We've been able to work around some aspects of the Amazon Web Services issue and add more resources to the Canvas infrastructure manually. These are improving performance for users who have been most severely impacted by the AWS problems this morning.
AWS is reporting progress on fixing the root cause of the incident. In the meanwhile, we will continue to take measures to address performance hotspots.
Posted Sep 20, 2015 - 09:46 MDT
Identified
Amazon reports that the AWS service that supports file uploads is operating as normal again. In our testing, users can again upload files to Canvas.
We're still tracking other issues related to the AWS issues this morning, including login problems, slowness, and error messages in the application.
Posted Sep 20, 2015 - 08:44 MDT
Investigating
Canvas users are seeing a wide range of problems in the application currently. These are related to issues with Amazon Web Services: http://status.aws.amazon.com/. Our DevOps team is exploring options to restore Canvas functionality as soon as possible. We will post updates at half-hour intervals; more frequently if we learn something significant.
Among the problems users might experience currently:
- Unable to log in (affecting some users, but not all)
- Unable to upload a file or submit an assignment
- Unable to access or download a file
- Unable to export a course or resources from a course