The affected servers from the earlier incident seem to be behaving normally. We will continue to monitor performance, but will mark this issue as resolved. We appreciate your patience with the situation.
Posted Nov 30, 2016 - 13:55 MST
Monitoring
Our engineers have completed the recovery process, and access to Canvas should now be restored for all users. We are continuing to closely monitor performance levels of the cluster, and our engineers are taking steps to ensure performance remains consistent as the server load increases. We apologize for the trouble experienced and we have provided additional updates to admins at those institutions affected.
Posted Nov 30, 2016 - 11:21 MST
Update
Our engineering team have implemented a plan to resolve this situation as-soon-as-possible. Currently the recovery plan is on track, and we'll continue to post updates as the timeframe becomes clearer. Thank you for your patience.
Our DevOps team continues to work on the issue causing significant slowness and error messages for about 2% of Canvas users. They’ve identified the root of the problem and are taking steps to fix it as quickly as possible. We’ll continue to provide detailed updates to admins at affected institutions by email every 30 minutes until the problem is solved.
Posted Nov 30, 2016 - 10:32 MST
Update
Our DevOps team continues to work on the issue causing significant slowness and error messages for about 2% of Canvas users. They’ve identified the root of the problem and are taking steps to fix it as quickly as possible. We’ll continue to provide detailed updates to admins at affected institutions by email every 30 minutes until the problem is solved.
Posted Nov 30, 2016 - 09:50 MST
Update
Our engineers have identified the cause of the slow performance and error messages affecting some Canvas users this morning. They are working to improve performance and fix the root cause of the issue as quickly as possible. Unfortunately, this may take some time. We sincerely apologize for the trouble this is causing for users. We’ll continue to post brief updates here and send more detailed messages to admins at affected institutions.
Posted Nov 30, 2016 - 09:04 MST
Identified
Our engineers have identified the cause of degraded service on certain Canvas servers. This was a result of a standard upgrade to the clusters, but certain re-builds on this particular server are taking longer than normal.
We'll continue to send updates as they become available.
Posted Nov 30, 2016 - 08:37 MST
Investigating
2% users are experiencing degraded performance this morning. As a result, these users are experiencing time-out error pages. Our engineers are investigating the cause of the pages errors, and are currently working to resolve this situation as soon as possible.