The issue causing slow jobs performance for selected institutions this morning is resolved. The backlog of jobs that built up during the affected period is clear, and jobs are now processing at a normal rate. We apologize for the hassle this caused for some of you this morning.
Posted Jan 23, 2015 - 09:10 MST
Monitoring
Our DevOps team has identified the source of the trouble (a steady stream of a specific kind of job clogging the queue for jobs of all kinds) and has implemented a solution. Jobs are flowing at a normal rate again. We anticipate that the backlog of jobs that built up over the last couple of hours will clear in the next 15-20 minutes. We'll post another update then.
Posted Jan 23, 2015 - 08:34 MST
Investigating
Some jobs-type transactions are not processing in Canvas currently. This issue is affecting certain SIS imports, notifications, content migrations, reports, and other asynchronous functions. Our DevOps team began seeing error messages related to the problem at about 4:30 AM Mountain Time this morning. They're actively investigating. We'll provide an update in 30 minutes or sooner.