Comment by blahedo
10 hours ago
Perspective from the trenches: I teach at a university that uses Canvas. We are in our final exams period right now.
We got our first email (from Academic Affairs) notifying us that it was down at 5:17pm EDT this afternoon, with little info; followup emails were sent at 6:24 and 6:57 with more info, but mostly about how we would be compensating for it and not about what actually was going on (other than, "nationwide shutdown" and "cybersecurity attacks", no further detail). I don't get a sense that they know much more than that, not that I would expect them to.
A perhaps telling detail: they're instructing us to have students email us directly with any work that had been submitted via Canvas. That suggests that they have no particular confidence that it will come back up soon.
I personally am only slightly affected; as a CS professor a lot of my students' work is done on department machines, and submitted that way, and I do the actual exams on paper. More importantly, I've never liked or trusted Canvas's gradebook, and so although I do upload grades to Canvas so students can see them, my primary gradebook is always a spreadsheet I maintain locally.
But I have a lot of colleagues for whom this is catastrophic at a level of "the whole building burnt down with all my exams and gradebooks in it"---even many of those that teach 100% in person have shifted much or all of their assessment into Canvas (using the Canvas "quiz" feature for everything up to and including final exams), and use the Canvas gradebook as their source-of-truth record. We've been encouraged to do so by our administration ("it makes submitting grades easier"). For faculty in that situation, they have few or zero artifacts that the students have produced, the students themselves don't have the artifacts to resubmit via email because they were done in Canvas in the first place, and they have no record of student grades or even attendance (because they managed that all inside Canvas). I guess they have access to the advisory midterm grades from March, if they submitted them (most do, some don't), but that might be it.
My gut feeling on this is that this is either resolved in hours (they have airgapped backups and can be working as soon as they can spin up new servers), or weeks (they don't). Very little in-between. And if that's true and we wake up tomorrow with this unresolved, I really have no idea what a lot of professors at my university and across the country are going to do to submit grades that are fair and reasonable. In the extreme case, they may have to revert to something we did in the pandemic semester (and before that, at my school, in the semester that two major academic buildings actually did burn to the ground a week before finals): let classes that normally count for a grade just submit grades as pass-fail. Because what else can you do?
(Well, one thing you can do is not put your eggs all in one basket, and not trust "the cloud" quite so much, but that ship's already sailed. I do wonder if in the longer term, anybody learns any lessons from this....)
UPDATE: As of 11:45pm EDT, my university's canvas instance is up and running! Here's hoping it stays (but I'll be downloading some stuff just in case...)
> the students themselves don't have the artifacts to resubmit via email because they were done in Canvas
It’s so simple to send an e-mail to the student with relevant records on completion of a quiz or whatnot. They don’t do it, because they want to control the data. (And universities don’t insist on it for who knows what reason.)
I've never used Canvas before, but all the LMSes that I've used allow students to enable emails whenever anything is updated, including when grades are posted. This is off by default because it's often 10+ emails a day, because many teachers post notes once a day, and with 5 classes, that adds up pretty quick. I personally have it enabled because it's pretty manageable with some custom Outlook rules, but setting this up is well beyond the capabilities of most students.
Canvas will send emails when grades are posted, but not what the grade is. Or at least that’s the way in the configurations I’ve seen. So, that wouldn’t help in a case where no one can access the canvas gradebook.
Setting up custom email filters is beyond the capabilities of most students? What are they learning? Where will they be qualified to work?
26 replies →
Students having records of what their score was doesn't prove to the professor / university what score they received. "FWD: Exam 1 Results" is not especially auditable.
Emails from Canvas saying a grade is available do not currently include the actual grade in the email, so that feature would have to be implemented first. And it's probably not implemented quite intentionally because of FERPA.
If only we had some way of signing messages
> Students having records of what their score was doesn't prove to the professor / university what score they received
It's better than nothing. (And good training for the real world.)
Also, most universities (and many schools now) issue academic e-mail addresses to students. In those cases, the email is definitive proof.
DKIM signature could be used to verify that Canvas' server sent the email with the given content
3 replies →
As opposed to a screenshot of a website? Presumably the professor has a spreadsheet of all assignment grades that is submitted to the school?
9 replies →
Presumably the system will be back up eventually, so there's not much benefit to lying here, since at best you'll raise your grade in a few classes for a couple months, while taking on a pretty big risk of getting caught.
You forget things can be signed, with the key owned by the school. It can be done.
4 replies →
Makes me glad I've always avoided doing my work on web platforms. When we used to have to make presentations in Google Slides I used to do them in Org-mode, then export to Sheets. I still have all those assignments sitting on my disk. Sure, there's versions of them on Google Drive, but I always make sure that the canonical version is the one on my disk.
>It’s so simple to send an e-mail to the student ...
What seems easy on hobby projects gets way more difficult at scale. Source: experience.
For what they charge for these LMSs, they should definitely be able to sent some emails.
> “My gut feeling on this is that this is either resolved in hours (they have airgapped backups and can be working as soon as they can spin up new servers)”
What good is having airgapped backups and spinning them up, if they are instantly vulnerable to the same attack again?
It does depend on what the attack is, but how do people approach that scenario?
Just to add one more data point, we also use Canvas at my university. The deadline for submitting who are eligible (i.e. passed compulsory assignments and labs) to take the exam was yesterday, and I couldn’t meet that deadline because Canvas went down. I usually do corrections offline so I have backups of my own evaluations, but these are courses with many teachers and many TAs, so Canvas is the way we sync our assessments.
I guess what surprises me the most is that it’s even legal for schools to outsource the core of what they do to some random tech company.
Either way, they were under no obligation to adopt this garbage technology regardless of whether it’s available, so this is 110% on them.
The alternative would be that each school develop their own platform for this, which also isn't very good use of their time and money?
Edit: No idea why this was down voted so much. I'm not defending Canvas, just wondering what the alternative would be.
5 replies →
I work in the Education sector as IT. We don't know much else either.
Everything we know has come from reddit threads / hackernews threads. There has been 0 official communication today indicating this was an attack, yet the login page was defaced by ShinyHunters.
I don't understand what's the panic and doomerism about. Any competent IT team has backups and will be up and running as they go back to a state before the breach. This is HN. I'm disappointed that everyone is talking about losing grades and going back to pen and paper. I don't see how that could happen in 2026.
And from the hacker's message itself, it's clear they want money in exchange for not releasing private info, not for the data itself.
Do we live in a fear based culture? Why the panic? Even if everything was hosted on Instructure's infrastructure, it's all AWS. I'd be VERY surprised if there aren't multiple way to go back to a previous state.
Most of the work and delay is to make sure they figure out where the breach occurred.
I'm sure you're right. Across tens (hundreds?) of thousands of institutions worldwide, each one is exercising its well-written incident runbook that not only gets updated regularly but also is rehearsed constantly, just in case something like this happens. After all, what university IT department DOESN'T prepare obsessively for the moment when they need to restore all grades on all assignments for all courses from backup and fall over to the backup system for final exam administration in any required format specified by any professor, in the second week of May, on a non-negotiable schedule? There's absolutely nothing to worry about here.
Schools don't have competent IT teams.
Here in the Netherlands a data center's power source (not even the machines) burnt down, data center is offline and University of Utrecht, one of the biggest universities here, is closed. Access passes don't work, work from home environment doesn't work, student information system is down, system for grading doesn't work. No failover for any of them (or maybe it was in the same DC?)
https://nos.nl/artikel/2613485-storingen-in-hele-land-door-b...
Sometimes it is very hard to recover from the offlining of essential systems: https://www.bbc.co.uk/news/articles/cy9pdld4y81o (Jaguar Land Rover, estimated cost in the billions)
I fully agree. What really pisses me off is that these "hacker" groups always spout off how they are doing it to screw the man but then threaten the average person. Millions of them. It just goes to show how uneducated, low-class, and simple these people really are.
> Any competent IT team has backups
Backups can be sabotaged (turned off or schedules manipulated) or compromised (say, by lateral movement).
> Even if everything was hosted on Instructure's infrastructure, it's all AWS.
AWS Backup isn't foolproof. Get your hands on administrator credentials as an attacker and suddenly the only thing between everything being gone for good and unrecoverable even for AWS is remembering to have put a permanent deletion protection on all resources in AWS Backup.
All these articles listing the American schools affected, "nationwide" outage reported, meanwhile hundreds of millions in the rest of the world affected.
Does anyone have a list of affected schools?
I don't have a list, but I can tell you the University of Iceland is affected.
Maybe a hybrid approach. Scramble to create a final exam/project and give them the option to do pass/fail or a real grade, their choice.
And then wish for the death of saas and a day where you can deploy your own software you can control and modify as you need.
Universities are not going to write their own software, and no they can’t use ‘agents’ to write and maintain it for them either.
What is the strategic response then? Assuming I'm a student and my grades are gone, and I want to graduate, shouldn't I pick pass/fail?
Does a future employer look at pass/fail vs the grade? do they care? Are there even jobs that matter enough to care out there for them?
This seems like, solving the problem but without actually seeing the broader goal or trajectory education is supposed to follow.
Most jobs I've had didn't care about a transcript in the slightest. It matters for future education and a small selection of jobs, and even them a few pass/fail courses won't cause any issues. It's not great if important, major-specific coursework is pass/fail, but usually you're not allowed to do that, so when it does come up you'll just have somebody ask what absurd situation (like this canvas thing) caused it.
> day where you can deploy your own software you can control and modify as you need.
Canvas is mostly FOSS
https://github.com/instructure/canvas-lms
To my European ears this just sounds like a disaster like this waiting to happen. God bless the annoying privacy OSS advocates and bureaucrats, I guess.
> they have airgapped backups and can be working as soon as they can spin up new servers
... and assuming they have a documented, tested, and trusted restore process.
Reminds me of the incident last year when a South Korean government's server room caught fire, which contained the government equivalent of Google Drive, and the only backup was in the same room, and they all burnt down together.
Some data was permanently lost, and then officers told reporters that multi-regional backup was not yet built because it was too hard at such a massive scale... of 858 TB.
> it was too hard at such a massive scale... of 858 TB
There are probably many S3 buckets in existence that are bigger than that.
Not saying that they should've used S3, but it's definitely possible configure multi-regional backup (and a government can afford it).
1 reply →
Ah yes the “recovery” part of the continuity plan. We tested that right? Right?
Backups are definitely helpful in ransomwares, but before systems can be restored and brought back online, victim organizations still need to assess the scope of the breach, find the initial access vector, identify compromised accounts, and evict the threat actor. That can take time.
I’m not certain, but it appears you’re giving Instructure a pass here, as if this is the first time they were hacked. But, it’s the second, by the same group.
As a parent of kids who are impacted by this, I’m not super concerned about the data being held for ransom, but I sure as fuck am concerned about how much it’s going to cost the district to move to another provider.
> I sure as fuck am concerned about how much it’s going to cost the district to move to another provider
Does Canvas have cybersecurity insurance?
Not at all; standard IR procedure is scope -> containment -> eradication -> recovery. There is a fog right now; we don't know all the details. It seems to me that it's just as likely they weren't fully kicked out before or that the initial vulnerability wasn't remediated. You can't recover until the threat actor has been removed.
> let classes that normally count for a grade just submit grades as pass-fail. Because what else can you do?
Schedule a single exam and that's your grade for that subject? That's how it should work anyway, credits for work during semester (or worse attendance) are not needed to evaluate if someone learned the material, give them an exam and done.
That's just bad outdated practice. It leads to cramming and less remembering than of the demand is for students to do work and show learning and effort throughout the year.
Most courses I've taken have obligatory assignments that are pass/fail, and you have to pass a certain amount during the semester to take the final exam. But the grade is determined entirely of the final exam.
Which to me seems the best way, you still have to learn throughout the year. Especially to avoid cheating this works nice. And as an aside, most people I know that did a year abroad in the US got 1-2 grades higher, as it was quite easy to just farm extra credits.
It has been my observation that most of the better students were the ones who would not put in work during the semester/year and cram at the end.
1 reply →
That's maybe something a school can do if exams are next week, or after.
At my school, tomorrow is the last day of exams. Most of the students have left campus. There's no time or mechanism to schedule an(other) exam.
Then you're testing how good someone is at exams as much as anything
Exams have performance variance. Otherwise you're only getting a pass/fall signal in any case.
Exams are the only fair way to evaluate if someone knows something (written or oral, in person). Take homes and attendance are just window dressing.
[flagged]
[dead]
[dead]