Comment by smcl
3 years ago
Last night I opened this, saw the HTTP 429 and figured "ah too many requests, I'll check the comments and try in the morning". The comments were all people swooning in shock about why some non-specific they (S3? Amazon? Someone else?) didn't use ".well-known" and others complaining about Mastodon and/or the fediverse. I had to read multiple comments to piece together the story, I swear it was like Elden Ring[0].
What this is actually about: BlueSky is Jack Dorsey's new Twitter clone, it is eventually intended to be some sort of fediverse thing but it's not there yet and it's not the source of the fediverse gripes here. You can authenticate your BlueSky user as the owner of a given domain or subdomain by placing a certain file with a given content somewhere under that domain/subdomain. However that "somewhere" was just a location one of the devs at BlueSky chose, rather than somewhere relatively standardised, like under the ".well-known" path (which you might recognise from things like OpenID Connect where the configuration doc is located @ example.com/.well-known/openid-configuration). So one user exploited this and became the "owner" of that Amazon S3 domain by setting up a storage account on Amazon S3 and following BlueSky's setup instructions. That is the main story here - some non-Amazon rando is now officially the Amazon S3 guy on Bluesky.
The next part is that someone posted about it on this https://chaos.social Mastodon instance, which got overwhelmed, the owners decided to save their server by electing to return a 429 response for that specific post if users don't belong to chaos.social, and that is why people are upset about Mastodon.
Interesting story, but I'm not interested in Dorsey's version of Twitter 2.0 unless it actually allows you to signup[1] and brings something compelling that Twitter didn't and Mastodon doesn't.
[0] - game with an intricate story that does its damndest to not actually tell you. If you want to know the story you have to piece it together yourself by picking up dozens of items scattered throughout the game and reading all their descriptions. Or you can do what I did - watch a video on YouTube.
[1] - they're doing an open beta and letting a little trickle of users on, who post about it on their Twitter/Mastodon/whatever. Feels a bit deliberate, like they're trying to build anticipation and frankly I detest little manipulative things like that so I'm out
> [1] - they're doing an open beta and letting a little trickle of users on, who post about it on their Twitter/Mastodon/whatever. Feels a bit deliberate, like they're trying to build anticipation and frankly I detest little manipulative things like that so I'm out
Frankly this cynicism feels unwarranted. Bluesky is not a finished product — it is still being built and, even with the small number of invited users so far, there have been problems that have needed attention. The moderation story is still being developed, the feeds are still being tweaked, the app still has bugs, federation still doesn't work yet. Having some users makes for a valuable feedback loop but the team would rapidly become inundated and burnt out (and the platform would possibly turn into a wild-west hellscape with irreversible reputational damage) if they were to open the floodgates entirely at this stage.
To clarify - it felt like this was an attempt to replicate the mid-00s play of building interest by restricting who can join and making it exclusive (Facebook did this by rolling it out uni-by-uni, Gmail was for a while invite-only and invites were highly valued) and therefore desirable.
Maybe that's in my head but layering this feeling on top of BlueSky being yet another microblogging service with a few other things that I don't love contributes to my impression of Bluesky being simply "meh". If it becomes the next thing that everyone uses, I'll inevitably have to check it out, I'm not going to be an early adopter though.
Your feelings resonate with me too. My attitude these days is that if a platform wants to make me feel excluded (in order to induce FOMO), then I accept being excluded. They win, I guess?
> The next part is that someone posted about it on this https://chaos.social Mastodon instance, which got overwhelmed, the owners decided to save their server by electing to return a 429 response for that specific post if users don't belong to chaos.social, and that is why people are upset about Mastodon.
It's like all these newfangled webapps don't understand the concept of caching static pages for anonymous users. There is absolutely no reason that something like this should result in more than one request (plus a handful more for static resources) handled entirely by the frontent webserver's in-memory cache for each person linked from other sides. But instead its all dynamic and the page shoots off more API requests before being able to show anything.
So the thing is that in one respect they actually do get caching, almost to a fault. One of the complaints I've seen among some Mastodon instance operators is that they end up storing some pretty hefty amounts of data locally as their instance caches remote posts, images and profiles from other instances that its members follow. One source of problems, which may have been resolved, was that even though there's a job that cleans out this cache the banner images from external profiles stick around. I saw this a while back and it seems like an easy fix so I imagine it's been addressed.
I don't think I am equipped to diagnose what the root cause was here. It's even possible that this instance wasn't intended to have viral posts (i.e. high profile posts that get would get shared to many external users) and they didn't want to invest in hardware/services to facilitate this.
I think the GP was referring to caching on the other end, caching static html that can be raised for all anonymous users.
The question is whether the server was having issues with a flood of new posts being sent in and stored, or a flood of anonymous users clicking a link and blogging down when the same html was getting rendered over and over.
Knowing Mastodon, I have a bunch of was the latter with the server coming out on all the new data it was trying to store locally
Archived version of the original Mastodon post: https://archive.is/fM06z
You should still follow Jonty (the poster at that Mastodon instance) wherever your socials are, because he's awesome, and posts about awesome things. He's also the organiser of EMFCamp (https://www.emfcamp.org/), which is a nerd/hacker camping festival.
Thanks, will do! I'm always on the lookout for interesting people on Mastodon!
> However that "somewhere" was just a location one of the devs at BlueSky chose, rather than somewhere relatively standardised, like under the ".well-known" path
I've not looked into BlueSky's domain based identity thing in any detail so I might be missing a point somewhere, but… If someone can manipulate its special location what would there be to stop the same someone being able to manipulate content under .well-known?
Are we just relying on .well-known having some extra protection (in this case by Amazon having created a bucket called .well-known so no one else could)? If so then .well-known is little safer than any other arbitrary location in this respect (because you are relying on every domain owner who might be spoofed to take an action to protect against this, rather than the protocol failing safe if nothing is done by the domain owner) and perhaps using DNS would be better.
> Are we just relying on .well-known having some extra protection [...] ? If so then .well-known is little safer than any other arbitrary location in this respect.
If .well-known had just been invented, that would be true. It's fairly well established at this point, though. For example, if someone can create arbitrary files in .well-known, they are also able to pass http-01 ACME challenges and thus issue TLS certs for your domain (modulo CAA) and MITM you. At this point, allowing users to modify .well-known is about as good an idea as allowing them to receive mail for postmaster@ or accepting incoming packets from 192.168.0.0/16 into your LAN.
Amazon S3 specifically would not be vulnerable because bucket names can’t have dots in them; same for every other service that doesn’t allow those. Neither would services that prefix usernames with ~ or @ or similar, nor services that already use http-01 ACME challenges to get certs thus are already using that path.
I’d be much happier if proving domain control were only done through DNS challenges, but that ship has sailed.
Good point with other common services like certificates via ACME.
Though MitM that way requires more steps than faking identity this way as you need to somehow get in the middle or redirect traffic towards you.
> I’d be much happier if proving domain control were only done through DNS challenges, but that ship has sailed.
Agreed.
Aaaah! Thanks! Great write up!!
Just wanna share that I found your comment a lot of fun to read, even if I had already pieced the story together from other comments. Thanks!
this is everything I wanted to know and more, all in one comment—thank you
[flagged]
> They should've picked a proper language that's able to perform and handle those requests like Go or Rust. Building a social network, especially distributed, and using a low performance language... what were they thinking?
Even if a Go or Rust variant of exactly the same implementation would have been two or three times faster, it would not have survived the same onslaught. The way to survive that sort of accidental DDoS is not to change language but to improve algorithm choices where possible (where that makes a difference in order, not just a small difference in scale), make sure you are scaling efficiently over the CPU cores (possible in all those languages in various ways) available per node without busting memory limits, and by scaling out once that becomes a limiting factor.
This was effectively a DDoS situation (not an attack, but from the server's PoV there is no difference between a malicious DDoS and a hug-of-death from a mass of interested parties) and using Go or Rust instead of Ruby would likely have made no difference what so ever.
(the rest of your post is not discussing technical matters pertaining to the rest of the thread so I'll not encourage the off-topic-mess by responding in any detail, other than to say that sort of complaint wrt handling potentially contentious issues is common on social networks and not Mastodon specific)
Or they should have just put configured caching of static content. This has nothing to do with Ruby, nor even the architecture of Mastodon (the software) which is not great, but about a server not being set up properly - any static cache or, even better, fronting it by a CDN, will trivially beat the most optimised compiled dynamic content generating framework.
Someone on one server trying to get a user on another server is maybe a bit much, but I don't know the content of the posts in question. For me it'd have to be something pretty extreme to take those actions, otherwise if I saw something that irritated me I'd just consider ignoring or blocking that person depending on the severity.
Remember though that the war is a pretty delicate subject and that a position that from your perspective seems perfectly peaceful could be seen differently by others. So to a German the message "no weapons to Ukraine!" could be seen as de-escalatory, but to a Ukrainian it could be seen as a betrayal or it may remind them of bad faith pro-Z/Putin trolling even if that doesn't describe you or your intent whatsoever.
Barring context, I'm going to guess that this was simply regular users on chaos.social ticking the box in the reporting interface that forwards the report to the originating server - an option that's vital in a federated system when reporting anything that you see as bad enough rather than just a violation of your own instances rules, but of course will be used for things not against either instances rules all the time on any larger instance.
> Remember though that the war is a pretty delicate subject
That’s good advice for a dinner party, not a valid defence of censorship on a social network.
1 reply →