Last night I opened this, saw the HTTP 429 and figured "ah too many requests, I'll check the comments and try in the morning". The comments were all people swooning in shock about why some non-specific they (S3? Amazon? Someone else?) didn't use ".well-known" and others complaining about Mastodon and/or the fediverse. I had to read multiple comments to piece together the story, I swear it was like Elden Ring[0].
What this is actually about: BlueSky is Jack Dorsey's new Twitter clone, it is eventually intended to be some sort of fediverse thing but it's not there yet and it's not the source of the fediverse gripes here. You can authenticate your BlueSky user as the owner of a given domain or subdomain by placing a certain file with a given content somewhere under that domain/subdomain. However that "somewhere" was just a location one of the devs at BlueSky chose, rather than somewhere relatively standardised, like under the ".well-known" path (which you might recognise from things like OpenID Connect where the configuration doc is located @ example.com/.well-known/openid-configuration). So one user exploited this and became the "owner" of that Amazon S3 domain by setting up a storage account on Amazon S3 and following BlueSky's setup instructions. That is the main story here - some non-Amazon rando is now officially the Amazon S3 guy on Bluesky.
The next part is that someone posted about it on this https://chaos.social Mastodon instance, which got overwhelmed, the owners decided to save their server by electing to return a 429 response for that specific post if users don't belong to chaos.social, and that is why people are upset about Mastodon.
Interesting story, but I'm not interested in Dorsey's version of Twitter 2.0 unless it actually allows you to signup[1] and brings something compelling that Twitter didn't and Mastodon doesn't.
[0] - game with an intricate story that does its damndest to not actually tell you. If you want to know the story you have to piece it together yourself by picking up dozens of items scattered throughout the game and reading all their descriptions. Or you can do what I did - watch a video on YouTube.
[1] - they're doing an open beta and letting a little trickle of users on, who post about it on their Twitter/Mastodon/whatever. Feels a bit deliberate, like they're trying to build anticipation and frankly I detest little manipulative things like that so I'm out
> [1] - they're doing an open beta and letting a little trickle of users on, who post about it on their Twitter/Mastodon/whatever. Feels a bit deliberate, like they're trying to build anticipation and frankly I detest little manipulative things like that so I'm out
Frankly this cynicism feels unwarranted. Bluesky is not a finished product — it is still being built and, even with the small number of invited users so far, there have been problems that have needed attention. The moderation story is still being developed, the feeds are still being tweaked, the app still has bugs, federation still doesn't work yet. Having some users makes for a valuable feedback loop but the team would rapidly become inundated and burnt out (and the platform would possibly turn into a wild-west hellscape with irreversible reputational damage) if they were to open the floodgates entirely at this stage.
To clarify - it felt like this was an attempt to replicate the mid-00s play of building interest by restricting who can join and making it exclusive (Facebook did this by rolling it out uni-by-uni, Gmail was for a while invite-only and invites were highly valued) and therefore desirable.
Maybe that's in my head but layering this feeling on top of BlueSky being yet another microblogging service with a few other things that I don't love contributes to my impression of Bluesky being simply "meh". If it becomes the next thing that everyone uses, I'll inevitably have to check it out, I'm not going to be an early adopter though.
> The next part is that someone posted about it on this https://chaos.social Mastodon instance, which got overwhelmed, the owners decided to save their server by electing to return a 429 response for that specific post if users don't belong to chaos.social, and that is why people are upset about Mastodon.
It's like all these newfangled webapps don't understand the concept of caching static pages for anonymous users. There is absolutely no reason that something like this should result in more than one request (plus a handful more for static resources) handled entirely by the frontent webserver's in-memory cache for each person linked from other sides. But instead its all dynamic and the page shoots off more API requests before being able to show anything.
So the thing is that in one respect they actually do get caching, almost to a fault. One of the complaints I've seen among some Mastodon instance operators is that they end up storing some pretty hefty amounts of data locally as their instance caches remote posts, images and profiles from other instances that its members follow. One source of problems, which may have been resolved, was that even though there's a job that cleans out this cache the banner images from external profiles stick around. I saw this a while back and it seems like an easy fix so I imagine it's been addressed.
I don't think I am equipped to diagnose what the root cause was here. It's even possible that this instance wasn't intended to have viral posts (i.e. high profile posts that get would get shared to many external users) and they didn't want to invest in hardware/services to facilitate this.
You should still follow Jonty (the poster at that Mastodon instance) wherever your socials are, because he's awesome, and posts about awesome things. He's also the organiser of EMFCamp (https://www.emfcamp.org/), which is a nerd/hacker camping festival.
> However that "somewhere" was just a location one of the devs at BlueSky chose, rather than somewhere relatively standardised, like under the ".well-known" path
I've not looked into BlueSky's domain based identity thing in any detail so I might be missing a point somewhere, but… If someone can manipulate its special location what would there be to stop the same someone being able to manipulate content under .well-known?
Are we just relying on .well-known having some extra protection (in this case by Amazon having created a bucket called .well-known so no one else could)? If so then .well-known is little safer than any other arbitrary location in this respect (because you are relying on every domain owner who might be spoofed to take an action to protect against this, rather than the protocol failing safe if nothing is done by the domain owner) and perhaps using DNS would be better.
> Are we just relying on .well-known having some extra protection [...] ? If so then .well-known is little safer than any other arbitrary location in this respect.
If .well-known had just been invented, that would be true. It's fairly well established at this point, though. For example, if someone can create arbitrary files in .well-known, they are also able to pass http-01 ACME challenges and thus issue TLS certs for your domain (modulo CAA) and MITM you. At this point, allowing users to modify .well-known is about as good an idea as allowing them to receive mail for postmaster@ or accepting incoming packets from 192.168.0.0/16 into your LAN.
Amazon S3 specifically would not be vulnerable because bucket names can’t have dots in them; same for every other service that doesn’t allow those. Neither would services that prefix usernames with ~ or @ or similar, nor services that already use http-01 ACME challenges to get certs thus are already using that path.
I’d be much happier if proving domain control were only done through DNS challenges, but that ship has sailed.
> They should've picked a proper language that's able to perform and handle those requests like Go or Rust. Building a social network, especially distributed, and using a low performance language... what were they thinking?
Even if a Go or Rust variant of exactly the same implementation would have been two or three times faster, it would not have survived the same onslaught. The way to survive that sort of accidental DDoS is not to change language but to improve algorithm choices where possible (where that makes a difference in order, not just a small difference in scale), make sure you are scaling efficiently over the CPU cores (possible in all those languages in various ways) available per node without busting memory limits, and by scaling out once that becomes a limiting factor.
This was effectively a DDoS situation (not an attack, but from the server's PoV there is no difference between a malicious DDoS and a hug-of-death from a mass of interested parties) and using Go or Rust instead of Ruby would likely have made no difference what so ever.
(the rest of your post is not discussing technical matters pertaining to the rest of the thread so I'll not encourage the off-topic-mess by responding in any detail, other than to say that sort of complaint wrt handling potentially contentious issues is common on social networks and not Mastodon specific)
Or they should have just put configured caching of static content. This has nothing to do with Ruby, nor even the architecture of Mastodon (the software) which is not great, but about a server not being set up properly - any static cache or, even better, fronting it by a CDN, will trivially beat the most optimised compiled dynamic content generating framework.
Someone on one server trying to get a user on another server is maybe a bit much, but I don't know the content of the posts in question. For me it'd have to be something pretty extreme to take those actions, otherwise if I saw something that irritated me I'd just consider ignoring or blocking that person depending on the severity.
Remember though that the war is a pretty delicate subject and that a position that from your perspective seems perfectly peaceful could be seen differently by others. So to a German the message "no weapons to Ukraine!" could be seen as de-escalatory, but to a Ukrainian it could be seen as a betrayal or it may remind them of bad faith pro-Z/Putin trolling even if that doesn't describe you or your intent whatsoever.
> 1) I feel that /host-meta is too casual of a name and prone to collisions. It matches /^[\w\-]+$/, which I think is a subset of a fair number of sites' usernames."
...
> i.e. put something ugly and weird in there, like a semicolon, to minimize
the chance that it interferes with people's existing URL structure.
This proposal made complete sense and the counter argument was of such a tiring kind. I see that from time to time in these old standards mail threads. I can't put my finger on it, but perhaps it is that they only saw the "now", not the future. Where the "now" was time when a webmaster were in charge of every website, and had complete personal control.
Things to learn about the FediVerse from the 429 error:
* The FediVerse is lots of WWW sites. Some are WWW-hosting companies showing off, with all of the acoutrements of high-end WWW sites, including CloudFlare protection and lots of tweaking of the back end stuff. Others are one-person sites where someone has just set up the vanilla Mastodon/Pleroma/Pixelfed/Friendica/whatever software on a cheap hosted VM somewhere. There are lots of in-betweens. I have an account on two sites, at each of the aforementioned extremes, one with well over 20,000 users and the other with around 40.
* It's really easy to deny service to the one-person sites, and many of the low-end ones.
* Chaos.Social's about page explains that it's a couple of people running a WWW site in their spare time on spare hardware. That's a little misleading, as they've upgraded the hardware a bit. But it's still 2 people, with ~5800 users. For more, start at https://meta.chaos.social/resources .
* There's nothing global in the FediVerse. Nothing gets sent everywhere. Some commenters here can see the post cached by their local WWW sites where they have accounts. But I'm in the opposite situation: None of the places where I have accounts have cached that post, and since the Chaos.Social sysop put the 429 error in place to combat the server overloading, they actually cannot pull that post with just its URL entered directly, although simple tricks like searching for @jonty@Chaos.Social instead and reading the user timeline work just fine.
* There's nothing global in the FediVerse. Using the aforementioned trick, I see a different view of the thread from Mastodon.Scot to what I see from Toot.Wales, and both of those are different to what's seen from other places.
Most of us are just browsing for interesting light reading anyway, so blocking us if they can’t serve a Hackernews worth of users seems basically… like an appropriate amount of robustness, for light reading.
Maybe the FediVerse is just not friendly to the idea of a “global top among thousands of users” rating.
The aforementioned trick wouldn't work if it wasn't. But before the 429 error was put into place the entire site was affected across the board by Hacker News. See https://chaos.social/@ordnung/110312020977014678 .
Thank you for the explanation, that actually makes sense. But I still think that serving a 429 is some kind of backwards old-school sysop kind of response. "It's the right HTTP-Code! hohoho!". It's obviously running nginx and serving a static copy of that url or setting up caching would take the same amount of time as serving the 429. It's 2023, it's possible to serve some thousand requests to static content on pretty much everything now.
Mastodon is written in Ruby on Rails and there are some inherent performance issues with that, it generates a huge number of Sidekiq jobs that can bog down a server quite easily. There are other, non-Ruby implementations aiming for compatibility with the Mastodon API though, so I’m curious to see how it will all shake out.
Further context: Bluesky lets you use a domain name you own as a user handle.
The official method is to set a TXT record, but apparently their "AT protocol" also lets you confirm a domain by serving `GET your.domainname.com/xrpc/com.atproto.identity.resolveHandle`
Microsoft funded Apple to keep another OS vendor alive, so it's not about discouragement. It's probably a lot cheaper to fund a competitor than paying the gov't and getting tagged as a recognized monopoly
Right now, federation is not turned on for the Bluesky instance.
There are differences in both, however. I'm not going to speak about my impressions of the Mastadon vs Bluesky teams because frankly, Mastadon never really caught on with me, so they're probably biased. ('they' being my impressions, that is, I just realized that may be ambiguous.)
At the protocol level, I haven't implemented ActivityPub in a decade, so I'm a bit behind developments there personally, but the mental model for AT Protocol is best analogized as git, honestly. Users have a PDS, a personal data server, that is identified by a domain, and signed. The location of the PDS does not have to match the domain, enabling you to do what you see here: a user with a domain as their handle, yet all the PDS data is stored on bluesky's servers. You can make a backup of your data at any time, and move your PDS somewhere else with ease (again, once federation is actually implemented, the path there is straightforward though). This is analogous to how you have a git repository locally, and on GitHub, and you point people at the GitHub, but say you decide you hate GitHub, and move to GitLab: you just upload your git repo there, and you're good. Same thing, except since identity is on your own domain, you don't even need to do a redirect, everything Just Works.
This analogy is also fruitful for understanding current limitations: "delete a post" is kind of like "git revert" currently: that is, it's a logical deletion, not an actual deletion. Enabling that ("git rebase") is currently underway. Private messaging does not yet exist.
Anyway if you want to know more the high-level aspects of the docs are very good. Like shockingly so. https://atproto.com/guides/overview They fall down a bit once you get into the details, but stuff is still changing and the team has 10,000 things to do, so it's understandable.
There is also another funny thing in the image, the user posting about is sending one from "retr0-id.translate.goog", which is odd. Somehow he has got https://retr0-id.translate.goog/xrpc/com.atproto.identity.re... to redirect to his page, and gotten that handle as well.
Eh, it’s worse than just funny; it’s concerning, because they should have known about and easily avoided this kind of vulnerability, it’s standard stuff you have to think about. So what else have they missed?
This is a private beta. Nobody is suggesting that any of this be used for anything serious just yet. Development happens out in the open, you can go find out what else they've missed by doing the work, or by waiting until others you trust have done so.
I myself have had an account for like a month now, but only started really using it a week ago, because that calculus changed for me, personally.
Like, it's not even possible to truly delete posts at the moment. This all needs to be treated as a playground until things mature.
This isn't even the first "scandal" related to this feature already!!!! There is another hole in what currently exists that allowed someone to temporarily impersonate a Japanese magazine a few weeks back.
For me, the worst thing about it is that they didn't just use webfinger. So webfinger isn't perfect, but it's there and in use. When they choose to invent new mechanisms for things there are perfectly serviceable options for, it makes me instantly sceptical of the rest.
Wouldn't be funny if it was a public beta that they want people to use for serious stuff. But it's neither serious, a beta or public, but basically a private alpha for playing around, so i'd be a bit lenient on screwups.
Google Translate recently moved translated web pages to domains like this. If you plug a webpage into GT it will put the translated content under <domain>-<tld>.translate.goog. This user's actual domain is https://retr0.id
Wait - nobody had ever created a bucket named xrpc before, ever? I would have imagined that short s3 buckets were squatted similar to domain names. (Or maybe they were, and it's this person who did so!)
The server itself seems to work fine. It only seems to be this specific post that's being 429'd.
I'm guessing it's some kind of anti-DDoS setup kicking in.
Mastodon is also quite heavy to host, my single user instance will easily gobble up several gigabytes of memory if you let it. There are more efficient ActivityPub servers but specifically Mastodon seems to be written for running efficiently on huge servers.
unfortunately this is exactly why mastodon won't work
I already don't trust mastodon links because 9 times out of 10 they simply don't work. Everyone's tiny hobby server falls over when one post gets big, and obviously not everyone is going to scale their servers to support the load of a viral post that might happen once every 6 months and will be 100x their base load
Just look at it failing the HN hug of death. If this one can't survive techies on a orange site rushing to the site then it cannot possibly survive a lotus of users from Twitter or TikTok rushing into any post on Mastodon, bringing it flat on to the floor.
I can only see Mastodon centralizing to cope with the load. But a server going down on this load from HN tells us it is no where near ready to handle an insurmountable amount of users or even begin to challenge Twitter which hosts 220M+ users every single day.
Mastodon user count has mostly been a steady growth. So far it hasn't really failed at a high level. We aren't in Bitcoin territory, the network isn't really slower than it was a year ago even if number of user is much higher. It's mostly distributed among many instances.
This is mind-blowing. Last I checked, the front page of HN sends tens of requests per second to each link. There are humans who can pack envelopes faster than the typical mastodon server can answer GETs. I'd love to see someone benchmark the top servers for a few seconds to see what it takes to break a reasonable latency SLA.
Sites like HN and that other one should track this. It could work like flagging - if enough people mark it as hugged to death (HTD), it would say so next to the link. Maybe it could even redirect to an archive if it’s currently HTD.
This is why mastodon , webfinger and ACME uss .well-known uri prefix. .well-known is reserved and you can't e.g. make a bucket named .well-known
It's funny the bluesky devs say they implemented "something like webfinger" but left out the only important part of webfinger that protects against these attacks in the first place. Weird oversight and something something don't come up with your own standards
> This is why mastodon , webfinger and ACME uss .well-known uri prefix
This is not how Mastodon does verification (at least not the main method). Mastodon doesn't just link users -> domain. It can link user -> webpage, for example to link social profiles between sites.
If you have a website with user generated content, and a user can set an arbitrary html attribute (rel="me") in a link pointing back to their profile, they can claim ownership of the page on Mastodon. Likewise, if they can set a link tag in the head element of the page for some reason.
Presumably this is somewhat harder to exploit than a (new, poorly thought out) dependency on a static file under /xrpc, but Mastodon does introduce more authentication footguns for sites than just .well-known! https://docs.joinmastodon.org/user/profile/#verification
Edit: authentication -> verification, since Mastodon distinguishes between the two (see below)
You're thinking of how Mastodon does verified links. You could do something similar, provide a verified link on your profile to a file in an S3 bucket, but there's very utility (or risk) in that.
What about serving the challenge file from the root or a near-root of the fully qualified url? Like www.domain.com/mastodon.txt or abc.freehost.com/mastodon.txt?
Maybe I'm old but what are some popular use cases for webfinger? (I'm just learning about it now)
The /.well-known/ path prefix is the standard name to use (https://www.rfc-editor.org/rfc/rfc8615) so that any sort of “we’ll host user content from our domain” thing can block it. (Hosting user content from the user’s domain is fine and doesn’t need this restriction.)
A few things are effectively grandfathered in due to their vintage: /favicon.ico, /sitemap.xml and /robots.txt are the three that occur to me—so if you’re running something vaguely like S3, you’ll want to make sure users can’t create files at the top level of your domain matching at least those names.
But nothing new should use anything other than /.well-known/ for domain-scoped stuff, or else you run into exactly this problem.
We're talking about folks setting up a custom domain for a personal social media presence. If they can handle nameservers and DNS records, they can handle a folder with a dot in the name.
The interesting thing to consider is that someone who is popular to read may well be as much of a headache for the local sysop as someone who is a frequent target for attacks. How long until they have a 'bot that just 429s all of your posts on sight, do you think? (-:
On a more serious note: Given that many user communities in the FediVerse are people conscious of their privacy and of suffering pile-ons by large majorities of outsiders, I wonder how many more examples of this there will be before people start asking the developers for out-of-the-box defaults that simply blacklist all requests that come in with referrer headers containing news.ycombinator.com and other similar "Slashdotting" sites.
This is a terrible implementation of domain verification. dns-01 and http-01 are more or less standardized at this point. Use them, and don't roll your own. Reference: https://letsencrypt.org/docs/challenge-types/.
They definitely should have used HTTP-01 if they’re doing verification on the web, but since this is about using a domain as identity this really belongs in DNS.
The issue with DNS-01 (and HTTP-01 to a lesser extent) as someone else mentioned is that the user friction is really high.
I’ve been working on a solution to this that I’ve been meaning to post to HN and this seems like as good an opportunity as any so here it is: [1]
It’s a method of storing a hashed (and optionally salted) verifiable identifier (think email or mobile) at a subdomain to prove authority for a domain.
And the primary way of identifying yourself is in fact DNS.
> I’ve been working on a solution to this
Your solution is almost identical to the BlueSky one: put a TXT record at _atproto.<domain> that resolves to a DID. The difference is that they mandate the DID spec and you do not. Which is totally fine! Just figured I'd let you know :)
I don't get http-based verification in general. If you want to really prove someone owns a domain, make them change an authoritative DNS record. Everything else feels like it is begging for edge cases to crop up. Why should my social media or SSL certificate vendor care about my web servers?
I worked on a product that required DNS changes to set up. Especially for business accounts, the level of friction was STUNNING. We had it take months to get set up because the contact had to submit a ticket to IT, write up the business justification, get director level approval, get security approval, and so on before it could get done. We had customers who couldn't even figure out which group in their company managed DNS. Yeah, you can argue that those companies are broken, but as an outsider I have no influence over that. The result was just that they couldn't use our product. On the flip side, we had consumer and small business customers who had purchased domains through simple webhosting things that didn't give them the required level of access to create a record (and/or they couldn't figure out how to do it). We eventually added an HTTP option and the success rate and time to success both improved hugely.
Both http and dns verification are stupid. Neither of them prove you own the domain.
http verification proves you temporarily control IP space relative to a viewer. dns verification proves you temporarily control name resolution relative to a viewer.
Both are trivially hacked, multiple ways. By the time someone finds out you did it (if they closely monitor CT logs, which nobody does) you've already had hours, days, weeks to run a MITM on any domain you want. The attack only has to work once, on any of 130+ CAs.
The solution is registrar-level proof. Cert request signed by the private key of the domain owner, sent to the registrar to verify, the registrar signs it if its true, it's sent to the CA who can see the registrar signed it. Now you know for a fact the domain owner asked for the cert. The only possible attack is to steal all three of the owner's private key, the registrar's private key, and a CA's private key.
I have been shouting about this for 10 years, none of the industry incumbents care. The internet is run by morons.
> If you want to really prove someone owns a domain, make them change an authoritative DNS record.
You're not wrong (ignoring how easy it is to hack DNS), but at the same time it's hard enough to get people to buy their own domain name, nevermind understand the system well enough to add a TXT record.
It's a strategy that's fine to implement when your target audience is server admins. It's a terrible strategy when your target audience is everyday users who you hope own their own domain. Doubly so in a world where owning your own domain is so rare for individuals.
Sure, adding a TXT record to verify domain ownership is fairly common and lots of tools still use it. But you either have to self host DNS (yet another container to maintain) or use your provider's API (yet another credential, yet another mailing list to subscribe to for inevitable breaking changes to the API).
In contrast, HTTP based verification often has built-in support with your webserver (Caddy) or only requires copy-pasting a few lines to your docker compose file.
There are edge cases, but they're also widely exploited so you won't run into them if you follow best practices.
I think people don't want to put DNS admin credentials in places where they might get leaked. Would be cool if a DNS server or provider offered credentials that could only do ACME challenges and not change any other records.
DNS challenge is required for wildcards on LE at the very least.
But the reason for HTTP is pretty simple - it's extremely easy to implement. You only need to tell your ops to redir a subdomain to your app and you're done, you don't need DNS with API that have narrow enough permission to allow that one team in whole company to generate ACME stuff; most providers ACLs on DNS end at "this client have acesss to that domain via API".
It's not about proving ownership, if it was about proving ownership we would do this via something at the registrar level.
It's about proving /control/. If a domain name is pointed to me (my IP/CNAME) I control it and it is reasonable to allow that person to issue an SSL certificate for a domain (or subdomain) under their control. If you, as the domain owner, want to restrict that, CAA exists as your tool to do so.
HTTP-01 works really well for when you host a custom domain on a SaaS application. The domain ownership stays with the customer, and all they have to do is CNAME/ANAME to your server. No messing with DNS TXT or NS records, 3rd party DNS provider API keys, etc.
Bluesky was started within Twitter as a federated social protocol in 2019. Spun out in 2021 as a standalone company and it was also being used in the Crypto group at Twitter until Elon came in.
I'm sure just because of its age and principals involved it's been heavily influenced by the crypto crowd.
But suddenly, without anything else really stepping in to fill the void, it's the Twitter alternative of the day. Anyway not surprised to find some gotcha's like this all things considered.
> I'm sure just because of its age and principals involved it's been heavily influenced by the crypto crowd.
It builds off of several specifications that came from the crypto crowd. It does not use a proof of stake or proof of work blockchain, though, so depending on how you use the words "crypto" and "blockchain," it either is or is not those things.
If you can upload a custom file to a domain/subdomain, bluesky social (Jack Dorsey's new twitter) uses it to verify you are the owner of the domain. Chaz uploaded his custom file to their Amazon s3 bucket and now since he was the first one to do it, his account is now associated with Amazon S3.
It'd be nice as an extra precaution, but please don't build things that rely on the Public Suffix List for security (this list by its nature is only a laggy incomplete approximation of the actual use of domains).
What is the easy fix? Use the .well-known/ standard instead of the current mechanisms, and roll back verification for anyone who’s already been verified with the flawed approach?
I created a rich text system for posts to handle things like links and mentions with the eventual goal of it being the basis for all kinds of rich text (bolding, italics, spoiler tags, etc)
the flexibility bit us on the butt. people started faking mentions via the APIs and one user figured out he could pack 1000 mentions into one "@everyone" and cause us all to get notified. pretty predictable in hindsight but I dropped the ball there
Given that the AT Protocol is not compatible with ActivityPub, I don't see how the first step, let alone the second or third, is an accurate description of the dynamics at play here.
> And also how long until Twitter 2.0 or X app uses AT
Ironically, pre elon-buys-twitter, it did in fact look like Twitter was going to end up implementing AT, if all went to plan. But then post acquisition, if anything, that looks less likely.
There is something really endearing about this. It takes me back to those days where we were building something new and bugs just popped up and we had fun with them, doing silly things and then fixing them. Morale was high, team was fun, we were all learning and building something new.
If you want to prove domain ownership, you have to do it at the domain level.
The ability to serve a file “www.example.com” in no way demonstrates ownership of “example.com”; it demonstrates that you control www.example.com.
If you want to prove ownership of a second level domain you must do it through a record in DNS, or through demonstrating control of something that is publicly known to control the domain such as the administrative contact emails.
This really is a solved problem in the PKI space; they should have borrowed that rather than invent their own.
The "primary" way doesn't really matter if a user checks their app and sees that it was verified.
Unless the UI makes it clear it was verified with "non-primary" methods so users can be cautious, any method of verification is essentially "primary" from the user POV.
Everything is moving very quickly, this is just generally the case for anything related to at protocol/bluesky (which is fine, they've been quite busy).
is blueskyweb.xyz an official site for them? I've only seen bsky.social which now redirects to.. bsky.app? (if so couldn't the web stuff live on.. bsky.app?)
edit: bsky.app links to blueskyweb.xyz so I guess so. weird.
Mastodon is a bunch of individual instances of varying power but maybe they should build in something that detects load and archived itself and redirects to the archive.
OK, this is a very cool method of verification for a social network but they goofed. Everyone goofs. In this thread folks being like "this is a bunch of jokers", yeah, I think they learned their lesson about rolling their own domain verification and everyone had some fun.
That doesn't work either, it just redirects back to the 429'd page on chaos.social. It's because the admins of the site put this up temporarily to deal with HN load.
Isn't this how over a decade ago people "hacked" Google by hosting a similar file on google sites/blogs when that was a thing? How's anyone still using this method for verification in this day and age :/
It looks a bit comical how Twitter rivals are doing based on this:
-Post points out mistake in BlueSky's tech kind of comparable to "rolling one's own crypto" in concept (not as dangerous but not the kind of mistake one wouldn't expect that profile of an outfit to make)
-Post is inaccessible because it's on mastodon and that specific mastodon instance seems borkd for whatever reason
All after lots of rage about twitter's tech being fragile in the sphere and probably going down any time soon as they fired most of their staff - but it's chugging along fine
I mean I think you could also do this on Mastodon but it would just show a "verified" highlight for amazon s3 in your profile, not that it would get verified as your username.
I don’t know what specifically you’re speaking of, but for the stuff I know of, Mastodon uses WebFinger, which puts the important stuff inside /.well-known/, and .well-known should be blacklisted as a “username” in any of these sorts of systems, for this very reason. (https://www.rfc-editor.org/rfc/rfc8615 specifies the /.well-known/ path prefix.)
Without understanding all the context, why not just serve files directly from your smartphone? You can approve your own contact list and share keys.
Yes, of course bandwidth is a concern, but then again there should be our way to monetize, and that's entirely possible using direct payments which already exist.
A little bit of cashing, CDN, and your phone as the initial file server. Tag a photo or video has shared.
I think this is not about using s3 to serve files, but someone having verified owning s3 on bsky by putting some challenge file in his bucket. My guess, also missing context.
Last night I opened this, saw the HTTP 429 and figured "ah too many requests, I'll check the comments and try in the morning". The comments were all people swooning in shock about why some non-specific they (S3? Amazon? Someone else?) didn't use ".well-known" and others complaining about Mastodon and/or the fediverse. I had to read multiple comments to piece together the story, I swear it was like Elden Ring[0].
What this is actually about: BlueSky is Jack Dorsey's new Twitter clone, it is eventually intended to be some sort of fediverse thing but it's not there yet and it's not the source of the fediverse gripes here. You can authenticate your BlueSky user as the owner of a given domain or subdomain by placing a certain file with a given content somewhere under that domain/subdomain. However that "somewhere" was just a location one of the devs at BlueSky chose, rather than somewhere relatively standardised, like under the ".well-known" path (which you might recognise from things like OpenID Connect where the configuration doc is located @ example.com/.well-known/openid-configuration). So one user exploited this and became the "owner" of that Amazon S3 domain by setting up a storage account on Amazon S3 and following BlueSky's setup instructions. That is the main story here - some non-Amazon rando is now officially the Amazon S3 guy on Bluesky.
The next part is that someone posted about it on this https://chaos.social Mastodon instance, which got overwhelmed, the owners decided to save their server by electing to return a 429 response for that specific post if users don't belong to chaos.social, and that is why people are upset about Mastodon.
Interesting story, but I'm not interested in Dorsey's version of Twitter 2.0 unless it actually allows you to signup[1] and brings something compelling that Twitter didn't and Mastodon doesn't.
[0] - game with an intricate story that does its damndest to not actually tell you. If you want to know the story you have to piece it together yourself by picking up dozens of items scattered throughout the game and reading all their descriptions. Or you can do what I did - watch a video on YouTube.
[1] - they're doing an open beta and letting a little trickle of users on, who post about it on their Twitter/Mastodon/whatever. Feels a bit deliberate, like they're trying to build anticipation and frankly I detest little manipulative things like that so I'm out
> [1] - they're doing an open beta and letting a little trickle of users on, who post about it on their Twitter/Mastodon/whatever. Feels a bit deliberate, like they're trying to build anticipation and frankly I detest little manipulative things like that so I'm out
Frankly this cynicism feels unwarranted. Bluesky is not a finished product — it is still being built and, even with the small number of invited users so far, there have been problems that have needed attention. The moderation story is still being developed, the feeds are still being tweaked, the app still has bugs, federation still doesn't work yet. Having some users makes for a valuable feedback loop but the team would rapidly become inundated and burnt out (and the platform would possibly turn into a wild-west hellscape with irreversible reputational damage) if they were to open the floodgates entirely at this stage.
To clarify - it felt like this was an attempt to replicate the mid-00s play of building interest by restricting who can join and making it exclusive (Facebook did this by rolling it out uni-by-uni, Gmail was for a while invite-only and invites were highly valued) and therefore desirable.
Maybe that's in my head but layering this feeling on top of BlueSky being yet another microblogging service with a few other things that I don't love contributes to my impression of Bluesky being simply "meh". If it becomes the next thing that everyone uses, I'll inevitably have to check it out, I'm not going to be an early adopter though.
1 reply →
> The next part is that someone posted about it on this https://chaos.social Mastodon instance, which got overwhelmed, the owners decided to save their server by electing to return a 429 response for that specific post if users don't belong to chaos.social, and that is why people are upset about Mastodon.
It's like all these newfangled webapps don't understand the concept of caching static pages for anonymous users. There is absolutely no reason that something like this should result in more than one request (plus a handful more for static resources) handled entirely by the frontent webserver's in-memory cache for each person linked from other sides. But instead its all dynamic and the page shoots off more API requests before being able to show anything.
So the thing is that in one respect they actually do get caching, almost to a fault. One of the complaints I've seen among some Mastodon instance operators is that they end up storing some pretty hefty amounts of data locally as their instance caches remote posts, images and profiles from other instances that its members follow. One source of problems, which may have been resolved, was that even though there's a job that cleans out this cache the banner images from external profiles stick around. I saw this a while back and it seems like an easy fix so I imagine it's been addressed.
I don't think I am equipped to diagnose what the root cause was here. It's even possible that this instance wasn't intended to have viral posts (i.e. high profile posts that get would get shared to many external users) and they didn't want to invest in hardware/services to facilitate this.
1 reply →
Archived version of the original Mastodon post: https://archive.is/fM06z
You should still follow Jonty (the poster at that Mastodon instance) wherever your socials are, because he's awesome, and posts about awesome things. He's also the organiser of EMFCamp (https://www.emfcamp.org/), which is a nerd/hacker camping festival.
Thanks, will do! I'm always on the lookout for interesting people on Mastodon!
> However that "somewhere" was just a location one of the devs at BlueSky chose, rather than somewhere relatively standardised, like under the ".well-known" path
I've not looked into BlueSky's domain based identity thing in any detail so I might be missing a point somewhere, but… If someone can manipulate its special location what would there be to stop the same someone being able to manipulate content under .well-known?
Are we just relying on .well-known having some extra protection (in this case by Amazon having created a bucket called .well-known so no one else could)? If so then .well-known is little safer than any other arbitrary location in this respect (because you are relying on every domain owner who might be spoofed to take an action to protect against this, rather than the protocol failing safe if nothing is done by the domain owner) and perhaps using DNS would be better.
> Are we just relying on .well-known having some extra protection [...] ? If so then .well-known is little safer than any other arbitrary location in this respect.
If .well-known had just been invented, that would be true. It's fairly well established at this point, though. For example, if someone can create arbitrary files in .well-known, they are also able to pass http-01 ACME challenges and thus issue TLS certs for your domain (modulo CAA) and MITM you. At this point, allowing users to modify .well-known is about as good an idea as allowing them to receive mail for postmaster@ or accepting incoming packets from 192.168.0.0/16 into your LAN.
Amazon S3 specifically would not be vulnerable because bucket names can’t have dots in them; same for every other service that doesn’t allow those. Neither would services that prefix usernames with ~ or @ or similar, nor services that already use http-01 ACME challenges to get certs thus are already using that path.
I’d be much happier if proving domain control were only done through DNS challenges, but that ship has sailed.
1 reply →
Aaaah! Thanks! Great write up!!
Just wanna share that I found your comment a lot of fun to read, even if I had already pieced the story together from other comments. Thanks!
this is everything I wanted to know and more, all in one comment—thank you
[flagged]
> They should've picked a proper language that's able to perform and handle those requests like Go or Rust. Building a social network, especially distributed, and using a low performance language... what were they thinking?
Even if a Go or Rust variant of exactly the same implementation would have been two or three times faster, it would not have survived the same onslaught. The way to survive that sort of accidental DDoS is not to change language but to improve algorithm choices where possible (where that makes a difference in order, not just a small difference in scale), make sure you are scaling efficiently over the CPU cores (possible in all those languages in various ways) available per node without busting memory limits, and by scaling out once that becomes a limiting factor.
This was effectively a DDoS situation (not an attack, but from the server's PoV there is no difference between a malicious DDoS and a hug-of-death from a mass of interested parties) and using Go or Rust instead of Ruby would likely have made no difference what so ever.
(the rest of your post is not discussing technical matters pertaining to the rest of the thread so I'll not encourage the off-topic-mess by responding in any detail, other than to say that sort of complaint wrt handling potentially contentious issues is common on social networks and not Mastodon specific)
Or they should have just put configured caching of static content. This has nothing to do with Ruby, nor even the architecture of Mastodon (the software) which is not great, but about a server not being set up properly - any static cache or, even better, fronting it by a CDN, will trivially beat the most optimised compiled dynamic content generating framework.
Someone on one server trying to get a user on another server is maybe a bit much, but I don't know the content of the posts in question. For me it'd have to be something pretty extreme to take those actions, otherwise if I saw something that irritated me I'd just consider ignoring or blocking that person depending on the severity.
Remember though that the war is a pretty delicate subject and that a position that from your perspective seems perfectly peaceful could be seen differently by others. So to a German the message "no weapons to Ukraine!" could be seen as de-escalatory, but to a Ukrainian it could be seen as a betrayal or it may remind them of bad faith pro-Z/Putin trolling even if that doesn't describe you or your intent whatsoever.
3 replies →
Here's the original email where I proposed .well-known:
https://mailarchive.ietf.org/arch/msg/apps-discuss/1_a06NU8z...
> 1) I feel that /host-meta is too casual of a name and prone to collisions. It matches /^[\w\-]+$/, which I think is a subset of a fair number of sites' usernames."
...
> i.e. put something ugly and weird in there, like a semicolon, to minimize the chance that it interferes with people's existing URL structure.
And later, how the semi became the dot: https://mailarchive.ietf.org/arch/msg/apps-discuss/j6KWTSTVC...
Fun bit of history!
This proposal made complete sense and the counter argument was of such a tiring kind. I see that from time to time in these old standards mail threads. I can't put my finger on it, but perhaps it is that they only saw the "now", not the future. Where the "now" was time when a webmaster were in charge of every website, and had complete personal control.
Thank you!
Things to learn about the FediVerse from the 429 error:
* The FediVerse is lots of WWW sites. Some are WWW-hosting companies showing off, with all of the acoutrements of high-end WWW sites, including CloudFlare protection and lots of tweaking of the back end stuff. Others are one-person sites where someone has just set up the vanilla Mastodon/Pleroma/Pixelfed/Friendica/whatever software on a cheap hosted VM somewhere. There are lots of in-betweens. I have an account on two sites, at each of the aforementioned extremes, one with well over 20,000 users and the other with around 40.
* It's really easy to deny service to the one-person sites, and many of the low-end ones.
* Chaos.Social's about page explains that it's a couple of people running a WWW site in their spare time on spare hardware. That's a little misleading, as they've upgraded the hardware a bit. But it's still 2 people, with ~5800 users. For more, start at https://meta.chaos.social/resources .
* There's nothing global in the FediVerse. Nothing gets sent everywhere. Some commenters here can see the post cached by their local WWW sites where they have accounts. But I'm in the opposite situation: None of the places where I have accounts have cached that post, and since the Chaos.Social sysop put the 429 error in place to combat the server overloading, they actually cannot pull that post with just its URL entered directly, although simple tricks like searching for @jonty@Chaos.Social instead and reading the user timeline work just fine.
* There's nothing global in the FediVerse. Using the aforementioned trick, I see a different view of the thread from Mastodon.Scot to what I see from Toot.Wales, and both of those are different to what's seen from other places.
Most of us are just browsing for interesting light reading anyway, so blocking us if they can’t serve a Hackernews worth of users seems basically… like an appropriate amount of robustness, for light reading.
Maybe the FediVerse is just not friendly to the idea of a “global top among thousands of users” rating.
Just FYI, that 429 was explicitly placed on the specific URL HN links to. The remainder of chaos.social is up and running perfectly fine.
[Edit: already mentioned above, I just misread it.]
The aforementioned trick wouldn't work if it wasn't. But before the 429 error was put into place the entire site was affected across the board by Hacker News. See https://chaos.social/@ordnung/110312020977014678 .
1 reply →
Thank you for the explanation, that actually makes sense. But I still think that serving a 429 is some kind of backwards old-school sysop kind of response. "It's the right HTTP-Code! hohoho!". It's obviously running nginx and serving a static copy of that url or setting up caching would take the same amount of time as serving the 429. It's 2023, it's possible to serve some thousand requests to static content on pretty much everything now.
So server software is not written with high performance in mind?
I've read somewhere that federation is done via regular HTTP requests which ends up really hogging down servers if someone has a lot of followers.
Mastodon is written in Ruby on Rails and there are some inherent performance issues with that, it generates a huge number of Sidekiq jobs that can bog down a server quite easily. There are other, non-Ruby implementations aiming for compatibility with the Mastodon API though, so I’m curious to see how it will all shake out.
6 replies →
For those not getting the context(like me), this seems to be about the Bluesky Social(https://bsky.app/), a twitter alternative.
Further context: Bluesky lets you use a domain name you own as a user handle.
The official method is to set a TXT record, but apparently their "AT protocol" also lets you confirm a domain by serving `GET your.domainname.com/xrpc/com.atproto.identity.resolveHandle`
and `xrpc` was available as an S3 bucket name :)
Yikes, why didn't they use a /.well-known/ address instead of inventing a new directory? This is entirely on Bluesky, not AWS.
2 replies →
Stunning that there are (were) any 4-char bucket names left.
16 replies →
Sometimes it feels like companies fund weak competitors to discourage / drown out competition.
There's enough imperfection in the world that no conspiracy is required.
1 reply →
Microsoft funded Apple to keep another OS vendor alive, so it's not about discouragement. It's probably a lot cheaper to fund a competitor than paying the gov't and getting tagged as a recognized monopoly
2 replies →
How does Bluesky compare to Mastodon? (Other than letting you register S3 as your user handle)
Here's how I think about it:
* ActivityPub -> AT Protocol (https://atproto.com/)
* Mastadon -> Bluesky (https://blueskyweb.xyz/)
Right now, federation is not turned on for the Bluesky instance.
There are differences in both, however. I'm not going to speak about my impressions of the Mastadon vs Bluesky teams because frankly, Mastadon never really caught on with me, so they're probably biased. ('they' being my impressions, that is, I just realized that may be ambiguous.)
At the protocol level, I haven't implemented ActivityPub in a decade, so I'm a bit behind developments there personally, but the mental model for AT Protocol is best analogized as git, honestly. Users have a PDS, a personal data server, that is identified by a domain, and signed. The location of the PDS does not have to match the domain, enabling you to do what you see here: a user with a domain as their handle, yet all the PDS data is stored on bluesky's servers. You can make a backup of your data at any time, and move your PDS somewhere else with ease (again, once federation is actually implemented, the path there is straightforward though). This is analogous to how you have a git repository locally, and on GitHub, and you point people at the GitHub, but say you decide you hate GitHub, and move to GitLab: you just upload your git repo there, and you're good. Same thing, except since identity is on your own domain, you don't even need to do a redirect, everything Just Works.
This analogy is also fruitful for understanding current limitations: "delete a post" is kind of like "git revert" currently: that is, it's a logical deletion, not an actual deletion. Enabling that ("git rebase") is currently underway. Private messaging does not yet exist.
Anyway if you want to know more the high-level aspects of the docs are very good. Like shockingly so. https://atproto.com/guides/overview They fall down a bit once you get into the details, but stuff is still changing and the team has 10,000 things to do, so it's understandable.
7 replies →
Solution is also on the works like use /.well-known/, so this is more like funny, rather than a big problem.
Key to trick was to have bucket named "xrpc" and store a file there: https://s3.amazonaws.com/xrpc/com.atproto.identity.resolveHa...
There is also another funny thing in the image, the user posting about is sending one from "retr0-id.translate.goog", which is odd. Somehow he has got https://retr0-id.translate.goog/xrpc/com.atproto.identity.re... to redirect to his page, and gotten that handle as well.
Eh, it’s worse than just funny; it’s concerning, because they should have known about and easily avoided this kind of vulnerability, it’s standard stuff you have to think about. So what else have they missed?
This is a private beta. Nobody is suggesting that any of this be used for anything serious just yet. Development happens out in the open, you can go find out what else they've missed by doing the work, or by waiting until others you trust have done so.
I myself have had an account for like a month now, but only started really using it a week ago, because that calculus changed for me, personally.
Like, it's not even possible to truly delete posts at the moment. This all needs to be treated as a playground until things mature.
This isn't even the first "scandal" related to this feature already!!!! There is another hole in what currently exists that allowed someone to temporarily impersonate a Japanese magazine a few weeks back.
29 replies →
For me, the worst thing about it is that they didn't just use webfinger. So webfinger isn't perfect, but it's there and in use. When they choose to invent new mechanisms for things there are perfectly serviceable options for, it makes me instantly sceptical of the rest.
Wouldn't be funny if it was a public beta that they want people to use for serious stuff. But it's neither serious, a beta or public, but basically a private alpha for playing around, so i'd be a bit lenient on screwups.
Google Translate recently moved translated web pages to domains like this. If you plug a webpage into GT it will put the translated content under <domain>-<tld>.translate.goog. This user's actual domain is https://retr0.id
Oof. This will not be the last time that decision causes a problem.
Reminds me of people taking the username “admin” or “hostmaster” at a free email service and being able to get domain verification emails.
Wait - nobody had ever created a bucket named xrpc before, ever? I would have imagined that short s3 buckets were squatted similar to domain names. (Or maybe they were, and it's this person who did so!)
There's an account bucket limit, so you'd need to create a huge number of AWS accounts with no immediate benefit.
> 429 Too Many Requests
Aight, level with me: Is every mastodon server running on a Raspberry Pi?
From the instance admins:
> We just started serving a http 429 error on the exact url of the post. So everything should go back to normal now.
So to answer GP's question, yes.
If you go to the user's profile and then to the post, it seems to be okay. So perhaps also looking at Referrer.
1 reply →
for the time being, archive.org has a snapshot of it: https://web.archive.org/web/20230504185520/https://chaos.soc...
The server itself seems to work fine. It only seems to be this specific post that's being 429'd. I'm guessing it's some kind of anti-DDoS setup kicking in.
Mastodon is also quite heavy to host, my single user instance will easily gobble up several gigabytes of memory if you let it. There are more efficient ActivityPub servers but specifically Mastodon seems to be written for running efficiently on huge servers.
Or running efficiently never maybe?
1 reply →
Not really: hhttps://leah.is/posts/scaling-the-mastodon/ They just got 6 times the normal requests: https://chaos.social/@ordnung/110312089838674624
unfortunately this is exactly why mastodon won't work
I already don't trust mastodon links because 9 times out of 10 they simply don't work. Everyone's tiny hobby server falls over when one post gets big, and obviously not everyone is going to scale their servers to support the load of a viral post that might happen once every 6 months and will be 100x their base load
8 replies →
You can also see the same "toot" as a "tweet": https://twitter.com/jonty/status/1653915932677271552
And the original tooter is apparently Google Translate?
This post is on the front of HN. Many a larger website have succumbed to HN's warm embrace.
Isn't HN pretty small? This post has <400 upvotes over 3 hours. There can't be 1000x that amount of lurkers can there?
10 replies →
Just look at it failing the HN hug of death. If this one can't survive techies on a orange site rushing to the site then it cannot possibly survive a lotus of users from Twitter or TikTok rushing into any post on Mastodon, bringing it flat on to the floor.
I can only see Mastodon centralizing to cope with the load. But a server going down on this load from HN tells us it is no where near ready to handle an insurmountable amount of users or even begin to challenge Twitter which hosts 220M+ users every single day.
Mastodon user count has mostly been a steady growth. So far it hasn't really failed at a high level. We aren't in Bitcoin territory, the network isn't really slower than it was a year ago even if number of user is much higher. It's mostly distributed among many instances.
Twitter is pretty sluggish, to be fair. 7 seconds to load and render a single tweet on mobile.
https://pagespeed.web.dev/analysis/https-twitter-com-realDon...
Mastodon.social is actually much faster on this particular benchmark. So maybe there is hope.
No trouble viewing it from another Mastodon server:
https://hachyderm.io/@jonty@chaos.social/110307532115312279
EDIT: Ah I guess if you're not logged into a hachyderm.io account, you get forwarded. So probably don't use the above link.
That just redirected me to too many requests.
Maybe, but the admin commented it was intentional for that specific post, it was slowing down the entire site.
> slowing down the entire site
This is mind-blowing. Last I checked, the front page of HN sends tens of requests per second to each link. There are humans who can pack envelopes faster than the typical mastodon server can answer GETs. I'd love to see someone benchmark the top servers for a few seconds to see what it takes to break a reasonable latency SLA.
5 replies →
one alt-social crumbling from throwing rocks at another alt-social
This is peak Web 5.0 right here.
Aren’t federated services grand?
[flagged]
26 replies →
Ironically, I've seen pages make it to the front of HN that were run on RPis or weaker.
There was a guy running a site off of a single core 32bit ARM SoC that was able to handle the HN frontpage.
chaos.social is run by the chaos computer club, you can assume that they configured it that way on purpose.
my profile, on the same server, loads fine.
This was both funny and ironic at the same time.
Upon clicking reply fastidiously, i got the hn 429
"Sorry, we're not able to serve your requests this quickly."
Wow.
Looks like we hugged it to death
Sites like HN and that other one should track this. It could work like flagging - if enough people mark it as hugged to death (HTD), it would say so next to the link. Maybe it could even redirect to an archive if it’s currently HTD.
lightly blew in its general direction to death
chaos.social is run on four dedicated servers
https://leah.is/posts/scaling-the-mastodon/
Because it is webscale.
This is why mastodon , webfinger and ACME uss .well-known uri prefix. .well-known is reserved and you can't e.g. make a bucket named .well-known
It's funny the bluesky devs say they implemented "something like webfinger" but left out the only important part of webfinger that protects against these attacks in the first place. Weird oversight and something something don't come up with your own standards
> This is why mastodon , webfinger and ACME uss .well-known uri prefix
This is not how Mastodon does verification (at least not the main method). Mastodon doesn't just link users -> domain. It can link user -> webpage, for example to link social profiles between sites.
If you have a website with user generated content, and a user can set an arbitrary html attribute (rel="me") in a link pointing back to their profile, they can claim ownership of the page on Mastodon. Likewise, if they can set a link tag in the head element of the page for some reason.
Presumably this is somewhat harder to exploit than a (new, poorly thought out) dependency on a static file under /xrpc, but Mastodon does introduce more authentication footguns for sites than just .well-known! https://docs.joinmastodon.org/user/profile/#verification
Edit: authentication -> verification, since Mastodon distinguishes between the two (see below)
Neither of these are 'authentication'
You're thinking of how Mastodon does verified links. You could do something similar, provide a verified link on your profile to a file in an S3 bucket, but there's very utility (or risk) in that.
Mastodon also allows you to be discoverable via a custom domain, using .well-known as parent mentioned https://docs.joinmastodon.org/spec/webfinger/ https://www.hanselman.com/blog/use-your-own-user-domain-for-...
2 replies →
What about serving the challenge file from the root or a near-root of the fully qualified url? Like www.domain.com/mastodon.txt or abc.freehost.com/mastodon.txt?
Maybe I'm old but what are some popular use cases for webfinger? (I'm just learning about it now)
The /.well-known/ path prefix is the standard name to use (https://www.rfc-editor.org/rfc/rfc8615) so that any sort of “we’ll host user content from our domain” thing can block it. (Hosting user content from the user’s domain is fine and doesn’t need this restriction.)
A few things are effectively grandfathered in due to their vintage: /favicon.ico, /sitemap.xml and /robots.txt are the three that occur to me—so if you’re running something vaguely like S3, you’ll want to make sure users can’t create files at the top level of your domain matching at least those names.
But nothing new should use anything other than /.well-known/ for domain-scoped stuff, or else you run into exactly this problem.
12 replies →
That is basically the idea of .well-known
Webfinger is when you want to multiplex multiple identities on a single domain
E.g. https://example.com/.well-known/webfinger?resource=nick@exam...
Will serve the challenge proving your handle is @nick@example.com
.well-known is basically the same idea https://datatracker.ietf.org/doc/html/rfc5785
Or why not just serve it from www.domain.com/.well-known so we only have one thing to block. :p
s3 supports my-bucket.s3-us-west-2.amazonaws.com style URLs as well
1 reply →
I use S3 for a simple app landing page and I use .well-known for deeplinks... Unless I'm misunderstanding your comment which I probably am.
Yes, but you don't have access to s3.amazonaws.com/.well-known, only to yourdomain.s3.amazonaws.com/.well-known.
1 reply →
Nostr too
.well-known seems unintuitive
Also the penalty isn't very high here. Someone impersonated a domain on a burgeoning protocol for a short while. So what?
> .well-known seems unintuitive
We're talking about folks setting up a custom domain for a personal social media presence. If they can handle nameservers and DNS records, they can handle a folder with a dot in the name.
6 replies →
This is the second time one of my posts has caused issues for the chaos.social admins. I am so, so sorry.
The hacker news DDOS is real.
Previously: https://news.ycombinator.com/item?id=34691489
The interesting thing to consider is that someone who is popular to read may well be as much of a headache for the local sysop as someone who is a frequent target for attacks. How long until they have a 'bot that just 429s all of your posts on sight, do you think? (-:
On a more serious note: Given that many user communities in the FediVerse are people conscious of their privacy and of suffering pile-ons by large majorities of outsiders, I wonder how many more examples of this there will be before people start asking the developers for out-of-the-box defaults that simply blacklist all requests that come in with referrer headers containing news.ycombinator.com and other similar "Slashdotting" sites.
It's not beyond the bounds of possibility. Compare https://github.com/mastodon/mastodon/issues/15431 .
1 reply →
Everything has happened before and will happen again: https://www.wired.com/2015/11/how-instagram-solved-its-justi...
Sorry for what, offering a free lesson in cache optimization? :)
This is a terrible implementation of domain verification. dns-01 and http-01 are more or less standardized at this point. Use them, and don't roll your own. Reference: https://letsencrypt.org/docs/challenge-types/.
They definitely should have used HTTP-01 if they’re doing verification on the web, but since this is about using a domain as identity this really belongs in DNS.
The issue with DNS-01 (and HTTP-01 to a lesser extent) as someone else mentioned is that the user friction is really high.
I’ve been working on a solution to this that I’ve been meaning to post to HN and this seems like as good an opportunity as any so here it is: [1]
It’s a method of storing a hashed (and optionally salted) verifiable identifier (think email or mobile) at a subdomain to prove authority for a domain.
1. https://www.domainverification.org
> this really belongs in DNS.
And the primary way of identifying yourself is in fact DNS.
> I’ve been working on a solution to this
Your solution is almost identical to the BlueSky one: put a TXT record at _atproto.<domain> that resolves to a DID. The difference is that they mandate the DID spec and you do not. Which is totally fine! Just figured I'd let you know :)
1 reply →
I don't get http-based verification in general. If you want to really prove someone owns a domain, make them change an authoritative DNS record. Everything else feels like it is begging for edge cases to crop up. Why should my social media or SSL certificate vendor care about my web servers?
I worked on a product that required DNS changes to set up. Especially for business accounts, the level of friction was STUNNING. We had it take months to get set up because the contact had to submit a ticket to IT, write up the business justification, get director level approval, get security approval, and so on before it could get done. We had customers who couldn't even figure out which group in their company managed DNS. Yeah, you can argue that those companies are broken, but as an outsider I have no influence over that. The result was just that they couldn't use our product. On the flip side, we had consumer and small business customers who had purchased domains through simple webhosting things that didn't give them the required level of access to create a record (and/or they couldn't figure out how to do it). We eventually added an HTTP option and the success rate and time to success both improved hugely.
15 replies →
Both http and dns verification are stupid. Neither of them prove you own the domain.
http verification proves you temporarily control IP space relative to a viewer. dns verification proves you temporarily control name resolution relative to a viewer.
Both are trivially hacked, multiple ways. By the time someone finds out you did it (if they closely monitor CT logs, which nobody does) you've already had hours, days, weeks to run a MITM on any domain you want. The attack only has to work once, on any of 130+ CAs.
The solution is registrar-level proof. Cert request signed by the private key of the domain owner, sent to the registrar to verify, the registrar signs it if its true, it's sent to the CA who can see the registrar signed it. Now you know for a fact the domain owner asked for the cert. The only possible attack is to steal all three of the owner's private key, the registrar's private key, and a CA's private key.
I have been shouting about this for 10 years, none of the industry incumbents care. The internet is run by morons.
11 replies →
> If you want to really prove someone owns a domain, make them change an authoritative DNS record.
You're not wrong (ignoring how easy it is to hack DNS), but at the same time it's hard enough to get people to buy their own domain name, nevermind understand the system well enough to add a TXT record.
It's a strategy that's fine to implement when your target audience is server admins. It's a terrible strategy when your target audience is everyday users who you hope own their own domain. Doubly so in a world where owning your own domain is so rare for individuals.
Convenience. DNS is routinely not automatable by API, or inconvenient to automate. HTTP, however, is normally easy to work with.
1 reply →
Sure, adding a TXT record to verify domain ownership is fairly common and lots of tools still use it. But you either have to self host DNS (yet another container to maintain) or use your provider's API (yet another credential, yet another mailing list to subscribe to for inevitable breaking changes to the API).
In contrast, HTTP based verification often has built-in support with your webserver (Caddy) or only requires copy-pasting a few lines to your docker compose file.
There are edge cases, but they're also widely exploited so you won't run into them if you follow best practices.
If you control a domain's DNS entry but I can serve arbitrary content to users from its servers, who really owns the domain?
I think people don't want to put DNS admin credentials in places where they might get leaked. Would be cool if a DNS server or provider offered credentials that could only do ACME challenges and not change any other records.
2 replies →
DNS challenge is required for wildcards on LE at the very least.
But the reason for HTTP is pretty simple - it's extremely easy to implement. You only need to tell your ops to redir a subdomain to your app and you're done, you don't need DNS with API that have narrow enough permission to allow that one team in whole company to generate ACME stuff; most providers ACLs on DNS end at "this client have acesss to that domain via API".
It's not about proving ownership, if it was about proving ownership we would do this via something at the registrar level.
It's about proving /control/. If a domain name is pointed to me (my IP/CNAME) I control it and it is reasonable to allow that person to issue an SSL certificate for a domain (or subdomain) under their control. If you, as the domain owner, want to restrict that, CAA exists as your tool to do so.
HTTP-01 works really well for when you host a custom domain on a SaaS application. The domain ownership stays with the customer, and all they have to do is CNAME/ANAME to your server. No messing with DNS TXT or NS records, 3rd party DNS provider API keys, etc.
Seems like the crypto crowd is moving on to creating flaky decentralized twitter clones now
Bluesky was started within Twitter as a federated social protocol in 2019. Spun out in 2021 as a standalone company and it was also being used in the Crypto group at Twitter until Elon came in.
I'm sure just because of its age and principals involved it's been heavily influenced by the crypto crowd.
But suddenly, without anything else really stepping in to fill the void, it's the Twitter alternative of the day. Anyway not surprised to find some gotcha's like this all things considered.
> I'm sure just because of its age and principals involved it's been heavily influenced by the crypto crowd.
It builds off of several specifications that came from the crypto crowd. It does not use a proof of stake or proof of work blockchain, though, so depending on how you use the words "crypto" and "blockchain," it either is or is not those things.
2 replies →
[flagged]
[flagged]
[flagged]
It got hugged to death too. Icing on the cake.
2 replies →
[flagged]
I have no idea what this means. Can someone explain?
If you can upload a custom file to a domain/subdomain, bluesky social (Jack Dorsey's new twitter) uses it to verify you are the owner of the domain. Chaz uploaded his custom file to their Amazon s3 bucket and now since he was the first one to do it, his account is now associated with Amazon S3.
It's ridiculous that this is not in the title.
11 replies →
>Chaz uploaded his custom file to their Amazon s3 bucket
their is who exactly? and why does bsky associate it with the s3 domain if it's just a file in a random bucket?
1 reply →
I would argue this is worse (and more hilarious) than Musk buying and giving out checkmarks for people.
I did over here https://news.ycombinator.com/item?id=35820670
Any domain on the public suffix list should just be ignored I suppose. https://publicsuffix.org/list/
It'd be nice as an extra precaution, but please don't build things that rely on the Public Suffix List for security (this list by its nature is only a laggy incomplete approximation of the actual use of domains).
bluesky dev here. whoops.
as others mentioned, not a hard fix.
What is the easy fix? Use the .well-known/ standard instead of the current mechanisms, and roll back verification for anyone who’s already been verified with the flawed approach?
Hilarious though! I'm guessing this is the kind of stuff the Beta was supposed to find. Any other cool/funny bugs y'all have found?
I created a rich text system for posts to handle things like links and mentions with the eventual goal of it being the basis for all kinds of rich text (bolding, italics, spoiler tags, etc)
the flexibility bit us on the butt. people started faking mentions via the APIs and one user figured out he could pack 1000 mentions into one "@everyone" and cause us all to get notified. pretty predictable in hindsight but I dropped the ball there
1 reply →
Do you have a public test suite so we can see how solid your protocol is?
Can I ask what's your tech stack?
[flagged]
(not paul)
Given that the AT Protocol is not compatible with ActivityPub, I don't see how the first step, let alone the second or third, is an accurate description of the dynamics at play here.
> And also how long until Twitter 2.0 or X app uses AT
Ironically, pre elon-buys-twitter, it did in fact look like Twitter was going to end up implementing AT, if all went to plan. But then post acquisition, if anything, that looks less likely.
There is something really endearing about this. It takes me back to those days where we were building something new and bugs just popped up and we had fun with them, doing silly things and then fixing them. Morale was high, team was fun, we were all learning and building something new.
If you want to prove domain ownership, you have to do it at the domain level.
The ability to serve a file “www.example.com” in no way demonstrates ownership of “example.com”; it demonstrates that you control www.example.com.
If you want to prove ownership of a second level domain you must do it through a record in DNS, or through demonstrating control of something that is publicly known to control the domain such as the administrative contact emails.
This really is a solved problem in the PKI space; they should have borrowed that rather than invent their own.
As said multiple times in this thread, the primary way of identifying yourself in this protocol is a TXT record in DNS.
The "primary" way doesn't really matter if a user checks their app and sees that it was verified.
Unless the UI makes it clear it was verified with "non-primary" methods so users can be cautious, any method of verification is essentially "primary" from the user POV.
1 reply →
Unless it's been recently updated their help article only lists TXT record for verification https://blueskyweb.xyz/blog/4-28-2023-domain-handle-tutorial
Everything is moving very quickly, this is just generally the case for anything related to at protocol/bluesky (which is fine, they've been quite busy).
is blueskyweb.xyz an official site for them? I've only seen bsky.social which now redirects to.. bsky.app? (if so couldn't the web stuff live on.. bsky.app?)
edit: bsky.app links to blueskyweb.xyz so I guess so. weird.
Mastadon has no chance if every time something becomes a little bit viral it's instance dies over the traffic.
Mastodon is a bunch of individual instances of varying power but maybe they should build in something that detects load and archived itself and redirects to the archive.
Let’s be honest… Mastodon was never going to make it outside niche technical circles, regardless of scaling. It’s a fun project though.
[flagged]
OK, this is a very cool method of verification for a social network but they goofed. Everyone goofs. In this thread folks being like "this is a bunch of jokers", yeah, I think they learned their lesson about rolling their own domain verification and everyone had some fun.
It's not just this incident, like not reading your own ToS to know what is in there, they seem to be sloppy
Good thing nobody uses bluesky, this would be really awkward otherwise.
429 Too Many Requests - hackernews hug of death?
https://archive.is/fM06z
Here is a link that (currently) doesn't break: https://mastodon.gamedev.place/@jonty@chaos.social/110307532...
Original has a 429, alternative link for this post thanks to the Fediverse!
https://mastodon.social/@jonty@chaos.social/1103075321453803...
That doesn't work either, it just redirects back to the 429'd page on chaos.social. It's because the admins of the site put this up temporarily to deal with HN load.
Isn't this how over a decade ago people "hacked" Google by hosting a similar file on google sites/blogs when that was a thing? How's anyone still using this method for verification in this day and age :/
The same mastodon post was also posted to twitter:
https://twitter.com/jonty/status/1653915932677271552
It looks a bit comical how Twitter rivals are doing based on this:
-Post points out mistake in BlueSky's tech kind of comparable to "rolling one's own crypto" in concept (not as dangerous but not the kind of mistake one wouldn't expect that profile of an outfit to make) -Post is inaccessible because it's on mastodon and that specific mastodon instance seems borkd for whatever reason
All after lots of rage about twitter's tech being fragile in the sphere and probably going down any time soon as they fired most of their staff - but it's chugging along fine
Does anyone have a screenshot? I'm getting a 429.
https://web.archive.org/web/20230504185520/https://chaos.soc...
Thx. Also. Yikes!
https://archive.is/fM06z
Yet another link to a mastodon instance that can't handle more than 10 concurrent users...
They'll just commandeer it later...
This is absolutely hilarious.
Am getting a 429 error. Can someone please fill me in on what was here?
Kiss of death
They switched it to make the page redirect to the Web Archive for now.
I see just error 429 can someone please explain?
They switched it to make the page redirect to the Web Archive for now.
429 too many requests
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
It's not that hard to modify code.. and I'd argue it's new enough that people should expect growing pains if they want to be early adopters...
"Death" of a product for one bug? ....
This was a simple thing, the fact they got it so wrong only makes me wonder what other bad choices have been made.
1 reply →
I mean I think you could also do this on Mastodon but it would just show a "verified" highlight for amazon s3 in your profile, not that it would get verified as your username.
I don’t know what specifically you’re speaking of, but for the stuff I know of, Mastodon uses WebFinger, which puts the important stuff inside /.well-known/, and .well-known should be blacklisted as a “username” in any of these sorts of systems, for this very reason. (https://www.rfc-editor.org/rfc/rfc8615 specifies the /.well-known/ path prefix.)
No, you can’t. To own s3.amazon.com the HTML on that exact domain and not a sub path needs to have a specific link back to your profile on it.
Without understanding all the context, why not just serve files directly from your smartphone? You can approve your own contact list and share keys.
Yes, of course bandwidth is a concern, but then again there should be our way to monetize, and that's entirely possible using direct payments which already exist.
A little bit of cashing, CDN, and your phone as the initial file server. Tag a photo or video has shared.
Really it's just an RSS feed and CDN.
I think this is not about using s3 to serve files, but someone having verified owning s3 on bsky by putting some challenge file in his bucket. My guess, also missing context.
That is correct.
1. Bluesky allows you to use a domain as a handle by creating a TXT record on an _atproto subdomain of the domain you wish to use (see https://mxtoolbox.com/SuperTool.aspx?action=txt%3a_atproto.s... for mine)
2. You can also serve up your DID by having the URL "https://<handle>/xprc/com.atproto.identity.resolveHandle" return the DID.
3. AWS buckets have the URL structure http://s3.amazonaws.com/[bucket_name]/
4. register "xrpc" as an S3 bucket, drop a file named "com.atproto.identity.resolveHandle" with the correct JSON in it
5. boom! your username can now be s3.amazonaws.com
Hope that helps.
5 replies →
Like Pied Piper? If yes, there is an entire show on it.