Comment by LinuxBender
3 years ago
This sounds like the methodology for blockchain and multiple ledgers. Ultimately however crt.sh is hosted somewhere and while there may be multiple controllers that have access to logs on the front-end, someone hosting that site could be compelled or blackmailed into tinkering with the levers behind the scene to exclude activity on a domain. I'm not suggesting that is what is happening, just that it could and Apple, Google and others would have plausible deniability. On the other hand having active probes distributed around the world on multiple ISP's looking for fingerprint changes would be much harder to hide though more expensive to operate along similar lines to archive.is or using distributed Nagios NPRE agents or using ThousandEyes probes.
> Ultimately however crt.sh is hosted somewhere
So is it possible to run my own copy of crt.sh? How demanding it is (e.g. data size)?
The crt.sh code is all open on GitHub, so can be hosted yourself. Last I checked a few months ago, the main ‘certificates’ table and indexes etc was close to 20TB, and there’s more than just that. It’s big, but has everything. A slimmed down database of just some lightweight info on the certs and issuer (notBefore, not after, SANs, serials etc) runs maybe a TB in bigquery.
A log monitor? Or specifically the crt.sh codebase?
Yes, the logs are public and I ran equivalent monitoring for a previous employer before the pandemic. You will need to consume records at the rate they're created to keep up, Let's Encrypt have stats you can look at, if you can cope with twice their typical daily throughput you'll be fine on average, but peaks will swamp you temporarily so design for that.
You can choose whether to store everything, or just stuff you consider interesting, and you can choose whether to care forever or only until expiry (so 398 days)
If you want everything (full certificates), indefinitely, that's a lot of data. Um, several terabytes per year maybe?
this is a very very silly take. Apple and Google have armies of lawyers who spend all day telling cops with subpoenas from all around the world to fuck off.