Comment by Jenda_
3 years ago
> Ultimately however crt.sh is hosted somewhere
So is it possible to run my own copy of crt.sh? How demanding it is (e.g. data size)?
3 years ago
> Ultimately however crt.sh is hosted somewhere
So is it possible to run my own copy of crt.sh? How demanding it is (e.g. data size)?
The crt.sh code is all open on GitHub, so can be hosted yourself. Last I checked a few months ago, the main ‘certificates’ table and indexes etc was close to 20TB, and there’s more than just that. It’s big, but has everything. A slimmed down database of just some lightweight info on the certs and issuer (notBefore, not after, SANs, serials etc) runs maybe a TB in bigquery.
A log monitor? Or specifically the crt.sh codebase?
Yes, the logs are public and I ran equivalent monitoring for a previous employer before the pandemic. You will need to consume records at the rate they're created to keep up, Let's Encrypt have stats you can look at, if you can cope with twice their typical daily throughput you'll be fine on average, but peaks will swamp you temporarily so design for that.
You can choose whether to store everything, or just stuff you consider interesting, and you can choose whether to care forever or only until expiry (so 398 days)
If you want everything (full certificates), indefinitely, that's a lot of data. Um, several terabytes per year maybe?