So they optimized from Vec to Box, but they're still using Box all over and spending 16 bytes on it? The things they're boxing need 2 bytes for length, and their memory use is low enough that they could cram the pointers into 4 bytes. Trying to pack that into 6 bytes is probably too much fuss for the benefit, but I see no reason to use more than 8 bytes.
Produce working product first, validate the idea, stabilize the business, start generating profit, and then you can start optimizing your costs.
In fact optimization is by far the easiest part of the process because there are many system programming experts on this HN thread who consider these optimizations to be trivial.
Its a yes if you do not know the domain space, query patterns well enough and also if the cost of optimization or time for optimization may have detrimental impact to business. In this case it most likely means that the crowd in the room did not anticipate much on this in early phases and no one in the room pointed these things out. The irony is that these performance and disk numbers are heavily discussed as a part of system design interviews.
> In fact optimization is by far the easiest part of the process because there are many system programming experts on this HN thread who consider these optimizations to be trivia
This is a misconception when you including roll out as a part of the change too, changing data once its running in production is hard, changing the data structure is even harder and when you talk about making changes in cache which is at the hot path its probably the hardest. Looking at the graph at the end it looks like it took them 4+ months to roll out the changes after optimization.
You're never going to get promoted with that attitude!
I'm joking...but not entirely. It sounds impressive on a promo packet when you say you've saved 100 TB of RAM / $$$ through whatever technique. But it sounds a lot less impressive when you say if this system grows to this size in x years, I will have saved 100 TB, especially when no one yet knows how large the system will really be in that time or what the cost of RAM will be. I dunno, maybe if you say that x years ago, I made a decision that now is saving us 100 TB, that's kinda impressive, but you're also getting credit for it x years after you did the work. It also doesn't have the implication that you did better than some other smart person who chose the other way. And there is a bias to care more about recent accomplishments. So I don't really think it'd be valued the same at all.
Also, in general big tech (at least Google) prefers growing the userbase over improving efficiency. Periodically efficiency is rewarded, e.g. when RAM cost suddenly balloons or some big must-have feature has suddenly used up capacity planned for something else. You get rewarded for doing efficiency work on demand, not eagerly.
I once got a $100 peer bonus for finding 100,000 cores that were essentially stranded by an accounting error in another team's migration script.
This assumes that you have plenty of cash to burn in the process, which is approximately correct for VC-backed ventures, and for offshoots of large corporations that play a lomg game.
> start generating profit, and then you can start optimizing your costs
Good thing they jumped on that as soon as they were profitable instead of burning cash. Oh wait...
I think a distinction to draw here is that Cloudflare had relatively large capital raises and were almost immediately profitable¹. They had the luxury of throwing away money. Judicious optimisation makes sense for scrappy start-ups, especially when trivial optimisations like these could easily be farmed off to an agent.
This reasoning is largely centered around the runway being finite. You obviously can't have costs so high you are making a huge loss, but also there's little value in improving margins past profitability until you actually have a stable segment of the market.
Looks like they're missing the obvious optimisation of putting the record data right after the CacheEntry members instead of allocating memory separately though. But that might just be me as a C-programmer talking and not be all that easy in Rust.
For the curious, this is technically possible in Rust using a dynamically sized type [1], but in practice is difficult and doesn't really play nice with the rest of the language. The nomicon entry concludes with "Yes, custom DSTs are a largely half-baked feature for now." [2]
Unfortunately, Rust is not a good choice for this kind of tricks. This is where Zig shines. In Rust, you can’t even use proper arenas, which can help a ton with allocations.
Cloudflare started to pick Zig recently, for projects, that have memory constraints.
You definitely can and this is done a lot. What you might mean is that you can't use standard library's collections with them (this is getting stabilized soon!) and have to use third-party, but that is a different thing than "can't use arenas".
> Rust is not a good choice for this kind of tricks.
Rust can do those tricks, but it's true that it is hard than in C or Zig. That said there are often crates to help.
Depends on how the CacheEntry is stored, it's probably stored in a slice of &[CacheEntry] which precludes storing the record data alongside it as the size of each entry must be fixed.
I wish more programming languages implemented record types as seen in databases, where dynamically sized fields are packed into a contiguous area of memory.
The CloudFlare manually implemented a clumsy version of this.
Wouldn’t it be nice for the compiler to manage this for you in the same way that your database engine does when it saves a “row”?
> dynamically sized fields are packed into a contiguous area of memory
Are you able to explain this? Do you mean an N sized array where each entry is either a value or a pointer to a value where the 'pointed-to' values are after the end of the array?
I'm trying to underatnd how you'd do this without having to parse M-1 elements to get the Mth entry if you did a [{size0, value0}, ....., {sizeN, valueN}] arrangement
With my own MaraDNS, I aggressively optimized the memory usage of blacklist entries by having a single really big malloc() to allocate the memory for the entries, then traversing that memory block for potentially blacklisted entries.
When I was using one malloc() per entry, a large blacklist took up 237 megabytes of memory. The same blacklist, once optimized to be loaded with a single malloc() call, only took up 9.5 megabytes of memory.
These seem like some fairly standard approaches for reducing memory usage. I can't help to think that the approach of joining several distinct list into a single one in some way undercuts Rust's safety guarantees.
If you previous had three distinct Vec objects, then Rust would guarantee that you can't index out of bounds. If you now put all those objects into a single Vec and rely on offsets, then you now open the door to indexing out of range of these sub-slices without any panics.
It's a minor point, and it doesn't really invalidate the optimization, but I'm surprised the article didn't mention it.
I think it’s more of a time vs code tradeoff, if done properly.
For example in the Vec case, you could theoretically build an alternative which encodes the “three sections” property internally, and ensures correctness at construction time for the pointers. Not as completely safe as a Vec, but you can still get similar benefits for the “business logic”.
But I agree, just having a custom structure that does not provide a safe wrapper around this would be sacrificing standard guarantees.
It's the exact thing Rust is made to protect against, on a more local scale. Every memory corruption bug is just an out-of-bounds index that wasn't protected against.
Sure, and usually one of the ways Rust serves us is with safety guarantees.
Which isn’t to say this optimization is a bad idea, just to say it’s sort of a straw man to imply coding in Rust to take advantage of safety guarantees is “serving Rust”
Not sure what they use to hold the cache key and entry. If a hashmap is used, then a radix tree (adaptive radix tree) would be better in saving memory space. Most of content of the qname field of the CacheKey is hostname, like www.site.com. The reverse version com.site.www fits nicely in navigation path of a radix tree. The common prefixes like "com." are shared and compressed in the parent nodes of the tree.
Even a BTree with compressed prefix keys can save space in the qname.
Why do people seem to think that optimization is something you only have to deal with once the software scales so much that 100s of TB of memory or disk space (or thousands of hours of processing time) are being wasted.
It is almost like nobody even thought during the design phase about what might happen down the road.
This is why so much software is bloated and often buggy. Just gets something that half-way works out the door ASAP and worry about the rest later (too often, never).
Funny thing about cloudflare. I have a dns warming script that uses their top 1k or 10k addresses. Then when my master starts up it warms the entire cache. Everything else uses memcache so the cluster is nice and toasty. As far as I can tell no one else releases domain statistics like them.
I've run into issues with using public wifi when I override my MacBook's DNS server to 1.1.1.1 or 8.8.8.8. I believe this is because captive portals require custom resolution of the name captive.apple.com. And external DNS servers will not resolve that correctly to the local gateway's authorization page.
AFAIK (at least it worked like that some 10 years ago) the captive portal just intercepts the HTTP page load and inserts its own content (most often a 302). So it just has to be a http web page. Firefox uses http://detectportal.firefox.com/canonical.html
Edit: ah, yes, DNS can be hijacked too (requires intercepting outgoing traffic on port 53 therefore incompatible with DoH), that may require fewer computing resources. Still need http otherwise the server cannot use the correct cert chain.
Edit 2: Wikipedia says both methods are used: https://en.wikipedia.org/wiki/Captive_portal and also mentions RFC 8910. I suspected something like that existed, hence my initial disclaimer.
My point was: that domain is not treated any differently from other domains.
I've had reliable success by using http://neverssl.com to force a basic HTTP connection for kickstarting a public WiFi portal login, although I have to disable NextDNS (iOS) too.
Can we take a minute to appreciate how utterly broken this state of affairs is? The dogged over centralization of DNS is an endless source of problems.
General theme: A programming language's native in-memory object format is typically optimized for random access, uniformity, and mutability (fields at fixed offsets, etc). Serialization formats for network or disk tend to be designed explicitly to be more compact. But you can design your own in-memory representation too, with the properties you need.
That’s the old school of thought. These days, designers of newer serialization formats realize that designing a more compact format doesn’t really buy much on modern CPUs and modern networks. See for example Cap’n Proto (whose inventor, kentonv, also works at Cloudflare) and flatbuffers.
this applies to more than DNS caches. In 1998 I mailed Microsoft a proposal to replace search engine crawlers with a push-based filesystem monitor (detect change → extract → compress → push to index). Got a 5-line rejection letter. They built the same thing 20 years later as IndexNow. Full story with the original letter: https://dev.to/andrew_vl/in-1998-i-proposed-push-based-searc...
The Record struct contains rtype and data where RecordData is a tagged union. Aren’t those two always in sync? Not a DNS expert, just wondering if this is redundant or there is a reason both are there. Doesn’t matter anymore if they store it already serialized but I would be interested why it was this way.
The most interesting result to me is that the richer parsed representation was not necessarily the faster one. If the hot path is mostly “read from cache and serialize back to DNS,” parsing everything upfront only to serialize it again can become unnecessary work and hurt locality....
Frankly weird that they were resorting to high level containers for this in the first place. Also, this line struck me as odd
> Big Pineapple uses jemalloc, an allocator designed for multithreaded, allocation-heavy workloads.
jemalloc multithreaded performance is actually poor(ish) compared to other modern allocators, which makes it a weird choice. But even weirder is why they're even using an allocator in the first place compared to a va MAP_ANON | MAP_NORESERVE arena carveout approach? You can also do punning that way too, which I'm not even certain if Rust supports?
I would also have instinctively reached for a large VM reservation to exploit demand paging. I have used that pattern a lot in C++ but not in Rust, so I don't know how difficult it would be to implement there.
Obvious question: why wasn’t this done earlier? It looks like all the data was already available. At THAT scale, reducing memory usage is a must-have, not a nice-to-have. Weird.
Cloudflare talks about having datacenters in 300+ cities. Presumably they have at least a few servers per datacenter. They saved 130 servers worth of memory... not even the minimum number of servers they have (seriously though, they probably have a LOT of servers)... a few GBs of memory per server running the service. At that scale this is a nice-to-have.
You start, get the type & length, and then that is how many bytes you read.
Some issues with that when you deserialize, from a raw stream in to `[u8; 4096]` buffer, the alignment is only guaranteed to be on 1 byte, not 4 bytes.
In practice it is 4 bytes, but if you run those tests with Miri, you'll get yelled at. So the fix there is to declare the buffer with a type that mandates the alignment of the largest type that you're going to be deserializing.
So then you start your buffer as follows: `[u32; 1024]`, and with `slice::from_raw_parts` you get to turn that into `[u8; 4096]` with the expected alignment.
As an exercise I wrote a streaming parser for netlink, the current existing package serializes everything, all at once.
This kind of encoding[0] is ubiquitous in networking protocols. It scales down to small silicon well and enables the receiver to estimate resource requirements or skip parts of a serial byte stream without storing it in memory first. These encodings usually aren't aligned by design.
It's called TLV encoding - tag/length/value. It's very common in all sorts of network protocols and serialisation formats. It allows you to skip unidentified tags. Sometimes, like in the PNG file format, there's a fixed bit in the tag that tells you whether it's safe to skip or if you have to reject the whole thing because you don't understand this tag.
Hey dang can I get my rate limit turned off pretty please?
One question the article doesn't answer is: why are they cacheing at all? If your cache is that big it isn't a cache. How much bigger is the dataset in question? There are 250 billion entries. Assuming 80/20, that implies 1.25 trillion records?
What's the speed of service/response time relative to the data source?
At that point it might be enough to replace your multiple caches with fewer in-RAM databases?
Maybe I'm misunderstanding, but this powers 1.1.1.1, it doesn't front an internal dataset. A cache miss hits a nameserver. Which is to say, the dataset is "every DNS record in the world"
I think the question is probably more along the lines of - why not do a database with 100 TB of storage/records instead of a cache? tomato / tomato.. especially with smart caching in front of database. 100TB of flash is a good bit cheaper than 100TB of memory
You have to cache, cloudflare doesn't know all the records ahead of time, they have to do recursive lookups to the authoritative servers that own the records and that is only good for the period of the TTL of the record. There is no "global" DNS record database or something like that.
It's a recursive resolver. The global DNS dataset is not something you could collect to serve directly vs caching from observations.
The data source is authoritative name servers operated by third parties, some of which are slow on their own, some of which are behind slow or lossy networks. Origin response times vary between probably 1 ms and 2 seconds +/- origins that never respond.
The simple answer is that if you didn't cache, DNS traffic would skyrocket, and the load would pile up on the authoritative servers, which were intended to be small, and during the early days of the Internet, were frequently on bandwidth-constrained links.
DNS is designed to distribute query load to the edge as much as possible, and that's enabled by caching. It just so happens that "the edge" is now becoming concentrated among a small set of providers because they wanted to make a business out of it.[1] They knew that this would be expensive going in, though.
[1] Nobody has to use 8.8.8.8 or 1.1.1.1. Most people can use their ISP's cache or a local cache instead without any noticeable difference in behavior.
The problem is there is a noticable difference in behavior because the ISP cache is overloaded so queries take longer. Sure, that's not everyone's experience, but there's a reason people chose to use alternate servers.
Rule 1. You can't tell where a program is going to spend its time. Bottlenecks occur in surprising places, so don't try to second guess and put in a speed hack until you've proven that's where the bottleneck is.
Rule 2. Measure. Don't tune for speed until you've measured, and even then don't unless one part of the code overwhelms the rest.
Rule 3. Fancy algorithms are slow when n is small, and n is usually small. Fancy algorithms have big constants. Until you know that n is frequently going to be big, don't get fancy. (Even if n does get big, use Rule 2 first.)
Rule 4. Fancy algorithms are buggier than simple ones, and they're much harder to implement. Use simple algorithms as well as simple data structures.
Rule 5. Data dominates. If you've chosen the right data structures and organized things well, the algorithms will almost always be self-evident. Data structures, not algorithms, are central to programming.
> Don't tune for speed until you've measured, and even then don't unless one part of the code overwhelms the rest.
Genuine question, is software performance really linear like that, that one can and should only fight the tightest bottleneck, one workload at a time? Never really sounded right.
It also sounds like the typical sleight of hand where the difficult bit is simply laundered a layer up, in this case the choice of what workload one investigates.
It is often not worth optimising in the early days. You don't know how popular it will become, you might not know how many DNS records you will hold, it was possibly written in an earlier language and ported as-is.
At the point someone queries the 100TB of RAM, then maybe it is worth revisiting but even that has risks. You have to design the migration path, have fallback mechanisms etc.
Premature optimization argument fits right in. Now that memory is up to 10x more expensive it is worth considering optimizing programs with large memory footprint.
Discussing trivial optimizations is a waste of valuable design time. You're never going to "forget" an optimization. The running system will remind you when the optimization is actually needed.
So they optimized from Vec to Box, but they're still using Box all over and spending 16 bytes on it? The things they're boxing need 2 bytes for length, and their memory use is low enough that they could cram the pointers into 4 bytes. Trying to pack that into 6 bytes is probably too much fuss for the benefit, but I see no reason to use more than 8 bytes.
This is the right way to deliver software.
Produce working product first, validate the idea, stabilize the business, start generating profit, and then you can start optimizing your costs.
In fact optimization is by far the easiest part of the process because there are many system programming experts on this HN thread who consider these optimizations to be trivial.
Its a yes if you do not know the domain space, query patterns well enough and also if the cost of optimization or time for optimization may have detrimental impact to business. In this case it most likely means that the crowd in the room did not anticipate much on this in early phases and no one in the room pointed these things out. The irony is that these performance and disk numbers are heavily discussed as a part of system design interviews.
> In fact optimization is by far the easiest part of the process because there are many system programming experts on this HN thread who consider these optimizations to be trivia
This is a misconception when you including roll out as a part of the change too, changing data once its running in production is hard, changing the data structure is even harder and when you talk about making changes in cache which is at the hot path its probably the hardest. Looking at the graph at the end it looks like it took them 4+ months to roll out the changes after optimization.
Or optimize a bit earlier and prevent having to scale out to a bazillion systems.
The way I usually prevent having to scale out to a bazillion systems is never getting more than 10 users.
11 replies →
You're never going to get promoted with that attitude!
I'm joking...but not entirely. It sounds impressive on a promo packet when you say you've saved 100 TB of RAM / $$$ through whatever technique. But it sounds a lot less impressive when you say if this system grows to this size in x years, I will have saved 100 TB, especially when no one yet knows how large the system will really be in that time or what the cost of RAM will be. I dunno, maybe if you say that x years ago, I made a decision that now is saving us 100 TB, that's kinda impressive, but you're also getting credit for it x years after you did the work. It also doesn't have the implication that you did better than some other smart person who chose the other way. And there is a bias to care more about recent accomplishments. So I don't really think it'd be valued the same at all.
Also, in general big tech (at least Google) prefers growing the userbase over improving efficiency. Periodically efficiency is rewarded, e.g. when RAM cost suddenly balloons or some big must-have feature has suddenly used up capacity planned for something else. You get rewarded for doing efficiency work on demand, not eagerly.
I once got a $100 peer bonus for finding 100,000 cores that were essentially stranded by an accounting error in another team's migration script.
You can build foundations that aren't extermely optimal but have future optimisations in mind.
This assumes that you have plenty of cash to burn in the process, which is approximately correct for VC-backed ventures, and for offshoots of large corporations that play a lomg game.
> start generating profit, and then you can start optimizing your costs
Good thing they jumped on that as soon as they were profitable instead of burning cash. Oh wait...
I think a distinction to draw here is that Cloudflare had relatively large capital raises and were almost immediately profitable¹. They had the luxury of throwing away money. Judicious optimisation makes sense for scrappy start-ups, especially when trivial optimisations like these could easily be farmed off to an agent.
¹ https://timeline.www.cloudflare.com/
This reasoning assumes you have access to infinite runway. You don't.
Exactly, and you need to start turning a profit before the end of that runway. Even if that means running code that is suboptimal.
6 replies →
This reasoning is largely centered around the runway being finite. You obviously can't have costs so high you are making a huge loss, but also there's little value in improving margins past profitability until you actually have a stable segment of the market.
we are all perfectly smooth, round, and filled with an incompressible liquid
Every startup is one bet in a Martingale strategy played by the class of people who remain solvent when you bust.
1 reply →
Only if you have loads of capital
> Produce working product first, validate the idea, stabilize the business, start generating profit,
not everybody is so lucky to be able to go in that order? The first part requires upfront capital/investment?
So obviously you start at optimization
[dead]
This is why system programming still matters.
Looks like they're missing the obvious optimisation of putting the record data right after the CacheEntry members instead of allocating memory separately though. But that might just be me as a C-programmer talking and not be all that easy in Rust.
For the curious, this is technically possible in Rust using a dynamically sized type [1], but in practice is difficult and doesn't really play nice with the rest of the language. The nomicon entry concludes with "Yes, custom DSTs are a largely half-baked feature for now." [2]
[1] https://doc.rust-lang.org/reference/dynamically-sized-types....
[2] https://doc.rust-lang.org/nomicon/exotic-sizes.html
> putting the record data right after the CacheEntry members
I assumed they couldn't do that because they're using it with some kind of generic HashMap<K, V>. In that situation, can "V" be dynamically sized?
A dynamically sized "V" would mean you can't have an array of them, which might preclude some hash map implementations.
> All type parameters have an implicit bound of Sized. The special syntax ?Sized can be used to remove this bound if it’s not appropriate.
, which HashMap does not do, i.e. the keys and values have to have a statically known size.
Unfortunately, Rust is not a good choice for this kind of tricks. This is where Zig shines. In Rust, you can’t even use proper arenas, which can help a ton with allocations.
Cloudflare started to pick Zig recently, for projects, that have memory constraints.
> In Rust, you can’t even use proper arenas
You definitely can and this is done a lot. What you might mean is that you can't use standard library's collections with them (this is getting stabilized soon!) and have to use third-party, but that is a different thing than "can't use arenas".
> Rust is not a good choice for this kind of tricks.
Rust can do those tricks, but it's true that it is hard than in C or Zig. That said there are often crates to help.
1 reply →
I'd like to know why I can't use arenas in rust? Especially considering that I have used them before in rust.
1 reply →
System programming always matters. Things are cheap until they aren't one day.
things are cheap until you reach a scale.
2 replies →
Depends on how the CacheEntry is stored, it's probably stored in a slice of &[CacheEntry] which precludes storing the record data alongside it as the size of each entry must be fixed.
This is where hand-rolled intrusive data structures, as are traditional in C, really shine.
less ergonomic, but still totally doable
I wish more programming languages implemented record types as seen in databases, where dynamically sized fields are packed into a contiguous area of memory.
The CloudFlare manually implemented a clumsy version of this.
Wouldn’t it be nice for the compiler to manage this for you in the same way that your database engine does when it saves a “row”?
> dynamically sized fields are packed into a contiguous area of memory
Are you able to explain this? Do you mean an N sized array where each entry is either a value or a pointer to a value where the 'pointed-to' values are after the end of the array?
I'm trying to underatnd how you'd do this without having to parse M-1 elements to get the Mth entry if you did a [{size0, value0}, ....., {sizeN, valueN}] arrangement
With my own MaraDNS, I aggressively optimized the memory usage of blacklist entries by having a single really big malloc() to allocate the memory for the entries, then traversing that memory block for potentially blacklisted entries.
When I was using one malloc() per entry, a large blacklist took up 237 megabytes of memory. The same blacklist, once optimized to be loaded with a single malloc() call, only took up 9.5 megabytes of memory.
https://samboy.github.io/blog/entries/MaraDNS.html#BlogEntry...
Why do I always find interesting new Twitter accounts just as the person is leaving :)
These seem like some fairly standard approaches for reducing memory usage. I can't help to think that the approach of joining several distinct list into a single one in some way undercuts Rust's safety guarantees.
If you previous had three distinct Vec objects, then Rust would guarantee that you can't index out of bounds. If you now put all those objects into a single Vec and rely on offsets, then you now open the door to indexing out of range of these sub-slices without any panics.
It's a minor point, and it doesn't really invalidate the optimization, but I'm surprised the article didn't mention it.
I think it’s more of a time vs code tradeoff, if done properly.
For example in the Vec case, you could theoretically build an alternative which encodes the “three sections” property internally, and ensures correctness at construction time for the pointers. Not as completely safe as a Vec, but you can still get similar benefits for the “business logic”.
But I agree, just having a custom structure that does not provide a safe wrapper around this would be sacrificing standard guarantees.
> I can't help to think that the approach of joining several distinct list into a single one in some way undercuts Rust's safety guarantees.
Not really. You just need to make the underlying fields private and provide methods to get slices to the data you need.
It's the exact thing Rust is made to protect against, on a more local scale. Every memory corruption bug is just an out-of-bounds index that wasn't protected against.
is dangling pointers reuse memory corruption bug from out of bound index?
you could always do a .get into the vector and handle the error, it doesn't necessarily need to panic.
Thank being said in this case it should be impossible to index out of bounds so maybe a panic is warented.
Tools exist to serve us, not the other way around.
Sure, and usually one of the ways Rust serves us is with safety guarantees.
Which isn’t to say this optimization is a bad idea, just to say it’s sort of a straw man to imply coding in Rust to take advantage of safety guarantees is “serving Rust”
Not sure what they use to hold the cache key and entry. If a hashmap is used, then a radix tree (adaptive radix tree) would be better in saving memory space. Most of content of the qname field of the CacheKey is hostname, like www.site.com. The reverse version com.site.www fits nicely in navigation path of a radix tree. The common prefixes like "com." are shared and compressed in the parent nodes of the tree.
Even a BTree with compressed prefix keys can save space in the qname.
Why do people seem to think that optimization is something you only have to deal with once the software scales so much that 100s of TB of memory or disk space (or thousands of hours of processing time) are being wasted.
It is almost like nobody even thought during the design phase about what might happen down the road.
This is why so much software is bloated and often buggy. Just gets something that half-way works out the door ASAP and worry about the rest later (too often, never).
Funny thing about cloudflare. I have a dns warming script that uses their top 1k or 10k addresses. Then when my master starts up it warms the entire cache. Everything else uses memcache so the cluster is nice and toasty. As far as I can tell no one else releases domain statistics like them.
We're finally seeing more appreciation for this kind of engineering. Not everything needs to be solved by throwing more hardware at the problem
I've run into issues with using public wifi when I override my MacBook's DNS server to 1.1.1.1 or 8.8.8.8. I believe this is because captive portals require custom resolution of the name captive.apple.com. And external DNS servers will not resolve that correctly to the local gateway's authorization page.
AFAIK (at least it worked like that some 10 years ago) the captive portal just intercepts the HTTP page load and inserts its own content (most often a 302). So it just has to be a http web page. Firefox uses http://detectportal.firefox.com/canonical.html
Relevant support page, though light in details: https://support.mozilla.org/en-US/kb/captive-portal
Edit: ah, yes, DNS can be hijacked too (requires intercepting outgoing traffic on port 53 therefore incompatible with DoH), that may require fewer computing resources. Still need http otherwise the server cannot use the correct cert chain.
Edit 2: Wikipedia says both methods are used: https://en.wikipedia.org/wiki/Captive_portal and also mentions RFC 8910. I suspected something like that existed, hence my initial disclaimer.
My point was: that domain is not treated any differently from other domains.
I've had reliable success by using http://neverssl.com to force a basic HTTP connection for kickstarting a public WiFi portal login, although I have to disable NextDNS (iOS) too.
2 replies →
Can we take a minute to appreciate how utterly broken this state of affairs is? The dogged over centralization of DNS is an endless source of problems.
That’s a Mac bug if so—it should be always using dumb udp/53 for captive detection, not some fancy DoH thing.
General theme: A programming language's native in-memory object format is typically optimized for random access, uniformity, and mutability (fields at fixed offsets, etc). Serialization formats for network or disk tend to be designed explicitly to be more compact. But you can design your own in-memory representation too, with the properties you need.
That’s the old school of thought. These days, designers of newer serialization formats realize that designing a more compact format doesn’t really buy much on modern CPUs and modern networks. See for example Cap’n Proto (whose inventor, kentonv, also works at Cloudflare) and flatbuffers.
It's weird that it took so long for these trivial optimizations but it might just be that they were working on optimizing other stuff.
this applies to more than DNS caches. In 1998 I mailed Microsoft a proposal to replace search engine crawlers with a push-based filesystem monitor (detect change → extract → compress → push to index). Got a 5-line rejection letter. They built the same thing 20 years later as IndexNow. Full story with the original letter: https://dev.to/andrew_vl/in-1998-i-proposed-push-based-searc...
The Record struct contains rtype and data where RecordData is a tagged union. Aren’t those two always in sync? Not a DNS expert, just wondering if this is redundant or there is a reason both are there. Doesn’t matter anymore if they store it already serialized but I would be interested why it was this way.
> 56% A records, 25% AAAA, and 19% TXT
And they say nobody uses IPV6.
It’s finally gaining some traction…
https://www.google.com/intl/en/ipv6/statistics.html
no doubt because of scraping and the cost of IPv4
The most interesting result to me is that the richer parsed representation was not necessarily the faster one. If the hot path is mostly “read from cache and serialize back to DNS,” parsing everything upfront only to serialize it again can become unnecessary work and hurt locality....
Frankly weird that they were resorting to high level containers for this in the first place. Also, this line struck me as odd
> Big Pineapple uses jemalloc, an allocator designed for multithreaded, allocation-heavy workloads.
jemalloc multithreaded performance is actually poor(ish) compared to other modern allocators, which makes it a weird choice. But even weirder is why they're even using an allocator in the first place compared to a va MAP_ANON | MAP_NORESERVE arena carveout approach? You can also do punning that way too, which I'm not even certain if Rust supports?
Rust supports punning via pointer casting, but you'll want to use #[repr(C)] on any data types used
I would also have instinctively reached for a large VM reservation to exploit demand paging. I have used that pattern a lot in C++ but not in Rust, so I don't know how difficult it would be to implement there.
How much is this in euro or do we measure money in ram now?
Obvious question: why wasn’t this done earlier? It looks like all the data was already available. At THAT scale, reducing memory usage is a must-have, not a nice-to-have. Weird.
Cloudflare talks about having datacenters in 300+ cities. Presumably they have at least a few servers per datacenter. They saved 130 servers worth of memory... not even the minimum number of servers they have (seriously though, they probably have a LOT of servers)... a few GBs of memory per server running the service. At that scale this is a nice-to-have.
probably agents going through tech debt or finding wins
every dept knows what they could do with more budget, the budget for those things just never comes
now agents have utilized budget more effeftively, unbottlenecking many things, including engineering blogs
I wonder at their scale, why wouldn’t it make sense to store the entries lightly compressed in memory?
> we store the records as a single Box<[u8]> containing each record encoded as a 2-byte length prefix followed by its raw bytes.
Interestingly this is exactly how netlink works-ish: https://manpages.ubuntu.com/manpages/focal/man3/netlink.3.ht...
You start, get the type & length, and then that is how many bytes you read.
Some issues with that when you deserialize, from a raw stream in to `[u8; 4096]` buffer, the alignment is only guaranteed to be on 1 byte, not 4 bytes.
In practice it is 4 bytes, but if you run those tests with Miri, you'll get yelled at. So the fix there is to declare the buffer with a type that mandates the alignment of the largest type that you're going to be deserializing.
So then you start your buffer as follows: `[u32; 1024]`, and with `slice::from_raw_parts` you get to turn that into `[u8; 4096]` with the expected alignment.
As an exercise I wrote a streaming parser for netlink, the current existing package serializes everything, all at once.
This kind of encoding[0] is ubiquitous in networking protocols. It scales down to small silicon well and enables the receiver to estimate resource requirements or skip parts of a serial byte stream without storing it in memory first. These encodings usually aren't aligned by design.
[0] https://en.wikipedia.org/wiki/Type–length–value
It's called TLV encoding - tag/length/value. It's very common in all sorts of network protocols and serialisation formats. It allows you to skip unidentified tags. Sometimes, like in the PNG file format, there's a fixed bit in the tag that tells you whether it's safe to skip or if you have to reject the whole thing because you don't understand this tag.
Hey dang can I get my rate limit turned off pretty please?
I'll buys some spare RAM you now have. I only need 64GB.
The Art of Production.
Now put the 100 terabytes of memory back to the market. Stop hoarding RAM.
DNS is important and 100TB of RAM is effectively nothing.
[dead]
[dead]
[dead]
[dead]
[dead]
One question the article doesn't answer is: why are they cacheing at all? If your cache is that big it isn't a cache. How much bigger is the dataset in question? There are 250 billion entries. Assuming 80/20, that implies 1.25 trillion records?
What's the speed of service/response time relative to the data source?
At that point it might be enough to replace your multiple caches with fewer in-RAM databases?
It's an interesting problem.
Maybe I'm misunderstanding, but this powers 1.1.1.1, it doesn't front an internal dataset. A cache miss hits a nameserver. Which is to say, the dataset is "every DNS record in the world"
I think the question is probably more along the lines of - why not do a database with 100 TB of storage/records instead of a cache? tomato / tomato.. especially with smart caching in front of database. 100TB of flash is a good bit cheaper than 100TB of memory
4 replies →
You have to cache, cloudflare doesn't know all the records ahead of time, they have to do recursive lookups to the authoritative servers that own the records and that is only good for the period of the TTL of the record. There is no "global" DNS record database or something like that.
>that is only good for the period of the TTL of the record.
Not really, TTLs are often short, but IPs might not change for years.
You can probably generate your own TTL, at scale, and avoid many DNS requests.
11 replies →
It's a recursive resolver. The global DNS dataset is not something you could collect to serve directly vs caching from observations.
The data source is authoritative name servers operated by third parties, some of which are slow on their own, some of which are behind slow or lossy networks. Origin response times vary between probably 1 ms and 2 seconds +/- origins that never respond.
The simple answer is that if you didn't cache, DNS traffic would skyrocket, and the load would pile up on the authoritative servers, which were intended to be small, and during the early days of the Internet, were frequently on bandwidth-constrained links.
DNS is designed to distribute query load to the edge as much as possible, and that's enabled by caching. It just so happens that "the edge" is now becoming concentrated among a small set of providers because they wanted to make a business out of it.[1] They knew that this would be expensive going in, though.
[1] Nobody has to use 8.8.8.8 or 1.1.1.1. Most people can use their ISP's cache or a local cache instead without any noticeable difference in behavior.
The problem is there is a noticable difference in behavior because the ISP cache is overloaded so queries take longer. Sure, that's not everyone's experience, but there's a reason people chose to use alternate servers.
They’re adding the cache consumed across all of their servers. It’s not one giant deep cache.
> Once we store a DNS response in the cache, however, we never modify it again. The capacity field serves no purpose, but still costs 8 bytes per Vec
Were there no design discussions/reviews when the system was setup to catch trivial things like this?
Rob Pikes 5 Rules of Programming:
Rule 1. You can't tell where a program is going to spend its time. Bottlenecks occur in surprising places, so don't try to second guess and put in a speed hack until you've proven that's where the bottleneck is.
Rule 2. Measure. Don't tune for speed until you've measured, and even then don't unless one part of the code overwhelms the rest.
Rule 3. Fancy algorithms are slow when n is small, and n is usually small. Fancy algorithms have big constants. Until you know that n is frequently going to be big, don't get fancy. (Even if n does get big, use Rule 2 first.)
Rule 4. Fancy algorithms are buggier than simple ones, and they're much harder to implement. Use simple algorithms as well as simple data structures.
Rule 5. Data dominates. If you've chosen the right data structures and organized things well, the algorithms will almost always be self-evident. Data structures, not algorithms, are central to programming.
https://web.archive.org/web/20260314210910/https://users.ece...
> Data structures, not algorithms, are central to programming
So you agree that they should've designed the system to use the appropriate data structure from the beginning?
5 replies →
> Don't tune for speed until you've measured, and even then don't unless one part of the code overwhelms the rest.
Genuine question, is software performance really linear like that, that one can and should only fight the tightest bottleneck, one workload at a time? Never really sounded right.
It also sounds like the typical sleight of hand where the difficult bit is simply laundered a layer up, in this case the choice of what workload one investigates.
It is often not worth optimising in the early days. You don't know how popular it will become, you might not know how many DNS records you will hold, it was possibly written in an earlier language and ported as-is.
At the point someone queries the 100TB of RAM, then maybe it is worth revisiting but even that has risks. You have to design the migration path, have fallback mechanisms etc.
It's also often that you can avoid all those future migration/fallback risks and pains if you invest a little bit of design thinking upfront.
So how would you decide which path to take in situations like this?
3 replies →
Premature optimization argument fits right in. Now that memory is up to 10x more expensive it is worth considering optimizing programs with large memory footprint.
Using obviously better data structures the first time isn't premature optimization.
3 replies →
How does that fit? What would be the evil of not wasting memory for many years at 1x?
7 replies →
Discussing trivial optimizations is a waste of valuable design time. You're never going to "forget" an optimization. The running system will remind you when the optimization is actually needed.
Boxed slice isn't really the most well known type/optimization, There usually aren't that many vec's that it makes a big difference.
it was working so no one thought to check