A few years back I patched the memory allocator used by the Cloudflare Workers runtime to overwrite all memory with a static byte pattern on free, so that uninitialized allocations contain nothing interesting.
We expected this to hurt performance, but we were unable to measure any impact in practice.
Everyone still working in memory-unsafe languages should really just do this IMO. It would have mitigated this Mongo bug.
Recent macOS versions zero out memory on free, which improves the efficacy of memory compression. Apparently it’s a net performance gain in the average case
A few years back I patched the memory allocator used by the Cloudflare Workers runtime to overwrite all memory with a static byte pattern on free, so that uninitialized allocations contain nothing interesting.
Note that many malloc implementations will do this for you given an appropriate environment, e.g. setting MALLOC_CONF to opt.junk=free will do this on FreeBSD.
Zeroing memory should absolutely be the default behavior for any generic allocator in 2025.
If you need better performance, write your own allocator optimized for your specific use case — it's not that hard.
Besides, you if you don't need to clear old allocations, there are likely other optimizations you'll be able to find which would never fly in a system allocator.
> OpenBSD uses 0xdb to fill newly allocated memory and 0xdf to fill memory upon being freed. This helps developers catch "use-before-initialization" (seeing 0xdb) and "use-after-free" (seeing 0xdf) bugs quickly.
I like this. The only information leaking is whether the memory range was previously used. I suppose you may want to control for that. I'd be surprised if OpenBSD didn't provide a flag to just freed memory to the same value as never allocated.
You know, I never even considered doing that but it makes sense; whatever overhead that's incurred by doing that static byte pattern is still almost certainly minuscule compared to the overhead of something like a garbage collector.
IMO the tradeoff that is important here is a few microseconds of time sanitizing the memory saves the millions of dollars of headache when memory unsafe languages fail (which happens regularly)
Non-deterministic latency is a drawback, but garbage collection is not inherently slower than manual memory management/reference counting/etc. Depending on the usage pattern it can be faster. It's a set of trade-offs
FYI, at least in C/C++, the compiler is free to throw away assignments to any memory pointed to by a pointer if said pointer is about to be passed to free(), so depending on how you did this, no perf impact could have been because your compiler removed the assignment. This will even affect a call to memset()
I patched the free() implementation itself, not the code that calls free().
I did, of course, test it, and anyway we now run into the "freed memory" pattern regularly when debugging (yes including optimized builds), so it's definitely working.
The author seems to be unaware that Mongo internally develops in a private repo and commits are published later to the public one with https://github.com/google/copybara. All of the confusion around dates is due to this.
I was definitely unaware. I suspected something like this may be up when I talked about the zero-review of the apparent PR "I’m not aware of Mongo’s public review practices". This is great to know though. Updating the piece now to mention this and explain the date discrepancy
In something like a database zeroing or poisoning on free is probably a good idea. (These days probably all allocators should do it by default.)
Allocators are an interesting place to focus on for security. Chris did amazing work there for Blink that eventually rolled out to all of Chromium. The docs are a fun read.
To the extent that any of this was ever true, it hasn’t been true for at least a decade. After the WiredTiger acquisition they really got their engineering shit together. You can argue it was several years too late but it did happen.
NoSQL is used for high availability of data at scale - iMessage famously uses it for message threads, EA famously uses it for gaming matchmaking.
What you do is have both SQL and NoSQL. The NoSQL is basically caches of resources for high availability. Imagine you are making a social media app... Yes of course you have a SQL database that stores all the data, but you maintain API caches of posts in NoSQL.
Why? This gets to some of your other black vs white insults: NoSQL is typically WAY FASTER than SQL. That's why you use it. It's way faster to read a JSON file from a hard drive than it is to query a SQL database, always has been. So why not use NoSQL for EVERYTHING? Well, because you have duplicated data everywhere since it's not relational, it's just giant caches essentially. You also will get slow queries when the documents get huge.
Anyway you need both. It's not an either/or thing. I cannot believe this many years later people do not know the purpose of SQL and NoSQL and do not understand that it is not a competition at all. You want both!
Because nobody uses mongo for the reasons you listed. They use redis, dynamo, scylla or any number of enriched KV stores.
Mongo has spent its entire existence pretending to be a SQL database by poorly reinventing
everything you get for free in postgres or mysql or cockroach.
Yeah fair, I was being a bit lazy here when writing my comment. I've used nosql professionally quite a bit, but always set up by others. When working on personal projects I reach for SQL first because I can throw something together and don't need ideal performance. You're absolutely right that they both have their place.
That being said the question was genuine - because I don't keep up with the ecosystem, I don't know it's ever valid practice to have a nosql db exposed to the internet.
What they wrote was pretty benign. They just asked how common it is for Mongo to be exposed. You seem to have taken that as a completely different statement
A highly cited reason for using mongo is that people would rather not figure out a schema. (N=3/3 for “serious” orgs I know using mongo).
That sort of inclination to push off doing the right thing now to save yourself a headache down the line probably overlaps with “let’s just make the db publicly exposed” instead of doing the work of setting up an internal network to save yourself a headache down the line.
> A highly cited reason for using mongo is that people would rather not figure out a schema.
Which is such a cop out, because there is always a schema. The only questions are whether it is designed, documented, and where it's implemented. Mongo requires some very explicit schema decisions, otherwise performance will quickly degrade.
It could be because when you leave an SQL server exposed it often turns into much worse things. For example, without additional configuration, PostgreSQL will default into a configuration that can own the entire host machine. There is probably some obscure feature that allows system process management, uploading a shell script or something else that isn't disabled by default.
The end result is "everyone" kind of knows that if you put a PostgreSQL instance up publicly facing without a password or with a weak/default password, it will be popped in minutes and you'll find out about it because the attackers are lazy and just running crypto-mine malware, etc.
No one, if you aren't in the administration's good graces and something shitty happens unrelated to you, you've put a target on your back to be suspect #1.
I'm still thinking about the hypothetical optimism brought by OWASP top 10 hoping that major flaws will be solved and that buffer overflow has been there since the beginning... in 2003.
I mean giving everyone footguns and you'll find that is unavoidable forever. Thoughts and prayers to the Mongo devs until we migrate to a language that prevents this error.
Evidence of no exploitations? It's usually hard to prove a negative, except when you have all the logs at your fingertips you can sift through. Unless they don't, of course. In which case the point stands: they don't actually know at this point in time, if they can even know about it at all.
Specifically, it looks like the exflitration primitive relies on errors being emitted, and those errors are what leak the data. They're also rather characteristic. One wouldn't reasonably expect MongoDB to hold onto all raw traffic data flowing in and out, but would absolutely expect them to have the error logs, at least for some time back.
is it true that ubisoft got hacked and 900GB of data from their database was leaked due to mongobleed, i am seeing a lot of posts on social media under the #ubisoft tags today. can someone on HN confirm?
Details are still emerging, update in the last hour was that at least 5 different hacking groups were in ubisoft's systems and yeah some might have got their via bribes rather than mongodb https://x.com/vxunderground/status/2005483271065387461
Almost always when you hear about emails or payment info leaking (or when Twitter stored passwords in plaintext lol) it's from logs. And a lot of times logs are in NoSQL because it is only ever needed in that same JSON format and in a very highly available way (all you Heroku users tailing logs all day, yw) and then almost nobody encrypts phone numbers and emails etc. whenever those end up in logs.
There's basically no security around logs actually. They're just like snapshots of the backend data being sent around and nobody ever cares about it.
Anyway it has nothing to do with the choice to use NoSQL, it has more to do with how neglected security is around it.
Btw in case you are wondering in both the Twitter plaintext password case and in the Rainbow Six Siege data leak you mention were both logs that leaked. NoSQL backed logs sure, but it's more about the data security around logging IMO.
If it is, it's less fluffy and empty than most of LLM prose we're usually fed. It's well explained and has enough details to not be overwhelming.
Honestly, aside from the "<emoji> impact" section that really has an LLM smell (but remember that some people legit do this since it's in the llm training corpus), this more feels like LLM assisted (translated? reworded? grammar-checked?) that pure "explain this" prompt.
I did some research with it, and used it to help create the ASCII art a bit. That's about it.
I was afraid that adding the emoji would trigger someone to think it's AI.
In any case, nowadays I basically always get at least one comment calling me an AI on a post that's relatively popular. I assume it's more a sign of the times than the writing...
A few years back I patched the memory allocator used by the Cloudflare Workers runtime to overwrite all memory with a static byte pattern on free, so that uninitialized allocations contain nothing interesting.
We expected this to hurt performance, but we were unable to measure any impact in practice.
Everyone still working in memory-unsafe languages should really just do this IMO. It would have mitigated this Mongo bug.
Recent macOS versions zero out memory on free, which improves the efficacy of memory compression. Apparently it’s a net performance gain in the average case
I wonder if Apple Silicon has hardware acceleration for memory zeroing... Knowing Apple, I wouldn't be surprised.
1 reply →
A few years back I patched the memory allocator used by the Cloudflare Workers runtime to overwrite all memory with a static byte pattern on free, so that uninitialized allocations contain nothing interesting.
Note that many malloc implementations will do this for you given an appropriate environment, e.g. setting MALLOC_CONF to opt.junk=free will do this on FreeBSD.
Zeroing memory should absolutely be the default behavior for any generic allocator in 2025.
If you need better performance, write your own allocator optimized for your specific use case — it's not that hard.
Besides, you if you don't need to clear old allocations, there are likely other optimizations you'll be able to find which would never fly in a system allocator.
> OpenBSD uses 0xdb to fill newly allocated memory and 0xdf to fill memory upon being freed. This helps developers catch "use-before-initialization" (seeing 0xdb) and "use-after-free" (seeing 0xdf) bugs quickly.
Looks like this is the default in OpenBSD.
I like this. The only information leaking is whether the memory range was previously used. I suppose you may want to control for that. I'd be surprised if OpenBSD didn't provide a flag to just freed memory to the same value as never allocated.
Is this the same as enabling `init_on_free=1` in the kernel?
You know, I never even considered doing that but it makes sense; whatever overhead that's incurred by doing that static byte pattern is still almost certainly minuscule compared to the overhead of something like a garbage collector.
IMO the tradeoff that is important here is a few microseconds of time sanitizing the memory saves the millions of dollars of headache when memory unsafe languages fail (which happens regularly)
4 replies →
Non-deterministic latency is a drawback, but garbage collection is not inherently slower than manual memory management/reference counting/etc. Depending on the usage pattern it can be faster. It's a set of trade-offs
FYI, at least in C/C++, the compiler is free to throw away assignments to any memory pointed to by a pointer if said pointer is about to be passed to free(), so depending on how you did this, no perf impact could have been because your compiler removed the assignment. This will even affect a call to memset()
see here: https://godbolt.org/z/rMa8MbYox
I patched the free() implementation itself, not the code that calls free().
I did, of course, test it, and anyway we now run into the "freed memory" pattern regularly when debugging (yes including optimized builds), so it's definitely working.
However, if you recast to volatile, the compiler will keep it:
24 replies →
The author seems to be unaware that Mongo internally develops in a private repo and commits are published later to the public one with https://github.com/google/copybara. All of the confusion around dates is due to this.
I was definitely unaware. I suspected something like this may be up when I talked about the zero-review of the apparent PR "I’m not aware of Mongo’s public review practices". This is great to know though. Updating the piece now to mention this and explain the date discrepancy
The author of this post is incorrect about the timeline. Our Atlas clusters were upgraded days before the CVE was announced.
thanks! updated
In something like a database zeroing or poisoning on free is probably a good idea. (These days probably all allocators should do it by default.)
Allocators are an interesting place to focus on for security. Chris did amazing work there for Blink that eventually rolled out to all of Chromium. The docs are a fun read.
https://blog.chromium.org/2021/04/efficient-and-safe-allocat...
https://chromium.googlesource.com/chromium/src/+/master/base...
How often are mongo instances exposed to the internet? I'm more of an SQL person and for those I know it's pretty uncommon, but does happen.
From my experience, Mongo DB's entire raison d'etre is "laziness".
* Don't worry about a schema.
* Don't worry about persistence or durability.
* Don't worry about reads or writes.
* Don't worry about connectivity.
This is basically the entire philosophy, so it's not surprising at all that users would also not worry about basic security.
To the extent that any of this was ever true, it hasn’t been true for at least a decade. After the WiredTiger acquisition they really got their engineering shit together. You can argue it was several years too late but it did happen.
1 reply →
Although interestingly, for all the mongo deployments I managed, the first time I saw a cluster publicly exposed without SSL was postgres :)
Not only that, but authentication is much harder than it needs to be to set up (and is off by default).
I'm sure there are publicly exposed MySQLs too
1 reply →
Most of your points are wrong. Maybe only 1- is valid'ish.
Ultimate webscale!
Are you guys serious with these takes?
You very often have both NoSQL and SQL at scale.
NoSQL is used for high availability of data at scale - iMessage famously uses it for message threads, EA famously uses it for gaming matchmaking.
What you do is have both SQL and NoSQL. The NoSQL is basically caches of resources for high availability. Imagine you are making a social media app... Yes of course you have a SQL database that stores all the data, but you maintain API caches of posts in NoSQL.
Why? This gets to some of your other black vs white insults: NoSQL is typically WAY FASTER than SQL. That's why you use it. It's way faster to read a JSON file from a hard drive than it is to query a SQL database, always has been. So why not use NoSQL for EVERYTHING? Well, because you have duplicated data everywhere since it's not relational, it's just giant caches essentially. You also will get slow queries when the documents get huge.
Anyway you need both. It's not an either/or thing. I cannot believe this many years later people do not know the purpose of SQL and NoSQL and do not understand that it is not a competition at all. You want both!
Because nobody uses mongo for the reasons you listed. They use redis, dynamo, scylla or any number of enriched KV stores.
Mongo has spent its entire existence pretending to be a SQL database by poorly reinventing everything you get for free in postgres or mysql or cockroach.
2 replies →
Yeah fair, I was being a bit lazy here when writing my comment. I've used nosql professionally quite a bit, but always set up by others. When working on personal projects I reach for SQL first because I can throw something together and don't need ideal performance. You're absolutely right that they both have their place.
That being said the question was genuine - because I don't keep up with the ecosystem, I don't know it's ever valid practice to have a nosql db exposed to the internet.
What they wrote was pretty benign. They just asked how common it is for Mongo to be exposed. You seem to have taken that as a completely different statement
2 replies →
A highly cited reason for using mongo is that people would rather not figure out a schema. (N=3/3 for “serious” orgs I know using mongo).
That sort of inclination to push off doing the right thing now to save yourself a headache down the line probably overlaps with “let’s just make the db publicly exposed” instead of doing the work of setting up an internal network to save yourself a headache down the line.
> A highly cited reason for using mongo is that people would rather not figure out a schema.
Which is such a cop out, because there is always a schema. The only questions are whether it is designed, documented, and where it's implemented. Mongo requires some very explicit schema decisions, otherwise performance will quickly degrade.
1 reply →
I would have hoped that there would be no important data in mongoDB.
But now we can at least be rest assured that the important data in mongoDB is just very hard to read with the lack of schemas.
Probably all of that nasty "schema" work and tech debt will finally be done by hackers trying to make use of that information.
10 replies →
The article links to a shodan scan reporting 213K exposed instances https://www.shodan.io/search?query=Product%3A%22MongoDB%22
It could be because when you leave an SQL server exposed it often turns into much worse things. For example, without additional configuration, PostgreSQL will default into a configuration that can own the entire host machine. There is probably some obscure feature that allows system process management, uploading a shell script or something else that isn't disabled by default.
The end result is "everyone" kind of knows that if you put a PostgreSQL instance up publicly facing without a password or with a weak/default password, it will be popped in minutes and you'll find out about it because the attackers are lazy and just running crypto-mine malware, etc.
My university has one exposed to the internet, and it's still not patched. Everyone is on holiday and I have no idea who to contact.
No one, if you aren't in the administration's good graces and something shitty happens unrelated to you, you've put a target on your back to be suspect #1.
"Look at me. I'm the DBA now"
-JS devs after "Signing In With Facebook" to MongoDB Atlas
AKA me
Sorry guys, I broke it
For a long time, the default install had it binding to all interfaces and with authentication disabled.
often. lots of data leaks happened because of this. people spin it up in a cloud vm and forget it has a public ip all the time.
I'm still thinking about the hypothetical optimism brought by OWASP top 10 hoping that major flaws will be solved and that buffer overflow has been there since the beginning... in 2003.
I mean giving everyone footguns and you'll find that is unavoidable forever. Thoughts and prayers to the Mongo devs until we migrate to a language that prevents this error.
> On Dec 24th, MongoDB reported they have no evidence of anybody exploiting the CVE
Absence of evidence is not evidence of absence...
What would you prefer them to say?
Evidence of no exploitations? It's usually hard to prove a negative, except when you have all the logs at your fingertips you can sift through. Unless they don't, of course. In which case the point stands: they don't actually know at this point in time, if they can even know about it at all.
Specifically, it looks like the exflitration primitive relies on errors being emitted, and those errors are what leak the data. They're also rather characteristic. One wouldn't reasonably expect MongoDB to hold onto all raw traffic data flowing in and out, but would absolutely expect them to have the error logs, at least for some time back.
4 replies →
Why is anyone using mongo for literally anything
Easy replication. I suppose it's faster than Postgres's JSONB, too.
I would rather not use it, but I see that there are legitimate cases where MongoDB or DynamoDB is a technically appropriate choice.
because it is "web scale"
ref: https://www.youtube.com/watch?v=b2F-DItXtZs
Whenever anyone writes about mongodb or redis I hear it in that voice.
Right? When they came out, it was all about NoSQL, which then turned out only mean key-value database, whom are plentiful.
This is a nasty ad repositorium datorum argumentation which I cannot tolerate.
I laughed.
Every time someone posts about NoSQL a thousand "programmers" reveal they have never had to support a lot of traffic lol
Nah, this time it was just you.
is it true that ubisoft got hacked and 900GB of data from their database was leaked due to mongobleed, i am seeing a lot of posts on social media under the #ubisoft tags today. can someone on HN confirm?
I read that hack was made possible by Ubisoft’s support staff taking bribes.
Details are still emerging, update in the last hour was that at least 5 different hacking groups were in ubisoft's systems and yeah some might have got their via bribes rather than mongodb https://x.com/vxunderground/status/2005483271065387461
1 reply →
TLDR: Blame logs not NoSQL.
Almost always when you hear about emails or payment info leaking (or when Twitter stored passwords in plaintext lol) it's from logs. And a lot of times logs are in NoSQL because it is only ever needed in that same JSON format and in a very highly available way (all you Heroku users tailing logs all day, yw) and then almost nobody encrypts phone numbers and emails etc. whenever those end up in logs.
There's basically no security around logs actually. They're just like snapshots of the backend data being sent around and nobody ever cares about it.
Anyway it has nothing to do with the choice to use NoSQL, it has more to do with how neglected security is around it.
Btw in case you are wondering in both the Twitter plaintext password case and in the Rainbow Six Siege data leak you mention were both logs that leaked. NoSQL backed logs sure, but it's more about the data security around logging IMO.
MongoDB has always sucked... But it's webscale (sic)
Do yourself a favour, use ToroDB instead (or even straight PostgreSQL's JSONB).
This has many similarities to the Heartbleed vulnerability: it involves trusting lengths from an attacker, leading to unauthorized revelation of data.
Related:
MongoBleed
https://news.ycombinator.com/item?id=46394620
Have all Atlas clusters been auto-updated with a fix?
yes. apparently before Dec 19 too
> In C/C++, this doesn’t happen. When you allocate memory via `malloc()`, you get whatever was previously there.
What would break if the compiler zero'd it first? Do programs rely on malloc() giving them the data that was there before?
That's what calloc() is for
It takes time to zero out memory.
"MongoBleed Explained by an LLM"
If it is, it's less fluffy and empty than most of LLM prose we're usually fed. It's well explained and has enough details to not be overwhelming.
Honestly, aside from the "<emoji> impact" section that really has an LLM smell (but remember that some people legit do this since it's in the llm training corpus), this more feels like LLM assisted (translated? reworded? grammar-checked?) that pure "explain this" prompt.
I didn't use AI in writing the post.
I did some research with it, and used it to help create the ASCII art a bit. That's about it.
I was afraid that adding the emoji would trigger someone to think it's AI.
In any case, nowadays I basically always get at least one comment calling me an AI on a post that's relatively popular. I assume it's more a sign of the times than the writing...
2 replies →
Though the source article was human written, the public exploit was developed with an LLM.
https://x.com/dez_/status/2004933531450179931
[dead]
[dead]