They wrote a JSON “parser” using sscanf. sscanf is not bulletproof! Just use an open source library instead of writing something yourself. You will still be a real programmer, but you will finish your game sooner and you won't have embarrassing stories written about you.
Yes, but now it's in the realm of ~3 minutes, and not ~8 minutes even on a top-spec PC, right? I really liked the game, but waiting 8 minutes to load just to get griefed by hackers within seconds of walking outside... I don't understand how that game makes any money.
Putting the (very valid) reasons for not having human-readable game saves aside, are you sure it's worse than using a 3rd party library that's built to accept semi-valid input values, possibly evaluates user input in some way and has difficult to debug bugs that occur only under certain inputs? I agree that writing a stable and safe parser for a binary data file isn't easy, but there's less things that can go wrong when you can hardcode it to reject any remotely suspicious input. Third party XML/JSON libraries OTOH try to interpret as much as possible, even when the values are bogus. Also no need to deal with different text encoding bugs, line endings...
You misunderstood. Game developers should use a _good_ third–party library, not a _bad_ one. At a minimum they should be able to read the source code so that they know it is good. Thus open source libraries should be at the top of the list.
If you don't know what “good” looks like, take a look at [Serde](https://serde.rs/). It’s for Rust, but its features and overall design are something you should attempt to approach no matter what language you’re writing in.
After finishing the article I immediately did ctrl+f "rust" and was disappointed to not see any of the results I wanted, but actually this comment is more hilarious than anyone saying "why didnt rockstar use rust in 2004!!!1111!!???" it's a bit more of a sophisticated joke since there's an IYKYK factor but it is no less hilarious. Bravo sir, bravo.
To u/db48x whose post got flagged and doesn't reappear despite me vouching for it as I think they have a point (at least for modern games): GTA San Andreas was released in 2004. Back then, YAML was in its infancy (2001) and JSON was only standardized informally in 2006, and XML wasn't something widely used outside of the Java world.
On top of that, the hardware requirements (256MB of system RAM, and the PlayStation 2 only had 32MB) made it enough of a challenge to get the game running at all. Throwing in a heavyweight parsing library for either of these three languages was out of the question.
The comment reappeared, and while you're right about using proper libraries to handle data, it doesn't excuse the "undefined behavior (uninitialized local variables)" that I still see all the time despite all the warning and error flags that can be fed to the compiler.
Most of the time, the programmers who do this do not follow the simple rule that Stroustrup said which is to define or initialize a variable where you declare it (i.e. declare it before using it), and which would solve a lot of bugs in C++.
While it doesn't excuse the bad habits, we do have to keep in mind C++98 (or whatever more ancient was used back then) didn't have the simple initializers we now take for granted. You couldn't just do 'Type myStruct = {};' to null-initialize it, you had to manually NULL all nested fields. God forbid you change the order of the variables in the struct if you're nesting them and forget to update it everywhere. It was just considerably more practical to do 'Type myStruct;' then set the fields when needed.
> it doesn't excuse the "undefined behavior (uninitialized local variables)" that I still see all the time despite all the warning and error flags that you can feed to the compiler.
Yeah but we're talking about a 2004 game that was pretty rushed after 2002's Vice City (and I wouldn't be surprised if the bug in the ingestion code didn't exist there as well, just wasn't triggered due to the lack of planes except that darn RC Chopper and RC plane from that bombing run mission). Back then, the tooling to spot UB and code smell didn't even exist or, if at all, it was very rudimentary, or the warnings that did come up were just ignored because everything seemed to work.
JSON didn't save them: This is the same studio that handrolled a JSON parser with accidentally quadratic time complexity, making most players wait 3 to 10 minutes to load GTA Online, for 7 years, until a player got tired and found the root cause.
You’re not entirely wrong, but a library doesn’t have to be “heavyweight” in order to be bulletproof. And you can load the library during startup and then unload it after; it doesn’t have to stick around for the whole run time of the game. Modern OSes will reclaim the pages after you stop using them, if there is memory pressure. Of course the PS2 didn’t do that I am sure.
Meanwhile, in a certain modern OS, unloading a library is too broken to the point that people are discouraged to do so... Try to unload GLib [0] from your process :p
Easier for internal development. Non- or less technical team members can tweak values without having to rebuild these binary files. Possibly also easier for lightweight modding externally as well.
This isn't that uncommon - look at something like Diablo 2 which has a huge amount of game data defined from text files (I think these are encoded to binary when shipped but it was clearly useful to give the game a mode where it'd load them all from text on startup).
Video games are made by a lot of non-programmers who will be much more comfortable adjusting values in a text file than they are hex editing something.
Besides, the complaint about not having a heavyweight parser here is weird. This is supposed to be "trusted data", you shouldn't have to treat the file as a threat, so a single line sscanf that's just dumping parsed csv attributes into memory is pretty great IMO.
Definitely initialize variables when it comes to C though.
Wow I had no idea YAML was that old. I always thought it was created some time around when CI/CD became popular. Now I'm really curious how it ended up as a superset of JSON.
The flaw isn't the language. The issue is a 0.5x programmer not knowing to avoid sscanf() and failing to default and validate the results. This could be handled competently with strtok() parsing the lines without needing a more complicated file format.
Worked fine on the target machines and the "0.5x programmer" got to see their family for winter holiday. Or are you saying they should have defensively programmed around a bug manifesting 21 years later and skip seeing their family during crunch time?
To be honest, I just don't like how you disparaged the programmer out-of-context. Talk is cheap.
I’ll be the first to defend the greybeards I’ve befriended and learned from in AAA, but having seen codebases of that age and earlier, the “meta” around game development was different back then. I think the internet really changed things for the better.
Your average hire for the time might have been self-taught with the occasional C89 tutorial book and two years of Digipen. Today’s graduates going into games have fallen asleep to YouTube lectures of Scott Meyers and memorized all the literature on a fixed timestep.
funnily enough, there was an infamous bug in GTA5 for a long time that was related to using JSON : https://nee.lv/2021/02/28/How-I-cut-GTA-Online-loading-times...
I remember that one too :)
They wrote a JSON “parser” using sscanf. sscanf is not bulletproof! Just use an open source library instead of writing something yourself. You will still be a real programmer, but you will finish your game sooner and you won't have embarrassing stories written about you.
Loading times are still absurd, fwiw.
Yes, but now it's in the realm of ~3 minutes, and not ~8 minutes even on a top-spec PC, right? I really liked the game, but waiting 8 minutes to load just to get griefed by hackers within seconds of walking outside... I don't understand how that game makes any money.
6 replies →
Putting the (very valid) reasons for not having human-readable game saves aside, are you sure it's worse than using a 3rd party library that's built to accept semi-valid input values, possibly evaluates user input in some way and has difficult to debug bugs that occur only under certain inputs? I agree that writing a stable and safe parser for a binary data file isn't easy, but there's less things that can go wrong when you can hardcode it to reject any remotely suspicious input. Third party XML/JSON libraries OTOH try to interpret as much as possible, even when the values are bogus. Also no need to deal with different text encoding bugs, line endings...
You misunderstood. Game developers should use a _good_ third–party library, not a _bad_ one. At a minimum they should be able to read the source code so that they know it is good. Thus open source libraries should be at the top of the list.
If you don't know what “good” looks like, take a look at [Serde](https://serde.rs/). It’s for Rust, but its features and overall design are something you should attempt to approach no matter what language you’re writing in.
There are no good third party libraries
4 replies →
> Putting the (very valid) reasons for not having human-readable game saves aside,
I don't follow. What would the reasons be?
A human-readable game save file is presumably human-editable.
8 replies →
Mostly to prevent people and programs from editing them, obfuscating implementation details, reducing file sizes (say had they used XML vs. binary)...
Higher barrier to cheating.
2 replies →
Telling a 3d engine programmer not to have opinions on data formats? Good luck with that.
After finishing the article I immediately did ctrl+f "rust" and was disappointed to not see any of the results I wanted, but actually this comment is more hilarious than anyone saying "why didnt rockstar use rust in 2004!!!1111!!???" it's a bit more of a sophisticated joke since there's an IYKYK factor but it is no less hilarious. Bravo sir, bravo.
[dead]
To u/db48x whose post got flagged and doesn't reappear despite me vouching for it as I think they have a point (at least for modern games): GTA San Andreas was released in 2004. Back then, YAML was in its infancy (2001) and JSON was only standardized informally in 2006, and XML wasn't something widely used outside of the Java world.
On top of that, the hardware requirements (256MB of system RAM, and the PlayStation 2 only had 32MB) made it enough of a challenge to get the game running at all. Throwing in a heavyweight parsing library for either of these three languages was out of the question.
The comment reappeared, and while you're right about using proper libraries to handle data, it doesn't excuse the "undefined behavior (uninitialized local variables)" that I still see all the time despite all the warning and error flags that can be fed to the compiler.
Most of the time, the programmers who do this do not follow the simple rule that Stroustrup said which is to define or initialize a variable where you declare it (i.e. declare it before using it), and which would solve a lot of bugs in C++.
While it doesn't excuse the bad habits, we do have to keep in mind C++98 (or whatever more ancient was used back then) didn't have the simple initializers we now take for granted. You couldn't just do 'Type myStruct = {};' to null-initialize it, you had to manually NULL all nested fields. God forbid you change the order of the variables in the struct if you're nesting them and forget to update it everywhere. It was just considerably more practical to do 'Type myStruct;' then set the fields when needed.
4 replies →
And it did not matter at all. The game shipped and was a success.
8 replies →
> it doesn't excuse the "undefined behavior (uninitialized local variables)" that I still see all the time despite all the warning and error flags that you can feed to the compiler.
Yeah but we're talking about a 2004 game that was pretty rushed after 2002's Vice City (and I wouldn't be surprised if the bug in the ingestion code didn't exist there as well, just wasn't triggered due to the lack of planes except that darn RC Chopper and RC plane from that bombing run mission). Back then, the tooling to spot UB and code smell didn't even exist or, if at all, it was very rudimentary, or the warnings that did come up were just ignored because everything seemed to work.
JSON didn't save them: This is the same studio that handrolled a JSON parser with accidentally quadratic time complexity, making most players wait 3 to 10 minutes to load GTA Online, for 7 years, until a player got tired and found the root cause.
https://nee.lv/2021/02/28/How-I-cut-GTA-Online-loading-times...
You’re not entirely wrong, but a library doesn’t have to be “heavyweight” in order to be bulletproof. And you can load the library during startup and then unload it after; it doesn’t have to stick around for the whole run time of the game. Modern OSes will reclaim the pages after you stop using them, if there is memory pressure. Of course the PS2 didn’t do that I am sure.
Meanwhile, in a certain modern OS, unloading a library is too broken to the point that people are discouraged to do so... Try to unload GLib [0] from your process :p
[0] https://docs.gtk.org/glib/
1 reply →
Why weren't binary files used like I would expect in the 1990's DOS game? fread into a struct and all that
By the 2000s, portability was a concern for most titles. Certainly anything targeted at a rapidly changing console market back then.
1 reply →
Easier for internal development. Non- or less technical team members can tweak values without having to rebuild these binary files. Possibly also easier for lightweight modding externally as well.
This isn't that uncommon - look at something like Diablo 2 which has a huge amount of game data defined from text files (I think these are encoded to binary when shipped but it was clearly useful to give the game a mode where it'd load them all from text on startup).
Video games are made by a lot of non-programmers who will be much more comfortable adjusting values in a text file than they are hex editing something.
Besides, the complaint about not having a heavyweight parser here is weird. This is supposed to be "trusted data", you shouldn't have to treat the file as a threat, so a single line sscanf that's just dumping parsed csv attributes into memory is pretty great IMO.
Definitely initialize variables when it comes to C though.
Wow I had no idea YAML was that old. I always thought it was created some time around when CI/CD became popular. Now I'm really curious how it ended up as a superset of JSON.
Vouching seems to be time-lagged and require more than one.
XML was everywhere.
The flaw isn't the language. The issue is a 0.5x programmer not knowing to avoid sscanf() and failing to default and validate the results. This could be handled competently with strtok() parsing the lines without needing a more complicated file format.
Worked fine on the target machines and the "0.5x programmer" got to see their family for winter holiday. Or are you saying they should have defensively programmed around a bug manifesting 21 years later and skip seeing their family during crunch time?
To be honest, I just don't like how you disparaged the programmer out-of-context. Talk is cheap.
4 replies →
I’ll be the first to defend the greybeards I’ve befriended and learned from in AAA, but having seen codebases of that age and earlier, the “meta” around game development was different back then. I think the internet really changed things for the better.
Your average hire for the time might have been self-taught with the occasional C89 tutorial book and two years of Digipen. Today’s graduates going into games have fallen asleep to YouTube lectures of Scott Meyers and memorized all the literature on a fixed timestep.
2 replies →