Comment by landr0id
14 hours ago
>This is probably a linear (lin) record of how the data was loaded unoptimized from the disk.
Yes, it's buried deep in the details but it's basically just every byte read being written in a linear stream to an output file.
I don't know which stage of grief this is, but since I wrote this blog post I've now ported my IDA debugger scripts to a dedicated QEMU plugin which logs all I/O operations and some other metadata. I tried using this technique to statically rewrite files by basically following DataLoad (with unique identifier) -> Seek -> Read patterns.
There's some annoying nuance to deal with (like seeking backwards implying that data was read, tested, then discarded) but I got this working. Unfortunately some object types encode absolute offsets in them that need to be touched up, so a couple of object types fail to load correctly in external tooling and the PC build of the game.
Now I'm just using this data to completely reimplement the game engine's loading logic from scratch using a custom IO stream which checks the incoming IO operation (seek/read) against what I logged from the game engine to ensure a 1:1 match with how the game loads data.
Have you done any analysis of what proportion of the lin file is being read in total?
You stated in the blog post, that your goal is to try and find unused content, however if as described, the file is just a record of how the game loads the data, then it won't contain any hidden unused assets, since unused assets would never have been read from the original unoptimised file, and thus never written to this optimized file.
I agree and don't think there's any unused data. For `common.lin` for instance my parser reads it basically to the end and there's some small amount of data that's unused. I never actually quantified the amount but I'm fairly certain it's <100 bytes. Probably a bug in there.
The goal post has shifted so far beyond my original intentional at this point. The devs working on the EnhancedSC mod have a strong desire to port some Xbox assets/maps to PC, so I'm mostly doing it at this point as an attempt to help them out.
*On second thought, there's definitely some unused scripting functionality. Script functions which are unused are still included in their parent classes and are loaded if the parent object is loaded, even if never directly called. Whether or not any of this is interesting is another story.
Textures and models though will definitely not be present unless they're used in some non-visible way.