Comment by WalterBright
15 hours ago
Back in the 1980s, text editors had configuration files. The configuration file would be read every time the editor was loaded. This was very slow on a floppy disk system.
I realized that, instead of a configuration file, I could configure the executable instead! So, any changes in configuration meant the editor would patch its own exe file!
This marvelous technique came to an end when attempts to stop malware got folded into the operating system.
Some early systems, like TeX and I believe some Lisps, took this to the extreme. Instead of patching, they loaded their config once and then dumped the configured process image to a file, which was used in subsequent invocations.
TeX was developed on DECsystem-10/20 machines (36-bit words, your choice of byte size). The various operating systems (Tops20, Sail/Waits, ITS) were superior to Unix in a few ways, one of which was that when a process was suspended (think control-Z, or even control-C) you could issue the built-in shell SAVE command that would save the entire state of the suspended process into a new executable, data segment as well as code. So, on these systems, you could run TeX, have it load a bunch of macros and such, and then SAVE the result as a real, pre-configured executable for the world to run. Easy-peasy.
This was true for the original TeX78 written in Sail, as well as the ultimate TeX82 written in Knuth's WEB macro language on top of Pascal (that nowadays typically gets transpiled to C). Other programs did similar stuff; the feature was in the OS way before TeX started.
(Gory detail: Actually, TeX went a little further, to free up all possible address space for the final executable: The version that could initialize various hash tables and hyphenation trie tables could dump (nee serialize) a binary file of the resulting data structures; then a slimmed-down version that didn't have that code would read the binary info back in to recreate the initial data structure state, and that's what you'd SAVE the production executable from.)
For the unix-y versions of TeX, there was effort made to mimic this sort of thing in user-land with "undump", but admirable as it was, it was a hack, I'm told. These days, everything is so fast, it's not clear that this feature would be worth it anyway.
Source: me; I was there.
I don't think undump is a hack except to the extent the whole thing is a hack. Linux shoves a lot of stuff to userspace that other operating systems put in kernel space; this is just another one of them.
The program loader is not magic - it just reads a list of things to mmap and then mmaps them. If you write a thing that writes a list of what's mmapped and call it the unloader... fine?
Emacs does this to create its image with all the added functionality above the minimum required to run elisp, then has a mechanism to pull in text files because elisp is just text anyway.
They just chucked the old system for a portable version of it, but until this last release, they still had to option of doing it the old school way.
Don’t think TeX ever did that, that was a “simple” Pascal program.
Lisp Machines though.. updating the operating system was by loading bunch of compiled files that replaced currently loaded functions in memory.
Then again, Lisp Machines where very proud of self modification — the CADR had a fun feature where it could modify the next instruction depending on things…
How the world has changed.
Pascal TeX to the best of my knowledge never did it but the web2c version does.
The description is a bit confusing because it can both dump only the warmed up interpreter image or the whole process, I think.
"With the program undump, you can use `core' to reconstitute a preloaded executable, which does not need to read a `.fmt' file to get started. Although preloaded executables save startup time, they have a big disadvantage: neither the disk space to store them nor their code segments (at runtime) can be shared. Therefore, if both tex and latex are running, twice as much memory will be consumed, to the general detriment of performance."
https://mirror.gutenberg-asso.fr/tex.loria.fr/texlive-htmldo...
> Don’t think TeX ever did that, that was a “simple” Pascal program.
Not exactly the entire process image, no, but essentially all of its data in a single chunk. It’s a peculiar Pascal program because it bypasses basically all of Pascal’s typing, records, etc., and instead builds its own from a set of (WEB) macros on top of a giant untyped array. Quoth Knuth in TeX: The Program §115:
> The dynamic storage requirements of TeX are handled by providing a large array mem in which consecutive blocks of words are used as nodes by the TeX routines. Pointer variables are indices into this array [...].
There are a few more areas designated for specific purposes, but at the end of the day (§1302) it works out about the way you’d expect:
On a floppy system, it's much faster to just patch it on disk than rewrite it!
Pharo [1] is a modern take.
[1] https://pharo.org/
emacs did this too.
And didn't stop doing this until 2020! (Specifically Emacs 27.1, which replaced "unexec", which dumped the process image, with the "portable dumper", which despite the name does not.)
1 reply →
IIRC early Turbo Pascal versions worked like that too, there was some "setup" program that let you configure colors, etc, by modifying the COM/EXE file itself.
I still run Borland's Turbo Pascal on CP/M systems, and many programs had setup/configuration programs which would rewrite binaries for specific input/output devices.
Choosing between ADM-3A or ANSI terminals by running "WINSTALL" would rewrite the main Wordstart executable WS.COM appropriately for example.
When I was learning Python I wrote a program that stored data in its own .py file. I felt pretty smart.
Was the advantage that the exe file's sectors were likely to be contiguous on disk, so the config data could likely be read in a single pass through sectors on the same track, as compared to probably having to wait for a full revolution and a track seek for the separate config file approach? Or was it that the config file was verbose but compressed to a much smaller image in memory, thus fewer bytes to read?