I've been recently working with Classic Mac OS programming[0] and just that memory model (also using dealing with the lack of virtual memory using opaque handles to memory that need to be locked when used) is painful enough[1] - having to deal with segment addressing on top of that does not sound like fun. Thank god for the Motorola 68000!
[1]The equivalent to HeapWalker I used was Metroweks ZoneRanger which was bundled with their compiler. It has a nice visualization of how fragmented the memory is https://bitbang.social/@kalleboo/116302075194704555
It wasn't really the processor architecture. Segmented addressing was actually fairly easy if the processor was used only in the way that protected mode was envisioned as working. As the headlined article observes, a lot of this stuff simply wasn't necessary in OS/2 1.x, even though that too had DLLs, callback window procedures, and the multiple tiny/small/medium/large/compact/huge memory models.
The differences were (a) that DOS+Windows was designed so that the same programs could run in both real mode, with overlaying, and 286 protected mode, with segmented virtual memory; and (b) that to really save on RAM DOS+Windows had ideas such as the data segments for DLLs being globally shared across all processes. These added all of the complications mentioned in the headlined article and more besides. It was the operating system, not the processor architecture.
I understood it as Windows developers had to manually deal with segment limitations since Windows supported running on pre-286 CPUs without protected mode (Wikipedia says Windows 1-3 all supported the 8088). OS/2 just made the 286 a minimum requirement so they could rely on a CPU with more modern features.
The 68k didn't come with an MMU like the 286 so MacOS couldn't rely on virtual memory like OS/2 did but at least the flat memory space meant you didn't have to juggle 64k segments
Sometimes I think that if it were the old days, I probably wouldn't have been able to program. I remember that these days we program on top of 64bit virtual addresses, but how did developers do it back then
It's easier when it's the only way to get things done. Think about how nobody who was learning programming before 2023 was seriously wishing "This would be so much easier if the computer wrote it all for me".
As someone who grew up coding after it was mostly 32-bit, I can't say this with certainty, but my gut feeling is that paradoxically you would have and it would've made you stronger.
I think the knowledge of underlying hardware is useful and good to know.
But also that sort of knowledge got dated pretty quickly in the early computer era. Further, the capabilities of things like optimizing compilers quickly got to a point where they'd outpace most hand written assembly. Today, it's basically just floating point operations where you can still do better than a compiler.
In the early days, you'd have the correct impression that the C compilers spat out utter garbage which was a lot slower than what you could hand craft. As optimization techniques got better and better, the work you did because the compiler was dumb ultimately would have gotten in the way.
Exactly. I'd argue that all those programming Gods and Gods because they went through that period. Whatever didn't kill them made them stronger. We should replicate that experience by deliberately writing in low level C and assembly for a few years.
Memory mapping/bank switching was fairly common on 8-bit and 16-bit systems, where a small memory window was used to select different memory banks, allowing a program to access more memory in chunks.
Game consoles like NES, SNES and Game Boy had additional hardware built in the cartridge to support memory mapping/bank switching.
For PCs, EMS (memory) provided a similar concept. It reserved a 64 kB window divided in 16 kB pages in the first 1 MB and allowed to map up to 32 MB.
I first found out about segmenting in 16 bit systems in 2016 by reading a lively explanation from an older edition of Duntemann's Assembly Language Step by Step (the newer editions focus largely on Linux and 32/64-bit systems).
> I think it was a lot more common for 8bit systems to allow for 16 bit addressing though.
The 6502 and Z80 could use 16 bit addressing to access up to 64kb of memory. The 6502 had various other addressing systems, including iirc 8 bits, but none of them were wider tha 16 bits.
You had to deal with two flavors of pointer, near and far. Far pointers came with segment selector, for accessing more than 64k. Your choice of memory model influenced the defaults. You might use near pointers for internal references in a module, and far pointers for external references.
I've been wondering about this lately. As a kid, I spent hour upon hour learning about computing: typing in Basic code from a magazine into a Commodore 64, playing with music on an Atari STe, learning my way around a DOS command line, dabbling with 3D modelling... just so much stuff that my own kids would never have the patience for.
I wonder if it's just that kids today (gods that makes me sound old!) are constantly surrounded by entertaining things to do - gaming, TV/films, music, social media.
I have been wondering how to train my 6-year old son and myself to increase my attention span.
Some rules are obvious -- cutoff mobiles and pads completely (he doesn't have access to them so it's for me), sit in the library and study from books (I believe this is even possible for programming topics as I can write on paper). Basically, cutting off everything electronics definitely helps -- even putting my phone in the bag improves productivity significantly.
But the problem is, my son is unruly. If I put him in the library, most likely he runs around and messes things up, which ends up we leave early without doing anything.
I think they learned by reading books such as Undocumented Windows or Windows Internals (not to be confused with Windows NT internals), and Microsoft documents.
In fact, I’d argue it was more fun than programming Javascript these days.
It wasn't really the 'Undocumented' and 'Internals' books. Pretty much everything in the headlined article was to be found in the SDK, Microsoft Press publications, and in many third party books about DOS+Windows programming.
Petzold's Programming Windows book, for example, devoted an entire chapter (chapter 7) to memory management, with diagrams and examples. In the 2nd edition (which I just pulled off the shelf to check) that chapter runs to some 40 pages.
16-bit x86 processors took 20-bit pointers, expressed as a 16-bit segment and a 16-bit offset. The segment was shifted four bits left and then the offset added. Which means there are lots of different segment:offset pointers that point to the same address. Segments are loaded into a segment register (one of CS, DS, ES, or SS) and then combined with an offset pointer in another register to create a pointer in this way. For example, 1e37:0008 would become 1e378.
It's complicated and janky as all get-out, but it made more sense if you were coming from 8080/Z80 development, as this was a scheme to ensure some degree of compatibility with 16-bit 8080 addressing while providing access to much more memory. 8086 was not binary compatible with 8080, but was designed so that 8080 programs could be machine converted to 8086 ones.
In languages like C, this took the form of three different types of pointers: NEAR, FAR, and HUGE. NEAR pointers were 16-bit offsets only, and dereferenced with respect to the current segment (usually in DS). FAR pointers were full segment:offset pairs but pointer arithmetic was only done on the offset which meant objects could be 64K max. HUGE pointers allowed for objects larger than 64k but at a significant performance cost.
In 1994 I was 2 years out of school. I'd written one windows shareware application and a whole lot of unix-y things. People were excited about the internet but most people didn't have access. Unix shell accounts via dialup were common though.
One day I was encouraged to write a Windows Sockets emulation layer for ordinary dial-up shell accounts like those offered by netcom. The idea was to allow the use of the recently released Mosaic browser without an actual internet connection. I figured sure, no problem. I'll use curl or some other tool in the shell account to do the actual fetching of URLs, transfer styles over zmodem, and simulate all the tcp/ip calls in the DLL.
I couldn't even get started. The reason is that I couldn't understand how the different Windows applications could all share memory allocated at runtime in the winsock.dll.
I asked a highly experienced ex Microsoft person, and he just said what are you talking about. There's no API to allocate shared memory.
So I gave up. 6 months later someone else did it.
Around then I realized the truth: Windows 3.1 had no memory protection at all. Specifically all global variables in DLLs were shared by default. The hard part wasn't sharing memory among users of a DLL. If anything, the hard part was having good discipline to avoid sharing it.
Since I'd only used multiuser Unix in school, and I knew Windows supported multitasking (even if only the cooperative kind), I just couldn't wrap my head around the idea that I'm multitasking operating system could exist without memory protection.
Pretty good detail in this article! But what really surprises me is how some ideas just keep coming back.
When I wrote a binary translator, I ended up having to keep a translated return stack to optimize RET opcodes. That put me in exactly the same position as the Win16 kernel with regard to having to patch pointers (in case of Win16, just the segment part) on stack.
Of course I did not have the benefit of my guests calling a lock function, so I ended up having to run a garbage collection operation to determine which pointers are in use & take exceptions on now-invalidated segments. Lots of extra work that Windows didn't need: it's nice to be king :-)
If you think programming in Win16 (or whatever we want to call it), you should try teaching people to do it. I worked as a commercial trainer on C and Windows way back when - C and the Windows API were no bed of roses, but the different memory models were mind-numbing for us tutors and the poor punters, many of whom didn't know C!
> Exports are used for application code which is externally called.
This was the magic moment for me, learning Windows 3.0 programming. The idea that my program is no longer master of it's world, but instead is just something that gets loaded and called by Windows.
Check the ID numbers (48410844 < 48424862) and bear in mind that Hacker News has this thing where sometimes submissions get re-cycled for attention. Yes, annoyingly it does seem to make the presented datestamps wrong.
People submit a lot of stuff all the time, very few people go through "New" and thus a new submission probably have a very short life time before it is drowned by newer submissions.
A submission to survive most likely needs some initial push from non-organic voting.
It probably helps if you share you submission early with your colleagues and in other sites.
I've had this experience a few times so I don't post submissions anymore either (including one of my own articles being flagged despite over a hundred comments). I know people will say vote rigging doesn't happen on HN, but I think it's naïve to think any site on the internet is impervious to vote rigging.
Win16 programming was an important formative phase in my career. There is a lot of wisdom in old solutions to thorny problems and knowing them often clues you to how one may adapt them to today's problem. For example, when CPU+GPU programming appeared i immediately imagined CPU memory accessed with "near" pointers and GPU memory accessed with "far" pointers with a switch to a pseudo-segment register.
It also conditioned a programmer to learn about various complexities involved and be careful in their programming i.e. it taught you discipline. You understood your compiler, OS and hardware better and how to write code keeping them all in mind. For example, i often say my study of embedded programming started with Win16!
I've been recently working with Classic Mac OS programming[0] and just that memory model (also using dealing with the lack of virtual memory using opaque handles to memory that need to be locked when used) is painful enough[1] - having to deal with segment addressing on top of that does not sound like fun. Thank god for the Motorola 68000!
[0]Made an AppleTalk chat client/server https://github.com/kalleboo/GlobalTalk-Chat
[1]The equivalent to HeapWalker I used was Metroweks ZoneRanger which was bundled with their compiler. It has a nice visualization of how fragmented the memory is https://bitbang.social/@kalleboo/116302075194704555
It wasn't really the processor architecture. Segmented addressing was actually fairly easy if the processor was used only in the way that protected mode was envisioned as working. As the headlined article observes, a lot of this stuff simply wasn't necessary in OS/2 1.x, even though that too had DLLs, callback window procedures, and the multiple tiny/small/medium/large/compact/huge memory models.
The differences were (a) that DOS+Windows was designed so that the same programs could run in both real mode, with overlaying, and 286 protected mode, with segmented virtual memory; and (b) that to really save on RAM DOS+Windows had ideas such as the data segments for DLLs being globally shared across all processes. These added all of the complications mentioned in the headlined article and more besides. It was the operating system, not the processor architecture.
I understood it as Windows developers had to manually deal with segment limitations since Windows supported running on pre-286 CPUs without protected mode (Wikipedia says Windows 1-3 all supported the 8088). OS/2 just made the 286 a minimum requirement so they could rely on a CPU with more modern features.
The 68k didn't come with an MMU like the 286 so MacOS couldn't rely on virtual memory like OS/2 did but at least the flat memory space meant you didn't have to juggle 64k segments
2 replies →
Sometimes I think that if it were the old days, I probably wouldn't have been able to program. I remember that these days we program on top of 64bit virtual addresses, but how did developers do it back then
It's easier when it's the only way to get things done. Think about how nobody who was learning programming before 2023 was seriously wishing "This would be so much easier if the computer wrote it all for me".
As someone who grew up coding after it was mostly 32-bit, I can't say this with certainty, but my gut feeling is that paradoxically you would have and it would've made you stronger.
I think it'd be mixed.
I think the knowledge of underlying hardware is useful and good to know.
But also that sort of knowledge got dated pretty quickly in the early computer era. Further, the capabilities of things like optimizing compilers quickly got to a point where they'd outpace most hand written assembly. Today, it's basically just floating point operations where you can still do better than a compiler.
In the early days, you'd have the correct impression that the C compilers spat out utter garbage which was a lot slower than what you could hand craft. As optimization techniques got better and better, the work you did because the compiler was dumb ultimately would have gotten in the way.
Exactly. I'd argue that all those programming Gods and Gods because they went through that period. Whatever didn't kill them made them stronger. We should replicate that experience by deliberately writing in low level C and assembly for a few years.
Memory mapping/bank switching was fairly common on 8-bit and 16-bit systems, where a small memory window was used to select different memory banks, allowing a program to access more memory in chunks.
Game consoles like NES, SNES and Game Boy had additional hardware built in the cartridge to support memory mapping/bank switching.
For PCs, EMS (memory) provided a similar concept. It reserved a 64 kB window divided in 16 kB pages in the first 1 MB and allowed to map up to 32 MB.
I first found out about segmenting in 16 bit systems in 2016 by reading a lively explanation from an older edition of Duntemann's Assembly Language Step by Step (the newer editions focus largely on Linux and 32/64-bit systems).
16 bit programs used 16 bit addresses, generally speaking.
Even with 32bit systems where you’d want more than 4GB RAM, application software still had 32 bit addresses (and thus 4GB memory limit).
I think it was a lot more common for 8bit systems to allow for 16 bit addressing though.
It’s been a while though. So hopefully I’m not misremembering things.
> I think it was a lot more common for 8bit systems to allow for 16 bit addressing though.
The 6502 and Z80 could use 16 bit addressing to access up to 64kb of memory. The 6502 had various other addressing systems, including iirc 8 bits, but none of them were wider tha 16 bits.
4 replies →
You had to deal with two flavors of pointer, near and far. Far pointers came with segment selector, for accessing more than 64k. Your choice of memory model influenced the defaults. You might use near pointers for internal references in a module, and far pointers for external references.
1 reply →
And the 32-bit 4GB limit was often really "just a bit under 2GB" depending on the hardware, OS, etc
1 reply →
Not really. 16-bit programs on x86 used 32-bit pointers (effectively 20-bit due to the segment mechanism).
8-bit microprocessors used 16-bit addresses.
Attention spans were longer.
I've been wondering about this lately. As a kid, I spent hour upon hour learning about computing: typing in Basic code from a magazine into a Commodore 64, playing with music on an Atari STe, learning my way around a DOS command line, dabbling with 3D modelling... just so much stuff that my own kids would never have the patience for.
I wonder if it's just that kids today (gods that makes me sound old!) are constantly surrounded by entertaining things to do - gaming, TV/films, music, social media.
5 replies →
I have been wondering how to train my 6-year old son and myself to increase my attention span.
Some rules are obvious -- cutoff mobiles and pads completely (he doesn't have access to them so it's for me), sit in the library and study from books (I believe this is even possible for programming topics as I can write on paper). Basically, cutting off everything electronics definitely helps -- even putting my phone in the bag improves productivity significantly.
But the problem is, my son is unruly. If I put him in the library, most likely he runs around and messes things up, which ends up we leave early without doing anything.
1 reply →
https://news.ycombinator.com/item?id=48435428
You had to figure out so much on your own back then - and reinvent the wheel.
For me it is fascinating how today I can learn a foreign language, or how to code by interacting with the LLM.
I think they learned by reading books such as Undocumented Windows or Windows Internals (not to be confused with Windows NT internals), and Microsoft documents.
In fact, I’d argue it was more fun than programming Javascript these days.
It wasn't really the 'Undocumented' and 'Internals' books. Pretty much everything in the headlined article was to be found in the SDK, Microsoft Press publications, and in many third party books about DOS+Windows programming.
Petzold's Programming Windows book, for example, devoted an entire chapter (chapter 7) to memory management, with diagrams and examples. In the 2nd edition (which I just pulled off the shelf to check) that chapter runs to some 40 pages.
1 reply →
16-bit x86 processors took 20-bit pointers, expressed as a 16-bit segment and a 16-bit offset. The segment was shifted four bits left and then the offset added. Which means there are lots of different segment:offset pointers that point to the same address. Segments are loaded into a segment register (one of CS, DS, ES, or SS) and then combined with an offset pointer in another register to create a pointer in this way. For example, 1e37:0008 would become 1e378.
It's complicated and janky as all get-out, but it made more sense if you were coming from 8080/Z80 development, as this was a scheme to ensure some degree of compatibility with 16-bit 8080 addressing while providing access to much more memory. 8086 was not binary compatible with 8080, but was designed so that 8080 programs could be machine converted to 8086 ones.
In languages like C, this took the form of three different types of pointers: NEAR, FAR, and HUGE. NEAR pointers were 16-bit offsets only, and dereferenced with respect to the current segment (usually in DS). FAR pointers were full segment:offset pairs but pointer arithmetic was only done on the offset which meant objects could be 64K max. HUGE pointers allowed for objects larger than 64k but at a significant performance cost.
In 1994 I was 2 years out of school. I'd written one windows shareware application and a whole lot of unix-y things. People were excited about the internet but most people didn't have access. Unix shell accounts via dialup were common though.
One day I was encouraged to write a Windows Sockets emulation layer for ordinary dial-up shell accounts like those offered by netcom. The idea was to allow the use of the recently released Mosaic browser without an actual internet connection. I figured sure, no problem. I'll use curl or some other tool in the shell account to do the actual fetching of URLs, transfer styles over zmodem, and simulate all the tcp/ip calls in the DLL.
I couldn't even get started. The reason is that I couldn't understand how the different Windows applications could all share memory allocated at runtime in the winsock.dll.
I asked a highly experienced ex Microsoft person, and he just said what are you talking about. There's no API to allocate shared memory.
So I gave up. 6 months later someone else did it.
Around then I realized the truth: Windows 3.1 had no memory protection at all. Specifically all global variables in DLLs were shared by default. The hard part wasn't sharing memory among users of a DLL. If anything, the hard part was having good discipline to avoid sharing it.
Since I'd only used multiuser Unix in school, and I knew Windows supported multitasking (even if only the cooperative kind), I just couldn't wrap my head around the idea that I'm multitasking operating system could exist without memory protection.
Pretty good detail in this article! But what really surprises me is how some ideas just keep coming back.
When I wrote a binary translator, I ended up having to keep a translated return stack to optimize RET opcodes. That put me in exactly the same position as the Win16 kernel with regard to having to patch pointers (in case of Win16, just the segment part) on stack.
Of course I did not have the benefit of my guests calling a lock function, so I ended up having to run a garbage collection operation to determine which pointers are in use & take exceptions on now-invalidated segments. Lots of extra work that Windows didn't need: it's nice to be king :-)
If you think programming in Win16 (or whatever we want to call it), you should try teaching people to do it. I worked as a commercial trainer on C and Windows way back when - C and the Windows API were no bed of roses, but the different memory models were mind-numbing for us tutors and the poor punters, many of whom didn't know C!
Thank god for the 386.
> Exports are used for application code which is externally called.
This was the magic moment for me, learning Windows 3.0 programming. The idea that my program is no longer master of it's world, but instead is just something that gets loaded and called by Windows.
I posted the same thing a few days ago:
https://news.ycombinator.com/item?id=48424862
I'll just stop posting on HN.
Check the ID numbers (48410844 < 48424862) and bear in mind that Hacker News has this thing where sometimes submissions get re-cycled for attention. Yes, annoyingly it does seem to make the presented datestamps wrong.
I think it's just bad timing.
It is always a matter of luck.
People submit a lot of stuff all the time, very few people go through "New" and thus a new submission probably have a very short life time before it is drowned by newer submissions.
A submission to survive most likely needs some initial push from non-organic voting.
It probably helps if you share you submission early with your colleagues and in other sites.
I've had this experience a few times so I don't post submissions anymore either (including one of my own articles being flagged despite over a hundred comments). I know people will say vote rigging doesn't happen on HN, but I think it's naïve to think any site on the internet is impervious to vote rigging.
This has happened at least 4 times to my posts just last month.
Good informative article.
Win16 programming was an important formative phase in my career. There is a lot of wisdom in old solutions to thorny problems and knowing them often clues you to how one may adapt them to today's problem. For example, when CPU+GPU programming appeared i immediately imagined CPU memory accessed with "near" pointers and GPU memory accessed with "far" pointers with a switch to a pseudo-segment register.
It also conditioned a programmer to learn about various complexities involved and be careful in their programming i.e. it taught you discipline. You understood your compiler, OS and hardware better and how to write code keeping them all in mind. For example, i often say my study of embedded programming started with Win16!
Another bit of cleverness was "Thunking" between 16-bit and 32-bit code. Here is Raymond Chen on how it worked there and Why can’t you thunk between 32-bit and 64-bit Windows? - https://devblogs.microsoft.com/oldnewthing/20081020-00/?p=20...
[flagged]