← Back to context

Comment by bonzini

17 hours ago

At the time of DOS, x86 didn't have multiple privileges. The system call instruction was typically INT, the software interrupt instruction.

Later on the 386 Intel added virtual 8086 mode which trapped to the kernel privileged instruction exception also for certain instructions that had to be virtualized, among them INT.

Yes, exactly.

We used a set of INT instructions in well-known low memory addresses that all jumped to the same place. We had an ASM file that you linked with, that had sixteen different address combinations for each.

The common entry point would look back on the stack and calculate from the return address which entry point had been called, and run the appropriate kernel call. We called it the CS:IP hack.

In the context of this post, the DOS INT10 and INTx(I forget) required the caller to load registers with the desired system call number, then perform the trap instruction in their code. Fortunately CTOS didn't need those particular software interrupts, so I could implement them for my purposes.

  • Windows 95 used a related hack. Whenever a v8086 program asked to create a call to protected mode code ("please give me a real mode address to call to, in order to start executing the protected mode routine at address 0x123456"), Windows would store the entry point in a table and hand out real mode addresses like FFD0:0, FFCF:10, FFCE:20, FFCD:30, FFCC:40 that all point to the same instruction (because the segment part is shifted left by 4 in real or v8086 modes).

    The routine at 0xFFD00 could then enter protected mode and use the code segment to build the index into a table of entry points: FFD0 goes to index 0, FFCF goes to index 1, and so on. But for extra kicks, the address isn't actually pointing to valid code. It points to a random "c" character in the BIOS, which is an ARPL instruction - which in turn is invalid in v8086 mode and therefore invokes the undefined opcode exception handler. The exception handler, which handily enough is already running in protected mode, then takes care of doing the 32-bit call.

    Related: https://news.ycombinator.com/item?id=45283085