Comment by amluto
2 days ago
In x86 land, there are, as a practical matter, four ISAs: real/v8086 mode, 16-bit protected mode, 32-bit protected mode, and 64-bit “long” mode. Machine code targeting one of these will be executed correctly by the CPU as long as the CPU is in the right mode. (Really it’s messier — there are the CS.D, CS.L, and SS.B bits plus the control bits for v8086, protected and long mode, but this barely matters.)
Sure, this is messy. But, critically, on x86, these are all modes, and any CPU that supports them makes them detectable and supports them in the same way. If you run long mode code outside long mode, some opcodes will be interpreted as the wrong instruction. But you will not find multiple different CPUs that decode valid instructions differently. If I run your weird old x86 code, either it will run correctly or it will fault.
Oh, and all these modes are older than RISC-V. To the extent that there are lessons to be learned, RISC-V should have learned them.
The fact that you can apparently find two RISC-V CPUs that decode some ordinary user mode instructions based on published standards as entirely different operations is bizarre, to say the least. The fact that the relevant CPU features can’t even be enumerated in user mode just makes it worse.
(There are edge cases in x86. Some invalid opcodes have different lengths on different vendors’ CPUs. This is not a problem in practice because, one way or another, they fault. There was also a little glitch in the 64-bit design where some really really old x87 FPU code that uses exceptions cannot be corrected handled by a kernel on a modern CPU.)
> But you will not find multiple different CPUs that decode valid instructions differently. If I run your weird old x86 code, either it will run correctly or it will fault.
Oh, that's not completely true. Intel 64 and AMD64 are not identical and they certainly have encodings that behave differently. As an example: f3 41 90 is pause on Intel, but xchg r8d, eax on AMD (granted, this is not a canonical instruction encoding). 66 e9 xx xx yy yy is a unconditional jump to a 32-bit relative offset on Intel, but on AMD, the offset is 16-bit only (yy yy are the start of the next instruction). x86-64 is typically used to refer to the very large common subset, but this doesn't mean the implementations behave identically.
There are also some weird corner cases where CPUs aren't 100% backwards compatible, just backwards compatible enough for the software that matters.
For those less familiar with x86, the example instruction encodings that are interpreted differently on Intel and AMD are not base encodings, but instruction encodings modified with prefixes.
The x86 ISA includes a great number of bytes that are used as instruction prefixes, many of which are obsolete. The problem is that the effect of prefixes upon instructions has never been completely defined in any Intel or AMD documentation. The prefix effects have been documented for some instructions, but they were left unspecified for most other instructions.
This has lead to divergent implementations in the unspecified cases. Well-behaved compilers should not generate such undocumented combinations of instruction prefixes with base encodings.
> The problem is that the effect of prefixes upon instructions has never been completely defined in any Intel or AMD documentation.
In case of the jump example, the effects are documented by Intel and AMD and they still differ. Point of the GP was that all CPUs don't decode valid instructions differently, which is not fully accurate as shown by the examples; and some of these differences are also explicitly documented.
It's also not accurate that most prefixes are obsolete when most see regular use today (66 size override for 16-bit operations, f2/f3 for string operations, 66/f2/f3 mandatory prefix for many (e.g. SSE) instructions, 64/65 fs/gs override for thread-local storage access and per-thread kernel storage, f0 lock for atomic operations, 3e (again) for branch-taken hint, 4x REX prefix for r8-r15 and 64-bit operand size; one can argue that the 67 address-size override is useless, and only the 26, 2e, and 36 are ignored; I don't count VEX/EVEX/REX2 as prefixes but more as opcode escapes).
Pretty sure 3DNow's instruction space has been repurposed too. There are a few like that, or worse extremely similar but ever so slightly different between vendors or even generations of the same vendor. I won't claim to be an expert, this is all in IIRC territory for me.
No, the 0f 0f opcode (most of 3DNow was encoding-wise a single RMI-encoded opcode where the immediate specified the operation) and the 0f 0e opcode (FEMMS) were not repurposed by either Intel or AMD.
Fair point.
66 E9 is not a practical compatibility problem, though: it’s not a useful encoding of a useful instruction on any CPU :)
> There are edge cases in x86
Some early 386 CPUs had XBTS/IBTS instructions (extract/insert bit string). They used 0F A6 and 0F A7 encodings.
Some early 486 CPUs had CMPXCHG encodings that reused the XBTS/IBTS encodings. That apparently screwed up some programs that tried to execute XBTS/IBTS to see if they were running on early (buggy) 386s so Intel moved CMPXCHG to different encodings (0F B0 and 0F B1).
0F A6 and 0F A7 are still left unused by Intel and AMD in their modern chips.
There's also the delightfully petty story about SYSENTER/SYSEXIT and SYSCALL/SYSRET for fast system calls. Intel came up with the first pair, AMD with the second. AMD of course had to support both and operating systems generally only supported Intel's pair.
Then AMD extended the x86 to 64 bits and of course required SYSCALL/SYSRET for 64-bit system calls (and did not support SYSENTER/SYSEXIT in 64-bit mode). Intel had to support AMD's instructions in 64-bit mode but decided to also support SYSENTER/SYSEXIT there (which I don't think any operating system has ever bothered to support).
To summarize: AMD supports both methods outside of 64-bit mode and only their own in 64-bit mode. Intel supports both methods in 64-bit mode and only their own in 32-bit mode. Or at least that's how it used to be. Maybe they've mellowed out by now.
Oh I remember SYSENTER/SYSEXIT, I think. I worked on CTOS, widely released by Burroughs/Unisys back then. CTOS used all the clever instructions. They performed badly. Later I learned from an Intel customer consultant on porting code, that we were the only ones who ever embraced fancy things like task gates etc. Everybody else just continued using regular push/pop and simple traps. Because they ran faster.
Yes, but x86 had a goal of running every old piece of code on the new thing. That’s not true with RISC-V. Whereas x86 accreted more and more features, RISC-V solves this with profiles that are not guaranteed to be compatible with each other. RISC-V doesn’t even support running 32-bit code on 64-bit processors without a recompile. In a sense, 32-bit RISC-V is a vaguely similar but incompatible ISA to 64-bit RISC-V. That would be a train wreck if there weren’t profiles to specify a group of features that must be there (and some others that might be optional, with a register to flag whether they are or not). The expectation is that profiles act a lot more like the x86 ISA, accreting features while preserving backward compatibility. It took me a while to realize this, too, and I remember several WTF episodes while reading through the specs. And of course profiles go beyond the ISA and specify system level arch as well (like the “PC architecture” did for Windows and Linux). You can certainly argue that it should have been done differently, but it’s not completely crazy given that there is not legacy RISC-V code needing to be run on newer systems for the most part. That won’t be true forever, and so profiles help manage that.
> The fact that the relevant CPU features can’t even be enumerated in user mode just makes it worse.
User-mode feature detection is usually used to select paths for acceleration instructions, like SIMD or crypto. The overlapping RISC-V instructions don't fit in that category: they're compressed versions of basic functions, mostly used in epilogs/prologs, which would be unconditionally compiled in.
There are no overlapping encodings in the 32-bit encoding space and I'm really hoping it stays that way.
> To the extent that there are lessons to be learned, RISC-V should have learned them.
Yeah, I think I agree with this. Also I wish I had been there when Andrew Waterman was writing his master's thesis so I could ask him not to include Whetstone in his size benchmarks, so that we might have left that encoding space free and avoided this conversation :-)
At least with user-mode feature detection there could be an assertion that the correct feature set is present.
Doesn't the ELF loader do that?