Comment by mrlonglong
16 days ago
the zero terminated string is I think is computing's biggest mistake. Pascal style strings were much safer.
16 days ago
the zero terminated string is I think is computing's biggest mistake. Pascal style strings were much safer.
Zero terminated string is a special case of sentinel value termination.
And sentinel value terminations make a lot of sense when you have punch cards and fixed length records that you need to carve into pieces.
Nobody expected any decisions they were making in the 1960s and 1970s to have any bearing on computing a half-century later. They all expected to have their mistakes long papered over by smarter people at some point.
But we ALL make the mistake of underestimating inertia.
There is a middle ground that Visual Basic (and then COM) took, with the BSTR type: It’s still a pointer to a zero-terminated char array, but there is a length field immediately preceding the first pointed-to byte. This is still compatible with a C string (assuming no embedded null characters), but BSTR-typed functions can take advantage of the length value.
> This is still compatible with a C string
Strictly speaking, it's not alignment compatible from CString to BSTR unless you declare all strings to be at most 255 characters or the cpu architecture doesn't require aligned access for multi-byte words (like x86). The BSTR alignment must match the alignment of the length word, meaning you can't convert a randomly-aligned C string to BSTR by simply attaching a prefix in-place.
Also, having the length embedded in the value rather than in the pointer makes it impossible to create BSTR (sub)slices without performing a memcpy. Fat pointers do not have this restriction.
A BSTR object is compatible with functions expecting a C string. The other direction obviously never holds, unless the C string is a BSTR to start with.
Yes, there is a trade-off between slices using the same format and having compatibility with C strings. Hence “middle ground”.
You can still use a string-slice type on top of BSTR, it just would be a separate additional type. Note that languages like Java also don’t have a singular type for strings and string slices.
These are great for "data smuggling" attacks where one layer of code assumes the length is 'x' and another layer assumes it is 'y'.
It makes hybrids like this very dangerous for anything even remotely security-adjacent, such as roles, tokens, etc.
This kind of thing caused the CVE-2009-2408 and CVE-2009-2510 "Null Truncation in X.509 Common Name Vulnerability."
FWIW pretty much all Pascals since the 90s use a similar approach.
In Free Pascal for instance, strings are pointers to the first character with a header in a "negative address" containing information about the length, reference count (strings are reference counted and use copy-on-write to avoid passing around copies all the time) and codepage (FP can convert strings between different encodings "transparently").
Nowadays replaced with HSTRING.
Partly agree but there would have been squabbling on the data type of the size, unless it was variable length. The latter would have had other issues too.
For a while, 16bit would probably have seemed too extravagant. Now 32bit would probably seem too small.
For a “strongly typed” language, C is pretty damn loose where would have mattered.
I like the D approach where arrays are just `struct { size_t length; T* ptr; }` internally --- and strings are just arrays of `immutable(char)`.
It has a big advantage over the Pascal approach in that you can do zero-copy slicing, since the length is separate from the actual data.
And `size_t` makes perfect sense for the length here. If your strings are longer than the address space (which `size_t` technically isn't, but is practically very strongly correlated to it), then you're going to have a problem regardless of the number of bits for the length anyway.
This only makes a difference in terms of memory size, not in terms of speed, because for decades processors and compilers have been optimized for moving bytes around.
But one would note that in order to gain memory for this particular case of slicing, one introduces 2 extra words (size and pointer) for every other cases. Like perhaps the second most common string operation, concatenation. In those other cases, the benefit is slightly negative.
I've had extensive experience with "counted strings" because I implemented a bunch of Forth interpreters which also uses this scheme. Including the common trick of using counted and zero-terminated strings, which is the worst of both worlds in the end. Forth is the kind of language that quickly show you how bad your choices are.
I eventually dropped all that and adopted ASCIIZ strings because they are generally more efficient (if you pay attention to the strlen() performance pitfalls) and having a dead simple interface with the rest of the world (OS, libraries) is more valuable.
1 reply →
C is a weakly typed language. It’s statically typed, which is a different thing
No, there would not have been and this is most likely not the reason. size_t exists for precisely this use case. It has existed since C89.
C is not really strongly typed.
"Strongly typed but weakly checked"
It turns out that the machine is much better at the sort of boring mechanical tasks where thoroughness counts and imagination doesn't and so languages which do more, and more, and more checking pay off very well. Rust's borrowck is the obvious first thought today but say WUFFS will check that you've proved certain key properties, WUFFS doesn't need to insert runtime bounds checks for example because you've proved, before the code would compile, that you don't have any bounds misses. You might have proved it by writing bounds checks yourself of course, or likely you have an inherent mathematical rationale for why your algorithm has no misses, but either way the compiler checked your work.
3 replies →
I don't think 32 bits for the size would be too small. That would max out at a 4GB string, which is large enough that even today it's a big red flag saying "what are you doing bro, reconsider your approach". I can conceive of a string larger than 4GB, but I can't conceive of a situation where it's reasonable to use one.
Zero terminated strings were the basis for an awful lot of useful software. Calling them the biggest mistake in computing is a bit OTT.
I haven’t programmed anything Pascal related for 30+ years but I dimly remember thinking at the time that I wished the string system wasn’t so hard to use.
That useful software would not have been less useful if the strings in it were represented as size + buf.
Oh really? Have you tried to rewrite anything to put your theory to the test? I don't think it's as straight forward as you think it is...
1 reply →
That argument isn’t valid. The argument would be “this string design enabled a whole lot of useful software” but that’s a different matter. (And it could very well be the case.)
Lead was the basis for an awful lot of useful gasoline. Doesn't mean it was the only solution or the best one.
A more accurate re-phrased version of the original is that they are the biggest mistake in the C language.
* https://news.ycombinator.com/item?id=1014533
zero terminated strings may cause a lot of bugs, but it also ends up in a lot of useful software, so, it;s impossible to say if its bad or not,
255 characters ought to be enough for everybody, right?
You mean bytes.. we have multi-byte characters now.
It was definitely an interesting way to allocate pointers. I did once have a very large project where devs didnt understand this and resolved hundreds or more off by one and memory overwrites in C due to this feature.
But at the same time, I think blaming the software was kind of a cop out. Devs were in a hurry and simply didnt respect the rules. Given todays software engineer at large. Nerfing programming languages so they cant destroy things might not be a bad idea. But AI will nerf everything.
why is AI gonna nerf everything? sure it could be used as the easy button, but I just spent two hours this morning learning about the neuroscience of how memory works in the brain that I didn't mean to and now I want to run studies on how memory works.
Why do you assume that AI is gonna nerf everything?
AGI might. AI? No way.
See, AI was trained on existing data - on all that existing C code out there (sure, and also on all the papers and articles saying what was wrong with that C code). Those bugs are in the training data, and often not marked as bugs. So when AI generates C code, is it going to avoid making the mistakes that human code made? No, it's going to generate the kind of code it was trained on. How could it be otherwise?
That's not going to nerf anything.
12 replies →
Clang and GCC both let you use Pascal strings in C if you would like (with `\p`). But Pascal strings aren't that useful today because the maximum length is too short.
Why would a pascal string be any shorter than a C string?
A C string is one pointer reaching all of memory, a Pascal string is two pointers reaching all of memory
A pascal string is a single byte with the length, followed by the data.
Some implementations use more bytes for the length data, such as Delphi which changed over to a 4 byte prefix length, though those aren't technically Pascal strings anymore. I can't find anything about a Pascal string being two pointers?
2 replies →
A Pascal string has a leading length byte. Because that is one byte, the text can't exceed 255 characters.
3 replies →
C strings (sentinel terminated strings) are infinite. Anything less than infinite is shorter than that. Anything with a known length is shorter than that. This makes them generic. Same type of constraint as you see in many algorithms.
Almost as bad as newline-terminated lines. ;)
What's bas about them and what are the alternatives, genuinely curious since never seen them spoken of
In addition to having to pick a size for the length counter and then, later, having to differentiate between lengths in bytes, codepoints, and glyphs, you can't subdivide a Pascal string using pointer arithmetic. To pass just the end of a string into a function, you have to either copy the tail of one Pascal-style string to another with a smaller size value, or your string has to be a struct with an integer and a pointer to the actual data instead of just an integer stuck on the beginning of the string. The first is a lot of copying in some cases, the second raises the specter of structs with invalid pointers. That's not to mention the potential problems that would cause with caches.
You can have a universal variable length field, for example 2 bytes for strings < 32768, then four bytes, 8 bytes etc. On the critical short string path, it costs just a single bit test. The glyph vs byte issues need to be dealt with in both formats.
The subdivision issue is a good perspective, but i would argue the performance impact of cloning substrings is dwarfed by the redundant full string reads to find length.
> You can have a universal variable length field, for example 2 bytes for strings < 32768, then four bytes, 8 bytes etc.
To hold the length of a string, I'd do something similar to unicode:
7-bits for size + 1-bit for continuation, then 15 bits for size + 1 bit for continuation, then 23-bits for size + 1 bit for continuation, etc.
Or maybe even do it exactly the same as unicode:
> On the critical short string path, it costs just a single bit test.
A few more clock cycles compared to NULL-termination, although my alternatives above require even more clock cycles.
If the hardware had instructions for sentinel values, things would be easier (Like how DOS calls used '$' termination for strings) and safer.
Load a sentinel byte into a register and have dedicated copy and compare instructions that take each two addresses (src and dst) and copies (or compares) src/dst until the terminator is reached (with copy copying the sentinel as well).
Considering that sentinel values are needed so often, and are so useful, it's surprising that this is not in any ISA. What we have now is kludgy workarounds in the HLL for this. It's hard to blame the HLL, because some workaround has to be implemented.
8 replies →
You could do 0xffff as a special case, and put another length+string/pointer to after the 255th byte.
The third option is to have a variable width length: the top most bit signals whether the next byte corresponds to the length or to the start of the string.
.. which is why you need a second type, the one dotnet calls "Span". A substring.
> Pascal style strings were much safer.
The limitations were brutal. Initially you could only have 255 bytes in a string. The length of a string and the size of the allocation are now separate and you may need to think about that unused memory in your design. The problem now doubles with the introduction of UTF-8. Your string size is in bytes and you need to track characters separately.
If you want to create an array of strings you either need to specify the length of all strings and accept the memory overhead or have an array of pointers to strings. If you use an array of pointers you may end up choosing to use the 'nil' value as a sentinel that means "end of list." So we're right back where we started.
--
Because someone decided to downvote this HN has limited the speed at which I can reply. This site is tragic and I'm fully done with it now. You can spread propaganda and poorly sourced zeitgeist and be among friends but if you try to have a genuine conversation about programming languages you are made to be unwelcome immediately. Screw this.
--
> No other data structure works like this.
The linked list.
> You can't mess this up in an array
C happily decomposes arrays into pointers. You can erase your length information from the type. This was an intentional decision.
> Strings are the only data structure that assume there will be a NULL at end.
Which is why almost every string API has a version that allows you to specify the maximum length. The fact that you can use a NUL doesn't mean you have to. Which is why the concept of "sentinel values" is broadly used in many types of applications you haven't considered here.
> You can spread propaganda and poorly sourced zeitgeist and be among friends but if you try to have a genuine conversation about programming languages you are made to be unwelcome immediately.
Indeed. And the ignorance of computing history in this discussion is particularly disturbing.
The context of this particular thread is "zero terminated string is ... computing's biggest mistake". This completely ignores the situation on the ground when C was developed. At the time, people were striving for a system programming language that sat above the level of assembly but was compact enough to run within the limited resources of the then emerging mini-computer systems. The PDP-11 on which C was developed was certainly not the first mini-computer, but it was among the earliest to have a regular enough instruction set and addressing model to make a general purpose, high-level system's language possible. These systems were extremely limited in memory; the PDP-11's instruction set is limited to directly addressing at most 64KiB (code and data) and many systems of the era were hardware limited to less than that. (Indeed, I regularly run an early version of Unix, including an early C compiler, on my PDP-11/05 which is maxed out at 56KiB [of actual core]). There was no way that even a brilliant engineer like Dennis Richie was going to be able to shoe-horn in "optional" types, or the mechanics of length-value strings into a compiler that has to run in such limited space, and produce code (e.g. the Unix kernel) that has to run in even less. The fact that strings and arrays are thin abstractions on top of pointers is both a brilliant compromise in design as well as a nod to then-prevalent assembly practice. It was the exactly kind of pragmatic decision that was needed to move computing along at the time. Of course the designs from this era are antiquated now. But they were not mistakes.
> This completely ignores the situation on the ground when C was developed.
A great many of those replying are many years short of having experienced anything like "the situation on the ground when C was developed". They simply have never known a day without hundreds of gigabytes or more of disk storage and 8G or more of RAM available for user processes after the OS consumes what it needs for its own work. They are "ignoring" because they simply have no basis for understanding.
The C code for strcpy is:
On a PDP-11 that is:
All those limitations were sorted out in 1978 with Modula-2 and open arrays, aka spans.
What about the UNIX and C folks propaganda of C being the first systems language, or always focusing on the original Pascal used for teaching and not everything else that followed up with Mesa, Modula-2, Ada, Object Pascal and friends, none of them with said limitations.
C was specifically developed to allow Unix to be ported.
It was a systems programming language and the first well known/successful one.
There was BCPL and then B before that, which is why the language is called "C".
Pascal was considered a teaching language, along with "Algorithms + Data Structures = Programs" by Wirth etc.
The UCSD P-system was one of the first "IDEs" and used Pascal and a bytecode interpreter of the compiled code.
Modula-2 was barely available in the early 1980s.
Ada was mired in MIL-SPEC and expensive compilers etc.
People used FORTRAN for scientific programming, C for most everything else in the non-IBM mainframe world.
10 replies →
> You can erase your length information from the type. This was an intentional decision.
Well yes, but given the number of security issues the argument is that it was in retrospect the wrong decision.
> Your string size is in bytes and you need to track characters separately
No worse than C strings then.
>The problem now doubles with the introduction of UTF-8. Your string size is in bytes and you need to track characters separately.
That isn't really a problem.
The problem with null-terminated strings is specifically what happens when you reach the end of the allocated array and there ISN'T a NULL character.
Every string function is designed to keep going until it finds the NULL character, so if a hacker gets rid of the NULL character, he can exploit pretty much any standard string manipulation function being used elsewhere in the program to manipulate whatever memory comes AFTER the string data structure.
No other data structure works like this. You can't mess this up in an array, because no function that manipulates arrays is just going to keep going until there is a null. That would be stupid because it would require users of the function to add a NULL to the end of their arrays before passing it to the function, so instead we just pass the size of the array to everything. Strings are the only data structure that assume there will be a NULL at end.
By the way, I read once that if you use UTF-32 every code point will be 4 bytes, constantly, but even then a single code point isn't necessarily a single character. Text is just complicated.
> No other data structure works like this.
In C most data structures work like this, you keep going until you find NUL (character) or NULL (pointer). E.g. Strings, array of pointers, linked lists, etc. Of course you can add length to most of those, but it isn't the canonical/traditional way of doing things.
2 replies →
> No other data structure works like this. You can't mess this up in an array, because no function that manipulates arrays is just going to keep going until there is a null.
This is patently false. Sentinel markers are used widely in array types. Consider GNU's getopt_long() function, a mainstay in GNU tools:
The argument longopts must be an array of [struct option] structures, one for each long option. Terminate the array with an element containing all zeros.
> Every string function is designed to keep going until it finds the NULL character, so if a hacker gets rid of the NULL character,
What sort of situation are you envisioning where a hacker can remove the sentinel (in the case of nul-termination) but not modify the length bytes (in the case of fat pointers)?
1 reply →
If I zero out the destination buffer of a strcpy, and the string is longer than the destination buffer I will run into a buffer overflow problem despite every byte being a zero byte. The absence or presence of the zero byte doesn't seem to be the deciding factor.
> the zero terminated string is I think is computing's biggest mistake.
No. They had trade-offs to make, and sentinel-based sequences are a needed thing, even outside of strings.
The mistake was that ISAs never looked at what HLL needed, then add the necessary instructions (I posted more about this below).
Even NULL is not a big mistake, when looked at in context of the time in which it was developed.
compared to Von Newman versus Harvard architecture for LLMs? I think that's a far bigger mistake.
Neumann, and .. what? In what way?
Prompt injection only works because there isn't two streams of input to give to the LLM. Von Neumann being the architecture with a single shared memory for both data and instructions. If there were a clean way for the LLM model to distinguish between system messages vs user messages, we wouldn't have that problem.
1 reply →
[dead]
Rust has "pascal style strings" (quotes because the concept is slightly different) so it's not a done deal
I think it was NULL itself. It was a long way until we realised we don't want invalid values and could use the type system to help us use special values safely.
The problem here is that null kinda is consequential of intentional design of the type system itself. In this way, I do think that null was discovered, rather than invented. Remember, C is a kinda "portable assembler" so the constructs in it are based relatively closely to how low level data structures are mapped out in memory.
This is, and continues to be, an incredibly useful feature that makes C and C structs immensely useful concepts. Part of that does need an invalid value[1]. NULL is convenient for this and although there are some very weird JavaScript-trinity-meme-style consequences for this[2], it's such a useful concept that basically all languages that have the ability to construct pointers have a null pointer[3].
The alternative world looks like everyone inventing their own invalid values. Invalid, non-null, pointers are typically MUCH worse than null pointers for debuggability and security. If you unintentionally read/write/execute memory at 0x0 (by far the most common value for NULL), most operating systems will trap this, whereas may not necessarily if 0x12345678 is your invalid value.
[1]: Stuff like IA64 had NaT bits which were effectively an extra bit for what I assume to be this sorta thing. The problem with this is that it costs an extra bit. I don't really know much about IA64, but presumably [NaT 1] + [don't care] would be your null pointers here. I think?
[2]: Really what the standard, in my opinion, should have done is probably not make use of the null pointer UB for many different functions. A lot of compilers took the UB surrounding that to make incredibly dubious "optimizations" that broke stuff with zero actual performance benefit whatsoever
[3]: Yes, even Rust. Although some (again in my opinion) unfortunate design decisions made it so that C-Rust FFI isn't zero cost because of how it treats spans/slices
>[3]: Yes, even Rust. Although some (again in my opinion) unfortunate design decisions made it so that C-Rust FFI isn't zero cost because of how it treats spans/slices
If Rust slices already make you sad, then the thing I'm cooking up will make you cry for days.
Compared to scripting languages with actual tagged types, C doesn't really have a type system, and that's readily apparent to anyone who has written C in the last 43 years and debugged a program written in it.
C pretends types exist with you, but once bytes hit the road, it's all real-life and segmentation faults.
C actually does have a type system and it's one of the bigger issues with the language. If it didn't, unaligned pointers and signed overflow would be totally fine.
6 replies →
By that logic, no natively-compiled language has a type system.
Though I should note that in a way, even some ISAs have one, what with e.g. separate float vs integer registers.
Genuinely curious, how would you handle cases where a value is unset without NULL? This is a legitimate case that happens a lot in eg data modeling
Sum types, of course.
23 replies →
The way we do it in modern languages with things like std::optional and even that is not the best example.
3 replies →
They already said:
> use the type system to help us use special values safely
... but this is not the place to explain what a type system is or what sum types/maybe/optional/etc. are.
Meh, I think NULL is fine in C. It's an extra, valid state to represent pointers at no cost. Unlike the more hand holdy languages, it's quite rare for a pointer in C to have the ability to be NULL since, more often than not, it's pointing at something known. It's actually quite rare to see NULL checks unless it's API code or something like that. I can see this being more of a problem in a managed language where anything can be NULL at any time.
NULL as a concept is fine. Inability to declare something as non-null is not.
There is a huge gap between developer expectation "it's pointing at something known" and hard reality confirmed by zillions of CVE. That's the reason optionality is prevalent in modern languages and type checkers (python, typescript), nowdays even Java has sane non-nullable types.
> to represent pointers at no cost
I wouldn't call "cause of bugs and security issues" "no cost".
> it's quite rare for a pointer in C to have the ability to be NULL
As a C programmer for more than 25 years, that is the exact opposite of my experience.
Struct foo has various members, including a bar*. But a foo may or may not be associated with a bar. If there's no associated bar, the bar* pointer is NULL. Seen and done this all the time
The problem with let's get rid of NULL is that it's a real, required state. The vast majority of computing is actually not binary: any real input generally has at least 3 possible states: not set, true and false.
In practice really 4 because "indeterminate" is a reasonable error condition you'd like to know about.
And it keeps increasing anyway: e.g. not set has subcategories: not set due to lack of user input, not set because we're loading state from the backend etc.
NULL is the first expression of that basic problem: it's definitely not enough to eliminate NULL because the first thing which happens is your non pointer default value takes it's place.
3 replies →
[flagged]
4 replies →