← Back to context

Comment by rswail

16 days ago

Things that have bugged me for 40 years...

* NUL terminated strings (and now, non UTF-8 encoded strings on input/output)

* Using LF or CR or CRLF as line terminators, and pipe/comma-delimited fields when there were other unambiguous ASCII characters that could have been used (eg, GS, FS, RS) that would have made the encoding/decoding of line termination an I/O thing keeping HT/VT/CR/LF/FF as literally print related codes.

33 comments

rswail

Reply
EvanAnderson  16 days ago

I did a project to translate data framed in the ASCII field/record separator characters and it was gloriously easy. All the ugly escaping considerations with comma-delimited data went away and it became much easier.

  • ptx  15 days ago

    What happens when the data contains the record or field separator characters?

    I suppose you could document that it's unsupported, and just drop or reject such values, but then the system couldn't be used to handle test data for such systems, for example.

    • EvanAnderson  15 days ago

      In the case of this system (a quasi-EDI interface used to move records from a fleet fueling point-of-sale system to the ERP software) those characters were forbidden by the source application. My code would have exploded in a fireball if they had been present, but the specification said they couldn't be.

    • rswail  15 days ago

      If it's purely binary data, then you can't.

      Otherwise you need to have some sort of escape mechanism, exactly like quoting strings in CSV. In fact, there's an ASCII code "ESC" for entirely that purpose. :)

      The problem is that those characters are non-printable, which means if you're just dumping the file out somewhere, you can't see them.

    • a96  15 days ago

      Same as any separator. Either it's not in acceptable set of non-separator input or there's an escape (that can also escape itself for the literal).

brewmarche  15 days ago

Now with Unicode we actually have even more:

NL Next line (from EBCDIC?)

LS Line separator (invented by Unicode)

PS Paragraph separator (same)

The Unicode standard says that in addition to CR, LF, CRLF and the above, vertical tabs and form feeds should also be treated as line separators.

flohofwoe  16 days ago

> non UTF-8 encoded strings on input/output

UTF-8 on stdin/stdout works perfectly fine (unless you are on Windows of course, which is stuck in in the early 90s when it comes to international text encoding).

> Using LF or CR or CRLF as line terminators

This is also an operating system convention, and it would be better if programming languages wouldn't try to "guess" the correct line endings, since this causes more problems than it solves - but again, this is mostly a Windows specific problem, and it's Microsoft's job to finally bring Windows into the current century.

  • rswail  16 days ago

    No, it was an Apple, Unix, and Microsoft problem.

    Unix used LF, Apple used CR, Microsoft used CRLF.

    They are all ASCII carriage movement codes, which is about driving the paper feed and print head of an ASR-33 or equivalent.

    So they all made the "wrong" decision about what to store in a file.

    They just chose different wrong characters.

    • kps  15 days ago

      > They just chose different wrong characters.

      Unix followed Multics. Multics chose right. ASCII/EMCA-6/ISO646 drafts discussed this at least as early as 1963¹: “For equipment which uses a single combination (called New Line) [...] NL will be coded at FE₂ [Field Effector 2 = 0x0A].”

      ¹ doi/10.1093/comjnl/7.3.197

      1 reply →

    • flohofwoe  16 days ago

      > Apple used CR

      Apple hasn't been using CR since the release of OSX (26 years ago). Microsoft could have made the switch at any time too (just as they could have switched to UTF-8 as universal text encoding on Windows), they just choose not to.

      In the end it's not the job of programming languages to clean up Microsoft's mess ;)

      4 replies →

  • skywhopper  16 days ago

    What programming languages try to guess line endings? Or are even aware of them?

    • flohofwoe  15 days ago

      Ok, technically not the programming languages, but their stdlibs. On MSVC at least, opening a file in text mode via fopen will translate CRLF into LF on read, and LF into CRLF on write, which has been a neverending source of confusion since at least the 1990s.

Parodper  16 days ago

LF makes the most sense, but they're all fine for text files. The issue is that CSV isn't text.

Last time I had to handle CSV files in bash, I converted them internally to RS and FS.

  • Ekaros  16 days ago

    Line feed resetting position really makes no sense. It should just continue text from where the cursor was but on next line. Like staircase. You need CR to go back to start.

    • a96  15 days ago

      It makes perfect sense when you consider text files. When line ends, next line obviously starts from column zero.

      CR is the only wrong choice. There's never a reason to go to start of line without erasing the line or moving to next line in a file. And even user interfaces will have smarter ways to do that. It's a completely useless concept outside of typewriters.

      Well, CRLF (or worse, LFCR) is also obviously a wrong choice because it's pointless to demand two characters and create problems when one of them is missing when one totally unambiguous character will do.