Comment by jnwatson

I may be the only person who ever understood every detail of C++, starting with the preprocessor. I can make that claim because I'm the only person who ever implemented all of it. (You cannot really know a language until you've implemented it.) I gave up on that in the 2000's. Modern C++ is simply terrifying in its complexity.

(I'm not including the C++ Standard Library, as I didn't implement it.)

This is a pretty standard document length. Modern microcontrollers have similar lengths (e.g. ATSAMD51 is ~2000 pages). Some of it is not software related, things like pin outs and electrical and mechanical descriptions.

It does take a huge amount of work to write and maintain. Typically the authors are not technical, so it also relies on the designers being available to answer questions as well. Then there’s a choice of how it’s written: narrative and potentially imprecise but readable, or terse and precise but hard to read. There’s both styles in the same document, terse for register descriptions.

Look up the Texas Instruments am3358. It's a tiny SOC, it was used in the beaglebone black. Its technical reference manual[1] is over 5000 pages, and it details all peripherals, all of the interconnects and every single register in the system. This, by contrast, is really just an overview.

Regards to (1), if you don't publish this information you're not selling a CPU, you're selling a very expensive chunk of sand. There is simply no way that a customer can guess at what your implementation looks like. Additionally, Intel barely has IP in the traditional sense. They hold patents, but their only real competitor in making x86 processors, AMD, has a long-standing mutual non-enforcement agreement wrt patents.

Regards to (2), I'm guessing a majority of this PDF can be generated sort of like you generate API documentation from doxygen comments.

[1]: https://www.ti.com/lit/ug/spruh73q/spruh73q.pdf?ts=175651560...

> Real question: Why do you think Intel does this?

I'm not sure large traditional silicon vendors like Intel, TI, et al re-evaluate the documentation requirements (and costs) on a chip by chip basis. It's probably done by chip class and for companies who've been selling chips by the millions over many decades to industries as diverse as defense, aerospace, automotive, etc there are classes of chips where robust, complete documentation is not only expected but often a required part of the RFP, compliance or conformance processes.

While this level of effort probably isn't needed for every chip in that class, it could be hard to reliably predict when a general purpose chip is still in the design phase which customers may be interested in it during its life (which for some of these chips might be decades). Many chips which conform to MIL-SPEC or other similar standards which can require extensive documentation are simply enhanced versions of standard chips, so the docs exist anyway. Finally, there's the organizational capabilities and culture aspect. Once the org needs to maintain the systemic ability to generate serious documentation at scale, you end up with a lot of managers and staff who think this way.

For datasheets that's normal. Might even be leaning towards smaller than average for the device in question.

For comparison, a data sheet for a single transistor can be around 12 to 30 pages. A data sheet for a tiny microcontroller is probably a few hundred pages.

I once wrote a driver for a flash chip and that had a data sheet of around 80 pages.

In terms of (2), I wonder if it's even possible to write a driver without such a document. In the end, the vendor is on the foot for the driver for major platforms (let's assume Linux) - if they can write a Linux driver without a similar spec to this doc, then the doc probably doesn't need to exist since the business wins from hobbyist drivers will be low. If they can't though, then it's just a matter of formatting an internal document for public consumption - the doc itself has to be maintained anyways so the cost seems lower and maybe reasonable. I have a feeling the doc is necessary but I am not specialized in the field.

Assumptions, fair or not, about (1) seems more likely somehow.

Probably CPU vendor culture? I forgot how large Intel's manual set is, but ARM's was ~11k pages the last time I checked. Intel's was smaller, but not that much smaller, certainly within an order of magnitude.

Id wager high frequency trading applications.

Good christ this is my current work laptop. It...mostly doesn't work. Plug in a USB camera and it'll just go. Several drivers, userspace utilities and other daemons and sometimes gstreamer works, but does Zoom work? Who knows!

I mean when the hobby OS wants to shut down, it can power the machine it's running on down. Not unlike what would happen if you clicked power off on your desktop OS menu.

Getting it to work on GCP meant properly driving something called the Intel PIIX4 controller which was emulated into the VM.

Separately from the OS being able to turn itself off, the OS needs to process a signal received by the hypervisor on this controller to support the hypervisor gracefully shutting it down. Otherwise GCP will wait 90 seconds after it has sent the shut down signal to give up and terminate the VM itself.

The problem I was trying to solve was (a) OS can shut itself down in GCP (b) restarts in GCP from the GCP console would be instant, rather than take 90+ seconds

> You are kinda wasting your time, the only way to make it work is to make the hobby OS Linux.

Not the parent, but of course they're wasting their time... That's the point of a hobby OS.

I'm working on a hobby OS, and I have no illusions that it's most likely fewer than 10 people will ever run it, and less than 100 will hear about it, but it lets me explore some interesting (to me) ideas, and forces me to learn a little more about random pieces of computing. If I ran on GCP, I'd want the reboot button to work. That sounds useful.

On the topic, I don't see why anyone would want to build a general purpose OS. There's enough already and even with the shrinking of hardware variety, there's a lot of stuff to support to make a general purpose OS work on enough hardware for people to consider using it. You can take Linux or a BSD and hack it up pretty good to explore a lot of OS ideas. Chances are you're going to borrow some of their drivers anyway, and then you'll end up with at least some similarity... may as well start there and save a lot of time. (My hobby OS has a custom kernel and custom drivers, but I only support a bare minimum of devices... (pc) console i/o, one real NIC, and virtio-net... that's all I need; I might add support for more NICs and more consoles later)

I write documentation as I'm writing the code. In my opinion, the code is only as good as its documentation -- they're two parts of the same thing. It's mostly comments at the top of files, and sometimes a markdown file in the same directory.

This way, good documentation is priced into my estimate for the project. I don't have a work item "spend a few days documenting." Nope, if I'm doing a foo then that includes documenting a foo at the same time.

With documentation one of the major hurdles is the maintainance. It is caring for a set of documents, created by people with different specializations, that describe the artefact from specific perspectives, but need to be kept in sync with the active creation and evolution of the artefact itself.

This is not impossible, but the effort and costs required are substantial and often lose out on a priority basis to just fixing or improving the product itself.

    > Look at Apple's documentation

To clarify for me: Is this good or bad?

You make a good argument, but let me take the other side:

What you describe is probably necessary for getting _fast_ Linux compatibility. However, if you are willing to take the overhead of a few layers of indirection, you can probably sandbox the Linux land somewhere, and not have it impact the rest of your design much.

Most hardware access doesn't have to be particularly efficient. And, yes, for the few pieces of hardware that you do want to support efficiently (eg your storage devices or networking, whatever you want to concentrate on in your design) these you can handle natively.

Btw, I would suggest that most people these days should write their toy operating systems to run as a VM on a hypervisor like Xen or similar. The surface to the outside world is smaller that way.

If you're going this route, I have found netBSD a better option for this sort of thing.

It has a rump kernel architecture which makes reusing the drivers almost trivial compared to reusing linus drivers with a new kernel.

  > you're never going to get there off of recycling linux's device drivers because that forces you to design your new OS as a linux clone in which cade you definitely did not need to write an entire new kernel from scratch.

thats in interesting point, and makes me wonder if some kind of open interface for drivers to write to (and os's could implement) wouldn't be worthwhile?

probably it would have to be very general in design, but something along the lines of driverkit or iokit might work?

Is this the old 'an OS is just a bag of buggy device drivers' argument?

    > custom Wifi board

Why didn't you use something more mainstream? Cost?

I don't like Meta, but there used to be a time where big corp used to spend 30% of its budget on R&D. It's how we got all the toys we have now, R&D labs of big Bell and others.

So please don't mock the spend. Big spends fail sometimes, and at least people were paid to do the work.

I don't know how you get paid for it, but if you want to write your own kernel, I'd start with an osdev tutorial. started with this one [1], but this one [2] has a promissing name... and I haven't really looked around.

It helps to have a concept to guide you too, but you can certainly make some progress on the basics before you figure out what you really want to do.

[1] https://wiki.osdev.org/User:Zesterer/Bare_Bones

[2] https://osdev.wiki/wiki/Multiboot1_Bare_Bones

Asahi Linux.

Isn't that what low level does on his YouTube channel teach people to reverse engineer stuff?

Yes, and then you get odd behavior you can't introspect because the card is a black box to you.

In some cases it is hidden though. USB has a lot of "generic" device classes where (at least in theory) the OS only needs to deal with a standardized interface and whatever actual hardware is behind it is driven by an embedded controller.

It's just that most internal hardware initially only cares only about Windows (or only Linux) so it makes more financial sense to develop a more complex driver than a complex firmware. The equation might change later on but by then you are stuck with the hardware design.

> No, the complexity usually isn't hidden. It's the driver's job to do that.

Why not, though? We used to have e.g. glass teletypes with microprocessors (8080/8051) in them that exposed a serial bus with very neat command protocol that we still use nowadays, that could boot up, init and self-test all on their own.

Incredible job security