Comment by TimTheTinker

Here's a HN comment from earlier this week that elaborates on some earlier operating systems that explored some really novel ideas: https://news.ycombinator.com/item?id=33768592

To summarize:

> ITS (of PDP-10 hacker fame) - processes could debug and introspect their child processes. The debugger was always available, basically. The operating system provided support for breakpoints, single-stepping, examining process memory, etc.

> KeyKOS (developed by Tymshare for their commercial computing services in the 1970s) - A capability operating system. If everything in UNIX was a file, then everything in KeyKOS was a memory page, and capabilities (keys) to access those pages.

In every operating system, the basic unit of abstraction will be a process -- which necessitates a scheduler, some form of memory protection, some way for the process to interact with the kernel, and the notions of "kernel space" and "user space". There is a lot of room for innovation there (see ITS), but I suspect most of the room for innovation is in how an OS abstracts/refers to various parts of the system.