Comment by PaulHoule

I've seen some systems with a lot of pieces where teams have attempted to avoid repetition and arranged to use a single source of schema truth to generate various other parts automatically, and it was generally more brittle and harder to maintain due to different parts of the pipeline owned by different teams, and operated on different schedules. Furthermore it became hard to onboard to these environments and figure out how to make changes and deploy them safely. Sometimes the repetition is really the lesser evil.

I keep pining for a stripped-down gRPC. I like the *.proto file format, and at least in principle I like the idea of using code-generation that follows a well-defined spec to build the client library. And I like making the API responsible for defining its own error codes instead of trying to reuse and overload the transport protocol's error codes and semantics. And I like eliminating the guesswork and analysis paralysis around whether parameters belong in the URL, in query parameters, or in some sort of blob payload. And I like having a well-defined spec for querying an API for its endpoints and message formats. And I like the well-defined forward and backward compatibility rules. And I like the explicit support for reusing common, standardized message formats across different specs.

But I don't like the micromanagement of field encoding formats, and I don't like the HTTP3 streaming stuff that makes it impossible to directly consume gRPC APIs from JavaScript running in the browser, and I don't like the code generators that produce unidiomatic client libraries that follow Google's awkward and idiosyncratic coding standards. It's not that I don't see their value, per se*. It's more that these kinds of features create major barriers to entry for both users and implementers. And they are there to solve problems that, as the continuing predominance of ad-hoc JSON slinging demonstrates, the vast majority of people just don't have.

Brb, I'm off to invent another language independent IDL for API definitions that is only implemented by 2 of the 5 languages you need to work with.

I'm joking, but I did actually implement essentially that internally. We start with TypeScript files as its type system is good at describing JSON. We go from there to JSON Schema for validation, and from there to the other languages we need.

It's not that we like it, it's just that most other solutions are so complex and difficult to maintain that repetition is really not that bad a thing.

I was however impressed with FastAPI, a python framework which brought together API implementation, data types and generating swagger specs in a very nice package. I still had to take care of integration tests by myself, but with pytest that's easy.

So there are some solutions that help avoid schema duplication.

My experience is that all of these layers have identical data models when a project begins, and it seems like you have a lot of boilerplate to repeat every time to describe "the same thing" in each layer.

But then, as the project evolves, you actually discover that these models have specific differences in different layers, even though they are mostly the same, and it becomes much harder to maintain them as {common model} + {differences}, than it is to just admit that they are just different related models.

For some examples of very common differences:

- different base types required for different languages (particularly SQL vs MDW vs JavaScript)

- different framework or language-specific annotations needed at different layers (public/UNIQUE/needs to start with a capital letter/@Property)

- extra attached data required at various layers (computed properties, display styles)

- object-relational mismatches

The reality is that your MDW data model is different from your Database schema and different from your UI data model (and there may be multiple layers as well in any of these). Any attempt to force them to conform to be kept automatically in sync will fail, unless you add to it all of the logic of those differences.

Having 12 different independent copies means nobody on your 30 people multi-region team is blocked.

SOAP, CORBA and such have a theory for everything (say authentication) It's hard to learn that theory, you have to learn a lot of it to be able to accomplish anything at all, you have to deal with build and tooling issues, but if you look closely there will be all sorts of WTFs. Developers of standards like that are always implementing things like distributed garbage collection and distributed transactions which are invariably problematic.

Circa 2006 I was working on a site that needed to calculate sales tax and we were looking for an API that could help with that. One vendor uses SOAP which would have worked if we were running ASP.NET but we were running PHP. In two days I figured out enough to reverse engineer the authentication system (docs weren't quite enough to make something that worked) but then I had more problems to debug. A competitive vendor used a much simpler system and we had it working in 45 min -- auth is always a chokepoint because if you can't get it working 100% you get 0% of the functionality.

HTTP never had an official authentication story that made sense. According to the docs there are basic, digest, etc. Have you ever seen a site that uses them? The world quietly adopted cookie-based auth that was an ad-hoc version of JSON Web Tokens, once we got an intellectually coherent spec snake oil vendors could spam HN with posts about how bad JWT is because... It had a name and numerous specifics to complain about.

Look at various modern HTTP APIs and you see auth is all across the board. There was the time I did a "shootout" of roughly 10 visual recognition APIs, I got all of them working in 20-30 mins except for Google where I had to install a lot of software on my machine, trashed my Python, and struggled mightily because... they had a complex theory of authentication which was a barrier to doing anything at all.

Worse is better.

Just use gRPC or ConnectRPC (which is basically gRPC but over regular HTTP). It's simple and rigid.

REST is just too "floppy", there are too many ways to do things. You can transfer data as a part of the path, as query parameters, as POST fields (in multiple encodings!), as multipart forms, as streaming data, etc.

Always thought that a standard like ISO8601 which always stores the date and time in UTC but appends the local time zone would beneficial.

ISO8601 is really broad with loads of edge cases and differing versions. RFC 3339 is closer, but still with a few quirks. Not sure why we can't have one of these that actually has just one way of representing each instant.

Related: https://ijmacd.github.io/rfc3339-iso8601/

That would be solved if JSON had a native date type in ISO format.

> He formalised a concept and gave it a snappy name, and then the concept got left behind and the name stolen away from the purpose he created it for.

I'm not sure the "name was stolen" or the zealot concept actually never got any traction in production environments due to all the problems it creates.