Comment by corytheboyd

For every million bricklayers, there may be one or two material scientists devising new clay formulations to make stronger, cheaper bricks.

The irony is that the bricklayers are often paid better, and they can be certain that their profession will remain in demand for the foreseeable future.

The material scientist may hit a dead end, or their company may axe their entire department to "cut costs".

One of the smartest human beings I've ever met ended up at Google, working on something very similar to CRTDs for the Google Wave project. The outcome was a commercial failure and his team was disbanded.

I did the same classes as him at University, and I use almost none of that knowledge in my rather pedestrian job. I make nearly twice what he does and I have nearly perfect job security. So there's that.

Author here. I love this comment.

> So, having said all of that, is there hope for adding things like this to my toolbelt?

Yes; but not yet. Right now if you want you can use yjs or sharedb. But the APIs aren't spectacular. Eventually I want these things integrated into svelte / react / swiftui and postgres / sqlite / whatever so it just sort of all works together and we get nice tutorials taking you through what you need to know.

We aren't there yet. Automerge is slow (but this is being worked on). Yjs is hard to use well with databases, and its missing some features. We'll get there. The point of all the mathematical formalisms is that if we do it right, it should just work and you shouldn't have to understand the internals to use it.

You are not a shitty software engineer who's been "coddled." What you likely are is dissatisfied with where your career is going (and who amongst us is not in some way) and nervous about not having a name brand pedigree so you are turning it inward and trying to understand it in ways that imply the cause and effect are localized to yourself. That's understandable but I wonder if it will get you anywhere. Stop beating yourself up and start exploring what you really want.

Do you want a brand name? Go work at a FAANG. Trust me, they're hiring. You'll get there and like me you'll realize they too have their problems and maybe the "over-paid Rails shit solutions" you "know will eventually leak" maybe weren't so bad after all.

Do you want to level up as an engineer? Go work at a hyper-growth startup (probably series B) that is falling over from its scale problems. You will probably end up solving some oddly challenging and novel engineering problem along the way.

Do you actually not mind this stuff but hate feeling like you have to always keep up with the Jones and it gives you anxiety? Go find a therapist that you have good chemistry with and see if you can't work out why you feel this way and how to fix it. To be honest, the corporate rat race thrives at making people who are otherwise doing quite well with their career progression feel like they're not because that makes them easier to exploit. Life is not an exam to be min-maxed. You don't take that fancy career or lifetime earnings with you when you die, so unless you're doing it for the intrinsic joy of doing it, there's an argument to be made that becoming better at engineering has no guarantee of making your life better.

See this blog post for just what it is -- a breathless discovery of a hammer and an author who now wants to use it for everything but may not be at the point where they realize it might not be a great tool for everything. You the person are probably a lot more competent and capable than you're letting on here. And if that's the case and you're unhappy for reasons in your control, hone in on why that is and see if you can't make life moves to change it. There's never been a better time to make a career move.

Good luck. I believe in you.

When this guy discusses OT vs CRDT he's discussing algorithms for some very specific use cases. Namely, concurrent document editing a la Google Docs. Realistically, there's maybe 50 people in the world who are versed in these kinds of algorithms at a really deep level. 99% of us are users. If I was pressed into it, I maybe have the chops to implement a reference implementation for something like this, but I've never come with a country mile of an organization that has the resources to invest in something like that. You can be a great programmer with a very successful career as someone who only knows how to read an API and use it. And unless you are tasked with creating a concurrent document editor, you will probably never need to understand the API for a CRDT.

I wrote an article for folks like you (because I also felt like this was my life at one point).

In this article I explain the basics of the simplest CRDTS the way a web developer without a deep knowledge of CS could relate to it: https://statagroup.com/articles/editing-shared-resources-crd... . I don't present the concept 100% fully, but I give a starting point for someone who "only knows rails". I found most online tutorials dive directly into the theory.

I included all the resources I used to learn the concept on the page too.

It's part of a series I wrote about editing shared resources ( https://statagroup.com/articles/editing-shared-resources ).

I feel ya, I did not graduate either, and didn't go for CS. After having Linux as a hobby, I made my way into a web dev job. The "good enough" work is killing me. I feel like I am pushing and pulling my team to the light, even simple things like DRY.

I think one thing I'm learning is where to spend my effort. Its definitely a balance, but I'm putting more time/passion into my personal projects/learning, whereas before I saw more opportunities for overlap with my passion and work. When your work doesn't recognize that effort or actively fights against those efforts, it's draining and frustrating.

I've also been thinking about dedicating some serious time to learning. I keep thinking about the idea of taking a month or two off from work. Just treat myself to my own mini-semester of homeschool. I'm somewhat confident I could learn enough to land a better job, but more importantly, one I enjoy more.

No, you have not locked yourself into anything, you can always grow. Comfort can breed complacency, so my initial advice is to quit your job and try to get a new one that's more demanding. It forces you to kick your own butt into learning new stuff.

I am an under-educated over-paid GED holding Glorified Sysadmin, and while a lot of the shit solutions I work on will eventually leak, I do spend a non-trivial amount of time looking for better solutions. I also try to keep my ears open for what other people are working on and read up on concepts I'm unfamiliar with, whether they crop up on HN or through work.

I learned about CDRTs several years ago at a former employer where a team mate was talking about how it was the future. However, we didn't end up using them. It turns out the more advanced your system gets, the more work goes into it, and the more of a pain in the ass it becomes to run [until it reaches a certain level of operational stability]. So I wouldn't say that using the most bleeding edge research is a good idea even if you know about it. Most people are still fucking up the basic stuff we were doing 10 years ago, and the neckbeards here probably have loads of stories about how much simpler and more stable distributed systems used to be.

> So, having said all of that, is there hope for adding things like this to my toolbelt? And where do I even start?

HN is a decent place to start. Now that you've read this article, you're aware of the general landscape of concurrent document editing algorithms.

Ultimately, all you really need is the language to a) describe your problem and b) compare potential solutions. I have a CS degree, and it's not like we spent semesters learning every algorithm under the sun. But what we did learn was the language to describe and compare the performance of algorithms (computational complexity theory). Beyond that, it's just about knowing enough terminology to be able to describe your problem, which you could learn just by skimming through an algorithms textbook (or random Wikipedia articles, honestly).

For example: a friend of mine was recently trying to devise an algorithm for ordering buff applications for a collectible card game. Once I was able to find the right language to describe the problem (topological sorting), finding an algorithm to use was easy.

The article mentions Martin Kleppmann. Go buy a copy of his "Designing Data-Intensive Applications". It may well have been titled "Practical Distributed Systems for the Everyday Developer". It is an absolutely fantastic book with a perfect ratio of theory/practice.

Extra points for buying a dead tree copy and reading it without a thousand alerts and internet temptations vying for your attention :)

Sounds like you've got an awareness of CRDTs now. That's a win and a differentiator from many other engineers.

I already knew about them, but there will be a stack of other stuff I don't know about. Just get Googling when you hit different problem areas.

I found out about CRDTs because I was working in a niche where a security device was used that breaks TCP between two hosting regions for an application. So I searched for issues to do with that and refreshed myself on the two general's problem, Byzantine fault tolerance, 2 and 3 phase commits and at some point OTs and CRDTs popped up. You'll find this stuff if you keep looking at a problem for long enough.

That was a while ago though and despite still working in that area I've still not implemented anything with them because I'm a contractor and I'm under the direction of people who don't have CRDTs in their toolbelt and don't want ideas from others! I guess my cheerful message is keep at it but know you can only do so much and there's plenty of other pits of despair to fall into even when you're out of the current one. Yay!

I never graduated, but that was back in the 80s. The important part is not getting "official" education, but learning how to self-learn. You don't need to necessarily understand the underlying maths, if you can understand the general theory.

And you're not a "shitty software engineer" if you are self-aware enough to know what you don't know. Read, start with Wikipedia, look up the references, play with stuff, explore.

If you need to understand certain things at a deeper level, then explore the classes and other guided tuition.

I've been doing this stuff for 30 years and I still learn every day. You will produce less than optimal work for ever. As a master craftsman (which is what you are, not an "engineer") you are always learning and improving your craft.

You can be satisfied when completing some work if you did your best to produce it and it works within the constraints you had. There will always be things you would do different or better, so do that next time.

CRDTs are just applied partial order theory. Partial orders are just mazes where you can drag your finger in a certain vague direcion, taking arbitrary turns in that vague direction, and still find the exit.

Get your intuition down really good and 10+ years is not required.

I can give a philosophical answer rather than a technical one. Our lives are filled with learning various things. And we all realize there is no end to it. But truly what makes you feel better about what you do is not this endless learning. It is when you can drop what you learnt and think freely about a problem that you can solve. Until you reach there, you are in a boat tossed by the waves trying to grab whatever you can to keep yourself afloat. Please disregard this answer if it is not of any help.

I might suggest starting with one thing at a time, pulling in more and more. No one learns it all at once.

I might suggest Martin Kleppman's Making Sense of Stream Processing freely available from Confluent [0].

[0] https://www.confluent.io/stream-processing/

Picking up Elixir is a decent way to get into backend distributed systems from a Ruby/Rails background.

I love the candor in your post but you're being too hard on yourself.

Learn functional programming, gradually. You'll get there. I like to think I reverted a decade of brain damage inflicted by enterprise java development in three years.

I've worked for multiple Rails shops with high quality engineers. The trick is to interview them too before accepting any offer. The best teams I've been on have been small. Past a team/project size, maintaining high quality code becomes much more difficult.

Check out http://www.roomservice.dev

Turns your front end app multiplayer with a library.

They manage the hard part and give you a simple API to get the benefits of realtime/syncing/diffing.

This is my daily struggle routine. I cannot get rid of the idea that proceeding my career only accumulate how much I earn, instead of making myself a better programmer.

> And where do I even start?

There are so many resources available to learn CS today. Are you really baffled about how to learn outside of school? Read. Practice. Repeat.

Hey are you me? Cause this sounds like me.

I am in this photo and I don't like it.