Comment by weatherlite
3 years ago
I think the problem of ageism wasn't mentioned at all and it should be. People who love the field have no problems keeping up to date. Sure some job posts will mention Hadoop or Kafka but whatever, a good dev will have no problem learning these in a few days. Does he get a chance though if he's 50?
Ironic thing here is that Hadoop is mostly already outdated.
Which is btw one of the depressing thing for a lot of data engineers: we used to play with those cool distributed processing frameworks, and now? We are mostly writing some terraform to deploy cloud resources, most of the distributed part being handled by those cloud providers.
> Which is btw one of the depressing thing for a lot of data engineers: we used to play with those cool distributed processing frameworks, and now? We are mostly writing some terraform to deploy cloud resources, most of the distributed part being handled by those cloud providers.
Sounds to me like switching one provider/tool by another - or are data engineers feeling bummed because the job has become too trivial / less fun?
I am only speaking for myself here, but I am really feeling the switch from "data engineering" to "data ops", for whatever that means.
In short, 5-10 years ago, writing mapreduce / spark jobs (or even debugging / optimizing hive jobs) was complex enough that it was often the job of the data engineer (and not the data analyst / scientist). And I do not only mean writing the data processing logic, but more importantly, properly configuring it so that the resource footprint was acceptable. This required a good understanding of the underlying framework, analyzing the job execution plan, tweaking the resource configuration, etc.
Now, writing distributed jobs is pretty trivial with most cloud providers, hence it is now purely done by data analysts and scientists. And the data engineers have switched to doing more of a devops kind of work, doing the plumbing between the various cloud components and the IaC required to provide those cloud resources to other data users. In short, you can be a data engineer and have absolutely no clue on how distributed systems are actually working, this will not be an issue in your daily job.
The Hadoop ecosystem is still very much ongoing. You must work in small tech/startups if you think that these are being replaced by managed services.
It is still ongoing, but the trend is really on managed services. Most shops that are still running hadoop distribution are doing it for legacy reasons (and I used to work in one). I mean, just look at job offers: how many offers do you see where hadoop experience is a plus VS cloud experience?
I'm not disputing the fact that there is ageism, as I'm sure there are thousands of examples of it, but there's so much demand and so many different companies. I've worked with plenty of over-50's. Maybe there's a sweet spot for growing companies where you need the experience, which has been where I've worked. Small companies don't need structure, and maybe want cheap employees. Large companies put all that structure in management and a few super senior folks. (though I saw plenty of over 50s in my large company experiences) Medium growing companies need experience. I dunno, just guessing since certain companies I've worked for seem to have a higher concentration of older folks.
Here, this rando website says that 46% of software engineers are 40+
https://www.zippia.com/software-engineer-jobs/demographics/
Now I'm curious, this stack overflow survey paints a grimmer picture:
https://insights.stackoverflow.com/survey/2018
That's a survey though, I'm curious what biases are going to exist in the data. We might assume that older software developers move around less? May be less likely to respond to surveys? May be less likely to visit stack overflow, especially if they do less hands on coding?
I'm with you, I don't know the answer this is definitely complex. It could be a combination of things - a) burnout due to ever changing tech b) ageism c) highly paid devs choosing to retire / switch professions early simply because they can financially
40 is definitely not that old anymore for tech I think. Well I'm 38 I'll find out soon.