Comment by woeirua

20 days ago

Here’s the thing though, most companies work with small data. The distribution of data set size follows a power law which means that few engineers get to work with petabyte sized datasets. That said, the job market definitely incentivizes people to have that kind of experience on their resume if they want to keep progressing in salary. This incentivizes over engineering.

2 comments

woeirua

Reply
benrutter  19 days ago

> the job market definitely incentivizes people to have that kind of experience on their resume

Yeah, this is sadly often true, but it's also another trap that people don't have to fall into.

I've been involved with hiring data engineers, and I see experience with distributed computing way more often than I see knowledge of simple profiling and debugging tools. But I'd personally value the latter a lot more when interviewing.

Companies that hire for skills they don't need are of course perpetuating this problem, but they're also paying a big "tax" in the sense that they're not hiring for the skills they actually do need.