The data engineer

I don't totally understand what that means

I’ve been running a data engineering department for 4 years in my company, and to be honest, I never looked at the actual role of the people I was hiring for that team. They passed a “simple” technical test and that’s it. BTW, any LLM nowadays is capable of solving the test in 3 seconds.

When working on the team, I never assumed they were different from regular developers and all the projects they were involved in had development tasks. For sure they had to understand the data and part of the business rules, but, as I said, nothing different than a regular developer.

You can argue the code (normally SQL + glue python code) and the data size is specific for a data engineer but if you have been working enough time as a developer you probably changed 5-6 times of the kind of tech you use, so again, nothing new.

The only difference I found is how well the good practices we learned over the last 30 years in software development are used. Most of the time version control, testing, continuous delivery, observability, monitoring, and others are just nice to have. BTW, I have this feeling with companies doing work based on LLMs.

And that’s worrying.