The good thing is that AI has increased our productivity already. Having worked with SQL for 20+ years, it was very easy to translate my SQL thinking into PySpark using Databricks AI assistant, for example. That didn't eliminate the need for me to study to understand the technology, but it definitely saved me much time from searching about the syntax, googling, etc. On the other hand I agree that we're not going to see AI completely replacing data engineers any time soon. The coding part is the less important piece of the work a data engineer does, which includes business requirements understanding, data modelling & architectural work, data flows and data quality rule framework design, etc. etc. So yes AI can write a bunch of lines in Python, but designing something that makes sense in the long term? I don't think so...
Agree with most of it. I wonder how we can evaluate AI generated content/solutions that may become ever more complex if we spend less time in the implementation details thus loosing our ability to assess.
I guess the answer is yes, in the long term at least. These LLM seem to improve at a quite impressive pace, also thanks to all of us who are training them with our prompts.
it's insanity to think any digital work won't be taken by AI at some point in time. Sure AI is just a tool at the moment but AI will massively replace humans successfully and it won't take long. AI can do even now complex computation and most likely is better than even the most senior software engineer on the planet. Any data workers who think AI won't take the boring and repetitive and tedious data work job are utterly crazy coconuts
I think the job is already changing fast, we are moving away from coding and more towards tooling and managing cloud services as they get smarter and more automated, its a different type of automation delivered by chat LLMs, the AI will be integrated inside our data lake and delivered by our cloud services. It happened before in the database administrator jobs (DBA). As the databases became more automated and companies migrated to the cloud, this once high-paying, high-demand job simply vanished and those people migrated to other positions such as System Administrators, Managers or even Data Engineers
The good thing is that AI has increased our productivity already. Having worked with SQL for 20+ years, it was very easy to translate my SQL thinking into PySpark using Databricks AI assistant, for example. That didn't eliminate the need for me to study to understand the technology, but it definitely saved me much time from searching about the syntax, googling, etc.
On the other hand I agree that we're not going to see AI completely replacing data engineers any time soon. The coding part is the less important piece of the work a data engineer does, which includes business requirements understanding, data modelling & architectural work, data flows and data quality rule framework design, etc. etc.
So yes AI can write a bunch of lines in Python, but designing something that makes sense in the long term? I don't think so...
Agree with most of it. I wonder how we can evaluate AI generated content/solutions that may become ever more complex if we spend less time in the implementation details thus loosing our ability to assess.
I guess the answer is yes, in the long term at least. These LLM seem to improve at a quite impressive pace, also thanks to all of us who are training them with our prompts.
Don't worry ✌🏻✊🏻
it's insanity to think any digital work won't be taken by AI at some point in time. Sure AI is just a tool at the moment but AI will massively replace humans successfully and it won't take long. AI can do even now complex computation and most likely is better than even the most senior software engineer on the planet. Any data workers who think AI won't take the boring and repetitive and tedious data work job are utterly crazy coconuts
You're more likely to get AI to replace itself 😂
That was a good one..replace itself like over engineering...cancer
I think the job is already changing fast, we are moving away from coding and more towards tooling and managing cloud services as they get smarter and more automated, its a different type of automation delivered by chat LLMs, the AI will be integrated inside our data lake and delivered by our cloud services.
It happened before in the database administrator jobs (DBA). As the databases became more automated and companies migrated to the cloud, this once high-paying, high-demand job simply vanished and those people migrated to other positions such as System Administrators, Managers or even Data Engineers