It is the reason why as a DevOps I’m learning Data Engineering. I see all companies asking for someone who knows code, cloud, data tools, scalability, monitoring… 3 professionals in 1… It is what it is…
The funny thing is that I wouldn't say I like Python for data analysis, but I'm forced to use it... but with a migration of awesome packages from R such as tidyverse, tidypolars, shiny, and some functional paradigm capabilities (chains, immutability...) in python my hate reduced a little bit. But for data engineering may be in the future Rust, Julia o other language more fail safe and faster will be mandatory inssted of python, right?
I don't see a real good Python alternative for Engineers right now. Everything else is niche. For analysts and Data Scientists R is the better choice (lots of people tell me that)
@andreaskayy Working on my Power BI portfolio, data analytics certification, & studying data engineering. BI Development & IT Management (for small companies) are some of my recent skills & duties. Many I speak with are experiencing the same thing.
For me it was not clear, what is the difference between AI engineer and data engineer. Or maybe AI engineer is more like a Python backend engineer in web development. Glueing together some LLM API with some frontend. But for those who are using already developed LLM, the benefits of linear algebra, statistics, machine learning, etc is unclear. About the market trend: I noticed some silence from recruiters on linkedin in the last 2 years. On the other hand, int the last couple of weeks it looks like more requests are arriving. But the companies are more pickier about who to choose, based on skills and lower salary expectations as well. You can be declined even after a 5-steps selection process…
Yeah, AI Engineer. People invented another job.. 🙄 Job market is really difficult right now, you are right. Especially since the beginning of the year. I'll have another video next week about that whole topic..
in realityy it is a very different thing. i am a tensorflow developer, it also can mean an ai engineer, but i do not know much about pipelines, i know how to clean data... hihihiihi The roles of an AI Engineer and a Data Engineer differ in focus, responsibilities, and skill sets, even though they often collaborate within data-driven projects. Here's a detailed comparison: AI Engineer Focus: Specializes in developing systems and applications that utilize artificial intelligence techniques, such as machine learning (ML), deep learning, and natural language processing (NLP). Builds models and algorithms to make predictions, automate tasks, or analyze complex data patterns. Key Responsibilities: Designing and implementing AI and ML models. Developing applications like chatbots, recommendation systems, or image recognition tools. Ensuring scalability and performance of AI systems. Collaborating with data engineers to gather and preprocess the data required for training models. Deploying AI models in production environments and monitoring their performance. Skill Set: Knowledge of AI and ML frameworks (e.g., TensorFlow, PyTorch). Programming languages such as Python, R, or Java. Understanding of deep learning, computer vision, NLP, and reinforcement learning. Familiarity with cloud platforms for AI deployment (AWS, Azure, Google Cloud). Typical Output: AI-powered systems, predictive models, automation tools. Data Engineer Focus: Specializes in designing, building, and maintaining the infrastructure and pipelines required to collect, store, and process data. Ensures data is accessible, reliable, and usable for analysis or AI/ML projects. Key Responsibilities: Building and managing data pipelines for ETL (Extract, Transform, Load) processes. Designing and optimizing data storage solutions (e.g., data warehouses, data lakes). Ensuring data quality, consistency, and security. Collaborating with data scientists and AI engineers to provide clean and structured data. Monitoring and maintaining data systems for performance and scalability. Skill Set: Proficiency in database technologies (SQL, NoSQL) and ETL tools. Knowledge of big data frameworks (Hadoop, Spark). Expertise in cloud services for data processing (AWS Redshift, Google BigQuery). Strong software engineering and scripting skills (Python, Scala, Java). Familiarity with DevOps and orchestration tools (Kubernetes, Airflow). Typical Output: Reliable, scalable data infrastructure and pipelines.
For most GenAI use-cases a pre-trained model is enogh. You're right that you don't really need a Data Scientist for that. Although for most non GenAI use-cases there are no pre-trained models, because every company works with different data and has different goals. You'll need someone who can really do ML.
It is the reason why as a DevOps I’m learning Data Engineering. I see all companies asking for someone who knows code, cloud, data tools, scalability, monitoring…
3 professionals in 1… It is what it is…
The more you know the more helpful you will be. Keep going.👍
Google "t shaped model"
dont forget security, data engineers often manage VPCs, subnets, trafic balancers and access to data bases.
@ yes, that's true. 👍
Sounds like you're working with AWS?
@@andreaskayy not only AWS, nowadays the companies use 2 or 3 providers… It is why be a generalist makes you a better fit than specialist…
@ very true. How difficult was it for you to add another cloud to your skills? They all kinda work the same.
The funny thing is that I wouldn't say I like Python for data analysis, but I'm forced to use it... but with a migration of awesome packages from R such as tidyverse, tidypolars, shiny, and some functional paradigm capabilities (chains, immutability...) in python my hate reduced a little bit. But for data engineering may be in the future Rust, Julia o other language more fail safe and faster will be mandatory inssted of python, right?
I don't see a real good Python alternative for Engineers right now. Everything else is niche. For analysts and Data Scientists R is the better choice (lots of people tell me that)
a gen ai engineer or ai engineer is definitely not a data engineer, both are crucial roles but they are very different
8 months unemployed. Not too encouraging.
Definitely Agree !! What's you preparing for?!
Yeah, what are you preparing for and what's your process for applying to jobs?
@andreaskayy Working on my Power BI portfolio, data analytics certification, & studying data engineering. BI Development & IT Management (for small companies) are some of my recent skills & duties. Many I speak with are experiencing the same thing.
@dipeshrathore8842 BI Development/Mgmt. Working on Data Analytics cert & eventually data engineering as well.
@@CinemaLover1900 Keep going brother✨, do little smart work. I'm sure you'll succeed in it 👏
For me it was not clear, what is the difference between AI engineer and data engineer.
Or maybe AI engineer is more like a Python backend engineer in web development.
Glueing together some LLM API with some frontend.
But for those who are using already developed LLM, the benefits of linear algebra, statistics, machine learning, etc is unclear.
About the market trend: I noticed some silence from recruiters on linkedin in the last 2 years. On the other hand, int the last couple of weeks it looks like more requests are arriving. But the companies are more pickier about who to choose, based on skills and lower salary expectations as well. You can be declined even after a 5-steps selection process…
Yeah, AI Engineer. People invented another job.. 🙄
Job market is really difficult right now, you are right. Especially since the beginning of the year. I'll have another video next week about that whole topic..
Huh? Do you mean what's the difference between an AI Engineer and ML Engineer?
AI Engineer, ML Engineer, Data Engineer.. It's getting complicated
in realityy it is a very different thing. i am a tensorflow developer, it also can mean an ai engineer, but i do not know much about pipelines, i know how to clean data... hihihiihi The roles of an AI Engineer and a Data Engineer differ in focus, responsibilities, and skill sets, even though they often collaborate within data-driven projects. Here's a detailed comparison:
AI Engineer
Focus:
Specializes in developing systems and applications that utilize artificial intelligence techniques, such as machine learning (ML), deep learning, and natural language processing (NLP).
Builds models and algorithms to make predictions, automate tasks, or analyze complex data patterns.
Key Responsibilities:
Designing and implementing AI and ML models.
Developing applications like chatbots, recommendation systems, or image recognition tools.
Ensuring scalability and performance of AI systems.
Collaborating with data engineers to gather and preprocess the data required for training models.
Deploying AI models in production environments and monitoring their performance.
Skill Set:
Knowledge of AI and ML frameworks (e.g., TensorFlow, PyTorch).
Programming languages such as Python, R, or Java.
Understanding of deep learning, computer vision, NLP, and reinforcement learning.
Familiarity with cloud platforms for AI deployment (AWS, Azure, Google Cloud).
Typical Output:
AI-powered systems, predictive models, automation tools.
Data Engineer
Focus:
Specializes in designing, building, and maintaining the infrastructure and pipelines required to collect, store, and process data.
Ensures data is accessible, reliable, and usable for analysis or AI/ML projects.
Key Responsibilities:
Building and managing data pipelines for ETL (Extract, Transform, Load) processes.
Designing and optimizing data storage solutions (e.g., data warehouses, data lakes).
Ensuring data quality, consistency, and security.
Collaborating with data scientists and AI engineers to provide clean and structured data.
Monitoring and maintaining data systems for performance and scalability.
Skill Set:
Proficiency in database technologies (SQL, NoSQL) and ETL tools.
Knowledge of big data frameworks (Hadoop, Spark).
Expertise in cloud services for data processing (AWS Redshift, Google BigQuery).
Strong software engineering and scripting skills (Python, Scala, Java).
Familiarity with DevOps and orchestration tools (Kubernetes, Airflow).
Typical Output:
Reliable, scalable data infrastructure and pipelines.
What about DataOps?
What about it?
If someone have to use chat gpt then why would someone learn ML. It is like going backwards.
Pretrained models are successor to ML
For most GenAI use-cases a pre-trained model is enogh. You're right that you don't really need a Data Scientist for that. Although for most non GenAI use-cases there are no pre-trained models, because every company works with different data and has different goals. You'll need someone who can really do ML.