I've just released my Free Data Engineering Course for Beginners on UA-cam! Make sure to check it out if you're interested in a career in data engineering! :) ua-cam.com/video/dvviIUKwH7o/v-deo.html
U r really amazing to starters. NB: I am just studying in 9th standard 😅 just checking how it works so that I don't have to look for this in more big way😁
I am Data Engineer and I describe myself as follows: "I am like a plumber in the data world and ensure data flows from one abstract system to another in a scalable, consistent and standardized manner to allow the business users to consume the data in a usable way" . My only advise is to ensure all your pipelines are robust and fault tolerance/notifications are built into your data layer.
Absolutely, a data engineer should always consider as many edge cases as possible, because they happen more often than not! I'm glad to read a comment from a fellow data engineer! ;)
And in finance make sure the pennies add up :-) I used to work on billing systems for telecoms multiple types of records multiple types of data sources - we used Map Reduce (back in the early 1980's ) with F77 and Pl/1G
That's amazing! Your speaking is very logic and concise, covering a lot of the most important concepts or sessions a data engineer does everyday. A very talented and graceful lady. Thanks for sharing and god bless you.
Your vlog feels real, honest and accurate. I am learning about data science, sql, python, etc. and I find your content very inspiring. Thanks for sharing your knowledge and experience. I am looking forward your future content. Greetings from Mexico.
1. Bring Data in from different vendors. Check 2. Data pipeline , data warehousing , and aws. Check 3. Table-valued function , Procs and Views. Check 4. Analytics, debug and Monitoring. Check 5. I will Help but Not going to be nice about it. Check Thank you this gives me confidence to change my career to data engineering . I Am not a Data Engineer but a BI developer. my company is kind of like a start up so we don't have a data engineer so I ended up doing all the pipeline to create my SSRS and Tableau report.
Just started my career as a data engineer in a professional environment as well and I can align with everything you say, Karolina! It's also very interesting because many people think Data Science is just that one single field but it can be divided in so many sub ones. I think you make the role of a DE clearer in this video! Great work. I am very excited to get to the same point and share the same experience with you.
How many tech stacks you need to have to become junior DE? Like you need to know Python, SQL or smth else? How many prerequisites should we have before go on to become junior DE?
I really felt that part when you mention to "adapt" for the next 5-10 years, because that's when you need to keep investing in your self. Thanks for sharing your opinion
So great to have such video lessons on youtube! I had an internship 2 and a half years ago where I did the data engineering task and now I want to continue my journey!! Now I study at university and I have much time to do that! Thank you!!!!!
Love it! How comes I'm just discovering you. Love your sense of humor. I'm working on the path of becoming a data engineer. This is encouraging as all the 5 points are things that excite me. Keep sharing great content. We'll keep supporting.
11:46 - 12:19 - 12:37 - 12:57 this part of the vid is the one i liked the most. not that it's something i already didnt had the idea of, but it felt nice to get reminded of the upcoming highs.
Super glad I came across your channel, Karolina! I'm currently a DA working to make some long-term transitional goals come true! -> data engineering -> data science. Thanks for your awesome insights! Your delivery is really engaging and easy to understand! Best wishes, Karolina!
Hi Karolina, thanks for sharing your thoughts about the subject, I starting a career as a data engineer and you video clarifies me and motivate me to do it as quickle as possible. Best regards.
Thanks for another interesting video Karolina! You know...as I was watching the video and hearing you talk about the different tasks data engineers have to perform, I kept thinking about the inevitability of all of those processes getting automated eventually. And of course you concluded with that interesting tidbit about having to learn new skills because you anticipated automation. Just goes to show, not only do you have to identify new opportunities and learn new skills in order to keep thriving, but you also have to develop a strategy to become as financially independent as possible so that you’re not caught all of a sudden in a situation where your skills just cannot support a decent wage anymore. If that doesn’t motivate someone to get off the daily grind for a few hours a night and just sit down, make new plans, pursue other interests, and just think about the meaning of it all, I don’t know what will!
My thoughts exactly, Juan! What you said about financial independence in particular. Sadly, in the face of automation, it is those who are not financially independent, or whose skills are not sufficiently transferable, that will be hit the most.
Thank you very much for this video. You could believe you are terrible helping other people installing software and things like that, but your videos are the most honest and down to earth description of the job of Data Engineering You are amazing.
Karolina, You are a pure heart 💓. (Evidence at 11:18) Keep uploading videos and share us your experience about Data engineering and it's prospects. May you achieve a Million Subscribers soon. Kudos, from India ✌️✌️
Thanks a lot from these kind of contents about Data Engineering, I'm a Java developer and I'm looking for conquer a new boundary because I'm a little bored doing micro-services and I want to try some related new topic. Greetings from Chile.
Hi, i also want to study data engg in an university but didn't knw there is any good universities and procedures for it.. If u can send me an email regarding that knwldge will be helpfull..... Divyyansh05@gmail.com
Great content! It's funny you said that the thing that you like the least is to help others and here you're creating videos to help others on how to become a data engineer. Plot twist. 😅 Nice job Karolina 👏
Is it possible to create a video(s) about data scientist vs data analyst vs data engineer? Or data jobs vs machine/deep learning/ai vs computer science jobs maybe a series of videos? Could u do a video about how much ai/machine/deep learning could impact jobs and which jobs will likely become automated or disappear vs which jobs will remain strong in the next 5-10 years? Also based on your last name, its great seeing someone else from a Polish or European background!!
Really liked your video! I work as a Data Scientist and can totally relate to the hard work our data engineering team puts in. Thanks, keep making more videos!
Can you let me know what is the career path for data engineer? Say I don't want to work as data scientist or machine learning engineer, I only like to do data engineering my my entire professional career. Can one go far just by working as data engineer only and doesn't move to other areas like ML or DS on the later stage?
Hi, absolutely! Data Engineering career path, like all career paths in technology, can take many shapes. But in majority of cases, you'll start out as a analyst/developer (building, fixing data pipelines), and then you will move on to assuming more responsibility, e.g. designing the entire data architecture of a company or managing teams.
Suppose a company wants to have an erp from oracle to sap, now past data is in oracle db, future will be in sap db, now CEO of the company would like to have all data in same format report to make a year on year comparison, then you would need data engineers.
Try to learn sap BW if you are so much interested to work in data engineering. But you could gain lot more insight if you would understand how business works. But then you donot appreciate economics much. So I think technical role is better for you.
Karolina Sowinska I have always like macro economics. Microeconomics sounded a bit technical and it was difficult. Development economics and international economics too were interesting to me.
I'm as a junior BI developer, studying and working with tableau, and I'm missing the programming. I'm going to start working on some projects on my own and see if that's really what I want to do, because of everything you said I liked it so much! The bad part is that where I live there is only Senior vacancy for data engineering
Thanks for the video and it's rly nice to see another female data engineer voicing their thoughts. I've been a data engineer for officially, I'd say 1.5 years and I really enjoyed the net result of ups and downs. Data engineers understand business better than software engineers, and understand tech better than data analysts/scientists. So to me, when I look at it as a role in translating data for business, I find it quite impossible to be obsolete in the next 5-10 years UNLESS the average data literacy within a company has increased (making better and more careful choices on data inputs) or people turn to machine-generated data more than human-generated data (or when NLP and Deep Learning married for good to successfully "guess" the right data). Otherwise the business would always need to pull in a data engineer to scale their data while preserving its correctness.
Hi there, nice to meet a fellow female data engineer! :) That's a very interesting and convincing perspective - perhaps you're right (and I definitely hope so!)
Always pleased to see experienced people talk about they work and life. According to this video, I guess your work is much more focus on the pre-process the data rather than implementing and training the model right?
Can one work as DE the entire career without having to move onto other relevant areas like MLE or DS later on? Say I love DE job so much, can I stick to it forever? What will be career path for DE? I don't like machine learning and data scientist role 😂
@@norpriest521 Hi, yes, you can! Typically the career progression is such that you eventually become the person responsible for the overall data flow in an organisation, eg. the Head of Data Engineering.
Very similar job to what Data Engineers do, but I guess Data Architects are typically higher up in the hierarchy/they're more experienced, so they can decide on the architecture, and not just build pipelines
To your last point about standardization and automation, I'd honestly love if Data Engineering and ML Engineering came closer once this happened. DE's already have to deal with all the requests from MLE's, might as well just put them in the same room. I see both getting closer to being abstracted away to standardized tools within the broader software engineering ecosystem. More successful and matured design patterns just need matured first. But once this happens, I very well could just see software engineering jobs focused on ML and/or Data experience being the new norm.
Hello I found your very helpful but I have some questions can you please answer them..? 1.) As you mentioned that if vendor change the name of coulum then we can make a system where we can take query take dynamic names as column changes. 2.) As it is automation process then soon demand for data engineer will be less ? 3) is data engineer a safe career to choose without hesitation that it won't automate or or it's demand will be less 3.) Is data engineering jobs and demands are at stagination stage as many people are moving into it. Is their a enough supply of people to this field like when the supply will be matched then it won't be hot job ? Thanks waiting for your reply
I'm a BI analyst who got approached for a data engineer job. I'm really hesitant about doing the move. I'm interested about learning more about the architecture and big data field but I'm afraid that I won't like it and won't be able to go back in an analyst position 🥺🤯
Probably gonna have to merge Data Analysis, Machine Learning theory wise like MSc paper reading, implementation wise, like X amount of frameworks, C++ production code, and Data Engineering Pipeline/MLOps, into a single title, and know all those stuff, to have a job in the future. I feel like the market has been a lot softer on older guys, and us young new ones are gonna need to raise 3x of their for same salary (or less). Also older guys are far away from being nice to young ones imho, and the technical debt will become greater and greater too!
Half a year? You know so much apparently for that period hahah. Anyways, I recently got my Physics degree and worked with Python in a few coursework and final year project. Next, I'll do a Masters in London but intend to start my career in the 'big data world' and I definitely consider data engineering (but mostly data science :D ). That video was quite helpful (although I don't know 10% of all programs you've mentioned lol ). Anyways, thumbs up! Thanks Karol. Ah, I hope I never need your help to install a driver or smth ahhaha
Physics, nice! Awesome foundation for data science! Oh, I also did a Masters in London - have you started looking at which unis you'd want to apply to? P.S. Data Science is probably more interesting (if you like statistics more than technology/coding) P.S.2 I also hope you won't need me to install drivers 😂
@@karolinasowinska I already applied to 3 unis, I was lucky enough to have 3 offers from Queen Mary, Imperial and UCL. I'll think I'll stick with UCL, although there's still a small chance to change my mind. I'm glad to know that Physics is a good foundation. I love almost everything related to mathematics, coding, statistics, etc (a true geek, ahah), but have no experience in data industry : / a bit scared of not finding any job. About P.S.2: if I do, I prefer to ask you to make a video about it ahahah. Thanks Karol.
Wow, congratulations, that's a lovely set of offers! I'm an Imperial graduate myself - if I can give you my biased opinion, it was a very good course (but also very, very tough). Don't worry, the industry loves geeks haha, you'll surely find a great job! A video about drivers you're saying... I'd give this one a long think! 😅
@@karolinasowinska ahahaha is better doing vids about your relationship with Apple ahhah. Imperial graduate, uow. Congrats!! I hope to go well through this toughness and start working soon (eager to learn :D ). Have an amazing evening. :D :D
It's great to know more about my degree (I am data enginerring student, I am in sixth quarter) and I think, oh my gosh, I need to learn a lot of topics because my teachers don't teach me but I am learning on my own. So, the question is: What are the main topics as a data engineer needs to know? and can you recommend me any page to learn about data engineering? More specific topics such as: Big data, distributed systems and data mining. Greetings from Mexico!
Karolina: Subscribed! Great video, really nice and simple down-to-earth explanations! One point though: I think at one time you talk of HADOOP and you seem to imply that HADOOP has to be on premise? Technically I think that is not true. So, your company might have their own on-premise HADOOP cluster(s), but it is also possible to get "HADOOP as a service" in the cloud, on AWS or other cloud providers. Still, that makes no big difference is what you explained, since the activities you described apply no matter if on premise or in the cloud, but I thought I would point it out.
Is there an increasing or existing trend of hosting Hadoop on cloud? I also feel Hadoop tends to be an on-premise thing. My company actually has been trying to migrate from on-premise Hadoop to Azure
Amazing video. simple and straight to the point. Karolina can you please explain if degree is needed to became a data engineer ? and if yes - what can i learn in home before (which skills)
Hi, thanks so much! A degree in a technical subject is a plus, but not a strict prerequisite. I've just released a free data engineering course here on youtube, so if you want to learn some relevant skills, check it out :) ua-cam.com/video/dvviIUKwH7o/v-deo.html
Agh, I have a huge choice to either pick between becoming a Power BI developer or Data Engineer, I'm proficient with sql and understand databases and table structures, but I know 0 in Python. This video definitely moved me towards data engineering though... agh decision decisions doesn't help I have to choose by tomorrow)... *BTW fantastic video, thank you for sharing your experience 😀
I'm glad it did! :) I also think that it is better to learn a broader, more sought after skills of data engineering. PowerBI is very useful, but sounds a bit narrow!
well in my country every single employee in most AI company is ML Engineer/data analyst/data engineer/Data scientist/dev ops....full stack data scientist
Which country are you working at? Your video is the clearest I've ever watched on DE role. I am a full stack developer(junior, have just 1 year experience), so can I ask how do you see the career path between DE and full stack developer? Thank you
Thanks so much! I live in the UK. I'd say DE is always a backend role, whereas full stack can involve some backend, and some frontend (like building GUIs). One more difference is that typically DE is responsible for bringing the data to a company for researchers to analyse. On the other hand, if SE does some back end work, this is typically to deal with data that their application generates. Of course everything will depend on a specific job. :)
Great questions! So imagine a company that invests money. That company needs to know the current state of the stock market, they also need to have the latest information about the companies, e.g. their latest financial performance. All of that information is data. That data must be bought from a vendor (like a stock exchange or a middleman like Bloomberg) and brought to your investment company. This is where you need some sort of a data pipeline that will ensure that this process is reliable, fast and secure! :)
where are the data vendors getting their data from? and what job does data vendors officially are call? How does someone become a data vendor? It is some1 gathering buying data from multiple business as a third party consumers for exemple w/ xyz vars ? or is a compagny selling a service and recycling business by selling datas? Do data enginnerer learn to program to and data scientist learn more in a particular science and ask you as a shortcut to triage second hand clothings to more reflect on a particular science to draw conclusion for future/act improvement and problems? what are the use of data use for?
Data vendors often *collect* the data. Let's say you're a government, and you record every time someone commits a crime. In the end, you end up having a huge database of crime incidents in your country. If you choose to make this data available (for purchase or free download) then you become a data vendor :) Of course this is just one example, but all other datasets get generated in this way - by recording and storing observations.
@@karolinasowinska Thank you. Was nice dreaming of you and hugging u. Some guy not happy. But why Alina do some1 buy this? What they do with it. What's the ultimate goal. AI?
What are the type of firm who buy data? financial educational governement ect ? Why your employer buys data? Ob to make or try to make money with it but can you give exemples?
@@karolinasowinska carrer first. Do you know any firms who buys data ahead for the sole purpose of re. Obv. this will disappear because of AI. (I had this revelation once in a dream but am not ready). Do you know who is the biggest data collector? -EuropA lol
Many thanks for the information, it’s really helpful. Can I please ask you with whom will the data engineers will be interacting most of the time. And what kind of information will be exchanged.
What about talend or SSIS ?? Data engineer should learn anyone of them or not ?? I can do etl using apache spark and airflow and Kafka and hive Are these enough to get job
I've just released my Free Data Engineering Course for Beginners on UA-cam! Make sure to check it out if you're interested in a career in data engineering! :)
ua-cam.com/video/dvviIUKwH7o/v-deo.html
@Profound Programing I don't, but you can also find me on Instagram: @karo_sowinska :)
@Profound Programing I'll consider setting it up at one point! :)
U r really amazing to starters. NB: I am just studying in 9th standard 😅 just checking how it works so that I don't have to look for this in more big way😁
@@annmaryjo3803 Thanks, and good luck on your journey! :)
What is your background and how you got into data engineering?
I am Data Engineer and I describe myself as follows: "I am like a plumber in the data world and ensure data flows from one abstract system to another in a scalable, consistent and standardized manner to allow the business users to consume the data in a usable way" . My only advise is to ensure all your pipelines are robust and fault tolerance/notifications are built into your data layer.
Absolutely, a data engineer should always consider as many edge cases as possible, because they happen more often than not! I'm glad to read a comment from a fellow data engineer! ;)
And in finance make sure the pennies add up :-) I used to work on billing systems for telecoms multiple types of records multiple types of data sources - we used Map Reduce (back in the early 1980's ) with F77 and Pl/1G
If you had to suggest an undergrad student between data science and data engineering, what would you suggest?
As a recruiter trying to understand job queries, this is gold ! Many thanks
My pleasure! :)
As a data engineer I couldn’t agree more with you!
I just got myself a job as a data engineer as my new career challenge. I appreciate your content, it essentially confirms what I'm getting into :)
That's amazing! Your speaking is very logic and concise, covering a lot of the most important concepts or sessions a data engineer does everyday. A very talented and graceful lady. Thanks for sharing and god bless you.
Thank you so much! I'm glad you like my delivery! :)
I wanted clarity on this profession for a while now. Nobody could explain this in an easy way. You did. Thank you! Very informative.
My pleasure! :)
Your vlog feels real, honest and accurate. I am learning about data science, sql, python, etc. and I find your content very inspiring. Thanks for sharing your knowledge and experience. I am looking forward your future content. Greetings from Mexico.
Thanks so much Luis, I'm glad that you're enjoying my videos! :) Good luck on your data science journey!
1. Bring Data in from different vendors. Check
2. Data pipeline , data warehousing , and aws. Check
3. Table-valued function , Procs and Views. Check
4. Analytics, debug and Monitoring. Check
5. I will Help but Not going to be nice about it. Check
Thank you this gives me confidence to change my career to data engineering . I Am not a Data Engineer but a BI developer. my company is kind of like a start up so we don't have a data engineer so I ended up doing all the pipeline to create my SSRS and Tableau report.
Sounds like you're already doing data engineering! Awesome stuff ;)
Just started my career as a data engineer in a professional environment as well and I can align with everything you say, Karolina!
It's also very interesting because many people think Data Science is just that one single field but it can be divided in so many sub ones. I think you make the role of a DE clearer in this video!
Great work. I am very excited to get to the same point and share the same experience with you.
Hey, thanks so much! I'm glad to hear that you've embarked on your DE journey, I'll be curious to hear how you're finding it! :)
How many tech stacks you need to have to become junior DE?
Like you need to know Python, SQL or smth else?
How many prerequisites should we have before go on to become junior DE?
I really felt that part when you mention to "adapt" for the next 5-10 years, because that's when you need to keep investing in your self. Thanks for sharing your opinion
Absolutely, we will change jobs more frequently than people did in the past, so one needs to expect that and learn to be more adaptable! :)
From a women in tech to a women in tech, great vdo. I can see the spark in your eyes mentioning your fav part.
Thanks! Great to hear from a fellow woman in tech! :)
Komentarz dla zasięgu ;)
Powodzenia w rozwoju kanału!
Dzięki! :)
So great to have such video lessons on youtube! I had an internship 2 and a half years ago where I did the data engineering task and now I want to continue my journey!! Now I study at university and I have much time to do that! Thank you!!!!!
lol the shade in calling data scientists 'not-technical folks', great video!
Love it! How comes I'm just discovering you. Love your sense of humor. I'm working on the path of becoming a data engineer. This is encouraging as all the 5 points are things that excite me. Keep sharing great content. We'll keep supporting.
This is one of the very good videos i have seen on data engineering, thanks.
Thank you, that means a lot. The saga is to be continued very soon!
11:46 - 12:19 - 12:37 - 12:57
this part of the vid is the one i liked the most. not that it's something i already didnt had the idea of, but it felt nice to get reminded of the upcoming highs.
Super glad I came across your channel, Karolina! I'm currently a DA working to make some long-term transitional goals come true! -> data engineering -> data science. Thanks for your awesome insights! Your delivery is really engaging and easy to understand! Best wishes, Karolina!
Thanks, I'm glad you enjoy the videos! :) And your career transition plan sounds great, good luck on your journey!
Hi Karolina, thanks for sharing your thoughts about the subject, I starting a career as a data engineer and you video clarifies me and motivate me to do it as quickle as possible.
Best regards.
Good luck on your journey and I hope to see you around here! ;)
Thanks for another interesting video Karolina! You know...as I was watching the video and hearing you talk about the different tasks data engineers have to perform, I kept thinking about the inevitability of all of those processes getting automated eventually. And of course you concluded with that interesting tidbit about having to learn new skills because you anticipated automation. Just goes to show, not only do you have to identify new opportunities and learn new skills in order to keep thriving, but you also have to develop a strategy to become as financially independent as possible so that you’re not caught all of a sudden in a situation where your skills just cannot support a decent wage anymore. If that doesn’t motivate someone to get off the daily grind for a few hours a night and just sit down, make new plans, pursue other interests, and just think about the meaning of it all, I don’t know what will!
My thoughts exactly, Juan! What you said about financial independence in particular. Sadly, in the face of automation, it is those who are not financially independent, or whose skills are not sufficiently transferable, that will be hit the most.
Knowing full well each job title is like a passing show these days, without doubt, I'm inspired! 👍
Thank you very much for this video. You could believe you are terrible helping other people installing software and things like that, but your videos are the most honest and down to earth description of the job of Data Engineering You are amazing.
I'm so happy that I could help some people with the videos! Thanks!
Thank you for sharing your experience and the great description. I love your accent. You have helped me decide my future career. ❣️
That's wonderful, I'm glad my video was so useful! ;))
Thanks for this video I’m happy to see an actual data engineer give her opinion unlike bigger cs career channels that just read articles
I like your clip, very well articulated. It's like you are reading my minds.
I am a mind reader indeed ;)
Karolina, You are a pure heart 💓.
(Evidence at 11:18)
Keep uploading videos and share us your experience about Data engineering and it's prospects.
May you achieve a Million Subscribers soon.
Kudos, from India ✌️✌️
Aw, I was just honest! Thanks so much! :)
This is extremely clear and concise! Thank you for sharing Karolina :)
Thank you! :)
Thanks a lot from these kind of contents about Data Engineering, I'm a Java developer and I'm looking for conquer a new boundary because I'm a little bored doing micro-services and I want to try some related new topic. Greetings from Chile.
I'm glad this was helpful, and best of luck! ;)
You helped me a lot Karolina. Continue to share your experience, so that we can learn from you.
I'm glad that the video was useful! :)
This was so helpful and informative. Thank you!
Thank you!! I just entered university in the career of data engineer, your video helped me clarify certain doubts
I'm glad you found the video useful! Good luck at university! ;)
Hi, i also want to study data engg in an university but didn't knw there is any good universities and procedures for it.. If u can send me an email regarding that knwldge will be helpfull..... Divyyansh05@gmail.com
Great content! It's funny you said that the thing that you like the least is to help others and here you're creating videos to help others on how to become a data engineer. Plot twist. 😅 Nice job Karolina 👏
Hello, I come from the @CodinEric channel. Your content is very interesting too, thanks for sharing
Thanks and welcome! ;) Sorry for the delay in responding, I've just noticed your comment!
Is it possible to create a video(s) about data scientist vs data analyst vs data engineer?
Or data jobs vs machine/deep learning/ai vs computer science jobs maybe a series of videos?
Could u do a video about how much ai/machine/deep learning could impact jobs and which jobs will likely become automated or disappear vs which jobs will remain strong in the next 5-10 years?
Also based on your last name, its great seeing someone else from a Polish or European background!!
Awesome suggestions - noted, they land on my list! And yes, you're right, I'm Polish! :)
This was super helpful, thank you so much!
My pleasure! :)
Welcome to my world! Best of luck 👍
And same to you! ;)
Really liked your video! I work as a Data Scientist and can totally relate to the hard work our data engineering team puts in.
Thanks, keep making more videos!
Can you let me know what is the career path for data engineer?
Say I don't want to work as data scientist or machine learning engineer, I only like to do data engineering my my entire professional career.
Can one go far just by working as data engineer only and doesn't move to other areas like ML or DS on the later stage?
Hi, absolutely! Data Engineering career path, like all career paths in technology, can take many shapes. But in majority of cases, you'll start out as a analyst/developer (building, fixing data pipelines), and then you will move on to assuming more responsibility, e.g. designing the entire data architecture of a company or managing teams.
Suppose a company wants to have an erp from oracle to sap, now past data is in oracle db, future will be in sap db, now CEO of the company would like to have all data in same format report to make a year on year comparison, then you would need data engineers.
Definitely! Data engineers are useful people to have on board! :)
Try to learn sap BW if you are so much interested to work in data engineering. But you could gain lot more insight if you would understand how business works. But then you donot appreciate economics much. So I think technical role is better for you.
@@candyfloss184 Economics has little bearing on reality. Thanks for your suggestions though.
Karolina Sowinska I have always like macro economics. Microeconomics sounded a bit technical and it was difficult. Development economics and international economics too were interesting to me.
Was a good video. Thank you for this one!
gorgeous as always miss karolina. very imformative.. thanks dear :)
I'm glad you found it useful :)
I'm as a junior BI developer, studying and working with tableau, and I'm missing the programming. I'm going to start working on some projects on my own and see if that's really what I want to do, because of everything you said I liked it so much! The bad part is that where I live there is only Senior vacancy for data engineering
Maybe some junior positions will pop up soon! Definitely go for what you enjoy!
@@karolinasowinska
Very interesting! Keep up the great work!
Thanks so much! :)
I work for an IT consultancy that primarily migrates businesses databases to the cloud. The quantity of work seems endless.
100%, I can imagine that, it's a very trendy thing to do at the moment!
working on cloud computing makes life easy and flow
A good one to explain what data engineers are supposed to do
Tried my best! :)
Thanks for the video and it's rly nice to see another female data engineer voicing their thoughts. I've been a data engineer for officially, I'd say 1.5 years and I really enjoyed the net result of ups and downs. Data engineers understand business better than software engineers, and understand tech better than data analysts/scientists. So to me, when I look at it as a role in translating data for business, I find it quite impossible to be obsolete in the next 5-10 years UNLESS the average data literacy within a company has increased (making better and more careful choices on data inputs) or people turn to machine-generated data more than human-generated data (or when NLP and Deep Learning married for good to successfully "guess" the right data). Otherwise the business would always need to pull in a data engineer to scale their data while preserving its correctness.
Hi there, nice to meet a fellow female data engineer! :) That's a very interesting and convincing perspective - perhaps you're right (and I definitely hope so!)
@irishuang A very insightful comment , thanks for being so clear and concise about what goes on behind the scenes in the world of data.
Przydatny film dla studenta I roku Data Engineering!
O, super! Fajnie, że znalazłeś taki konkretny kierunek, na większości uniwersytetów jest tylko Computer Science. Powodzenia! :)
very cool thanks for posting - i'm learning about data analytics/python/sql wondering if data engineering is a better fit with my IT background
i'm thinking of a career shift to a data engineer. thank you for this :)
My pleasure, and good luck! :)
2021 and still worthy.
Not much has changed since I recorded this! :) Perhaps the accelerated move to the cloud!
Perfect description
thank you for the insights! = )
Always pleased to see experienced people talk about they work and life.
According to this video, I guess your work is much more focus on the pre-process the data rather than implementing and training the model right?
100% true :)
Thank you for the video. Very clear
My pleasure, I'm glad you found it useful! :)
Great explanation of this role, thanks for sharing!
Thanks! :)
@@karolinasowinska you're welcome!
being a data engineer is cool.
But deep learning is much more interesting.
Thank you!
I agree that deep learning is more intelectually challenging!
Can one work as DE the entire career without having to move onto other relevant areas like MLE or DS later on?
Say I love DE job so much, can I stick to it forever?
What will be career path for DE?
I don't like machine learning and data scientist role 😂
@@norpriest521 Hi, yes, you can! Typically the career progression is such that you eventually become the person responsible for the overall data flow in an organisation, eg. the Head of Data Engineering.
Apache Airflow looks like a good fit for this sort of pipeline, really powerfull
Thx! That's worked very well to me, cause I'm already work with data but I didn't know what I was lol.
wonderful insights!! keep creating :)
Thank you, will do! ;)
Very very good video! Awesome.
Thanks! :)
thank you so much , keep up !
Would love to hear your opinion about what "Data Architects" do.
Very similar job to what Data Engineers do, but I guess Data Architects are typically higher up in the hierarchy/they're more experienced, so they can decide on the architecture, and not just build pipelines
Appreciate your explanation.
I'm glad it was helpful! :)
To your last point about standardization and automation, I'd honestly love if Data Engineering and ML Engineering came closer once this happened. DE's already have to deal with all the requests from MLE's, might as well just put them in the same room.
I see both getting closer to being abstracted away to standardized tools within the broader software engineering ecosystem. More successful and matured design patterns just need matured first. But once this happens, I very well could just see software engineering jobs focused on ML and/or Data experience being the new norm.
Hello I found your very helpful but I have some questions can you please answer them..?
1.) As you mentioned that if vendor change the name of coulum then we can make a system where we can take query take dynamic names as column changes.
2.) As it is automation process then soon demand for data engineer will be less ?
3) is data engineer a safe career to choose without hesitation that it won't automate or or it's demand will be less
3.) Is data engineering jobs and demands are at stagination stage as many people are moving into it. Is their a enough supply of people to this field like when the supply will be matched then it won't be hot job ?
Thanks waiting for your reply
Thank you for the good content! You are awesome o/
I appreciate that!
I have been enjoying data engineering and cloud engineering over data science recently
Awesome video.
Thank you! More videos on data engineering are coming soon :)
Super helpful!
I'm an unspoken data engineer lol. I didn't know what ETL was...read about it and realized I have experience in it lol.
Haha I can completely relate - I wrote my Master's thesis on this topic, and only later did I realise that was ETL!
@Jared Dudley or data engineering specifically! ;)
I'm a BI analyst who got approached for a data engineer job. I'm really hesitant about doing the move. I'm interested about learning more about the architecture and big data field but I'm afraid that I won't like it and won't be able to go back in an analyst position 🥺🤯
If you enjoy being an analyst, don't necessarily jump at the opportunity!
I don't know why but I started loving this girl.
I am on a date with this girl through youtube videos.
Keep posting more videos for your lovable fan.
That's very flattering, thanks!:)
Probably gonna have to merge Data Analysis, Machine Learning theory wise like MSc paper reading, implementation wise, like X amount of frameworks, C++ production code, and Data Engineering Pipeline/MLOps, into a single title, and know all those stuff, to have a job in the future. I feel like the market has been a lot softer on older guys, and us young new ones are gonna need to raise 3x of their for same salary (or less). Also older guys are far away from being nice to young ones imho, and the technical debt will become greater and greater too!
Perhaps that's true. One benefit that the younger generation of developers have is knowledge/information availability!
Half a year? You know so much apparently for that period hahah. Anyways, I recently got my Physics degree and worked with Python in a few coursework and final year project. Next, I'll do a Masters in London but intend to start my career in the 'big data world' and I definitely consider data engineering (but mostly data science :D ). That video was quite helpful (although I don't know 10% of all programs you've mentioned lol ). Anyways, thumbs up! Thanks Karol. Ah, I hope I never need your help to install a driver or smth ahhaha
Physics, nice! Awesome foundation for data science! Oh, I also did a Masters in London - have you started looking at which unis you'd want to apply to? P.S. Data Science is probably more interesting (if you like statistics more than technology/coding) P.S.2 I also hope you won't need me to install drivers 😂
@@karolinasowinska I already applied to 3 unis, I was lucky enough to have 3 offers from Queen Mary, Imperial and UCL. I'll think I'll stick with UCL, although there's still a small chance to change my mind. I'm glad to know that Physics is a good foundation. I love almost everything related to mathematics, coding, statistics, etc (a true geek, ahah), but have no experience in data industry : / a bit scared of not finding any job. About P.S.2: if I do, I prefer to ask you to make a video about it ahahah. Thanks Karol.
Wow, congratulations, that's a lovely set of offers! I'm an Imperial graduate myself - if I can give you my biased opinion, it was a very good course (but also very, very tough). Don't worry, the industry loves geeks haha, you'll surely find a great job! A video about drivers you're saying... I'd give this one a long think! 😅
@@karolinasowinska ahahaha is better doing vids about your relationship with Apple ahhah. Imperial graduate, uow. Congrats!! I hope to go well through this toughness and start working soon (eager to learn :D ). Have an amazing evening. :D :D
Great video! Good info! Sincerely done! Appreciated! Thank you! Liked! Subscribed!
I'm glad you enjoyed watching! :)
Thanks a lot. You explained everything very well.I'm starting my career as a Data Engineer, so wish me luck.
That's amazing, good luck! :)
Really interesting video!
I Don't Know why in my country there aren't girl that talk about it carrers. 😅
Cya from Italy!
Thank you! Tech careers seem to attract guys a bit more than girls, but I think it's slowly changing now, and many girls do coding! :) Ciao!
Thank you so much :)
Hi Karolina, your content is amazing and informative. I just curious that what to learn to get started with data engineering. Thank you
Thank you so much! I'm soon going to release a series of videos on data engineering, so stay tuned! :)
Karolina Sowinska That sounds interesting, Thank you
It's great to know more about my degree (I am data enginerring student, I am in sixth quarter) and I think, oh my gosh, I need to learn a lot of topics because my teachers don't teach me but I am learning on my own. So, the question is: What are the main topics as a data engineer needs to know? and can you recommend me any page to learn about data engineering? More specific topics such as: Big data, distributed systems and data mining. Greetings from Mexico!
hahah "each data vendor, they are not nice people, they want data in certain format" hahahah true! I can relate so much
Totally! haha
Karolina: Subscribed! Great video, really nice and simple down-to-earth explanations! One point though: I think at one time you talk of HADOOP and you seem to imply that HADOOP has to be on premise? Technically I think that is not true. So, your company might have their own on-premise HADOOP cluster(s), but it is also possible to get "HADOOP as a service" in the cloud, on AWS or other cloud providers. Still, that makes no big difference is what you explained, since the activities you described apply no matter if on premise or in the cloud, but I thought I would point it out.
Yes, you're right, I only later realised that Hadoop might as well be done in the cloud! Thanks! :)
Is there an increasing or existing trend of hosting Hadoop on cloud? I also feel Hadoop tends to be an on-premise thing. My company actually has been trying to migrate from on-premise Hadoop to Azure
thanks Karolina.
how we are gonna learn Hadoop? (if it is your favorite part :) )
Good question, I'll look for a suitable course to recommend:)
@@karolinasowinska Nice, wait for that.
beside that, you can get affiliation from them.
Amazing video. simple and straight to the point. Karolina can you please explain if degree is needed to became a data engineer ? and if yes - what can i learn in home before (which skills)
Hi, thanks so much! A degree in a technical subject is a plus, but not a strict prerequisite. I've just released a free data engineering course here on youtube, so if you want to learn some relevant skills, check it out :) ua-cam.com/video/dvviIUKwH7o/v-deo.html
@@karolinasowinska thank you so much
Useful information, thanks🌹
I'm glad it was helpful! ;)
Beautiful, funny, smart and data engineer like me! IM IN LOVE!!🥰
Agh, I have a huge choice to either pick between becoming a Power BI developer or Data Engineer, I'm proficient with sql and understand databases and table structures, but I know 0 in Python. This video definitely moved me towards data engineering though... agh decision decisions doesn't help I have to choose by tomorrow)... *BTW fantastic video, thank you for sharing your experience 😀
I'm glad it did! :) I also think that it is better to learn a broader, more sought after skills of data engineering. PowerBI is very useful, but sounds a bit narrow!
This is very helpful!!!
I'm glad you found this helpful! ;)
well in my country every single employee in most AI company is ML Engineer/data analyst/data engineer/Data scientist/dev ops....full stack data scientist
I can imagine that! Where is that, by the way? :)
Can you take a look on datacamp data engineer path and give us your feedback
Sure, I'll have a look soon and let you know!
@@karolinasowinska could you take a look at data engineering path in dataquest as well?
@@victor5265 sure, I'll try to talk about it soon!
Can you make a video on how should one go about learning data engineering?
That's a highly requested video - I think I'm going to make it very soon! Stay tuned! :)
Which country are you working at? Your video is the clearest I've ever watched on DE role. I am a full stack developer(junior, have just 1 year experience), so can I ask how do you see the career path between DE and full stack developer? Thank you
Thanks so much! I live in the UK. I'd say DE is always a backend role, whereas full stack can involve some backend, and some frontend (like building GUIs). One more difference is that typically DE is responsible for bringing the data to a company for researchers to analyse. On the other hand, if SE does some back end work, this is typically to deal with data that their application generates. Of course everything will depend on a specific job. :)
What about data architect? What’s the difference between that and data engineer
Why Data comes to company?
What exactly is Data?
Great questions! So imagine a company that invests money. That company needs to know the current state of the stock market, they also need to have the latest information about the companies, e.g. their latest financial performance. All of that information is data. That data must be bought from a vendor (like a stock exchange or a middleman like Bloomberg) and brought to your investment company. This is where you need some sort of a data pipeline that will ensure that this process is reliable, fast and secure! :)
@@karolinasowinska Thanks understood!
where are the data vendors getting their data from? and what job does data vendors officially are call? How does someone become a data vendor? It is some1 gathering buying data from multiple business as a third party consumers for exemple w/ xyz vars ? or is a compagny selling a service and recycling business by selling datas? Do data enginnerer learn to program to and data scientist learn more in a particular science and ask you as a shortcut to triage second hand clothings to more reflect on a particular science to draw conclusion for future/act improvement and problems? what are the use of data use for?
Data vendors often *collect* the data. Let's say you're a government, and you record every time someone commits a crime. In the end, you end up having a huge database of crime incidents in your country. If you choose to make this data available (for purchase or free download) then you become a data vendor :) Of course this is just one example, but all other datasets get generated in this way - by recording and storing observations.
@@karolinasowinska Thank you. Was nice dreaming of you and hugging u. Some guy not happy. But why Alina do some1 buy this? What they do with it. What's the ultimate goal. AI?
What are the type of firm who buy data? financial educational governement ect ? Why your employer buys data? Ob to make or try to make money with it but can you give exemples?
@@EmilianButoi2 You need data to make well-informed decisions. For example, investment decisions or marketing decisions.
@@karolinasowinska carrer first. Do you know any firms who buys data ahead for the sole purpose of re. Obv. this will disappear because of AI. (I had this revelation once in a dream but am not ready). Do you know who is the biggest data collector? -EuropA lol
Hi! Could you please talk about Scala programming language.
9:40 - that smile hit me, you are so beautiful ❤
Should I learn to deploy big data in cloud ?? I saw the job requirements say aws or big query Google and azure ?? What about
on-premisses ...
Yes, cloud is a very popular choice for many companies, so it's worth learning AWS and the like
Many thanks for the information, it’s really helpful. Can I please ask you with whom will the data engineers will be interacting most of the time. And what kind of information will be exchanged.
My pleasure! Mostly with data scientists and data vendors:)
What about talend or SSIS ?? Data engineer should learn anyone of them or not ??
I can do etl using apache spark and airflow and Kafka and hive
Are these enough to get job