seems like a great tool! Combined with Mage that already has great integrations with dbt, it seems you can make a powerpul and easy-to-set-up data pipeline :-)
Yep we are orchestrator agostic so you can run it on mage or on whatever you want. We offer a dbt runner too, so you don't have to set up credentials and config twice.
If I let both examples run in Google Colab for all rows, not just the first 5 in example 3, I am getting about 10x speed up with example 3 vs. example 2. How can this be possible?
Because a full download at once is faster than a full download as stream. What we demonstrate in the timing is that one does a full download while the other only 5 rows.
The topic is too broad to write a code, he had like 1 hour to demonstrate 3 dlt features, make an introduction, show generators. It would have taken a few hours to write it. Moreover in the case of dlt the code itself is 5 lines and one has a notebook... I agree that it was sometimes hard to follow, but better so, than looking a few hours video (IMHO).
Yeah, well most of the time it was like reading loud what is written in the file. There was not really a use of free words and only a few examples where explained better than written in the text. Seems sufficient to read the file. But I was happy when Alex tried to ask some questions or explained some things differently. Apart from that I think there is no need to watch the video
@@tobiasfsdfsd This was a live workshop which had to be pre-prepared to fit in the time and cover everything effectively. There would be little benefit going off topic. A class is different and you are welcome to suggest it to us, if there is enough demand we will do it!
@@MegaTarino That was exactly what we were looking to do, convey the info effectively in a short amount of time, and as with any class there's an expectation that the learner dedicates 8x the time themselves to practice and comprehend, and for this we provided the notebook and homework assignment. We barely got started covering the topic too, since it's so broad. We are working on a more comrehensive ppeline building course, but that will take 6 hours and be similarly packed, with the expectation that the student invests at least 24h on their own. Covering an entire domain in a few hours is quite efficient I would say, there's a ton of possible complexity around ELT
Starts at 10:44
Classe really well made, i enjoy to have a complete .md file with all instruction writed. It's really convenient ! Thanks !
Thank you! the intention was that here the class is delivered in a human format, and the student can take their time to deep dive on their own.
seems like a great tool! Combined with Mage that already has great integrations with dbt, it seems you can make a powerpul and easy-to-set-up data pipeline :-)
Yep we are orchestrator agostic so you can run it on mage or on whatever you want. We offer a dbt runner too, so you don't have to set up credentials and config twice.
Amazing content and discussion, as data engineer myself looking forward to contribution in the project.
Thank you!
If I let both examples run in Google Colab for all rows, not just the first 5 in example 3, I am getting about 10x speed up with example 3 vs. example 2. How can this be possible?
Because a full download at once is faster than a full download as stream. What we demonstrate in the timing is that one does a full download while the other only 5 rows.
Might be a stupid question but in 33:25 you say that if we have 4gb in file, we have 8gb in memory. Why is that?
because it runs twice
You keep the contents of the 4GB file in memory twice, in both "data" and "parsed_data" variables, thus 8GB of memory is consumed
Besides that it was kept twice, we assume efficient data storage. In reality, if you load 4gb to a df, you might see much more ram usage.
lego 75192 millennium falcon?
Yes =)
Quite hard to watch & follow.
Workshop should be "writing code", step by step and not show ready code.
The topic is too broad to write a code, he had like 1 hour to demonstrate 3 dlt features, make an introduction, show generators. It would have taken a few hours to write it. Moreover in the case of dlt the code itself is 5 lines and one has a notebook... I agree that it was sometimes hard to follow, but better so, than looking a few hours video (IMHO).
Yeah, well most of the time it was like reading loud what is written in the file. There was not really a use of free words and only a few examples where explained better than written in the text. Seems sufficient to read the file. But I was happy when Alex tried to ask some questions or explained some things differently. Apart from that I think there is no need to watch the video
@@tobiasfsdfsd This was a live workshop which had to be pre-prepared to fit in the time and cover everything effectively. There would be little benefit going off topic. A class is different and you are welcome to suggest it to us, if there is enough demand we will do it!
@@MegaTarino That was exactly what we were looking to do, convey the info effectively in a short amount of time, and as with any class there's an expectation that the learner dedicates 8x the time themselves to practice and comprehend, and for this we provided the notebook and homework assignment. We barely got started covering the topic too, since it's so broad. We are working on a more comrehensive ppeline building course, but that will take 6 hours and be similarly packed, with the expectation that the student invests at least 24h on their own.
Covering an entire domain in a few hours is quite efficient I would say, there's a ton of possible complexity around ELT