Thank you for the video! Would it also be possible to setup notebook parameters and set those parameters through data pipelines for orchestrating notebook runs?
Thanks for your videos', Much appreciated. If I want to copy CSV source to parquet and that too be available under a single container(sub folder) hierarchically (year/Month/day) , which method would be better copy activity or note book ? In terms of performance and using json configured pipelines ?
I haven't personally done performance comparison that much but I would assume that a notebook (if done right) is faster and more efficient considering the capacity usage. I might do video about these topics in near future when I have had time to dig into those. :)
Yes, schemas can be also managed with those config files and if it seems that schemas make the config file very cluttered then one option is to have another config for them.
I covered this briefly in the video, but the reason is version control and deployment. In the real scenario I want to have my configurations in version control (git) and then deploy them via deployment pipeline to following environments and for that using files is better than using data in a table. Hope this answers to your question. :)
Thank you for the video! Would it also be possible to setup notebook parameters and set those parameters through data pipelines for orchestrating notebook runs?
Yes you can!
My video on notebook parameters and return values:
ua-cam.com/video/LOIQCE0Szsc/v-deo.html
On laatukamaa, kiitos!
Olkaa hyvä! :)
Thanks for your videos', Much appreciated. If I want to copy CSV source to parquet and that too be available under a single container(sub folder) hierarchically (year/Month/day) , which method would be better copy activity or note book ? In terms of performance and using json configured pipelines ?
I haven't personally done performance comparison that much but I would assume that a notebook (if done right) is faster and more efficient considering the capacity usage. I might do video about these topics in near future when I have had time to dig into those. :)
Hi, how do you usually manage schema in the schema-first approach? Do you store schemas for tables in the config files?
Yes, schemas can be also managed with those config files and if it seems that schemas make the config file very cluttered then one option is to have another config for them.
It's a very good tip.
But one question.
"Why do you prefer using config.json instead of delta tables for control?
I covered this briefly in the video, but the reason is version control and deployment.
In the real scenario I want to have my configurations in version control (git) and then deploy them via deployment pipeline to following environments and for that using files is better than using data in a table.
Hope this answers to your question. :)
@@AleksiPartanenTech tksss
lakehouse which u use in ur videos are they schemas enabled or not
Schema enabled! :)
@@AleksiPartanenTech thanks