Відео

Thank you!

Excellent video! Is there any parameter that would create a column for the 'CURRENT' flag? You can add a column for current_flag explicitly but I was curious if it could be generated automatically. The concept is well explained though.

thank you

Excellent videos, thanks for creating them!

Excellent videos, thanks for creating them!

Excellent videos, thanks for putting them!

Great Explanation. Kindly please share the code too.

🎉

🎉

🎉

Good video, thank you: Did you maybe forget to make stag_silver_table as a view instead of a table?

What will happen if source data has an additional column of values which has to be added to target records when matched. Will merge be able to do? Or will it give schema error?

Very informative videos on Databricks Optimization. Few questions on this video. 1. getNumPartitions give us number of partitions that run in parallel at any given time or the total number of partitions of the data frame based on cluster and spark configuration? 2. The available worker cores are 8-16 but the number of in memory partitions from step #2 are just 8. Why not 16? 3. After physically partitioning the file based on Country, the data in each partition gets reduced a lot. Even then there are 8 files under each partition. Is this because the minimum number of available cores are 8? why not 16 since max cores are 16?

Hi Sir , Can we add primary key and foreign key constraints in dlt meta tables ??

Can we bring in the parent relation ?

Hello, I have a question: Suppose I have a cluster with 8 cores and a dataset of 20 GB. Would it make sense to repartition the data into 20 or 21 partitions? After all, I only have 8 cores that can work in parallel, so shouldn’t that be the optimal number of partitions? I clearly understand the principle of coalesce, but I’m a bit confused about the idea of using repartition to create more partitions than the number of cores in the cluster. Thank you very much!

Thank you again for another great tutorial! I follow all your videos, and you explain the material exceptionally well. Based on the videos and Databricks documentation, cluster by is now considered the preferred method and is expected to replace z-order and optimize. Do you agree?

Great content, thank you so much

You helped me so much! Thanks

Thank you so much for the guide! I’m trying to understand-wouldn’t it make sense to create the partitions as 1GB from the start instead of 128MB? Also, once we’ve done the OPTIMIZE, will Spark know to directly read only from the large file? This is quite confusing with the whole concept of data skipping, because in such a case, it seems like data skipping wouldn’t be applicable.

resources are available for 2 hours in development mode.

Good video, thank you very much 😍

Nice crystal clear explanation. Can you add these notebooks , those will be helpful

Thank you it’s very useful! There is any way to estimate a df size?

Very nice and informative video

Thank you for your videos! very usefull. Can you make videos about how to implement CI/CD solutions using Azure Devops?

Thanks a lot

Keep inspiring ❤

Can this be used for Avro files?

any example to read data from catalog table rather cloudfiles

do we need always need streaming table, can't be static table to read

Need your source code, Please provide

Hi Bro how can we implement this in DLT pipeline?

Hello Sir, Could you please share the Parameterized code.

I can see only 4 colums steps, Action, Expected result, attachment. Can we add a customized column like the actual result etc???!

Thanks for the video. Will this resolve the error where the streaming table can only use append-only streaming sources?

Thanks but please how we use html file in automation , the file is existed in test result which was generated when click on capture actions when excute the test case

Hi, can you please send your contact information please

Wow, You are an excellent teacher. Subscribed your channel and looking forward to get more such beautiful explanations 🙏🏼

sir can u please share some sql queries for all these scenarios like verifying datatype and truncation wld like to see sample scenarios

15:24 Why u created second view with a dlt.readStream() despite the fact, that was a Materialized view, not a streaming table? shouldn't it be something like dlt.read() ?

Hi, THanks for videos on Delta live table.However in this playlist continuity is missing. I means vedios are shuffeled. Could you pls add numbers in vedios. so that we can follows vedios one after others. so we can get more clarity. Example Introduction vedio comes 2nd.

Great series!!! I have two questions. We notice that there were 2 sample files loaded in sequence and files stayed in the source folder. With DLT, how could we move old files from source folder to archive folder/storage at the end of pipeline? What is best/better practice for data archiving when using DLT? Another question is how we can perform data retention, e.g. say delete/remove data older than 30 days from the tables managed by DLT pipelines? It would be great if we can talk about more about managing data life cycle with DLT context in the coming videos. Thanks a lot!

everytime you are defining the schema explicitly, but what if we have 100's of files at the source location. you need to first start with historic load, then daily load, apply CDC for all these 100's of files into their respective tables. how do we handle such situation. Obviously it is not good idea to use multiple (100s of ) notebooks right

good video.

Useful content 🎉❤

meta store present in control plan or data plan?

I don't think this was helpful. It didn't really explain the WHY of anything. Why the view, WHY the readStream how do you know when you should do a materialized view or not?

@softwaredevelopmentenginee5650 could you fix the sequence

Your videos are very much beneficial, i have a use case where we need to process nested JSON and those nested have to saved into mulitple table.Also,we will receiving 2 different type of file like sales,purchase.each file is having different schema and transformation,would you be help on building end to end pipeline