here mount point can i give directly access key is secure or not. instead of directly without given access key is possible to create secret key by using key vault.
In our project we are using abfss path apart from wasbs. Most projects i had seen they are using abfss. What is the difference between abfss vs wasbs. Could you please explain 🙏
Hi, is there any other way or option to access files from ADLS inside databricks without mounting the storage account in databricks? Can you please provide your inputs on that. Thank you!
Account Key- if you provide access using it then user will get complete access on account. Like view/modify. SAS- it's useful if you want to share the access to resources for a specific period of time and only specific permission like view or create or modify or all.
Hi bro, today I attended the TCS interview. they asked me about realtime scenarios in adf. 1. how to create reusable pipeline for collecting the required columns from n no of files from adls to SQL. ex I have 10 files in every file I have 20 columns but I want only 15 columns. I need to do this activity repeatly so you can create a pipeline for reusable. 2. in adls I have different CSV files in Adls ex India, aus, eng, sa cricket teams. I want only Indian cricket team realted files. for this purpose how to create a pipeline. Please create realtime scenario video for this thanks in advance
2.) in adls u have diff csv files , u want only india team related files ? ans: take getmetadata acitivity in this to fetch all files than take filter activity in this u have give condition on starts with indian files next copy activity ....then u get only indian team files.
2nd method ; fisrt u can take csv datasets and linked services on adls gen2 ,,take pipeline on copy activity to select source in source u give widcard file path. *.csv open dataset select indian folders only and finally in sink u create datasets and linkedservice on destination ..finally sink u give name on sink path...debug u can get output only indian files
Using Account Key:
dbutils.fs.mount(
source=‘wasbs://@.blob.core.windows.net/’,
mount_point=‘/mnt/’,
extra_configs={‘fs.azure.account.key..blob.core.windows.net’:’’})
Using SAS token:
dbutils.fs.mount(
source=‘wasbs://@.blob.core.windows.net/’,
mount_point=‘/mnt/’,
extra_configs={‘fs.azure.sas.< containerName >..blob.core.windows.net’:’’})
can i save this Account key and SAS token , can we use same syntax in Realtime works
It's very nice video, learning adb from you. Please upload more videos
Thanks 🙏
Crystal clear, really very helpful
Thanks for your appreciation,
It always motivate me
here mount point can i give directly access key is secure or not. instead of directly without given access key is possible to create secret key by using key vault.
Yes we can, watch below video:
ua-cam.com/video/BF_UNfRJrD4/v-deo.html
In our project we are using abfss path apart from wasbs. Most projects i had seen they are using abfss. What is the difference between abfss vs wasbs. Could you please explain 🙏
I don't have answer for this question now but let me check and confirm
Can we create pipeline in ADF to copy from input to output instead of mount process? Tq
Yes, in ADF you can do that.
Hi, is there any other way or option to access files from ADLS inside databricks without mounting the storage account in databricks? Can you please provide your inputs on that. Thank you!
I didn't see any other option except mount points.
Nice explanation 👌 👍 👏
Thanks 🙏
Superb video😍
Thanks 🙏
Thank you
Thanks
Please share to others
Keep learning and growing 💗
Hi What is the Difference between Account key and SAS token.
Account Key- if you provide access using it then user will get complete access on account. Like view/modify.
SAS- it's useful if you want to share the access to resources for a specific period of time and only specific permission like view or create or modify or all.
Hey It is nice video. Thank you
Thanks
Please share to others
Keep learning and growing 💗
Hi bro,
today I attended the TCS interview. they asked me about realtime scenarios in adf.
1. how to create reusable pipeline for collecting the required columns from n no of files from adls to SQL. ex I have 10 files in every file I have 20 columns but I want only 15 columns. I need to do this activity repeatly so you can create a pipeline for reusable.
2. in adls I have different CSV files in Adls ex India, aus, eng, sa cricket teams. I want only Indian cricket team realted files. for this purpose how to create a pipeline.
Please create realtime scenario video for this thanks in advance
2.) in adls u have diff csv files , u want only india team related files ?
ans: take getmetadata acitivity in this to fetch all files than take filter activity in this u have give condition on starts with indian files next copy activity ....then u get only indian team files.
2nd method ; fisrt u can take csv datasets and linked services on adls gen2 ,,take pipeline on copy activity to select source in source u give widcard file path. *.csv open dataset select indian folders only and finally in sink u create datasets and linkedservice on destination ..finally sink u give name on sink path...debug u can get output only indian files
Can you please explain more about 2nd question