- 4
- 6 936
Data Gravity
Приєднався 22 гру 2021
Hi, I am Vengatesh Parasuraman. I am a data geek and my specialty is Microsoft Fabric - your all-in-one analytical solution.
In this channel, you will watch and learn Microsoft Fabric with my how-to videos, deep dives, fundamental to advanced concepts covering a wide range of topics - data integration, data analytics, data engineering, data science, data warehousing and real time analytics. I will provide special focus on platform security, data security, administration and governance.
If you are a Microsoft Fabric user or an administrator, you are in the right place !!!
If you have any questions on Microsoft Fabric, feel free to comment and engage in discussions, I would love to interact with you here !!!
*** Official Disclaimer ***
I, Vengatesh Parasuraman, am a Microsoft Employee, but this UA-cam Channel, Data Gravity, is not associated with Microsoft or any of it's affiliates or partners. This is my personal channel and all opinions expressed are MY OWN.
In this channel, you will watch and learn Microsoft Fabric with my how-to videos, deep dives, fundamental to advanced concepts covering a wide range of topics - data integration, data analytics, data engineering, data science, data warehousing and real time analytics. I will provide special focus on platform security, data security, administration and governance.
If you are a Microsoft Fabric user or an administrator, you are in the right place !!!
If you have any questions on Microsoft Fabric, feel free to comment and engage in discussions, I would love to interact with you here !!!
*** Official Disclaimer ***
I, Vengatesh Parasuraman, am a Microsoft Employee, but this UA-cam Channel, Data Gravity, is not associated with Microsoft or any of it's affiliates or partners. This is my personal channel and all opinions expressed are MY OWN.
Loading on-prem DB to Fabric Lakehouse with Synapse Spark
In this video, you will learn how to load data from on-prem SQL Server database, securely and privately, to Microsoft Fabric Lakehouse using Synapse Spark, without having to poke holes in your on-prem firewalls.
Please find below all the shell scripts used for IP Forwarder:
1. To change current user to root
sudo su -
2. To enable IP Forwarding flag
echo "1" > /proc/sys/net/ipv4/ip_forward
3. Declare variable: Ethernet Interface
ETH_IF=eth0
4. Declare variable: Front-end port number
FE_PORT=1443
5. Declare variable: Destination IP
DEST_IP=10.0.4.7
6. Declare variable: Destination port number
DEST_PORT=1443
7. Create DNAT Rule
iptables -t nat -A PREROUTING -p tcp -i ${ETH_IF} --dport ${FE_PORT} -j DNAT --to ${DEST_IP}:${DEST_PORT}
8. Create SNAT Rule
iptables -t nat -A POSTROUTING -o ${ETH_IF} -j MASQUERADE
9. Check IP Tables Configuration
iptables -t nat -L
-----------------------
Follow me on Linked-In:
www.linkedin.com/in/vengat83/
Please find below all the shell scripts used for IP Forwarder:
1. To change current user to root
sudo su -
2. To enable IP Forwarding flag
echo "1" > /proc/sys/net/ipv4/ip_forward
3. Declare variable: Ethernet Interface
ETH_IF=eth0
4. Declare variable: Front-end port number
FE_PORT=1443
5. Declare variable: Destination IP
DEST_IP=10.0.4.7
6. Declare variable: Destination port number
DEST_PORT=1443
7. Create DNAT Rule
iptables -t nat -A PREROUTING -p tcp -i ${ETH_IF} --dport ${FE_PORT} -j DNAT --to ${DEST_IP}:${DEST_PORT}
8. Create SNAT Rule
iptables -t nat -A POSTROUTING -o ${ETH_IF} -j MASQUERADE
9. Check IP Tables Configuration
iptables -t nat -L
-----------------------
Follow me on Linked-In:
www.linkedin.com/in/vengat83/
Переглядів: 1 876
Відео
Loading on-prem DB to Fabric Lakehouse with Dataflows
Переглядів 2,8 тис.11 місяців тому
In this video, you will learn how to load data from on-prem Oracle database, securely and privately, to Microsoft Fabric Lakehouse using Dataflows and On-prem Data Gateways. Follow me on Linked-In: www.linkedin.com/in/vengat83/
Loading data from ADLS behind firewalls to Fabric Lakehouse
Переглядів 1,7 тис.11 місяців тому
As part of this Microsoft Fabric Security video, you will learn how to connect to your data sources that are behind firewalls, vnets, private endpoints such as Azure Data Lake Storage Gen 2 (ADLS Gen 2) and load them into Microsoft Fabric to start evaluating with your corporate data. Follow me on Linked-In: www.linkedin.com/in/vengat83/
Azure Synapse Analytics - Security Fundamentals
Переглядів 5742 роки тому
Want to learn all about how to secure your data in Azure Synapse Analytics ? In this video, I will walk you through the fundamental concepts of Synapse Security. You will learn about Synapse Workspaces, Workspace Endpoints and Synapse Studio. This video is part of a multi-part video series on Synapse Security. Stay tuned to watch the rest of the videos. Check out our Synapse Security White Pape...
Great content please create more videos in fabric
Hi Is there a way to connect to on-prem SQL server using Fabric Notebook?
I am not able to load large data from Oracle to fabric, Microsoft said they dont have that provision till date, is there any way to get 100million + data from Oracle on prem to fabric without dataflow ? Thanks
is it possible to set a public ip to fabric to be allowed in a firewall in front of my sql db?
Amazing content, Vengatesh! This video saved me a lot of time and effort, thank you so much. I am curious whether we can create a shortcut within datalake from ADLS2 storage container with the same approach? Its giving me error 'invalid credentials'.
This is exactly the information I was looking for, thank you!
very good we like to use immediately very happy we need some help how i connect u sir
How come is this considered connecting to an on-prem DB when you are actually connecting to a Azure VM? The backend in the case of on-prem should be an NIC connected to an on-prem, while during the creation you only see Azure VMs
Very clear explanation. I have followed the steps to connect with Microsoft SQL server with Fabric and once i have clicked on publish i am getting error and cannot load data into Fabric warehouse. Can you give me any ideas or work around this issue
Thanks for this video. I was looking for this solution (or work around). Can Azure Data Factory(classic) write directly to Fabric Lakehouse using AAD authentication? In that case we can read data with ADF and write to Lakehouse from the same.
Can we write the same python code in fabric notebook. Exexuting python code in azure synapse analytics will incure extra cost. In fabric no need to pay extra cost for python code. It is part of fabric subscription
Hi @dineshs9670 - Yes, eventually you will be able to use the same code in Fabric Spark (when we ship the equivalent of Managed VNets in Fabric). But at the moment, Fabric Spark clusters will have no way to reach your on-prem DB behind firewalls privately. In order to enable that connectivity, you may be required to open up your on-prm DB with a Public IP and allow inbound ports such as 1433 (or whatever the DB is listening on) for our spark clusters to connect - if your security allows that, then yes you can use Fabric Spark with the same approach. The workaround I showed with Synapse will pass your security gates.
@@datagravity3578 any idea when Managed VNets in Fabric will be available?
Awesome!! Thanks for that!
awesome! looking forward to the part 2.
This is great! But how do you connect to a on prem SQL SERVER DB/DW ? I've tried to use gateway, last week, and it wasn't working. Is there any way to do it? Thanks!
Thanks for Watching ! The method would be the same (except that you dont have to install the drivers, it comes with it) - you can refer to SQL Server data source in our on-prem data gateways here: learn.microsoft.com/en-us/power-bi/connect-data/service-gateway-enterprise-manage-sql
@@datagravity3578 Thanks! I'll try that when I got a chance :)
Awesome video!! 1. Could you please explain what AD user that is used for authenticating to Fabric, is it managed identity or a personal user? 2. How is Synapse able to access Fabric over the public internet when Synapse is using a managed private network? 3. Does Fabric and Synapse have to be in the same tenant? Thanks :)
Thanks for watching !!! 1. My Synapse Spark notebook is using my personal (logged in) AAD user credential to execute. So the authentication to Fabric is also happening seamlessly with that AAD token. If you were to use Managed Identity, it is very much possible to execute your notebook using MI and you have to provide the appropriate permissions for that MI in your Fabric Workspace - Note: For MI to work, Fabric and Azure Synapse needs to be on the same tenant only. 2. MAnaged VNet provides network isolation and secure outbound connectivity for spark workloads (and azure integration runtimes). When it comes to outbound connectivity, spark can connect to any public endpoint from a Managed VNet - all outbound calls are allowed. Only when we enabled "data exfiltration protection" for Synapse, we will block any outbound calls that do not fall under approved tenants and those don;t have a managed private endpoint. In my case, I have enabled Managed VNET but I have not enabled DEP. Watch my video to learn more about Managed VNets - ua-cam.com/video/4PJOuhFosLY/v-deo.htmlsi=ZGQjdYgdEOCbWAi9 3. For seamless authentication yes - but not mandatory. You can call the OneLake APIs with appropriate access tokens also inside your notebook: blog.fabric.microsoft.com/en-us/blog/connecting-to-onelake?ft=All:
Thank you@@datagravity3578! :) I really appreciate your prompt response and excellence explanation. This deserves a Sub, hope you keen making more content
brilliant idea!!!
Glad you think so!
Great video Vengatesh!
Glad you enjoyed it
Good Explaination Venkatesh ! Looking forward for next vidoes