Snowflake - Loading data from Azure
Вставка
- Опубліковано 20 гру 2024
- I can be reachable on jana.snowflake2@gmail.com
-----------------------------
Steps:
-------
1. You should have Snowflake trial account
2. You should have Azure trial account
3. Create storage account and containers in Azure
4. Upload the source files to these containsers
5. Create storage integration between Snowflake and Azure
6. Create stage objects using the storage integration object
7. Use copy commands to extract the data from files and load in snowflake tables.
// Creating Azure free trial account
• How to Create a Free A...
-----------------------------------
-- Create a storage integration object
CREATE STORAGE INTEGRATION snow_azure_int
TYPE = EXTERNAL_STAGE
STORAGE_PROVIDER = AZURE
ENABLED = TRUE
AZURE_TENANT_ID = 'Azure-Tenant_ID'
STORAGE_ALLOWED_LOCATIONS = ('azure://snowazureintg22.blob.core.windows.net/customerdatafiles', 'azure://snowazureintg22.blob.core.windows.net/snowazurefiles');
-- Describe integration object
DESC STORAGE INTEGRATION snow_azure_int;
-----------------------------------
// Create database and schema
CREATE DATABASE IF NOT EXISTS MYDB;
CREATE SCHEMA IF NOT EXISTS MYDB.file_formats;
CREATE SCHEMA IF NOT EXISTS MYDB.external_stages;
// Create file format object
CREATE OR REPLACE file format mydb.file_formats.csv_fileformat
type = csv
field_delimiter = '|'
skip_header = 1
empty_field_as_null = TRUE;
// Create stage object with integration object & file format object
CREATE OR REPLACE STAGE mydb.external_stages.stg_azure_cont
URL = 'azure://snowazureintg22.blob.core.windows.net/snowazurefiles'
STORAGE_INTEGRATION = snow_azure_int
FILE_FORMAT = mydb.file_formats.csv_fileformat ;
//Listing files under your azure containers
list @mydb.external_stages.stg_azure_cont;
// Create a table first
CREATE OR REPLACE TABLE mydb.public.customer_data
(
customerid NUMBER,
custname STRING,
email STRING,
city STRING,
state STRING,
DOB DATE
);
// Use Copy command to load the files
COPY INTO mydb.public.customer_data
FROM @mydb.external_stages.stg_azure_cont
PATTERN = '.*customer.*';
//Validate the data
SELECT * FROM mydb.public.customer_data;
Steps to Load data from Azure
------------------------------
Step 1: Create storage integration between Snowflake and Azure:
docs.snowflake...
Step 2: Create External Stage objects:
docs.snowflake...
Step 3: Copy command to load the data from Azure containers to Snowflake tables:
docs.snowflake...
This is the one exactly what I wanted to understand. Thanks a lot.
SUPER well explained! Just what I needed to know! MANY THANKS!
Hello janardhan
hope you are doing well
could you please make a video on notification alerts and i need to be notified through email when a table is dropped or truncated in the snowflake
by creating the notification alerts. Also when the ware house size is altered i need to be notified.
Please kindly make a video on this ASAP
thank you
Hi sir could you please do one video about snowflake future , current market
and prerequisites or things to learn for some one who what to change into Snowflake domain?
Pls upload video on iceberg tables and dynamic tables
great explain. how to get these files for a good practice.
Hi Jana ,Could you pls provide video on error notification when snowpipe skips the file due to errors
will try
Crisp and clear
Whats best way to do incremental load daily from container to Azure
Adf or snowflake task or something else?
When there is in-built functionality, why should we go for other tools with extra cost? And more over it depends on the technology stack your company is using.
Hello Jana, how can i upload files directly from github to azure container. I tried using shell but each time i am getting 403 error. This request is not authorized to perform this operation and not able to load files. Able to load it from local system but from git hub how to do it.
I never did it, just try this.
www.c-sharpcorner.com/article/uploading-files-from-a-git-repository-to-azure-storage-using-azure-cli/
Sir i took udamey snowflake database for your material is added in the udamey list
yes
Hi sir, do you provide online or offline classes?
No
Hi janardhan garu, it was very useful video ,, I have a requirement to extract the aws snowflake data ingested into azure storage using adf ..can i execute the same commands and assign the role ,,will that work ? Currently I am seeing an issue
What is this boss? I don't understand.
aws snowflake data ingested into azure storage into adf
We have aws snowflake instance ,,my requirement is to connect aws snowflake using Azure adf nd load the data into azure storage
I am using the copy activity in adf ..error is "snowflake fails to access remote file ".Can you please suggest on the same .
It seems credentials or blob storage permission issue. Read this page
community.snowflake.com/s/article/Permissions-error-during-COPY-INTO-from-Azure-Storage-Location
learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-snowflake
Hi sir , plsql course em iyna plan chestunara?
not now
Hi sir,I'm already working on monitoring batch jobs. I want to switch my career to snowflake side. Are you providing any online classes or udemy course is enough to get job on snowflake?
That Udemy course covers all topics needed for interviews
@@mrjana520 thanks for your reply. I'll take that course
Thank you sir❤
Please share data files also
how can I share them in youtube.
You can get them from my Udemy course.
My Snowflake Udemy Course:
www.udemy.com/course/snowflake-complete-course-for-clearing-interviews/?couponCode=A853DD251C99330498C0
Use the coupon below to get it for 499 rupees.
Coupon: A853DD251C99330498C0
What job can we look for after learning snowflake and sql.. I mean like snowflake developer? Or Etl developer?? Etc
There are jobs for Snowflake Developer
Correct, snowflake developer, data engineer etc.