Real-Time Ingestion And Transformation With Snowflake
Вставка
- Опубліковано 10 вер 2024
- Learn how Snowpipe Streaming and Dynamic Tables work together to address a very common use case: ingesting CDC events from source databases/applications and representing that data in Snowflake in multiple forms--current replica of the source tables, aggregation and summary tables, and full historical view of all table transactions using slowly changing dimensions.
These two new features bring highly scalable, near real-time streaming ingestion and single-row insert capabilities to Snowflake, plus a new level of simplicity to merge and transform all those events into consumable data sets with minimal latency, greatly reducing files and micro-batch approach. Over 50K events/second throughput from a laptop, at-source tokenization, AES encryption, and more.
Check out the resources linked in the video:
Snowpipe Streaming
API Docs: javadoc.io/doc...
Documentation:
docs.snowflake...
GitHub Sample Client Project:
github.com/sno...
Dynamic Tables
Article: www.snowflake....
Documentation:
docs.snowflake...
PII Quickstart:
quickstarts.sn...
We also invite you to watch the webinar "Batch and Streaming Data Ingestion Best Practices with Snowflake" to learn how to leverage COPY, Snowpipe, and Snowpipe Streaming to build batch and streaming data pipelines at scale. Register now at: tinyurl.com/2c...
Where can I find the .sh scripts that is being shown in video to generate data from local machine to snowflake?
Check out the links in the video description! Here is the sample client project: github.com/snowflakedb/snowflake-ingest-java
@@snowflakedevelopers Thank you,. is there any video about Snowflake kafka connector for confluent kafka. I need to write data to snowflake tables directly using snowflake kafka connector which pulls data from confluent kafka topic.I'm looking for setup instruction on Snowflake kafka connector and setup on snowflake environment.Appreciate if you can share any videos, current snowflake documentation is not good enough.
Are window functions still supported in dynamic tables? I believe it was not and was considered as a limiation.
Check this section out in the Dynamic Tables known limitations section of the docs: docs.snowflake.com/en/user-guide/dynamic-tables-limitations#unsupported-constructs-operators-and-functions
paste the links of articles of dynamic tables and snowpipe streaming which are in the ppt.
Thanks for your comment. The links were added to the video's description. Have fun building!
Please/thanks!