- 53
- 51 628
Decodable
United States
Приєднався 24 лют 2022
Decodable’s mission is to make streaming data engineering easy. Decodable delivers the first real-time data engineering service - that anyone can run. As a serverless platform for real-time data ingestion, integration, analysis, and event-driven service development, Decodable eliminates the need for a large data team, clusters to set up, or complex code to write.
To learn more, visit us at decodable.co
To learn more, visit us at decodable.co
Managing Streaming between Multiple External Resources with One Connection
Data movement is at the heart of the platform we’re building at Decodable-simplifying the movement of data between all kinds of systems, in real-time, with as little or as much processing as needed. With Multi-Stream Connectors (MSC) in Decodable, you can now employ a single connection to manage streaming data movement from multiple external resources with ease.
Join Decodable engineers Gunnar Morling and John MacKinnon for this tech talk and demo as they build a multi-stream data flow from a MySQL database to Snowflake.
* Limit operational overhead and cost by significantly reducing the number of source and sink connections you need for your data flows
* Set up multi-stream end-to-end data flows within minutes, including the automated scanning for source resources and the creation of required sink resources
* Handle the complexities of schema evolution in a guided way
____
Start free at app.decodable.co/-/accounts/create
Get started at docs.decodable.co/docs/web-quickstart-guide
Visit us at www.decodable.co
Join the community at decodablecommunity.slack.com/join/shared_invite/zt-uvow71bk-Uf914umgpoyIbOQSxriJkA#/shared-invite/email
Follow Decodable at decodableco & www.linkedin.com/company/decodable
Follow Gunnar at gunnarmorling & www.linkedin.com/in/gunnar-morling-2b44b7229/
Follow Eric at esammer & www.linkedin.com/in/esammer/
Join Decodable engineers Gunnar Morling and John MacKinnon for this tech talk and demo as they build a multi-stream data flow from a MySQL database to Snowflake.
* Limit operational overhead and cost by significantly reducing the number of source and sink connections you need for your data flows
* Set up multi-stream end-to-end data flows within minutes, including the automated scanning for source resources and the creation of required sink resources
* Handle the complexities of schema evolution in a guided way
____
Start free at app.decodable.co/-/accounts/create
Get started at docs.decodable.co/docs/web-quickstart-guide
Visit us at www.decodable.co
Join the community at decodablecommunity.slack.com/join/shared_invite/zt-uvow71bk-Uf914umgpoyIbOQSxriJkA#/shared-invite/email
Follow Decodable at decodableco & www.linkedin.com/company/decodable
Follow Gunnar at gunnarmorling & www.linkedin.com/in/gunnar-morling-2b44b7229/
Follow Eric at esammer & www.linkedin.com/in/esammer/
Переглядів: 221
Відео
Send Your Decodable Metrics to Datadog
Переглядів 186Рік тому
Join Saketh Kurnool to learn how to send data from Decodable to Datadog for downstream analytics. You can use this connector to send data from the Decodable _metrics stream into Datadog in order to monitor the state and health of your Decodable connections and pipelines. Start free at app.decodable.co/-/accounts/create Get started at docs.decodable.co/docs/web-quickstart-guide Visit us at www.d...
How To Enable Change Data Capture With Postgres On Amazon RDS--Data Streaming Quick Tips Episode #4
Переглядів 2,4 тис.Рік тому
Join Gunnar Morling for this episode of "Data Streaming Quick Tips" to learn how to set up a Postgres database on Amazon RDS for change data capture, so that you can emit data change streams to tools like Debezium or managed stream processing platforms such as Decodable. Resources: * Decodable documentation: docs.decodable.co/docs Please reach out to Decodableco or gunna...
Array Aggregation With Flink SQL-Data Streaming Quick Tips Episode #3
Переглядів 1,4 тис.Рік тому
In this episode of "Data Streaming Quick Tips", Gunnar Morling is showing you how to aggregate the elements of an array with Flink SQL, using the built-in function JSON_ARRAYAGG(), as well as a user-defined function for emitting a fully type-safe data structure. This then is used to ingest the results of a 1:n join as nested document structures into a search index in Elasticsearch. Resources: *...
Re-keying a Kafka Topic--Data Streaming Quick Tips Episode #2
Переглядів 680Рік тому
In this episode of "Data Streaming Quick Tips", Gunnar Morling is showing you how to re-key a Kafka topic Serverless style, using Decodable! Resources: * Decodable documentation: docs.decodable.co/docs Please reach out to Decodableco or gunnarmorling if you'd like us to discuss specific data streaming topics in a future DSQT episode. Happy streaming! Start free at app.de...
The Flink Upsert Kafka SQL Connector--Data Streaming Quick Tips Episode #1
Переглядів 1,6 тис.Рік тому
In this inaugural episode of "Data Streaming Quick Tips", Gunnar Morling is taking a look at how to use Flink's Upsert Kafka SQL connector for propagating events from a changelog stream (created via Flink CDC and Debezium) to a topic in a Redpanda cluster. Resources: * Full source code: github.com/decodableco/examples/tree/main/flink-learn/2-kafka-upsert * Upsert Kafka SQL Connector: nightlies....
Streaming Data into Snowflake with Decodable
Переглядів 331Рік тому
Learn how to reduce the cost and complexity of streaming data into Snowflake with Decodable. Start free at app.decodable.co/-/accounts/create Get started at docs.decodable.co/docs/web-quickstart-guide Visit us at www.decodable.co/ Explore our connector library at www.decodable.co/connectors Join the community at decodablecommunity.slack.com/join/shared_invite/zt-uvow71bk-Uf914umgpoyIbOQSxriJkA#...
Welcome to Decodable!
Переглядів 4,1 тис.Рік тому
Decodable is a real-time stream processing platform built on Apache Flink®, providing data architects, engineers, and analysts with a simple, easy-to-use developer experience leveraging SQL, making the power of Flink accessible to everyone and providing the fastest route to success. Start free at app.decodable.co/-/accounts/create Get started at docs.decodable.co/docs/web-quickstart-guide Visit...
Decodable at Solutions Review's Data Demo Day
Переглядів 111Рік тому
Join Doug Atkinson as he interviews Decodable CEO Eric Sammer. They discuss making streaming data engineering easier with Decodable's fully-managed stream processing platform, which allows for the real-time ingestion, integration, and transformation of data to support the development of event-driven applications and services. This helps put the power of real-time data engineering into the hands...
Introduction to Apache Flink and Flink SQL
Переглядів 5 тис.Рік тому
Join Gunnar Morling for a ten minute introduction to Flink and FlinkSQL, as you see him build a Flink pipeline to process data from one Kafka topic to another Kafka topic. In this example, he'll be using RedPanda's Kafka API compatible offering to stream data into and from Flink. Start free at app.decodable.co/-/accounts/create Get started at docs.decodable.co/docs/web-quickstart-guide Visit us...
Benefits of Real-Time Stream Processing
Переглядів 268Рік тому
Join David Fabritius as he explores the features and benefits of leveraging Decodable for your real-time stream processing needs. Decodable is a stream processing platform providing the simplest method for moving data anywhere with real-time speed, transformed to match the needs of its destination. As a fully managed stream processing service, Decodable provides pre-built connectors to external...
Security and how it plays in your data infrastructure
Переглядів 4702 роки тому
An Overview of How Security Works With Data In today’s cloud-first world, data can exist in a wide variety of locations with different security considerations than the previously locked-down private datacenter. Unfortunately, bad actors that want that data can also now inhabit these new locations. Privacy and data protection regulations are designed to enforce the duty of care, especially when ...
Interview: Change Data Capture With Apache Flink
Переглядів 1,6 тис.2 роки тому
In this interview, we talk about Change Data Capture with Debezium and Flink with experts in this field as our guests: Gunnar Morling, one of the creators of the Debezium project answers any about CDC in general, as well as Debezium-specific questions. For deep insights into the Flink CDC Connectors project, we have Leonard Xu and Jark Wu, two long-term Flink contributors and leads on the Flink...
Change Stream Processing With Apache Flink
Переглядів 2,3 тис.2 роки тому
In this demo-heavy webinar Gunnar Morling and Sharon Xie answer common questions on change stream processing: - What is Change Data Capture (CDC) and why should I care? - What do I gain when I integrate CDC with Apache Flink? After this introduction, they switch to code to show how it works in a demo that includes CDC sources, stream processing and delivery to an Elastic database for searching....
Interview: Deploying Flink With The New Kubernetes Operator
Переглядів 4,7 тис.2 роки тому
Apache Flink PMC chair, Robert Metzger, in conversation with Gyula Fóra and Mátyás Örhidi, the main contributors of the new Flink Kubernetes Operator launched earlier this year. The Flink Kubernetes Operator is an abstraction layer on top of Kubernetes that makes deployment and operation much easier than apply Flink directly to Kubernetes components. In this discussion we cover the motivation f...
Mirror Data from PostgreSQL to Snowflake
Переглядів 1,5 тис.2 роки тому
Mirror Data from PostgreSQL to Snowflake
Ingesting & Processing S3 changes via AWS Lambda
Переглядів 982 роки тому
Ingesting & Processing S3 changes via AWS Lambda
Realtime Join Between Confluent Cloud and PostgreSQL
Переглядів 1912 роки тому
Realtime Join Between Confluent Cloud and PostgreSQL
Configuring Apache Kafka with mTLS - mutual TLS authentication
Переглядів 4,3 тис.2 роки тому
Configuring Apache Kafka with mTLS - mutual TLS authentication
Top 3 Challenges Running Multitenant Flink At Scale
Переглядів 5522 роки тому
Top 3 Challenges Running Multitenant Flink At Scale
Ingesting COVID data into Imply Polaris
Переглядів 782 роки тому
Ingesting COVID data into Imply Polaris
Real-time Change Data Capture (CDC) Processing Part Two
Переглядів 1802 роки тому
Real-time Change Data Capture (CDC) Processing Part Two
Real-time Change Data Capture (CDC), Processing and Ingest using Decodable
Переглядів 4822 роки тому
Real-time Change Data Capture (CDC), Processing and Ingest using Decodable
MySQL CDC to Clickhouse using Decodable's Change Stream Capabilities
Переглядів 7352 роки тому
MySQL CDC to Clickhouse using Decodable's Change Stream Capabilities
Processing Bitcoin data from Kafka to S3
Переглядів 7582 роки тому
Processing Bitcoin data from Kafka to S3
Synchronizing MySQL Data To Clickhouse
Переглядів 1,6 тис.2 роки тому
Synchronizing MySQL Data To Clickhouse
Brilliant sir!!!
Hi sir how can I contact you can we connect once?
was it necessary for you to use snowpipe? why can't just decodable directly load to snowflake ?
Way tooo complicated just to move your data around, its time for industry disruptor to abstract all that junk and make it possible in a simple user driven experience. You're gong to lose half the audience.
Thanks for the informative interview. Its been an year so may be its time to do a refresher interview to understand the changes to flinkml in the last year as mentioned at the end of this interview
Please paste link of of commands and code
This superlative, very good stuff and detailed
Nice! Thanks for the explanation The audio in the first part of the video is amazing, I suppose if she uses the Mac Microphone should be better than the Bluetooth headset!
Great discussion!
A Great one! Thank you
Curious to learn more about us? Here are some great next steps! Start free at app.decodable.co/-/accounts/create Get started at docs.decodable.co/docs/web-quickstart-guide Visit us at www.decodable.co/ Explore our connector library at www.decodable.co/connectors Join the community at decodablecommunity.slack.com/join/shared_invite/zt-uvow71bk-Uf914umgpoyIbOQSxriJkA#/shared-invite/email Follow Decodable at twitter.com/decodableco & www.linkedin.com/company/decodable Follow Eric at twitter.com/esammer & www.linkedin.com/in/esammer/ Follow Gunnar at twitter.com/gunnarmorling & www.linkedin.com/in/gunnar-morling-2b44b7229/ Follow Robert at twitter.com/rmetzger_ & www.linkedin.com/in/metzgerrobert/
That's a nice demo (also apart from the topic), exceptionally well executed. How many takes did it take? Out of curiosity, why not using a proper group by and using a correlated subquery for that purpose?
Thanks you so much, means a lot to me coming from you! Not sure about the number of takes tbh, the entire video (including the development efforts for that UDF) took me about one and a half weeks, ca. three days of which were for the video. In regards to grouping vs. using a correlated subquery, that's a great question. I wanted to discuss it in the video actually, but then forgot about it, and it was too long already anyways. In a nutshell, GROUP BY would only work for a single ARRAY_AGGR call. For multiple ones (as in that second example), it won't work, as you can't group by the result of an aggregation function, also you can't nest aggregation function calls (say, there was another 1:n relation from order line to yet another table). Both can be done with the subquery approach.
@@GunnarMorling Sure nested aggregations will not work, but on the same level with the same grouping set? Sure, why not: select date_trunc('month', measured_on) as month, array_agg(import), array_agg(export) from measurements group by month;
Does Eclipse have presentation mode, or only modern IDEs, like IntelliJ IDEA, implement it?
Great thanks, it was extremely informative.
This is helpful thanks, do you have an example repo to follow along with the video?
Yes, you can find the complete source code from this video in the Decodable examples repo here: github.com/decodableco/examples/tree/main/flink-learn/1-quickstart
Awesome! Are those tables real one on disk? What if the incoming stream is infinite? Would such table grow Infinitely? Or is this just a view of the stream for some period?
Thanks for this great question, and sorry for the late response. "Are those tables real one on disk?" It depends on the type of table. If you are doing a lookup join between a stream containing cdc events and a stream of events, Flink will rebuild/materialize the table in its state backend for looking up the current values. If you are using the RocksDB statebackend, those tables will be on disk. "What if the incoming stream is infinite?" For a stream of cdc / change events from some source table, the number of unique rows is limited. If you are joining two infinite streams, you can define a TTL for the events in state, so that they will be evicted eventually. " Or is this just a view of the stream for some period?" This is what windowing functions are for. You can create windows over a stream where Flink is buffering all the data for a period of time (or a number of events) to analyze them as a batch.
Hi Great video!!!, do you know if flink stream can do streaming between two separate kafka clusters?
Thanks for your nice feedback! Yes, streaming between separate Kafka clusters can be done with Flink.
Came across your video today and I must say you have explained really well. May I know about any upcoming batches for Kafka Admin training if you take?
Very powerful tools - debezium and flink ! simple and very helpful UI for such complex works by decodable. Thanks for the explanation
The video is not long, but it clearly introduces the core features of 1.16 👍🏻
Hubert- very well explained as always !!