Ben Gamble, Field CTO at Ververica (the original creators of Apache Flink), and Sida Shen, Product Manager at CelerData, break down the essentials of real-time stream processing, sharing insights into its benefits, challenges, and practical ways to get started.
Take a closer look at VERA: The Path to Cloud-Native Apache Flink, and how it’s designed to support today’s real-time data needs. The session concludes with a live demo that showcases how CelerData (managed StarRocks) and Ververica (managed Flink) integrate to form a powerful data stack. If you're interested in real-time data processing and its practical applications, this is a perfect place to start!
----------------------------------------------------------------------------------------------------------------------
00:00 Introduction
02:30 What is Stream Processing?
04:15 Does Stream Processing Mean it is “Always On”? Does it Make it Expensive? How Do You Manage the Cost?
08:47 How Do I Know if Stream Processing is Actually Useful for My Business?
11:08 The Technology Complexity of Stream Processing
17:20 With the Technology Available Today, What Use Cases Can Stream Processing Specifically Excel In? What Kind of Use Cases Are There for Stream Processing?
22:16 Understanding Real Time: "Soft Real Time," "Near Real Time," and "Real Real Time"-What Real Time Means
24:36 Challenges in Real-Time Analytics in Analytical Settings and How Databases Like StarRocks Address These Challenges
28:19 The Evolution of Table Formats and Introduction to Apache Paimon
32:38 The Inherent Complexity Compared to Batch Processing and the Knowledge Required to Manage Workloads
35:22 Solutions to Managing the Complexity in Stream Processing: Apache Flink and Vera Cloud Native
40:27 Demo: Celerdata + Ververica
50:34 Q&A
50:40 When not to use Apache Flink
53:16 Does lineage work with custom Flink jobs or only with Flink SQL? If it does, how does it inspect the custom code to determine the lineage?
55:58 How do you get the first batch of data in place before turning on streaming? For example, in a MySQL database → Flink → StarRocks pipeline, what is the best method to load all historical data?
58:12 Data lake analytics is gaining interest, but it still seems too slow to fully support streaming/real-time. What does the future look like for streaming analytics as people shift towards data lakes?
59:21 From the Paimon perspective, what do you think will happen in this space, especially with Snowflake doubling down on IB and Databricks acquiring Tabular?
1:01:59 How do you make a streaming solution compatible with a transactional database like StarRocks? I understand that inserts in StarRocks generate both partitions and replicas, which could pose issues with frequent small inserts.
1:04:02 How does Flink/Ververica handle scaling (e.g., when a large volume of data reaches the initial PostgreSQL database) and resource sharing? Can I run multiple small jobs on the same cluster/VM?
---------------------------------------------------------------------------------------------------------------------
Learn more at celerdata.com/
Connect with us:
LinkedIn: www.linkedin.com/company/celerdata/
Twitter: celerdata
CelerData Website: celerdata.com/
StarRocks GitHub: github.com/StarRocks/StarRocks
StarRocks Website: www.starrocks.io/
Slack: starrocks.io/redirecting-to-slack
#DataEngineering #StreamProcessing #RealTimeAnalytics #ApacheFlink #DataAnalytics #CloudNative #ApachePaimon #BigData #StreamingData #DataPipelines #OpenSourceData #EventDrivenArchitecture #DataInfrastructure #DataStreaming #DataOps #DataInnovation #DataLineage #ModernDataStack #FlinkSQL #DataManagement #DataScalability #ETL #ChangeDataCapture #DataProcessing #DataIntegration