As data engineers, we frequently need to build scalable systems working with data from a variety of sources and with various ingest rates, sizes, and formats. This talk takes an in-depth look at how Apache Kafka can be used to provide a common platform on which to build data infrastructure driving both real-time analytics as well as event-driven applications.
Using a public feed of railway data it will show how to ingest data from message queues such as ActiveMQ with Kafka Connect, as well as from static sources such as S3 and REST endpoints. We’ll then see how to use stream processing to transform the data into a form useful for streaming to analytics in tools such as Elasticsearch and Neo4j. The same data will be used to drive a real-time notifications service through Telegram.
If you’re wondering how to build your next scalable data platform, how to reconcile the impedance mismatch between stream and batch, and how to wrangle streams of data-this talk is for you!
⏱️Time codes
===========
00:00:00 Introduction
00:01:14 Trains as a source of event streams
00:01:48 Streaming ETL - overview
00:02:47 Streaming ETL with rail data - live visualisation demo
00:06:27 ksqlDB introduction and brief overview
00:07:24 Streaming ETL - ingest - detail
00:07:39 Data sources
00:08:13 Streaming data from ActiveMQ into Kafka
00:09:34 Ingesting data from Amazon S3 into Kafka
00:10:29 Ingesting data from CSV into Kafka
00:11:54 Ingest - recap
00:12:11 Streaming ETL - Transformation overview
00:12:45 Modeling rail data in an Entity-Relationship-Diagram (ERD)
00:14:01 Extracting and wrangling ActiveMQ messages in Kafka
00:15:12 Using ksqlDB to process messages from ActiveMQ
00:16:03 Splitting multiple message types out from a single topic in Kafka using ksqlDB
00:16:58 Joining events to lookup data with ksqlDB (stream-table joins)
00:18:03 Building composite keys in ksqlDB
00:18:11 CASE statement in ksqlDB for decoding values
00:18:36 Schemas in stream processing, and using ksqlDB to define and apply them
00:20:37 The role of the Schema Registry in streaming ETL
00:21:18 Transformation - recap
00:21:40 Using the transformed data
00:22:33 Kafka Connect overview
00:22:59 Streaming from Kafka to Elasticsearch
00:23:36 Kafka to RDBMS (PostgreSQL)
00:24:28 Do you *actually* need a database?
00:25:00 Building materialised views in ksqlDB
00:25:42 Kafka to S3
00:27:02 Kafka to Neo4j
00:27:36 Building real-time alerting using Kafka
00:29:33 Monitoring and Maintenance
00:30:26 Conclusion & Summary
Confluent Cloud
============
Confluent Cloud provides fully managed Apache Kafka, connectors, Schema Registry, and ksqlDB. Try it out and use code RMOFF200 for money off your bill: www.confluent.io/confluent-cloud/tryfree/?.devx_ch.rmoff_xHV1mGXV5Ds&
Resources & Links
============
📓Slides: talks.rmoff.net/6GsyFX/on-track-with-apache-kafka-building-a-streaming-etl-solution-with-rail-data
👾Demo code: rmoff.dev/kafka-trains-code-01