Introduction to Stateful Stream Processing with Apache Flink • Robert Metzger • GOTO 2019
Вставка
- Опубліковано 28 вер 2024
- This presentation was recorded at GOTO Amsterdam 2019. #GOTOcon #GOTOams
gotoams.nl
Robert Metzger - PMC member of the Apache Flink project; Co-founder, Engineering Lead Ververica
ABSTRACT
Stream Processing has evolved quickly in a short time: only a few years ago, stream processing was mostly simple real-time aggregations with limited throughput and consistency. Today, many stream processing applications have sophisticated business logic, strict correctness guarantees, high performance, low latency, and maintain terabytes of state without databases. Stream processing frameworks also abstract a lot of the low-level details away, such as routing the data streams, taking care of concurrent executions, and handling various failure scenarios while ensuring correctness.
This talk will give an introduction into Apache Flink, one of the most advanced open source stream processors that powers applications in Netflix, Uber, and Alibaba among others. In particular, we will go through the use cases that Flink was designed for, explain concepts like stateful [...]
Download slides and read the full abstract here:
gotoams.nl/201...
/ gotoamst
/ goto-
/ gotoconference
#ApacheFlink #EventStreaming #DataProcessing #StreamProcessing
Looking for a unique learning experience?
Attend the next GOTO Conference near you! Get your ticket at gotocon.com
SUBSCRIBE TO OUR CHANNEL - new videos posted almost daily.
www.youtube.co...
Host's PPT slides were so vividly to illustrate the concept of Flink and its application. Its Awesome !
Robert Metzger has amazing teaching ability!...great talk
This was an amazing talk
“These notions of time mostly affect the latency…… and the correctness”
Veverica means squirrel in Slovenian... wondering if name Ververica has anything to do with that or is it just a coincidence?
No, this is not a coincidence ;)
Amazing talk!
ammm... this was a wonderful talk :-)
Wonderful talk
What does he mean by "Queries change faster than data"?
Select a,b from A; Select c,d from A and so on... here the query changes but its on the same table A which is the data
Queries change faster than data: You have a large dataset, and you are exploring it by running a lot of different queries on it.
Data changes faster than your queries: You have decided on a few queries that you want to run continuously (they are not changing frequently), but you have a lot of new incoming data.
in other words, Queries change faster than data == "heavy reads, low writes"
Good use of 31 min.
Amazing talk, learnt a ton. Moreover picked up some speaker notions, kudos Robert!
Single's Day :-)
I'm writing a streaming application with Flink and getting this error from Flink - org.apache.flink.runtime.io.network.exception.RemoteTransportException : Connection Unexpectedly closed by remote task manager xx1.xx2.xx3.xx4:abcd. This might indicate the that the remote task manager was lost.
The configurations for Flink are as follows
flink.parallelism.default = 60
flink.taskmanager.numberOfTaskSlots = 3
flink.taskmanager.memory.flink.size = 1G
flink.jobmanager.memory.jvm-metaspace.size = 512mb
flink.taskmanager.memory.jvm-metaspace.size = 2048mb
Could anyone please help me what is the cause for the error and how to fix it.
You need to check the logs of the TaskManager.
A default parallelism of 60 on 3 task slots will only work if you have 20 TaskManagers.