Kafka in 100 Seconds
Вставка
- Опубліковано 30 вер 2024
- Apache Kafka is a distributed event streaming platform used to handle large amounts of realtime data. Learn the basics of Kafka in this quickstart tutorial.
#programming #datascience #100SecondsOfCode
💬 Chat with Me on Discord
/ discord
🔗 Resources
- Kafka Homepage kafka.apache.org/
- Kafka Github github.com/apa...
- RabbitMQ in 100 Seconds • RabbitMQ in 100 Seconds
🔥 Get More Content - Upgrade to PRO
Upgrade at fireship.io/pro
Use code YT25 for 25% off PRO access
🎨 My Editor Settings
- Atom One Dark
- vscode-icons
- Fira Code Font
🔖 Topics Covered
- What is Apache Kafka?
- Who created Apache Kafka?
- What is Kafka used for?
- How do large apps handle streaming data?
- Apache Kafka basic examples
The thumbnail is unexplainably aesthetic, please dont stop this series , always learning something new from these
has the aethetic of the japanese prefecture flags
@@SpektralJoOsaka++
@@2ku4 yeah it looks very similar to osakas symbol
That’s the Kafka logo?
Aesthetic is not an adjective
hey this is not the kafka I am searching for, but thank you fireship
bro’s looking for stellaron hunters☠️
Dude slow down. You lost me at 00:01
Hey man, love your videos, but a small correction for what you mention at 1:45
"Kafka guarantees that any consumer of a given topic will always read the event in the exact same order"
That's true but at partition level. A topic is made of partitions, and a topic with 1 partition will guarantee that. But as soon as you need parallel consumption you need more partitions (one per concurrent consumer). When having multiple partitions, a single, non-concurrent consumer will get the events of any partition in the proper order but events from different partitions may be consumed with different interleaving
This is why it's so important to put a proper key for your messages, as it's the default value used for sending your message to different partitions (different events with the same key will always end up in the same partition)
Again, love your content, you explain super great. But I thought that that small point had to be addressed
😄
Yes that's an important detail to understand: order is guaranteed across one partition and all messages sharing a key are stored on the same partition, hence order is guaranteed for all messages sharing a key.
But order isn't guaranteed on a whole topic.
Thanks for the clarification, that's an important detail.
@@Fireship pin the comment my man
"small correction"
@@lonkhoi6764 humble man
Totally not here while searching for kafka Honkai star rail..😂😂😂
here i come
IM HERE
@@robloxislife318 🙂
me searching for Arknights's Kafka.
Same
Everyone is gangsta until you have to handle reset flow
Reset flow? What is that
@@HelloWorld-fg2nm when there is a failed process in Kafka, ideally you have to rollback every process that has been succeded before. Just like rolling back query in ORM. That is why you need separate flow for reset
Im sorry if you are confused by what im saying. Im still learning english
@@farhansangaji5029you can have a windowing topic for these cases and do reconciliation
so true!
There's a different Kafka that's even better 👀
*sigh* HSR fans are everywhere
Good time to be a coder and a Honkai star rail fan.
Lmao fr
let's goooo
HERE'S TO EVERYONE WINNING THEIR 50/50 RAHHHHHHHHH 🗣‼🔥💯
Please make comprehensive videos on topic like Kafka, I think 100s videos are good but they are getting boring now
I'd love to see a quick into to Event Sourcing on this channel!
That would be useful.
I'd love that.
Kafka is perhaps the best options for it. You just store the message forever and ur done. I recommend book Designing Event-Driven Systems by confluent CTO. It's awesome except it's a bit too bias for Kafka...tho nothing like Kafka so u can't blame the man.
Do your own research mate if you really want it.
I would like to see it with EventStoreDB
This series never fails to teach me about things I didn't know I needed to know about.
I'd love a 100 seconds on a CI tool like Jenkins, your videos are always so informative :)
Leeroy Jenkings
What do you get from searching Kafka?
Honkai kafka
Franz kafka
Kafkaqesue
Apache kafka
I have a theory about why Kafka the messaging system is named Kafka after Franz Kafka the writer.
'Das Schloss'(The Castle) is a Novel written by Franz Kafka in which he describes a castle having occupants structured as the ultimate bureaucracy.
Members of the Castle does not know each other and communicate via a paper based messaging system.
You don't want to deal with such a bureaucracy, but you certainly would want to design large complex systems like so: Independent services not knowing about each other communicationg via an asynchronous messaging system. I think that is why Kafka is named Kafka, although just a personal theory...
That particular writer was known for writing stories about enigmatic, baffling events where the characters have no idea what’s going on. Like one about a trial where the person on trial is not even told what the charges are.
And that's what I like about Kafka , you can literally read your own meaning to it, here a team of developers have used his stories to create a great service.
Pretentious nerds.
That's a way better explanation that "it's a system optimized for writing" especially considering Kafkas writing process beeing so chaotic and unorganized that he himself did not see the any meaning in his stories
I thought this was about the author. :(
Thought this was meme compilation about Kafka from Honkai Star Rail for a sec
I always learn something new from these vids they're truly great
Then let's hope he does a video on "they're vs their"
@@Djulio vs vs. vs.
@@Djulio why you so smart
I love honkai star rail 😨
You should make a "Qt in 100 seconds" :3
its a very very popular native Desktop Development framework that can compile for nearly any platform and is written in C++
Many of your production software like earlier iterations of Photo Shop or After Effects, currenr Davinci Resolve, OBS, Krita and many more are written with Qt
and still holds strong as one of the best native GUI solutions (native, aka. no web related stuff)
Qt is also supported in python. There is also QML (QT markup lanugage I guess) which also supports running javascript.
@@mkhuzaima But if u are going to use JS for any desktop application, html is wayy better as it is open standard. Also, you can use webview and write ur gui in HTML CSS JS and use C++, Java, Python or any other favorite language for the main logic of the app.
As someone who worked with pyQt I agree
I doubt that any Adobe product (PS..) is made with QT - no Linux version.
I used to be a QT fanboy, but not anymore. Licensing, enough said. And seemingly lagging behind on technology (C++20.. ) and tooling (QTcreator needed).
Adobe uses their own internal tool
2:20 in java, since you’re implementing single-method interfaces you can just make it a lambda and can omit the argument types
This is why you use IntelliJ for Java!
help i thought that you meant honkai: starrail kafka
As someone who works for Confluent and has worked on Kafka for some time now, I absolutely love your video. 🔥
Glad you love the content ! I'll like to introduce you to something new............👆👆✍️✍️
Kafka rerun in 2.0❤❤🔥🔥🔥 i wanna get her lc
Trueeeee, definitely pulling for her now that I have the chance to do so (lost pity to yanKING)
Fireship in 100 Seconds
Do you actually know Java? Because it would explain the lack of sanity
well, time to say bye.
boom.
As someone who uses Kafka at work, it is so shit and I wouldnt wish it on any dev
Was expecting a philosophical analysis
Imagine a Kafka topic is like a multi-lane highway, where each lane (partition) has cars (messages) driving in a single direction.
*Order Guarantee:*
Within each lane (partition), cars (messages) drive one after the other in a strict sequence.
If you're observing just one lane, you'll always see cars in the exact order they entered that lane.
*Multiple Lanes (Partitions):*
If you're watching the whole highway (the entire topic with multiple partitions), cars from different lanes might cross you at slightly different times.
So, while cars within each lane are in order, across lanes, they might appear mixed up.
*Why Lanes (Partitions) Matter:*
More lanes mean more cars can drive simultaneously, allowing for faster traffic flow.
In Kafka terms, this means more consumers can read messages concurrently, leading to faster data processing.
*Choosing the Right Lane with a Key:*
When a car (message) enters the highway (topic), it needs to pick a lane (partition). This choice is based on the car's license plate number (message key).
Cars with the same license plate number will always choose the same lane. In Kafka, messages with the same key always go to the same partition, ensuring they're read in order.
In summary, while Kafka keeps the order of messages within each partition, when you have multiple partitions, the order of messages across them can be mixed. Choosing the right key for your messages ensures they land in the expected partition.
Not sure about the developer but it's a good channel for a Product manager who can look at multiple techs from the top without going deep i to weeds
Glad you love the content ! I'll like to introduce you to something new............👆👆✍️✍️
not the kafka i was hoping for
mood
Great video! I would love to see a video of "Scala in 100 seconds" :)
Let's discuss about something bigger in this 2023✍️✍️👆♥️........ happy new year✍️✍️
Yeeesss
Please, do a video in beyond fireship about kafka :)
Just to you know, nice job
can u do AutoHotkey in 100 seconds? i think that would be cool. the release of v2 changes a lot of the syntax from v1.1
Automating a GUI is a recipe for despair.
One morning, when Gregor Samsa woke from troubled dreams, he found himself transformed in his bed into a horrible vermin.
What’s more is you can use consumer groups to split the workload if you have massive data to compute. And what is more is you can use different strategies to reassign partitions to new workers to increase ingestion throughput
I didn't think I'd be interested in Kafka but I'm glad I watched. You're like a tech magazine, broadening my horizon.
👆Glad you love the content ! I'll like to introduce you to something new............👆👆✍️✍️❤️
The statement about guaranteed order of reading the messages from a topic is misleading at best, as it only applies only to single partition topics. And topic partitioning is a big part of what makes kafka scalable.
yes , within a partition , the message ordering is guaranteed , that's why , we have to choose key for the message wisely , like to make sure all transactions from a user is going to same partition & maintain order , we should choose user id as partition key ... that way with partitioned topic , the parallelism & ordering both can be maintained ...
I swear your worse than Alexa. I was just talking to a buddy yesterday about kafka and here's the video... 🤣🤣Great video like always!!!!
You read my mind, I’ve been hoping you’d cover Kafka for months now!
Pretty sure more people are making new web libraries than actually building websites that people actually use at this point.
I’m honestly tired with all these frameworks
This is not framework
1:09 Rest in Peace party guy from that Radio Disney song from my childhood.
In an Iron Man movie, Kafka could have been useful in a scene where Tony Stark (Iron Man) needs to process and manage a large volume of real-time data or communications. Here's a hypothetical scenario where Kafka could play a role:
Scene: Tony Stark is in his high-tech lab, and he's remotely controlling his Iron Man suit, which is deployed in a distant location to handle a crisis. He needs to receive and process real-time data from various sensors on the suit, such as vital signs, telemetry data, and external environmental data, while also receiving live video feeds.
How Kafka could be useful:
1. **Real-time Data Ingestion**: Kafka could be used to ingest data from these sensors and video feeds in real-time. Each type of data (vital signs, telemetry, video) could be treated as a separate Kafka topic.
2. **Data Processing**: Tony needs to process this data for real-time decision-making. Kafka Streams, a component of Kafka, could be used to perform real-time data processing, such as analyzing vital signs for signs of distress, stabilizing the suit's functions, and identifying threats in the video feed.
3. **Reliability**: In a high-stakes situation like this, Kafka's reliability ensures that no data is lost. If there are network interruptions or delays, Kafka can buffer and replay messages, ensuring that Tony has access to all the critical data.
4. **Scalability**: If the crisis intensifies and more data needs to be processed, Kafka can scale horizontally by adding more Kafka brokers, allowing Tony to handle the increased data flow without performance issues.
5. **Monitoring**: Kafka provides extensive monitoring capabilities, which could be depicted in the movie as Tony monitoring the health of the data pipeline in real-time, ensuring that he has a clear view of the suit's status.
In this scenario, Kafka would enable Tony Stark to efficiently manage and respond to real-time data, enhancing his ability to control the Iron Man suit and handle the crisis effectively. It would add a layer of realism to the technological aspects of the movie.
Request for GTK(specifically GTK4 and libadwaita). Its really hard to understand for absolute beginners. And also how to read its documentation and how the documentation are relevant for PyGTK, or Relm or GTK-rs. For example, its really confusing for devs who started by learning web technologies first and also its quite different from XAML based winapps. So it will be really be helpful if you guide us.
Well, it is hard to make a tutorial for GTK because you can code it in C, Python, Rust, etc, there are many language bindings. Also there are two ways to create GTK user interfaces, either programmatically by calling functions/classes, or decoratively in a XML file.
Agree, a Python GTK tutorial would be good
Iced is a far easier gui library to use with Rust. It may not be as full fledged, but the way it's designed is easier to understand, even without a tutorial vs gtk4rs with a tutorial.
these aren't tutorials though, they're just an overview. You are not really understanding anything from the video unless you already know the technology beforehand, just getting why it exists
@@javierflores09 I think he has made tutorials before, 100s is just one of the series on the channel
I always thought it was called Kakfa because modern web development is a bewildering and occasionally infuriating jumble of nonsense
Got reminded of this item from _The Onion_ : “Prague's Kafka International Named Most Alienating Airport”.
👆Glad you love the content ! I'll like to introduce you to something new............👆👆✍️✍️❤️❤️
Also come here from the wrong Kafka. but great re-cap video after all.
It also used with frameworks as Apache Flink or Apache Spark combined together for distributed event system such as streaming , application to alert ⚠ on some event and now with many machine learning Apps. Great explanation consise love u're 100s @fireship
Glad you love the content ! I'll like to introduce you to something new............👆👆✍️✍️
Wow. I started working on Kafka and had no clue what it was. After months got a basic idea but this video makes it so helpful
I've watched this before doing the quickstart on apache kafka website, to have a general idea of what it was, and ended up with a really abstract idea of it. Then after doing the quickstart, and playing with it a little bit, I re watched this video and now it felt like the perfect summary, all became crystal clear!
I'll encourage anyone who feels like it is too abstract or is bit confused to follow similar steps :)
So kafka has more throughput than RabbitMQ? Thought it was the other way
RabbitMQ is best for transactional data, such as order formation and placement, and user requests. Kafka works best with operational data like process operations, auditing and logging statistics, and system activity. Both are the same speed if RabbitMQ is mostly empty. RabbitMQ that keeps events around will take far more resources. But you probably don't need Kafka. Most people don't.
Glad you love the content ! I'll like to introduce you to something new............👆👆✍️✍️❤️
@@JTWebMan The company that I am working for is using kafka with Spring Cloud Stream already. But I am always confused between the choice of RabbitMQ, Kafka, and maybe other similar tech(ActiveMQ, JMS, IBM MQ?, AWS SQS, AWS SES etc)
My dear Jeff, we wait prolog in 100 seconds, we want it NOW.
To use Kafka do we have to learn Java? If you only know Python or Go do you have to use like Azure event hubs instead?
No Emo Nymph, to use Kafka ,Java is not compulsory , you can use Python or other programming languages also ..
But why.?!
@Fireship I think topics like this should have a comparison on the why to use this VERSUS a simpler concept say a SQL database.
Kafka good for data immutability and distributed scaling?, SQL for data integrity and little data?
So SQL good for a single user todo app, Kafka for a collab project-mgmt with real time updates on todo completion...
_Stupid hard to tl;dr something like this_.
If you don't know what "Message Queueing for real-time distributed streaming data pipelines.. " means then even helpful introductory vids like this sound like the video for the Rockwell Retro Encabulator, or describing a monad. Good for inoculation though.
As we get deeper into development concepts and specialist niche software the WHY gets lost easily and dogma enters; such as assuming something like kafka can outright replace a database (aka "Streams Everywhere" makes everything look like a nail).
Along with the worse dev sin: Docs written for those who know so if you know you know thus the docs are pointless, if you don't know the docs make no sense.
Ight Imma Head Out To The Mountains
But why? ua-cam.com/video/zWH_9VRWn8Y/v-deo.html
Glad you love the content ! I'll like to introduce you to something new............👆👆✍️✍️
Please make a Kafka for Haters video where you mention people overusing Kafka like using it as a relational database replacement. Would be funny to show at my work. I feel like people start using Kafka for more than they should just because it's easy to put up a streams applications and do some filters. Not even a small company, bmw. And they have different pods calling each other to know who has what information because the data is partitioned across pods according to the Kafka partitions
Suggestion: OpenSSL in 100 seconds
Can we get a longer like 10 min long video, that explains it more.
Let's discuss about something bigger in this 2023✍️✍️👆♥️........ happy new year✍️✍️
First! Also as my request, please do Love2D in 100 seconds next!
Umm FYI you don't need zookeeper anymore? Also another FYI no one needs Kafka stop. Your startup doesn't need google scale. Unless you are already making 100's of millions of dollars please don't use Kafka!
Glad you love the content ! I'll like to introduce you to something new............👆👆✍️✍️
Hadoop in 100 seconds
excellent timing! Had kafka come up in a meeting at work last week and didn’t know anything about it.
As Fireship awoke one morning from uneasy dreams he found himself transformed in his bed into a gigantic insect
nice
Thanks for info! Can you please do a full video to teach js ?
Let's discuss about something bigger in this 2023✍️✍️👆♥️........ happy new year✍️✍️
this video was very Kafka-esque
how about a comparison between apache kafka and redpanda data?
I love how he didn't use lambdas just to make java appear worst (as if java doesn't have bad syntax already)
Can you make a video about Windows driver development? Beeing able to write programs to execute in the kernel sounds very powerful but there's not realy mouch documentation exept of Microsoft's own. I found this topic quite hard to get started with. A fireship video would be amazing.
Linux is easier.
1:42 are you my mummy?
Mommy Kafka is a nice villainous girl.
The Brazilian Central Bank uses Kafka in its PIX system, that makes money transactions between individuals or entities in real time between bank accounts in any Brazilian bank. If anything, that show how powerful Kafka can be (2 billion transactions per month, with almost R$1 billion - which is roughly US$200 million - moved between accounts in that period).
FYI most databases can handle 2 billion transactions a month so not sure that is a win. Good use case though for immutability.
Glad you love the content ! I'll like to introduce you to something new............👆👆✍️✍️
@@JTWebMan for a financial application nothing is ever that simple, specially since it has become such a key service in day-to-day use by the general population. But I do understand your point
twitter handles like 2 billion transactions per day bro, kafka is more powerful than you think, lol
Difference mqtt?
Day 1 Billion of asking what's the name of the background song 🤒😶🌫️
Been waiting for this video for a while maybe do couple deep dives on kafka?
Glad you love the content ! I'll like to introduce you to something new............👆👆✍️✍️
NATS is another great tool that does sort of the same thing as Kafka. it's JetStream module is amazing!
currently doing some r&d on nats for my company and so far it looks way less complicated than kafka
@@RedditFam check out NSQ as well, it's slightly different but also written in Go and a great option.
watching this channel has made me think creating languages is just a walk in the park lol
please make video on node server less Frame work
Sadly, no mention about partitions, keys, and group_ids
Let's discuss about something bigger in this 2023✍️✍️👆♥️........ happy new year✍️✍️
I just started at a new position and needed to understand Kafka. Wow, Fireship keeps reading my mind
👆Glad you love the content ! I'll like to introduce you to something new............👆👆✍️✍️❤️
If you want a faster drop in replacement for Kafka, then try Redpanda.
What does red panda has, that kafka does not ?
@@kukla-mukla6000 no zookeeper, no jvm, its 10x faster, etc
@@kukla-mukla6000 custom data transformation built in directly using WASM
@@kukla-mukla6000 instead of asking, why don't you just look it up? rofl
@@uziboozy4540 I looked up the claims of 10x faster and it's panda devs who make those claims. So I dunno.
What problems does "no zookeeper" solves ?
Less ram consumption is great if that is true.
Not sure about WASM either, looks like something that can slow down the overall perfomance of panda cluster.
I also read that panda has its own problems handling transactions...
For me panda stayed where it was - a kafka wanna be.
Nice! Please do Apache Spark next!
Let's discuss about something bigger in this 2023✍️✍️👆♥️........ happy new year✍️✍️
1:09 aaron carter came out of the blue
Glad you love the content ! I'll like to introduce you to something new............👆👆✍️✍️
I worry that it's harmful to suggest running ZooKeeper in CURRENT_YEAR.
Specifying CLUSTER_ID and NODE_ID instead would be better.
This feels like ROS, which could be an interesting 100 seconds video.
👆Glad you love the content ! I'll like to introduce you to something new............👆👆✍️✍️❤️❤️
Last time I was this early youtube had a different icon!!!
Understood nothing... I didn't get an overview of what is the problem that kafka is trying to solve.
It makes a lot of sense if you have a distributed system with services that need to process pieces of data that could be described as events or messages. If there are a lot of messages coming in and they have to be handled in some way then the source of the messages can just throw them into kafka without worrying about them further. Then you can have your service that consumes those and handles them however it needs to. If your consumer can't handle the amount of messages as quickly as they arrive then you can just deploy more instances of the consumer. Hope that was a good example.
@@YTNoobProductions Thanks a lot for this explanation ❤️
but what's the point of events? what need isbeing metbybeingable to reliably stream sequential data to multiple parties? can't this be done much more simply? or is scalability really that important?
Scale is important for big huge companies. Most system and people do not need Kafka!
Glad you love the content ! I'll like to introduce you to something new............👆👆✍️✍️
The short video idea is straight genius.
The default start script being a .sh feels super weird to me
In Hyperledger Fabric, which I use for our Thesis, it uses Kafka for consensus. Like, leader election and storing of data.
Let's discuss about something bigger in this 2023✍️✍️👆♥️........ happy new year✍️✍️
unreal engine in 100 seconds
👇
Is anyone using a Kafka jdbc connector successfully, I am always getting errors.
0:10
yesss, i was hoping that you will make this reference, and you did! yaay
also great video as always
👆Glad you love the content ! I'll like to introduce you to something new............👆👆✍️✍️❤️❤️
Do a video on Apache Thrift someday.
Glad you love the content ! I'll like to introduce you to something new............👆👆✍️✍️
Ordering is guaranteed only per Topic Partition.
Glad you love the content ! I'll like to introduce you to something new............👆👆✍️✍️
An extra detail. RabbitMQ has launched a Streams last year, improving massively its throughput
i was the crazy developer about 3 years ago for wanting to introduce Kafka in a project which was having load problems, great now Kafka is famous.
Please do a 100s on Kubernetes!
I think its called kafka because of the bug.
👆Glad you love the content ! I'll like to introduce you to something new............👆👆✍️✍️❤️
what is the app name that you use to start kafka and zookeper?
Glad you love the content ! I'll like to introduce you to something new............👆👆✍️✍️
I thought this was the algorithm
Let's discuss about something bigger in this 2023✍️✍️👆♥️........ happy new year✍️✍️