☕ If you liked this video and want to support my channel, please consider buying me a coffee. Your contribution helps me create more content like this: 👉 Buy Me a Coffee: www.buymeacoffee.com/andrey.fadeev 👉 Ko-fi: ko-fi.com/andreyfadeev Please also subscribe to my other resources: 👉 Substack: blog.andreyfadeev.com 👉 Telegram: t.me/andreyfadeevchannel I'm truly grateful for your support, and thank you for watching! 🙏
really great video, thanks for showing parallell ordering through key -based ordering. You can also add other processing order. Alos, this could be achieved through configuration
Thanks for the video and especially the demo! Clear one thing for me please: as you've said, with basic kafka consumer processing in case of failer we get an error loop, so we can't commit a message reading, lag is growing, users don't get updates - problem. If we use this library with "repartitioning"/ordering by exact key we still fail on the exact message, but the rest of the keys in other threads will be processed except will not be commited (as you've shown, the lag graph will be the same). After we fix the error and restart the application we still have to reprocess all the messages behind the failed one, but for the Second time? So either Processing has to be idempotent or we have to check if messages we already processed? Correct?
Hi, yeah, it's anyway a good idea to have the idempotency mechanism in the event processing (if side effects are critical in our use case). There is also a clever logic built-in the PC library, basically, it uses a bitmap to store the knowledge of processed and failed events (which is written as metadata on brokers) - but I wouldn't rely on that as the only idempotency mechanism as I've said, it's more a performance improvement (eg. consumer knows which messages were processed already, so we don't even need to check our own idempotency logic) after the restart. Check PC GitHub readme: 19.4. Offset Map for more details.
We are using it in production quite extensively, some bugs are fixed from time to time (check release notes in GitHub), but overall it just ticks all the boxes as we were looking for a reliable solution for key-based non-blocking processing. Not sure what you mean by SRE-related experience, the only shift is that some times that lag metric from Kafka doesn't actually mean that it's a lot of messages unprocessed (it just means there is a message that is failing it's blocking the offset commit) but the library is exposing metrics that could be used instead (N of messages in retry loop).
☕ If you liked this video and want to support my channel, please consider buying me a coffee. Your contribution helps me create more content like this:
👉 Buy Me a Coffee: www.buymeacoffee.com/andrey.fadeev
👉 Ko-fi: ko-fi.com/andreyfadeev
Please also subscribe to my other resources:
👉 Substack: blog.andreyfadeev.com
👉 Telegram: t.me/andreyfadeevchannel
I'm truly grateful for your support, and thank you for watching! 🙏
really great video, thanks for showing parallell ordering through key -based ordering. You can also add other processing order. Alos, this could be achieved through configuration
Thanks for the video and especially the demo!
Clear one thing for me please: as you've said, with basic kafka consumer processing in case of failer we get an error loop, so we can't commit a message reading, lag is growing, users don't get updates - problem. If we use this library with "repartitioning"/ordering by exact key we still fail on the exact message, but the rest of the keys in other threads will be processed except will not be commited (as you've shown, the lag graph will be the same).
After we fix the error and restart the application we still have to reprocess all the messages behind the failed one, but for the Second time?
So either Processing has to be idempotent or we have to check if messages we already processed? Correct?
Hi, yeah, it's anyway a good idea to have the idempotency mechanism in the event processing (if side effects are critical in our use case). There is also a clever logic built-in the PC library, basically, it uses a bitmap to store the knowledge of processed and failed events (which is written as metadata on brokers) - but I wouldn't rely on that as the only idempotency mechanism as I've said, it's more a performance improvement (eg. consumer knows which messages were processed already, so we don't even need to check our own idempotency logic) after the restart. Check PC GitHub readme: 19.4. Offset Map for more details.
@@andrey.fadeev Thanks for the answer, totally agree.
Does anyone use it on production? Can you please share SRE related experience?
We are using it in production quite extensively, some bugs are fixed from time to time (check release notes in GitHub), but overall it just ticks all the boxes as we were looking for a reliable solution for key-based non-blocking processing. Not sure what you mean by SRE-related experience, the only shift is that some times that lag metric from Kafka doesn't actually mean that it's a lot of messages unprocessed (it just means there is a message that is failing it's blocking the offset commit) but the library is exposing metrics that could be used instead (N of messages in retry loop).
Have we worked together at hh a long time ago? :)