The Geek Narrator
The Geek Narrator
  • 171
  • 587 711
DBOS internals - Build reliable backends 10x faster
The GeekNarrator memberships can be joined here: ua-cam.com/channels/_mGuY4g0mggeUGM6V1osdA.htmljoin
Membership will get you access to member only videos, exclusive notes and monthly 1:1 with me.
Here you can see all the member only videos: ua-cam.com/play/UUMO_mGuY4g0mggeUGM6V1osdA.html
------------------------------------------------------------------------------------------------------------------------------------------------------------------
About this episode:
------------------------------------------------------------------------------------------------------------------------------------------------------------------
In this episode we are talking to Peter and Qian, co-founders of DBOS. The conversation covers the challenges of creating fault-tolerant applications, the architecture of DBOS, and how it addresses reliability at multiple layers.
Chapters:
00:00 Introduction to the Geeknerder Podcast
00:29 Meet the Co-Founders of DBOSS
01:25 The Core Problem: Building Reliable Systems
02:05 How DBOSS Solves Reliability Issues
04:29 Understanding DBOSS Architecture
06:09 Deep Dive into DBOSS Library
08:36 Postgres and State Management
18:31 Handling Parallel Steps and Performance Concerns
26:00 Observability and Version Control
30:18 Running Multiple Code Versions
30:58 Managing Workflow Versions
32:03 Surgery on Workflow States
33:15 Library Annotations and Durable Execution
34:24 Migrating to the Cloud Version
37:23 Handling Email Workflows
42:41 Transactional Guarantees with Postgres
48:44 Technical Challenges and Multi-Tenancy
54:12 Real-World Use Cases and Benefits
59:45 Conclusion and Final Thoughts
Some important links:
- Main website: www.dbos.dev/
- DBOS docs: docs.dbos.dev/
- Open-source DBOS Transact libraries:
- Python: github.com/dbos-inc/dbos-transact-py
- TypeScript: github.com/dbos-inc/dbos-transact-ts
------------------------------------------------------------------------------------------------------------------------------------------------------------------
Like building real stuff?
------------------------------------------------------------------------------------------------------------------------------------------------------------------
Try out CodeCrafters and build amazing real world systems like Redis, Kafka, Sqlite. Use the link below to signup and get 40% off on paid subscription.
app.codecrafters.io/join?via=geeknarrator
------------------------------------------------------------------------------------------------------------------------------------------------------------------
Link to other playlists. LIKE, SHARE and SUBSCRIBE
------------------------------------------------------------------------------------------------------------------------------------------------------------------
If you like this episode, please hit the like button and share it with your network.
Also please subscribe if you haven't yet.
Database internals series: ua-cam.com/video/yV_Zp0Mi3xs/v-deo.html
Popular playlists:
Realtime streaming systems: ua-cam.com/play/PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA-.html
Software Engineering: ua-cam.com/play/PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17.html
Distributed systems and databases: ua-cam.com/play/PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_d.html
Modern databases: ua-cam.com/play/PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsN.html
Stay Curios! Keep Learning!
Переглядів: 421

Відео

The JVM Internals - Basics (Episode 1)
Переглядів 1 тис.21 день тому
Join as a member to get access: ua-cam.com/channels/_mGuY4g0mggeUGM6V1osdA.htmljoin Understanding JVM Internals: Episode 1 - Basics and Key Concepts Hello everyone. In this first episode of the JVM internal series, we look into the fundamentals of the Java Virtual Machine (JVM). We'll explore what the JVM is, why it's powerful, and cover essential concepts such as platform independence, byte-co...
Database Trends and More with Peter Zaitsev @percona
Переглядів 1,4 тис.28 днів тому
Deep Dive into Databases with Peter Zaitsev | The GeekNarrator Podcast Join host Kaivalya Apte and special guest Peter Zaitsev from Percona on this episode of the Geeknerder Podcast. They discuss Peter's fascinating journey into the world of databases, founding Percona, and the evolution of open source database solutions. Topics include the rise of PostgreSQL, the comparison between MySQL and P...
World of Stream Processing with Apurva Mehta and Yingjun Wu
Переглядів 2,1 тис.Місяць тому
Understanding Stream Processing: Trends, Challenges, and Future Directions with Apoorva Mehta and Yingjun In this episode, host Kaivalya Apte discusses stream processing, its evolution, current trends and future directions, with experts Apoorva Mehta, co-founder and CEO of Responsive, and Yingjun, co-founder of Rising Wave Labs. Topics include the benefits and challenges of using Kafka Streams,...
How would you design a database on Object Storage?
Переглядів 3,5 тис.Місяць тому
Join Kaivalya Apte and Simon Hørup Eskildsen from Turbopuffer as they talk about the complexities of building a database on top of object storage. Discover the key challenges, the nuances of various storage formats, and the critical trade-offs involved. Learn from Simon's rich experience, from his time at Shopify to creating Turbopuffer. This episode covers everything-from approaches to write-a...
Practical Systems Learning & Verification with Jack Vanlightly
Переглядів 17 тис.2 місяці тому
Practical Systems Learning & Verification with Jack Vanlightly Welcome to The GeekNarrator podcast! In this episode, host Kaivalya Apte goes deeper into the practical applications of formal methods with Jack Vanlightly, a principal technologist at Confluent. With years of experience in distributed systems, Jack discusses his journey and how formal methods have been instrumental in system design...
Database Internals - NileDB Postgres re-engineered for multitenant apps
Переглядів 19 тис.2 місяці тому
Database Internals - NileDB: Postgres Re-engineered for Multitenant Apps with Gwen Shapira Join us in this episode as we dive deep into the intricacies of NileDB, a groundbreaking database re-engineered for multi-tenant applications. Our special guest, Gwen Shapira, co-founder of NileDB and a notable figure in the database field, shares her insights and technical know-how on solving the common ...
AutoMQ: A New Kafka Alternative on S3 That Addresses Your Costs and Pain Points
Переглядів 1,5 тис.2 місяці тому
In this video, we dive into AutoMQ, an interesting streaming project that reinvent Apache Kafka’s storage layer to gain many attractive features, mainly including: - 10x cost-effective than Apache Kafka: AutoScaling、Support Spot Instance、Separate Storage to S3. All this make AutoMQ 10x cost-effective than Apache Kafka. - Easy to operate: No need to manage the cluster's capacity yourself. Statel...
Building a continuous profiler with Frederic from Polar Signals
Переглядів 15 тис.2 місяці тому
Building a Continuous Profiler with Frederic from Polar Signals | Geek Narrator Podcast In this episode we chat with Frederic from Polar Signals. We dive deep into the intricacies of building a continuous profiler, the challenges faced, and the unique solutions developed by Polar Signals. Frederic shares insights from his background in observability and discusses the innovations in FrostDB, a c...
Database Internals - SlateDB with Chris Riccomini
Переглядів 14 тис.3 місяці тому
Welcome back to another episode! Today, I have a special guest, Chris Riccomini, joining me to delve into the exciting world of databases. In this episode, we focus on SlateDB, a new and innovative database that's making waves in the tech community. We'll cover a wide range of topics, including the architecture of SlateDB, its internals, design decisions, and some fascinating use cases. Chris, ...
System Design the formal way with FizzBee
Переглядів 11 тис.3 місяці тому
In this video I talk to Jayaprabhakar Kadarkarai aka JP who is the founder of FizzBee. FizzBee is a design specification language and model checker to help developers verify their design before writing even a single line of implementation code. We have discussed where it is applicable, what are the benefits, how does it work and many other interesting challenges with examples. Chapters: 00:00 I...
Learnings from building Open Source Distributed Systems with Kishore Gopalakrishna
Переглядів 29 тис.5 місяців тому
In this episode of The Geek Narrator podcast, hosted by Kaivalya Apte, we welcome a special guest, Kishore Gopalakrishna from StarTree, co-author of Apache Pinot and other notable projects. Kishore shares his extensive experience in building real-time analytics and streaming systems, including Apache Pino, Espresso, Apache Helix, and Third Eye. The episode delves into the motivations and challe...
WarpStream: A drop-in replacement for Kafka
Переглядів 27 тис.6 місяців тому
In this episode of The GeekNarrator podcast, host Kaivalya Apte interviews Ryan and Richie, the founders of WarpStream. They discuss the architecture, benefits, and core functionalities of WarpStream, a drop-in replacement for Apache Kafka. The conversation covers their experience with Kafka, the design decisions behind WarpStream, and the operational challenges it addresses. They also delve in...
XTDB - An Immutable SQL Database
Переглядів 11 тис.6 місяців тому
Exploring XTDB with Jeremy Taylor & Malcolm Sparks: An In-Depth Dive into Immutability and Database Internals In this episode of the Geek Narrator Podcast, host Kaivalya is joined by Jeremy Taylor and Malcolm Sparks from Juxt to explore XTDB, an immutable database designed to handle complex historical and financial data with precision. They delve into the architecture, internal mechanics, and u...
Turso - SQLite for production
Переглядів 17 тис.6 місяців тому
Turso - SQLite for production
Testing Distributed Systems the right way ft. Will Wilson
Переглядів 9 тис.7 місяців тому
Testing Distributed Systems the right way ft. Will Wilson
Taking Postgres to the next level with Neon
Переглядів 18 тис.7 місяців тому
Taking Postgres to the next level with Neon
Scaling Derived Data for Planet-Scale Applications at Linkedin
Переглядів 5177 місяців тому
Scaling Derived Data for Planet-Scale Applications at Linkedin
SuperCharging PostgreSQL for Search and Analytics - ParadeDB (Philippe Noël)
Переглядів 18 тис.7 місяців тому
SuperCharging PostgreSQL for Search and Analytics - ParadeDB (Philippe Noël)
Modern OLAP Database System Design with FDAP (Andrew Lamb)
Переглядів 14 тис.7 місяців тому
Modern OLAP Database System Design with FDAP (Andrew Lamb)
The ultimate multi-model Database, SurrealDB with Pratim Bhosale
Переглядів 8098 місяців тому
The ultimate multi-model Database, SurrealDB with Pratim Bhosale
Demystifying Real-time Analytics, Search and Hybrid Search with Dhruba, CTO @Rockset
Переглядів 3,7 тис.8 місяців тому
Demystifying Real-time Analytics, Search and Hybrid Search with Dhruba, CTO @Rockset
Rapidly Simulate Production Traffic ft. Michael Drogalis
Переглядів 7098 місяців тому
Rapidly Simulate Production Traffic ft. Michael Drogalis
Demystifying Stream Processing Fundamentals
Переглядів 8038 місяців тому
Demystifying Stream Processing Fundamentals
High Performance with GraalVM - Alina Yurenko
Переглядів 5 тис.8 місяців тому
High Performance with GraalVM - Alina Yurenko
Taming TimeSeries Data with QuestDB - Javier Ramirez
Переглядів 4369 місяців тому
Taming TimeSeries Data with QuestDB - Javier Ramirez
Beat the CAP Theorem : Make Distributed consistency simple
Переглядів 1,6 тис.9 місяців тому
Beat the CAP Theorem : Make Distributed consistency simple
KuzuDB: The Future of Graph Databases (w/ The Founder of KuzuDB)
Переглядів 1,1 тис.9 місяців тому
KuzuDB: The Future of Graph Databases (w/ The Founder of KuzuDB)
Restate - making distributed systems simple with Stephan Ewen
Переглядів 8039 місяців тому
Restate - making distributed systems simple with Stephan Ewen
Volt Active Data: Low Latency Stream processing
Переглядів 50710 місяців тому
Volt Active Data: Low Latency Stream processing

КОМЕНТАРІ

  • @thatguyadarsh
    @thatguyadarsh 14 днів тому

    Very detailed and to the point discussion. I am in awe as this is just what I needed. Very grateful for this discussion. Thanks for sharing!!

  • @VahidOnTheMove
    @VahidOnTheMove 15 днів тому

    How about to edit/delete a record?

  • @arun553
    @arun553 16 днів тому

    absolute heat, thanks! I'm a big fan of the highlighted subtitles fwiw

  • @RandhirKrSingh-x9l
    @RandhirKrSingh-x9l 18 днів тому

    Thank you for creating this video Kaivalya! I learned so much about DynamoDB in just 90 minutes from you and Alex. Such valuable insights!

  • @arun553
    @arun553 21 день тому

    solid, ty! The subtitles came in handy, I was pretty much relying on those when playing at faster playback speeds!

  • @hariharapamarjane2215
    @hariharapamarjane2215 24 дні тому

    How many parts will be there?

    • @TheGeekNarrator
      @TheGeekNarrator 24 дні тому

      At least 5. But will keep adding to this series. There is just so much to talk about 😀

  • @Alpheus2
    @Alpheus2 24 дні тому

    Cool format. Pleasantly surprising how much info you can convey in a fun way in under 20mins.

  • @PraveenKumarBN
    @PraveenKumarBN 24 дні тому

    This is amazing... Keep coming up with more content, especially on Graph Databases

  • @hanamufidah5809
    @hanamufidah5809 25 днів тому

    Thank you both for the vid. It's a great discussion. 30:02, Instagram allows user to edit the post dynamic content (description, caption, location, tags, etc)

  • @VaibhavPatil-rx7pc
    @VaibhavPatil-rx7pc 27 днів тому

    Awesome explaination 🎉, thanks

  • @abhishekshrivastav7941
    @abhishekshrivastav7941 Місяць тому

    He has not spent more time in industry where extremely busy IC schedule open office hours so that their work doesn't get impacted by each and everyone disturbing them.

  • @souravdhar47
    @souravdhar47 Місяць тому

    ❤ this content is just awesome

  • @mkres
    @mkres Місяць тому

    am I the only one questioning how packing 10.000 business transactions into a single DB transaction works consistency wise? :)

  • @DurgeshGautam-d2b
    @DurgeshGautam-d2b Місяць тому

    Bb

  • @harshavardhanreddykhvr
    @harshavardhanreddykhvr Місяць тому

    How is Hdfs different from the above mentioned approach?

    • @TheGeekNarrator
      @TheGeekNarrator Місяць тому

      HDFS is a file system while we talked about object storage. File system is hierarchical while object storage is flat, which allows for infinite storage and scalability. File system on the other hand has limitations in that regards. File system might be better in terms of speed and performance but again depends on the total size etc. so yes the approach we discussed is purely on top of cloud object storage.

  • @KavirajKanagaraj
    @KavirajKanagaraj Місяць тому

    What is the rust library Simon talked about building flexible storage formats? I heard something like "archive". But failing to find the right one.

  • @manish-mk
    @manish-mk Місяць тому

    These podcasts are goldmine! Also I watched your only livestream today while I was in metro. Got many insights. You should do more livestreams.

    • @TheGeekNarrator
      @TheGeekNarrator Місяць тому

      Thanks 🙏🏻 I will make a note for livestreams.

  • @Md_sadiq_Md
    @Md_sadiq_Md Місяць тому

    Pushing the algorithm ❤

  • @himanshupandey9303
    @himanshupandey9303 Місяць тому

    Thanks for updating the video. Great content. Keep it up.

  • @AmeliaArdath
    @AmeliaArdath Місяць тому

    Hi! So, I just integrated DuckDB into my little open source distributed trace viewer and came here to learn more about its internals, but I have to stop the talk for a second and thank you profusely for having clear, amazing subtitles, right in the video. As a programmer with profound hearing loss, it's usually a struggle for me to get through technical videos with the jumpy auto-generated captions UA-cam provides, and it so rare that I get to just sit back and enjoy a talk like a hearing person would. Thank you for providing this experience!

    • @TheGeekNarrator
      @TheGeekNarrator Місяць тому

      This comment has made my day. Thanks a lot for sharing that. Unfortunately I got a lot of comments asking me to remove those subtitles as they were distracting so I had to remove them from other videos. Now I am motivated to find a way to make it work for both use cases.

  • @HimanshuKanodia
    @HimanshuKanodia 2 місяці тому

    👍

  • @HimanshuKanodia
    @HimanshuKanodia 2 місяці тому

    👍

  • @HimanshuKanodia
    @HimanshuKanodia 2 місяці тому

    👍

  • @HimanshuKanodia
    @HimanshuKanodia 2 місяці тому

    I don't build product directly, but in kind of integration work, but honestly even if I try to think from user perspective, I don't see people around me, in teams, willing to work like that or let others do. 🤣🤣

  • @HimanshuKanodia
    @HimanshuKanodia 2 місяці тому

    Nice podcast. Today only I started listening your podcast.

  • @VipulVaibhaw
    @VipulVaibhaw 2 місяці тому

    This was awesome!

  • @bbowjazz
    @bbowjazz 2 місяці тому

    Heikki rocks!

  • @aneksingh4496
    @aneksingh4496 2 місяці тому

    Your podcasts and the way you dig down to detailed level is revolutionary...please keep making such insightful videos ...this has enhanced my knowledge many folds .. let's catch up if you want

  • @anthonya880
    @anthonya880 2 місяці тому

    Please remove members only videos. It doesn't make sense yet for your channel.

    • @TheGeekNarrator
      @TheGeekNarrator 2 місяці тому

      Thanks for the feedback. I am happy you added “yet”, which tells you believe 😀🙏🏻

    • @anthonya880
      @anthonya880 28 днів тому

      @@TheGeekNarrator Yes your channel is different. It has lot of potential.

  • @RokijulIslam-sk4dt
    @RokijulIslam-sk4dt 2 місяці тому

    😅o

  • @D9ID9I
    @D9ID9I 2 місяці тому

    The proper way is to rewrite using native languages like C++ or Rust. Use of java again is a plain joke.

    • @kaimingwan526
      @kaimingwan526 2 місяці тому

      Language has never been an issue with Kafka. Embracing the Kafka ecosystem and being fully compatible with Kafka is an important trade-off.

    • @D9ID9I
      @D9ID9I 2 місяці тому

      @@kaimingwan526 Efficiency is the issue. It translates to number of resources you have to pay for monthly. That's why Redpanda exists.

  • @costathoughts
    @costathoughts 2 місяці тому

    Thank you Kaivaly for the video, it was really well explained and the animations are amazing. I would like to point just one thing the title "Reinventing Kafka the right way" I think is not precise once the part that AutoMQ are improving is the Storage Engine system and not the Message Middleware Broker (if I understood correctly based on this video). It was more an opportunity to enhance an isolated component from Kafka for cloud storage instead a full rewrite. An example of "Reinventing the right way" in my point of view would be the ScyllaDB which has backwards compatibility with Cassandra although it was rewritten using C++. I wish the best brow keep going!!!! EDIT 1: And they created a platform over na existent Kafka concept

    • @TheGeekNarrator
      @TheGeekNarrator 2 місяці тому

      Thanks for watching and the feedback. I appreciate it. I think you made a fair point, I guess re-engineered would have been a more precise word. I will change that.

  • @ShortGiant1
    @ShortGiant1 2 місяці тому

    Great video, learnt a lot!

  • @nosh3019
    @nosh3019 2 місяці тому

    Thanks for this interview. Great as always! One question: so if the WAL writes are batched and periodically flushed to storage, there in a window in which if the process crashes, the writes will be lost? I.e the client response that write is complete is not durable.

    • @TheGeekNarrator
      @TheGeekNarrator 2 місяці тому

      Thanks for watching. Yes, there are ways to configure the behaviour. This might help you slatedb.io/docs/faq/#what-happens-if-the-process-goes-down-before-slatedb-flushes-data-to-object-storage

    • @ShortGiant1
      @ShortGiant1 2 місяці тому

      Yeah that’s the “linger” stuff they spoke about. The client won’t get an ack about the write being successful until the flush happens. Therefore it would be as if the write request never happened (in practice the client would retry)

  • @harshrai456
    @harshrai456 2 місяці тому

    The storage architecture reminds me of AWS RDS (Aurora) Architecture with the control plane managing a muli tenant storage fleet and having write quoram for strong consistency. Great discussion, loved it.

  • @Jasin-p6w
    @Jasin-p6w 3 місяці тому

    Wonderful video

  • @jmitesh01
    @jmitesh01 3 місяці тому

    Information dense conversation! And isn't it expected that size is sufficiently large then it would increase number of put calls to object storage. Why don't you do buffering based on time plus size based criteria?

    • @TheGeekNarrator
      @TheGeekNarrator 3 місяці тому

      Good question. I think there are plans but haven’t really finalised yet. But maybe in the future a hybrid setting will be available. IIRC I asked this question to Chris.

  • @thrawn01
    @thrawn01 3 місяці тому

    Thank you for this, looking forward to assisting with the golang port! This was a great overview of the project, and a great way to get acquainted with some of the design decisions.

  • @plargyle
    @plargyle 3 місяці тому

    I love how you simplify complex topics in the crypto space.

  • @MarkHarrison-g7r
    @MarkHarrison-g7r 3 місяці тому

    Like all duckdb users, I really enjoy hearing Mark Raasveldt talk. You did a great job of guiding the conversation into intesting and informative areas!

    • @TheGeekNarrator
      @TheGeekNarrator 3 місяці тому

      Thanks a lot watching and sharing your thoughts 🙏🏻😀

  • @arkadutta744
    @arkadutta744 3 місяці тому

    This is super cool . Probably the only youtube video [> 1hr ] , which I watched in whole. I took a whole week to watch it ... made sure I understood every idea presented and discussed. Thank you so much. I have a question => The partition placement , is DDB still using consistent hashing , for deciding which partition goes to which node[physical server] ... or there is some other algorithm used now. The original Dynamo paper mentioned consistent hashing [I read half the paper] ... Precisely => 1. How is it decided which key goes to which partition ? 2. How is it decided which partition goes to which node[physical server] ? Consistent hashing ... works in a bit different way ... all of the people watching this video knows how . For example in normal consistent hashing I guess it is hard to keep a limit on partition size ... I have read some part of the original consistent hashing paper ... in my quest to understand that paper ... somehow I reached your video .. and I thank myself for that.

  • @imranfp
    @imranfp 3 місяці тому

    Well presented and explained well. The best part when Matthias use pointer to exactly mention what part in presentation is under discussion. Thanks 👏

  • @AFRINKHATOON-m1z
    @AFRINKHATOON-m1z 3 місяці тому