Solving distributed systems challenges in Rust

Поділитися
Вставка
  • Опубліковано 24 лис 2024

КОМЕНТАРІ • 122

  • @АртемЖмака-к2л
    @АртемЖмака-к2л Рік тому +129

    Still the best Rust related stuff! Really appreciate what you are doing!

  • @TimeConvolution
    @TimeConvolution Рік тому +58

    I hope you do more of these, especially with diagramming. It really helped to visualize the problems you were solving.

  • @rcjara
    @rcjara Рік тому +36

    Your content has always been fantastic, but this was my favorite video and the one I got the most out of. I think it’s because of the clear and digestibly sized problems. I don’t feel like I need to focus for a whole 3 hour block of time to fully get everything out of it. I can start and stop the video at my leisure. It also gives me a clear project I can do if I feel I want to practice what was just learned. Similar vibes to the time you worked through the rust macro book. Obviously, make the content you want to make, but you’ve definitely left me wanting more in this genre.

  • @ozturkberkayy
    @ozturkberkayy Рік тому +81

    Please do more distributed system videos like consensus algorithms, distributed databases etc

    • @ErickBuildsStuff
      @ErickBuildsStuff Рік тому +7

      I would second this motion. Consensus algorithms in rust. Also Byzantine Fault Tolerant State Machine Replication

    • @nceevij
      @nceevij Рік тому +1

      I too want the same

    • @daphenomenalz4100
      @daphenomenalz4100 6 місяців тому

      Yeah, he even has a phd in distributed systems, pls do more of it 🙏

  • @jRsqILVOY
    @jRsqILVOY Рік тому +18

    It'd be great to see more videos on distributed systems - like implementing RAFT and other consensus algorithms, query routing, distributed hash tables, distributed storage, etc.

  • @sugasheeze
    @sugasheeze Рік тому +28

    I'd be extremely interested to see how someone like you--with your literal doctorate in distributed systems--would build up a similar challenge.

  • @MegaPacoquinha
    @MegaPacoquinha Рік тому +4

    Now I have something to watch through easter! Ty Jon

  • @polares8187
    @polares8187 Рік тому +4

    It was so satisfying to see you pass the 3c without changing anything.

  • @BertPdeboy
    @BertPdeboy Рік тому +4

    Funny how enthusiastic you are about "Rustengan" :D
    I APPROVE!

  • @jitcorn1103
    @jitcorn1103 Рік тому +6

    I always learn many new things from your videos. Thank you so much!

  • @bilyzhuang9242
    @bilyzhuang9242 Рік тому

    Appreciate your rust coding video. I am a beginner for the rust and I really learn really practical rust knowledge by following your coding line by line. Thank you !!! Wish you a good life.

  • @htr-g8r
    @htr-g8r Рік тому

    Hey! I remember Jepsen for their Consistency Models page! It was really helpful to get me started on distributed systems and understanding data consistency, along with potentially building hand-wavy reasoning/logical system around it! Happy to see fly-io take the author far far away onto actual courses for distributed systems!

  • @vishalsharma-bp9zu
    @vishalsharma-bp9zu Рік тому +2

    Thanks for the stuff, high high quality, not sure what the 20 dislikers were expecting from this video.

  • @Lockeinfurry
    @Lockeinfurry Рік тому +6

    Great video as always! I think you are still seeing big Gossips in the end because you are making an assumption that all Gossiping happens bi-directionnally. If n1 Gossips to n2 but n2 never gossips to n1, n1 will always want to send the full thing as it doesn't knonw that n2 now knows the stuff it sent to it previously. That's where the GossipOk messages might come in handy so that n2 can tell n1 it saw the messages n1 Gossiped to it.

  • @topdeveloper4794
    @topdeveloper4794 Рік тому +6

    i hope you provide us a video which is continue to solving rest of the problems

  • @bsodmike
    @bsodmike Рік тому

    Awesome as always Jon, enjoyed this thoroughly. I appreciate that you are a master at articulating your thought process and reasoning behind various changes taken (and the pros/cons at those ultimately taken).

  • @YTCrazytieguy
    @YTCrazytieguy Рік тому +2

    I think your gossip optimization has a bug - the topology doesn't need to be symmetric, so a node doesn't necessarily know about messages that nodes in its neighborhood know about.
    Had a blast, thanks!

  • @random6434
    @random6434 Рік тому

    When you defined Body with a field rest: B and then rejected it, you could actually keep Body and use Body for two-step deserialization and Body for one step deserialization if you happen to know B should be. There's a slight bit of weirdness in the second case you end up with a redundant string but you could make a type with two fields, a hashmap of serde_json::Values and a string for the type: struct DynBody { ty: String, fields: HashMap}. But deserialization to a KnownType might need a little bit of custom code to check the type field and reject if it's not the expected type.
    Just some random thoughts.

  • @maywakeTV
    @maywakeTV 11 місяців тому

    Thanks for sharing! Just got interested in learning Rust again and found your book on Kindle and via that your UA-cam!

  • @JustSomeAussie1
    @JustSomeAussie1 Рік тому +37

    Never took Jon for a Naruto guy

    • @bsodmike
      @bsodmike Рік тому +1

      rustengan was an epic choice!

  • @chrissaltmarsh6777
    @chrissaltmarsh6777 Рік тому +1

    In relation to distributed (heterogeneous, asynchronus) systems, you might be interested in Petrinets. If you are actually building a system, this gives you a tool for catching some of the problems before you design them in.
    It is a bit like the relationship between ER modelling and database implementaion.
    I'm using Rust for my 'nodes' (made from 'Place' 'Transition' and 'Arc' in PN language) and Rust fits perfectly.

  • @LB767
    @LB767 Рік тому +2

    39:13 OMG the amount of times I've had this problem, and being a beginner had no clue how to fix, so I would end up trying to shuffle lines around or clone or god knows what, when dereferencing was all that's needed...
    This is why I love your vids, they contain so much helpful practical knowledge, they are just a joy to watch.

  • @beersheep0109
    @beersheep0109 Рік тому +4

    Amazing. Looking forward to more similar contents.

  • @bemofresh3117
    @bemofresh3117 Рік тому +50

    I literally asked the question “is there any material for distributed systems from the Rust perspective?” On LinkedIn, and was told that I was asking a stupid question! Can you believe that?

    • @mechanicalmonk2020
      @mechanicalmonk2020 7 місяців тому +12

      Yes. It's linked in. It's like the worst parts of Reddit and Twitter combined

    • @Tyradius
      @Tyradius 4 місяці тому +3

      You can ask questions on LinkedIn? Thought it was just for listing your certs and degrees.

    • @VictorAlexandre00
      @VictorAlexandre00 4 місяці тому

      ​@@mechanicalmonk202018:17

    • @VictorAlexandre00
      @VictorAlexandre00 4 місяці тому

      ​@@mechanicalmonk202018:31

    • @VictorAlexandre00
      @VictorAlexandre00 4 місяці тому

      ​@@mechanicalmonk202018:40 18:41

  • @DavidSmith-ef4eh
    @DavidSmith-ef4eh 2 місяці тому +1

    These tutorials are a great advertisement for golang.

  • @narigoncs
    @narigoncs Рік тому +2

    One thing that would have been interesting to see is what kind of an effect changing the gossip interval has on the latency to consistent, in combination with the 10% extra items sent per gossip!

  • @tommy_salami108
    @tommy_salami108 Рік тому +2

    Hey Jon,
    I think it would very interesting to implement eye tracking. For example, at 22:39, you tab back to the serde_json docs for just a moment, then find the answer of what you're looking for. Seeing exactly what you looked at on the page would clue us in to your thought process, and add another layer.
    -Tom

    • @TimeConvolution
      @TimeConvolution Рік тому

      That would be neat but I don’t know how easy eye tracking is to set up without dedicated hardware

  • @dmitrij34
    @dmitrij34 Рік тому +2

    Hi Jon, amazing as always!
    I had an idea for sync across nodes - can you form a probabilistic data structure(like a bloom filter) from your node’s values and send it to neighbours, so they can test for known values, to figure out the data that we might be interested in?
    Sure, there may be false positives(we don’t know about the value, but the bloom filter says we do), but I feel like they can be dealt with.
    Or use a lossy hash table for inverse behaviour(only false negatives)

    • @jonhoo
      @jonhoo  Рік тому

      It's a cool idea - worth trying out!

  • @saurabhshinde1855
    @saurabhshinde1855 Рік тому +4

    Hii John,
    Love your work !! Hugh respect!!
    What is your take on decisions made by The Foundation on "Rust" trademark and logo usage ??
    I felt little disappointed..

  • @aleksandr-belousov_1
    @aleksandr-belousov_1 Рік тому +3

    Man, I missed your content! :)

  • @berndeckenfels
    @berndeckenfels 8 місяців тому

    3:01:30 instead of random you can use a module counter so you send n, n+1 and n+2 additional alreadyknowns. Or you add the first x of all already now since in the next message they won’t be contained

  • @user-sn3tf7gf9r
    @user-sn3tf7gf9r Рік тому

    Thank you for this amazing walkthrough!

  • @ruijungao
    @ruijungao 11 місяців тому

    Great work! However, just wonder why not just return a Result, and let nodes forget serialization stuffs?

  • @narigoncs
    @narigoncs Рік тому +2

    Great video as usual!

  • @lightninginmyhands4878
    @lightninginmyhands4878 Рік тому +7

    I like that this guy looks up cool words to name his cargo

    • @bsodmike
      @bsodmike Рік тому +1

      He does this regularly and comes up with some really interesting ones. Defenestration was one he picked (I think). Do you realise how hard it is to name a lib well?
      "There are only two hard things in Computer Science: cache invalidation and naming things."

    • @TNH91
      @TNH91 6 місяців тому +1

      @@bsodmike I think you forgot off by one errors ;P

    • @bsodmike
      @bsodmike 6 місяців тому

      @@TNH91that and runtime segfaults!

    • @tenebrae711
      @tenebrae711 4 місяці тому

      @@bsodmike not anymore with (safe) Rust!

  • @ClineMusic00
    @ClineMusic00 Рік тому +1

    Would love to see the next challenges

  • @viniciusrolandcrisci272
    @viniciusrolandcrisci272 Рік тому +2

    Amazing content, Jon!
    I have a little (maybe dummy) question: At the end you added an eprintln! for debug the size of messages sent. Why that operation didnt break the reading from stdout? How Maelstrom knows what is a message response and what is a log from stdout? Sorry if I missed something 😅

    • @jonhoo
      @jonhoo  Рік тому +6

      Ah, so, `eprintln!` (note the `e`) prints to STDERR, not STDOUT. It's `println!` (no `e`) that prints to STDOUT. Since Maelstrom only checks STDOUT, you can safely use STDERR for other kind of printing/logging.

    • @viniciusrolandcrisci272
      @viniciusrolandcrisci272 Рік тому +1

      @@jonhoo Oh, that makes sense! Thanks! Keep up the excellent content!

  • @dandreani
    @dandreani Рік тому +1

    39:09 when even Jon get stuck with the borrow checker... i think i am safe

  • @JohnKoepi
    @JohnKoepi Рік тому

    There is a crate that implements fully RPC/KV spec if you need any kind of support to fully focus the most interesting challenges - Kafka / Tx.

  • @L0wPressure
    @L0wPressure Рік тому

    1:01:27 - hehe, Beavis, he said "generic over some ASS"
    I kinda understand the gist of the code, but i feel like it's still way over my level at the moment. Awesome content anyway :)

  • @bjugdbjk
    @bjugdbjk Рік тому

    Common need something like these, Bring rust to Backend stuff !!

  • @andydataguy
    @andydataguy Рік тому

    Glad to see you appreciate DnD 🐉🙌🏾

  • @levizin917
    @levizin917 Рік тому +1

    Hey I know this is not in the topic of Rust exactly but can you post a video of your coding setup? (ide, vim scripts ...)

  • @veryrare876
    @veryrare876 3 місяці тому +1

    FIrst things first, rustengan is a lengendary name. Second off, how do you get that rustc message in your neovim convig?

  • @alejrandom6592
    @alejrandom6592 Рік тому

    Hi, I woke up and this video was on. Nice stuff

  • @alexxi3955
    @alexxi3955 Рік тому

    Great vid, loved it!

  • @floriantavares1849
    @floriantavares1849 Рік тому +1

    I loved this stream ! Very interesting.
    Have you in mind to do an other stream focusing on consensus protocol, traditional leader based (Paxos or Raft) or even new leaderless alternatives (Epaxos, Atlas, Tempo, Accord) ? That would be awesome !

  • @lker-dev
    @lker-dev 11 місяців тому

    For the gossip section getting longer was there also the possibility based on provided topology that ‘n1’ talks to ‘n2’ but ‘n2’ doesn’t talk to ‘n1’? Or were they always bi directional without the need for the sync?

  • @saksmlz
    @saksmlz Рік тому

    Thank you so much for doing this ❤

  • @natenatters
    @natenatters Рік тому

    Very cool video! Thanks :)

  • @johannessandjaja7630
    @johannessandjaja7630 Рік тому

    hi, i was wondering on 39:14 in terms of ownership and borrowing is there any difference between
    `&mut *output`
    changing the function param to
    `... mut output: &mut StdoutLock...`
    and just call
    `... serde_json::to_writer(&mut output, &reply).context("serialize response")...`
    anyway great content, thanks!

  • @Augustine_354
    @Augustine_354 Рік тому

    This is gold.

  • @cooldude3010
    @cooldude3010 Рік тому

    Love this content. More of this.

  • @kopo88
    @kopo88 10 місяців тому

    In the echo example, would `println!("{}", serde_json::to_string(&reply)?);` have worked instead of explicitly working with the `StdoutLock`?

  • @topdeveloper4794
    @topdeveloper4794 Рік тому +1

    Thanks. Can you reupload eng subscription version?

  • @yoyoma7273_6
    @yoyoma7273_6 4 місяці тому

    Which editor is this guy using? I really love it.

  • @__J____ff
    @__J____ff 10 місяців тому

    whats that Panda symbol that shows up next to payload: Payload popup ? what vim plugin is this ? and whats the on the go errors from ?

  • @ZhanhuiLi
    @ZhanhuiLi Рік тому

    Very helpful, thanks。

  • @gimmypharel
    @gimmypharel Рік тому

    Hi Jon, awesome stream as always! Got a question about the "improving efficiency" part. Why dont you just keep track of which messages you've already sent to the peer and stop sending those? Is it because you are not sure whether those have arrived or failed due to network conditions?

    • @jonhoo
      @jonhoo  Рік тому

      That's exactly right!

    • @guilhermesoares7857
      @guilhermesoares7857 Рік тому

      ​@@jonhoo In this case, implementing a perfect link to mitigate this problem wouldnt benefit you in terms of performance ( not to mention that the program would have to be async ) right ? Watching you program/explain these distributed system challenges is giving me a new love for this topic!

    • @mishikookropiridze
      @mishikookropiridze Рік тому

      @@guilhermesoares7857 really perfect link can never exist

  • @nachiketkanore
    @nachiketkanore Рік тому

    Thanks for the inspiration

  • @zilianglin7417
    @zilianglin7417 Рік тому

    Hi Jon, thank you for the great streaming.
    At around 2:23:49 I notice there are proc macro errors after you gd into other crates. Are there ways to avoid those?

  • @jongxina3595
    @jongxina3595 Рік тому

    Working on 4. Im not sure I understand but they provide a service for kv storage so to replicate this in Rust it means I would just use a database like Redis right?

  • @ovidiu_nl
    @ovidiu_nl Рік тому +3

    In the unique ID generation it looks like you did a lot of work to capture your own ID from the "init" message and store in state. But... don't you get your own node ID with every message, including the "generate" message itself? Couldn't you have just used `input.dst`?
    This is as far as I got in the video, so maybe this will be addressed later on.

    • @jonhoo
      @jonhoo  Рік тому +4

      Oh, yeah, I suppose you could just grab out the dst field in the message! That only works under the assumption that there are no multicast messages though :p The more relevant response is that we need more of the init state for later challenges, so we might as well construct the flow for grabbing init state properly anyway.

    • @ovidiu_nl
      @ovidiu_nl Рік тому +2

      @@jonhoo Fair enough. Great topic, btw! I really enjoyed the video this far.

  • @Script_Alchemist
    @Script_Alchemist Рік тому

    Gosh this setup, I had nothing predownloaded. That took a while

    • @Script_Alchemist
      @Script_Alchemist Рік тому

      I'm not sure what happened. I created the project like you and my rust-analyzer broke. I debuged for 3 hours. Deleted the repo. Recreated it without the --bin and it works.
      Not sure that happened there, but I'll write that in case it helps anyone else

  • @ningzhi3415
    @ningzhi3415 11 місяців тому

    It's really an excellent stuff.

  • @seftondepledge3658
    @seftondepledge3658 Рік тому

    Does anyone know if there is still a suggestions website for future streams?

  • @simonfarre4907
    @simonfarre4907 Рік тому

    What I'm wonder is, what's there to gain by making everything this generic? I love the generic stuff in this video, as it's a particularly weak point in my skillset, so any practicing on that is great, but every time I am faced with problems I need to solve i just so rarely find myself going for a generic solution - because it's so rare that it actually needs that _level_ of re-use (and in generics case it's not so much re-use, there's some, but not entirely, like in the case of C++ templates for instance).

  • @benedyktjaworski9877
    @benedyktjaworski9877 Рік тому

    I don’t like the “reverse dst and src” logic in responses - because you’re given your node id in the init, my assumption would be that you might see a message intended for somebody else and should have some logic for ignoring it (so you should check if the dest field is actually your own id before replying to it).
    I guess the person who wrote the “you’re still reversing dst and src” comment in the chat at some point had something similar in mind. Since you know your node_id you should use that as the src of the response, instead of just blindly reversing the message’s fields.
    Of course if maelstorm only delivers messages to the intended destinations and there’s no multicast, it doesn’t really matter.

  • @DarkKnightUNS2008
    @DarkKnightUNS2008 Рік тому

    I only remembered Rasengan after 3 hour of video :)))))

  • @ElGnomistico
    @ElGnomistico Рік тому +1

    Do you have any interest in trying to implement a chess engine in Rust? I do and watching you do it would be very instructional.

  • @blob4000
    @blob4000 8 місяців тому

    Thank you so much!

  • @dunghdinh6268
    @dunghdinh6268 Рік тому

    Amazing !!!

  • @PettoMartino-s8z
    @PettoMartino-s8z Рік тому

    Very nice content. I think it's the first time I watch a live for so long. Does anyone know someone doing similar content in go?

  • @jly_dev
    @jly_dev 3 місяці тому

    Naruto whirlpools, Rasengan, I'm in 🚀

  • @matthewpublikum3114
    @matthewpublikum3114 Рік тому

    What plugin is that in vim? What's a good code completion plugin?

  • @rodrigo_t9
    @rodrigo_t9 Рік тому

    What drawing program is being used?

  • @rawsteelfsp
    @rawsteelfsp Рік тому

    Can someone tell me what drawing tool is being used here?

  • @ShikaIE
    @ShikaIE 8 місяців тому

    Would you continue this challenge in future?

  • @act0r399
    @act0r399 Рік тому +1

    Jon, what you think about create new video about Tokio and same field stuff? In many jobs requires Tokio, and still in youtube no cool video about Tokio and async in Rust? Pls))

  • @bjugdbjk
    @bjugdbjk Рік тому

    Ths s so cool !!

  • @JohnKoepi
    @JohnKoepi Рік тому

    Nice!

  • @felgenh399
    @felgenh399 Рік тому

    love the name

  • @ZacharyPrado
    @ZacharyPrado Рік тому

    (Noob Q) why are stack traces in Java ? Tyia

  • @Wyvernnnn
    @Wyvernnnn 11 годин тому

    Nice of you to have built in closed caption but the captions are way too long to be useful! You have to keep each at a maximum of 10 words

  • @bocckoka
    @bocckoka Рік тому

    r#type? we have a lot of that.

  • @tsalVlog
    @tsalVlog Рік тому

    Of all the streams for me to miss....

  • @myname2462
    @myname2462 Рік тому

    Best quality content at 7:30 😅

  • @rontman
    @rontman Рік тому

    Missed opportunity for “chidori”😅

  • @josephmerrill2686
    @josephmerrill2686 Рік тому

    Rust devil!

  • @simonfarre4907
    @simonfarre4907 Рік тому

    Won't the unique challenge fail? You ran it with only 1 node. Are the different nodes always creating unique ID:s regardless of if they have no knowledge of what the other nodes are spitting back to Maelstrom?

  • @reybontje2375
    @reybontje2375 Місяць тому

    I love how you struggle most with coming up with the name, rather than distributed systems programming. 😅

  • @ozanmuyes
    @ozanmuyes Рік тому

    Be careful Jon, we don't want you sued... 😒

  • @Stopinvadingmyhardware
    @Stopinvadingmyhardware Рік тому

    No! You will not use Erlang!

  • @Thisguyrocks518
    @Thisguyrocks518 Рік тому

    I call it seeeered

  • @pat_neal_codes
    @pat_neal_codes 9 місяців тому

    missed opportunity to name it Charybdis, for completing charybDistributed system challenges

  • @bfrancis9898
    @bfrancis9898 23 дні тому

    Dude please mute the drinking. 🙏Otherwise a good video but just can’t take that part. 🤯