Implementing (parts of) git from scratch in Rust
Вставка
- Опубліковано 5 сер 2024
- In this stream, we implement core pieces of git from scratch by following the CodeCrafters git "course" @ app.codecrafters.io/join?via=..., just like we did for BitTorrent in • Implementing (part of)... .
If you sign up with the link above, you get free access to the challenge (and all their challenges) for 7 days. Alternatively, you can access the content for free (albeit without the nice infrastructure) at github.com/codecrafters-io/bu....
Video isn't sponsored (as in, I get no money for making the video), though like last time the above is a referral link so that if you find it useful and end up paying that also helps me - maybe a good use for an employee learning budget!
Various links from the video:
My (well, J's) Discord:
discord.jonhoo.eu/
GitHub sponsors:
github.com/sponsors/jonhoo/
Missing Semester class on git:
missing.csail.mit.edu/2020/ve...
Code at the end:
github.com/jonhoo/codecrafter...
0:00:00 Introduction
0:02:39 Housekeeping
0:04:36 Questions before we start
0:08:39 Initializing the challenge
0:09:47 git init
0:25:27 git cat-file
0:58:21 Mitigating zipbombs
1:08:37 git hash-object
1:41:50 Organize into modules
1:47:41 git ls-tree
2:24:21 git write-tree
3:41:45 git commit-tree
3:59:06 git commit
4:23:56 Outro
Live version with chat: ua-cam.com/users/liveU8HeVjY9cig - Наука та технологія
Jon's channel is my favorite place on internet and earth thank you Jon please do more of these
Not only the best resource for learning Rust on the internet, also one of the most clear and easy to follow programming tutors on the internet. Thanks for all you do, Jon!
Hey man, just wanted to say I love your stuff.
This kind of stream actually sold me on streaming CodeCrafters myselft, and on playing with Rust as well.
Keep it up!
The cross-platform gymnastics git has to do is truly spooky.
Thank you for the video.
I am writing in the Rust from last 2 years for commercial company.
It's a coincidence that my name is previewing in your video @01:09 "mohitagnihotri", I have implemented this challenge own self (no external reference) and now comparing my solution with yours.
I learn not only from your way of Rust code implementation but I also take some lessons from your thought process.
Really very beneficial for me.
Thankful to you!!!
This blows me away. I thought there is more "magic" in git. Nice to learn how it works. I'm currently learning rust and had a good time following your explanations. You have a very good style of explaining things!
I watched all of your videos and they're so well made. Thanks so much for this content. You are amazing!
**british DJ voice** one of the best one of the best
more of these building complex systems long form videos please bossman
Another neat thing to implement might be an Entity Component System like FLECS / shipyard / etc
I second this
Great episode! Not only did I learn a bit about Rust, but I also learned quite a bit about git! Nice :)
Haven't watched it yet, but as someone still getting acquainted with rust and craving a deeper understanding of gits internals, I look forward to seeing what you've done.
Jon, a walk through the actix framework would be nice for someone like me trying to understand the actor model
Actix's actor model is not a great representation of most other actor models. It's got some really weird limitations and it's difficult to build proper supervision trees with it.
Nice content! My implementation used a lot of in-memory strings compared to your streaming implementation. This is definitely very educational!
Wait, you're one of the missing semester folks? Holy crap you're the goat
This is like crack to me, I can't get enough! Not only do I learn more Rust, I learn more stuff about git! I wish this challenge would continue until a fully featured git clone (😉) would be implemented!
My brain already hurts after watching just about a third of the video, but it _is_ amazing. Thanks for this!
3:19:10 The write-tree ordering was bugging me, so I looked at what git.git does. The ordering of the tree contents is just based on comparing bytes (strcmp), but they are sorting the entire index which has entries like "src/commands-rs" and "src/commands/write_tree-rs" (using dashes instead of dots to avoid becoming links). So the ordering in the tree object includes a phantom "/" at the end of a directory name.
Ignoring the index probably makes the examples easier, but make actually working like git much harder.
3:33:04 I think the comparison logic you looked up is trying to emulate that without building the index. But you can see the place new entries are added in add_index_entry_with_check in read-cache and that is just using strcmp. You can see the result with `git ls-files --stage` which shows that ordering.
Oh, thank you! Just found your comment. Had the same theory - thanks for looking it up!
Thanks for looking this up! Implicitly ending with a slash actually makes a lot of sense... though I'd prefer that the UI showed it!
I love to send this kind of material to people.
I'm thinking of doing my own spin on these types of challenges, where we start with a discussion of the problem and datastructure and dataflow design, and then build in a way that is *fast* by the time the tutorial is done. For example in this or the bittornent one, architecting for io_uring and mmap, instead of high overhead POSIX stuff.
Another voice in the aether: these videos are awesome. Thanks for making them!
I was inspired by this and did the redis challenge. It was super interesting and I learned a lot. It would be awesome to see Jon have a stab at it the "correct" way. I have a little bit of trouble of finding correct mental model of TCP and how to parse it incrementally without excessive cloning, it's hard to search youtube or google for good information. If you have any recommendations for material I would be interested!
Great content! Couldn't help but notice that you have your browser search bar on the bottom - can't say i have ever seen that 😅
dude you are amazing! 💝
Interesting note: not sure about the rest of the plumbing but `git cat-file` itself will ignore the size value and print the entire rest of file. I used the same basic idea you used for your type (i.e. read one byte extra and compare how much was read) and created an object with a mangled size for testing. Tried it on both, mine worked, proper git printed the entire file.
Great stuff
Thanks!
Maybe the directories in write-tree have a silent slash at the end?
Awesome
3:40:22 it seems that ultimately the comparison rule just says append / to directory names while sorting, and what the git function does is to ensure that no new memory allocation is needed
AFAIK Windows has no concept of an executable bit. On windows, under git-bash, it has a series of heuristics that it uses to guess if a file is executable, but they don't work well. So much so, there is a specific git subcommand to directly set its file-mode flags because you have no chmod on windows.
P.S. One of the heuristics it uses when you run ls -la on git-bash is to actually check for shebang by reading the file.
Surprised the tree sort didn't just use collect(). I suppose you can't distinguish the errors?
Hey, what is this theme?
Can i get it on VSCode?
What are you using for tab completion in terminal ?
I assume it's at least close to the default fish config, which is pretty nice.
can you share your vim config?
Could you maybe try to implement the basics of Vim/Neovim in Rust?
What colorscheme is that? Love it.
Gruvbox dark hard :)
I'll also wait one day with zig
Please do a decrusting rayon
Just fix Git, with this alternate version, on the way...
What does he mean "this allocates"?
3:31:07 - 3:31:15 --- Clip it. Meme it.
I stopped to comment when you arrived at the sorting issue. I'll make a guess that the answer is that they first list the elements of a directory, then the directory itself. Now that I commented this I'm looking forward to see if this is the answer to the problem.
OK I did not expect them to have that approach. I'm still trying to understand why go to such lengths just to sort stuff
Those who are struggling in Windows os let mode = if meta.is_dir() {
"40000"
} else if meta.is_symlink() {
"120000"
} else if is_executable(&path) { // use is_executable::is_executable
"100755"
} else {
"100644"
};
What is the inline hints for function arguments called prepended with a panda and the defeninition above?, for example at ua-cam.com/video/u0VotuGzD_w/v-deo.html
You know, I honestly couldn't tell you 😅 I think it shows the current argument type, but no idea what has enabled it!
Ah, looks like it's github.com/ray-x/lsp_signature.nvim specifically the "hint prefix"!
@@jonhoohuge thanks!! also thanks for all the videos, filling the void for us above beginner level but not professional rust users :D
God I hope Code Crafters adds a dark mode. Getting smacked in the face with white backgrounds when swapping between the editor and the browser is giving me the squints.
Also, never realized that std::fs::create_dir_all was a thing. That's very handy!
the darkreader extension
please do git clone
implement udp next time
8:40 - actual start of the video
thank me later :)
The chapter marks already have that time code in there (and other useful ones!) :)
high level, zero cost abstraction and yet we are reading and writing in buffers like cave men... doesn't feel high level at all, other than useless verbose keywords and method calls...
You mean passing the output to read calls? The whole point is these are the low level io interfaces used to reduce memory usage and improve throughout, if you want the high level APIs just fs::read(path) and you get all the bytes, or fs::read_to_string(path) for UTF-8; or use Read::bytes() to get an iterator you can use all the goodies there on.
It's not like this is any different to how, say, Node does this, either, with the exception that you don't have a resizable buffer type yet (they're adding one for WASM memory), so if you, say, wanted to read to end you need to end up doing all that reallocation and copying yourself, or keeping and array of buffers and concatenating then at the end. Not fun.
Can you say what verbose keywords and method calls you're referring to? Do you mean individual names or just that there's generally a lot more used than in other languages?
There's definitely an argument that Rust can get too verbose; but in my experience that's tricky to actually compare; for example adding .with_context(|| ...) to each call seems like a pain in the butt until you try to do the equivalent in other languages: most of the time it's a wrapping try .. catch throw of an error, and it's often tricky to get it to give readable output. JS did recently make this much nicer with Error accepting a cause option, which I really appreciate, but it still is missing expression throws and trys.
Thanks!
I wonder why your abstractions returned out so awesome. I notice that often when I make my own abstractions they often do not benefit me as much as i do see n this video
That may also be your case, but he doesn’t seem to abstract until needed. Your first guess it probably gonna fail, so just inline everything and wait for a clear second candidate to pop up, evaluate and implement. Especially with an already working state, it should also be easier to pull off.
The general answer is unfortunately "get good" - there's lots of rules of thumb and guidelines and blog posts and everything else telling you how to do it well: and none of them seem to really help all that much because you don't know which to use and when.
For me, it's all been hard-won experience over to decades, and I think I'm still not as good as Jon. Ugh. 😂