Implementing TCP in Rust (part 1)
Вставка
- Опубліковано 11 жов 2024
- In this stream, we started implementing the ubiquitous TCP protocol that underlies much of the traffic on the internet! In particular, we followed RFC 793 - tools.ietf.org... - which describes the original protocol, with the goal of being able to set up and tear down a connection with a "real" TCP stack at the other end (netcat in particular). We're writing it using a user-space networking interface (see www.kernel.org... and the Rust bindings at docs.rs/tun-tap/).
We got decently far, and are now able to both establish an incoming TCP connection (i.e., perform the TCP three-way handshake) and terminate the connection cleanly. You can find the current code at github.com/jon.... There's still a long way to go though, as we still need to add support for data segments, retransmissions, and timers; so that's what we'll cover in the next stream! Eventually, we will also want to implement RFC 1122 - tools.ietf.org... - which lays out a number of further enhancements that must be made in order to interoperate nicely on the internet. This is also discussed in RFC 7414: tools.ietf.org..., which we're also aiming to support.
You can see the live version with comments at • Implementing TCP in Ru... - Наука та технологія
MAN MUTED THE SOUND WHEN TYPING THE PGP PASSWORD. THIS IS AS REAL AS YOU CAN GET +1 SUB
Why yes of course, security research has made me paranoid.
54:55
@Raiden Miguel Fuck off filthy bot
damn i didn't notice
Especially when you went and told everyone exactly what model of keyboard you use. @@jonhoo
It's nice to see advanced programming tutorials for once in my life. Had to wait for youtube way to long to suggest this channel. Cheers!
Jack Mead another person with advanced programming streams is George Hotz. He works mostly in python but also does some c++ and random other Langs when needed.
His aren’t quite “tutorials” but he does explain things as he does them
@@techguy8275 george hotz is a little ad hoc :))
@@birdboat5647 i were just to comment that too , haha
Tsoding is another good one
Someone donate this guy a wizard hat.
And robe?
Wow -- I love seeing your thought process around 25:20 playing with setcap. ESPECIALLY at 26:00 when instead of doing what normally too busy people do of "copy paste go next," you proceed to look more into the man pages to understand the WHY of the CLI. A+ learner.
Thanks for sharing your knowledge Jon. I'm 40 minutes in and have already learned a ton about networking and rust.
Is this how twitter looked 4 years ago, wild. Great content
If only the world had more generous people like you! Hat off bro :)
Thank you for the video. A few suggestions:
There are several missed opportunities for missed abstractions that would increase clarity of the concepts. For example, whether the ack number is valid, using iph.destination_addr().octets() instead of constructing the tuple, using an std::io::Cursor when sending bytes to the nic.
You did a case analysis of the wrapping arithmetic. I had the same instinct when I first saw the problem. But because the condition is invariant under translation, you can wrapping_sub start from all of the quantities. In other words, you can shift things so that start = 0. The equation becomes a simple matter of comparison (x < end) with a small bit of fudging depending on which bounds you want to include (0 < x
Hi! octets() actually did not exist at the time when this was made. Cursor I believe I already point out in a later part of the video series. And someone else beat you to pointing out that fact about wrapping arithmetic (ua-cam.com/video/bzja9fQWzdA/v-deo.html&lc=UgxGnIzDfUu92NgVkTJ4AaABAg), which ultimately resulted in this PR: github.com/jonhoo/rust-tcp/pull/1 :)
This channel is my best discovery of 2020
Checking whether a is less than b in a cyclic domain is as simple as checking whether the difference is greater than half of the domain. So for u32, a is less than b if a-b > 2**31. Much simpler than the solution that you coded, I believe (at least, up to the point that I've watched).
Also, in future sessions, if you wouldn't mind, it would be nice if you would highlight (with your mouse) the tidbit of info that you identify when you switch to your browser window for reference. I'm often focusing on your mouse (which isn't doing anything) when you find what you need, switch back to your editor, and then I struggle to follow what you found.
I'm really enjoying and learning a lot from these sessions!
Hmm, I'm not entirely sure how that can be true? a < b isn't meaningful is a cyclic domain, since there is nothing in a < b that tells you whether it should be true for a = b - 1. The specific issue we're trying to solve here is a < b < c, which _does_ have a well-defined solution. That said, there are probably much better ways to code it up than I did (and I think I mentioned that in the stream), I just didn't want to stop and figure out what they were :)
As for highlighting what I"m reading, that's a good point! Someone also mentioned in another comment that they feel as though I'm sometimes switching too rapidly back and forth between terminal and browser. Highlighting what I'm reading would probably help mitigate that too. It's just very counter to how I _normally_ work, since I try to avoid the mouse as much as possible, so I'll have to overcome a lot of muscle memory. I'll try though!
@@jonhoo Sure it does. The key is how the difference between the two numbers compares to half of the domain. For example, assume we have a u4 domain [0..=15] with addition and subtraction that wraps. Let a = 14 and b = 1. Is a less than b? Yes, because a.wrapping_sub(b) == 13 > 2**3. Is b less than a? No, because b.wrapping_sub(a) == 3 < 2**3 . So, even though we wrap around from max to min, we can still compute this.
The Occam language had a construct that was used for 16- and 32-bit timers called "AFTER". This is how it was computed, because for a cyclic domain, you have to view the results with respect to half of the range of the type. See the section about AFTER (10.2.7) in www.wotug.org/occam/documentation/oc3refman.pdf. With a 1 us timer clock, the timer on a 32-bit transputer would wrap after 2147 seconds, but when comparing whether one time was AFTER another, you could only handle differences of 1073 seconds.
Yeah, I wish that I had watched the live stream. Viewing it after the fact was frustrating because I couldn't interject what I'm pretty sure is a correct (and simple) solution.
Why is a less than b in that case? Without an anchor, you have no way of saying. In circular arithmetic, b is just as much 1 as it is 17, so a < b isn't well defined. It sounds like you're thinking of some particular definition of a < b in circular arithmetic where things can only be larger if they're larger by at most half the width of the space?
Again, I don't see where your notion of half the space comes from. TCP sequence numbers have no such notion as far as I'm aware.
@@jonhoo I can't see how they wouldn't. Because they will wrap, they have to use some interval less than the full interval to define when something is "after". Using half gives you the most bang for your bits. I believe you even mention near the beginning of the stream about an expected time bound for sequence numbers (4.5 hrs at 2 Mb/s from the RFC).
But try it for yourself. Use a 4-bit sequence and then iterate enough to wrap-around. You will see that you can't reason correctly about which number would come later in the sequence unless you partition on a difference of 8.
You may have noticed my username (Occamish). This is a problem that I looked at back in the eighties (quite a bit) and the Inmos Transputer and Occam language used the "AFTER" concept heavily.
Here's a response on superuser.com/questions/966212/does-the-sequence-number-of-tcp-packet-headers-wrap-around/966225 which gives essentially the same algorithm:
"in cases where the maximum window will be less than 2^31, and if x and y are type uint32_t it's practical to define x
As an example, using a 4-bit space and with a=1 and c=14, is b=2 between a and c? Your implementation says yes, but if you consider that a may have come AFTER c by wrapping around past 15, then b is not between a and c. So, you have to make a definition that takes half of the domain into account, otherwise you can't properly reason about the wrapping of the sequence numbers.
In u32 terms, let a=1, c=0xffffffd, and b=2. If given those three sequence numbers, do you think that it would be reasonable to infer that we didn't wrap and that a didn't come after c and that b is between a and c? I believe the answer is no.
Brilliant technical video. Thanks a ton for time and effort. Very helpful!
This is such a gold. I really wish you keep doing this. Like I don't even care about the length of the video man. Big fan. I have learnt a lot. Thanks a ton.
this is great, thanks for uploading
Wow 10/10 you are the real deal
Thank you! This has been the best 4 hours I have ever spent learning a new language.
Finally, I've found a video of high technical quality. Thanks!
This is insane, thank you for the time and effort! I learned a lot from this video:)
This is beautiful. Thank you for sharing the knowledge.
It seems like the direct solution for the issue at 2:50:00 would be to first put the unambiguous check: you must be in one of 3 order cases, (u, a, n) or (a, n, u) or (n, u, a). Then you can add a subjective check: (0 < a - u < X) and (0 u), so you won't even need to check extra cases, but double check that.
I really think it would be simpler to use a wrapping_sub for the acceptable ack check, I'm thinking something like
ack.wrapping_sub(una).wrapping_sub(1) < nxt.wrapping_sub(una)
(this assumes that una < nxt (by wrapping comparison), which I'm not sure we can do, but I think so, since both are on our SND side)
Say that ack
Actually, all of that uses wrapping math, so you can wrap most of the TCP fields with a Wrapping to make arithmetic wrapping by default.
There's actually been quite some discussion about this elsewhere in the comments on this video, which resulted in this PR: github.com/jonhoo/rust-tcp/pull/1. It turns out that there's actually a well-defined de facto way of doing "wrapping less-than" comparisons that the RFC just didn't explicitly outline. See the PR for details :)
I’ve never seen such a good coder. It’s been a year since I started learning and I’m nowhere close. I’m curious, how long have you been doing this?
yeah lol. i'm just a beginner and didn't understand shit at all from this video but just watched some minutes and he uses the keyboard like jimi uses the guitar lol. changing more than one lines at once and indenting them etc. without using mouse is impossible for me lol. i dont know shit.
@@sundaymanali5854 Yeah, Jon is the 10x developer that every company wants. I have never seen anyone code like this.
Really cool tutorial - tun/tap works on WSL2 for those wanting to follow along on a windows machine
how awesome, I want to pick up rust again for this, if you have a patron account, I'd definitely pitch in
I'm so happy to hear you enjoyed it! Sadly I had to close my Patreon account a while back (twitter.com/Jonhoo/status/1047259437319278592), but I appreciate the sentiment. Hopefully one day it'll come back :)
I'd been curious about this for ages, thank you so much for your awesomeness.
Das ist wunderbar!
Thank you for the stream!
Regarding the wrapping comparisons, I find it much easier to do a case analysis on the endpoints:
- if start < end then the interval has not wrapped: x is ok if start
Thank you so much! I learned a lot. Please keep posting.
Thanks for sharing!
Hopefully someday someone with a super heart, make a Spanish translation of this channel
Immediately subbed. PLEASE keep the good videos coming!!
My english very bad, but I just need to write this comment for promotion this video in recomendations, thank you
like your videos, have to watch the next one live! @wrapping comparision (the 40min in the middle): just subtract the start everywhere (may wrap), then the comparision is 0 < x < end (without wrapping)
This was actually already brought up in some other comments, and have since been fixed! github.com/jonhoo/rust-tcp/pull/1
Oh thank goodness. I was stressing that whole segment! 😅
Definitely waiting for part 2
Thank you, hope to see more like this
Kjempe kult å se på deg kode. Keep up the good work :)
really a rocking start!!!
Thanks. Lot of learning for me
This is awesome. Thanks for creating this.
thanks for your vids! I'm learning Rust!
Thank you so much, John, learned so much, keep it up, please. I need you to learn more about rust and looking forward to see more and more videos ;)
A+ video. im so glad you have since stopped drinking liquids right into the mic 💀
It would be nice if the libraries exposed a state machine and you could do the whole thing without allocating a buffer.
What a genius!
Super interesting!! Thanks!
Hey Jon, nice videos! I am wondering which resource would you recommend for a beginner student in distributed systems. Preferably something that is really hands-on, with practical coding examples.
MIT's Distributed Systems class! All the coursework, lecture notes, and tests are freely available online: pdos.csail.mit.edu/6.824/
@@jonhoo I had no idea there were course videos beyond the OpenCourseWare ones, thank you!
awesome Jon, Great job and thanks a lot
u are my god now
I feel like I’m amazed at what he’s doing and I don’t understand much about it. it’s system programming, low-level programming or another area of programming that I’m not familiar with ?
Rust rocks!
This is gold!
Great stuff, I learned a lot. Thanks.
Thanks!
Very Helpful！Thank you！
Being a Phd student, How the hell do you find time to do this???? I'm an under graduate and I really like to do stuff like this but have no time.
You make time for what you love.
@@nobytes2 Exactly.. I sacrifice good grades for my own study
A PhD is like a 9-5 job, he does this in his downtime
@@honzasedlon3309bad
If I could make a quick observation, might I suggest that you read out (or tldr) what you are replying to on the chat screen since some of your viewers will not be watching the stream live. Otherwise, great job! Having fun watching and learning.
I try to do that, or at least to fold the relevant context into the answer, but it's definitely something I can get better at! :)
Awesome!
May I ask which keyboard you are using? Very great video btw!
Awesome! Finally something different. May I know how old are you? ;) thanks for this awesome video.
Glad you liked it! Something different in what sense though? I'm 29 :)
@@jonhoo Well, you can't find tonnes of well made tutorials on this subject or even poorly made tutorials. Networking is a hard nut to crack.
why was there some network activity when we created the tun interface? Was kernel using the newly created "tun0" for some purpose?? And at 48:25 how can one make sure that the kernel uses the "tun0" interface to connect through netcat and not any other interfaces?
Hmm, which OS do you use? That's a well set up interface that I haven't seen before.
Hi Jon! At 45:47, how did you automatically nest the eprintl!() macro with one argument per line?
I have rustfmt set to run on save, which formats things like the eprintln! macro :)
Congrats for getting 48k more followers!
you should do a video where you implement the minecraft protocol
41:58 How do you get the error message on the right side instead of at the bottom? Is it an coc configuration?
so good!~ thanks
Hi Jon, I'm very curious what app and equipment you're using to draw at 8:27. I'm guessing you're using some drawing tool that hooks up to your cursor? Either that or you're a wizard with a mouse.
The program is MyPaint, and I'm using a Huion drawing tablet :)
This is great. How did you separate the arguments onto separate lines at 38:45? A lot of cool code stuff that you do feels like magic and it's never clear what you're using/pressing to do it.
Most of that is just having rustfmt run automatically at save. All I'm doing is saving the file :p
haven't finished this vid, I wonder this vid is developed on linux or macos? I'm using macos while this tuntap seems not working on mac
Linux
I am 30 minutes into the video. So is the end goal to get this TCP to talk to hosts on the internet or just to keep it internal to the operating system? Anyway great work keep it up I'd like to see how far this is going to go.
The plan is for it to work against a "real" TCP implementation running elsewhere. It shouldn't be restricted to the same host.
has anyone done this on Win or Mac? I abandoned Windows since the tuntap stuff wasn't available there or on WSL, and on Mac i found the tun_tap_mac crate and some tun tap driver but that was giving me /dev/tunX (ie. /dev/tun0) interfaces instead of /dev/net/tun. I then tried linking /dev/net/tun -> /dev/net/tun0 but am now getting ioctl issues. Anyone else gone down this path?
I'm on mac, but I'm using vagrant to spin up a debian box
Wow you type so fast, Doing stuff like this is will take forever for me because I type so slow lol
Thanks for the content I am trying to implement my custom vpn protocol and toss is invaluable content
Btw what’s the software you use for drawing on the screen
Thanks! It's called MyPaint.
PART 2?!
Hi Jon. I'm following your tutorial (am at the 50 minute) but as I'm working with macos I'm transforming all stuff to be compatible and one thing bothers me. It looks like I'm receiving a Datagram (Network/Internet layer) and not a Frame (Data-link/Network Interface later) so I don't have to do the initial parsing as you do. I was scratching my head what could be the reason for that and still have no idea, but isn't it the TUN should operate on the Network/Internet later (OSI/TCPIP) and TAP on Data Link/Network Interface layer? Maybe it's related to the OS differences? It's bottering me a lot as a perfectionist to understand the issue.
I'm not sure I follow - we also use TUN (so IP-level packets), not TAP (Ethernet-level packets), so I'd be expecting you to get datagrams and not frames.
@@jonhoo apologies, now I understand what was happening. I missinterpreted the type of the data received as you were receiving the packet prepended with the packet information, and on macos it's not possible (as far as I see). Everything is clear now, thank you for your reply!
I did not understand the trick used to write out the headers. How do the writers know the last written position?
Thank you! Though i have to say that the quick window switching from editor to doc to another tab within the blink of an eye can be _really_ infuriating. If you want to read a little bit about what's on the screen, it feels like someone is just violently pulling that context away from you. E.g. 1:42:20
That's a good point - somehow that's just the working style I've gotten used to when I just have one monitor. I could try a side-by-side view next time, though it will make for a somewhat cramped space. Do you think that would be better?
@@jonhoo Honestly, i don't know, both versions have ups and downs. Your style is good 90% of the time when you're browsing the docs or just coding, because it's clear what you're working on. But in the intersection, when you're quickly switching windows, its just no good.
With the window tiled into editor/docs it indeed gets crammed and tricky when the line length gets close to 99 chars.
Another solution i could think of is in case you have a second screen, put the documentation on the second screen when you have to quickly check docs, but on the other hand labourious for you.
Maybe it's just me, so take it with a grain of salt :)
Sadly I don't have a second monitor at home, otherwise that might be a good solution. I do like that the viewers can also see what I'm looking at though. If I read something in the docs and it's not on screen, that could probably be confusing. I think what I'll try is just to be more mindful about quickly switching back and forth, and hopefully that'll help!
I think it's okay for you to do the violent swapping and stuff, since we viewers can rewatch the stream anytime. We can pause the video so it's okay if we want to read
Awsome
Great Work, Can you make a video on your OS and editor setup?
he did that already.
@@salman2070 I just got to know ;)
Hi，Jon，I'm a rust beginner from China,can I reprint your vedio?I will note with the link address of the original video.(Most of Chinese people can't use youtube.)
Hi! Yes, feel free to post the video elsewhere! I'd love it if you can link back here though, and also post the link to the video you post on this thread :)
@@jonhoo sure,the address is www.bilibili.com/video/av44590996
And this is the 1080P version: www.bilibili.com/video/av45568851
I know this is a relatively old video, but... Is that a topre keyboard? 0_o
Jesus, is that you? I dont think normal humans can be such good coders!
You need implement QUIC in next video! :)
We could! Though there's already a pretty solid one over here: github.com/cloudflare/quiche. And we already have several other stream topics to cover!
@@jonhoo Unfortunately, CloudFlare does not stream development proccess, unlike you! :)
Implementing TCP in Rust is one thing...doing it with Vim as your text editor is another... respect Jon!
Why? Vim is better than any IDE.
Hello from Cambodia, I want to ask you for some source about Networking with Rust Programming Language from beginner, because I try to watch this video and I think it's high level of me.
hey, if you're beginner then instead of looking at how to implement TCP, learn how to use it, and other protocols. Look into sockets, how to do them in rust (all should be in standard library, doc.rust-lang.org/std is your friend) for both TCP and UDP. Also you can look at Tokio, after you're familiar with working with blocking versions from standard library. Feel free to ask if you have any questions
Would a different way to do the wrapped check just be to say if start < end then do the regular check, and if start > end, then do the opposite check? To me that feels more clear, but maybe it's not to everyone.
I got
(start < x && (end < start || X < end)) ||
(start > x && x < end && end < start)
who set CARGO_TARGET_DIR 27:10 so you can read it?
could you do one for mac osx since its not been maintained there for n2n
when I run the program (state 31:16) and monitor it with tshark I keep seeing outgoing TCP requests (SYN and TCP retransmissions) to a certain IP (138.201.81.199) any ideas why that would be happening?
I see the same thing. It seems to be some Archlinux thing, from running `traceroute` on the IP address; but I don't know exactly what the requests are being made for. (I'm guessing you're running Arch too?)
Hmm, that's interesting.. I don't have a good sense of what that might be, since as far as I can tell there shouldn't be any other traffic going over the TUN. It can't be DNS either since it's going over TCP. I think you'd have the most luck looking at the destination port and packet contents to try to figure out what that traffic is.
@@94samrat yes thanks
It's the networkmanager package that checks for connectivity. See bbs.archlinux.org/viewtopic.php?id=223720
Incidentally, I had the same problem with an Ubuntu derivative; see www.omgubuntu.co.uk/2017/09/disable-network-connectivity-checking-ubuntu-17-10 for a fix
You're my doppelganger... weird
Hey Jon, just curious what is your actual profession?
Hi! I'm currently a graduate student at the Parallel and Distributed Operating Systems group at MIT: pdos.csail.mit.edu/
@@jonhoo can anybody join this group? i am interested.
@@VijayBambhaniya yes, if you can get an admission at MIT and want/can/worthy of doing research on those topics. Suffice to say very few people can/qualify to pursue that at MIT no less.
Lame question, but how did you get Vim to look like that? Oh, and you're gifted.
Take a look at ua-cam.com/video/ycMiMDHopNc/v-deo.html :)
awesome, how could you move the tabs to bottom?
I have a video on that over at ua-cam.com/video/ycMiMDHopNc/v-deo.html&lc=Ugw7mC8ZYKfWnJwL9Tx4AaABAg :)
how in the f did you learn how to do this lmao
is the code published somewhere?
It's linked in the description: github.com/jonhoo/rust-tcp
I wonder how mutable borrows wor... *dies*
does anyone know how he copied content from his terminal buffer with keyboard?
1:11:22 Receive the sin
please do a vim session
make a video about rust D-BUS please
salam kenal mas, sy jg penggemar rust dan pake scaproust utk IPC