Virtual Console and printf() //Source Dive// 003
Вставка
- Опубліковано 29 вер 2023
- In this installment of //Source Dive//, we're back in xv6, looking at how the operating system layers abstractions. The "console" is a great example, with a low-level driver rooted in the hardware, to an abstract console, to the printf function that eventually outputs characters to the screen.
=[ 🔗 Links 🔗 ]=
🐋 RISC-V Docker Image: github.com/francisrstokes/rv-...
🎥 Series Playlist:
🗣 Discord: / discord
⭐️ Patreon: / lowleveljavascript
💻 Github Repo: github.com/mit-pdos/xv6-riscv
This might be the only place on the whole internet where someone explains such a seemingly complicated thing like an OS kernel literally line by line, call by call and it's so eye opening. Please keep going like this
This is Ben Eater level technical explanation. Really well done, I’m loving it.
Personally I’m following quite easily but when I start to bog down because of esoteric naming conventions or a gap in my knowledge, you’ve already done the hard work and I’m pulled along nicely. Excellent work.
This series is amazing! As a C++ developer who studied electronic engineering I was always curious about how does the OS actually interacts with the CPU but all the resources I found about the Linux kernel were too complex to start with. This series is a perfect starting point! Looking forward to the next video!
Just wanted to say thank you for making this series, I am enjoying it immensely. For me, the level of detail you're going into is perfect and your explanations are spot-on. 👍
Was eagerly waiting for the part 3 , thanks for the series
I absolutely love this series! Thank you for putting time into this and presenting it so well! ❤
Great video. I’ve loved this type of low-level programming since the beginning of my career and you explained concepts really well. This is an excellent complement for OS Lectures, and actually I have found your videos more useful because you show real code.
Those two algorithms for printing numbers base 10 and 16, and pointers are brilliant.
Like many forks here said, this video is really amazing and very informative and unique. I absolutely love this series. Great job man, you are my hero.
It blows my mind that the decimal-to-ascii function is what blows your mind. I had to write an algo to do that in C 101 and it was almost identical, and I was a novice programmer. All the other stuff you cover is, to me, what is exceptional code.
It's less about how complex it is, and more about how elegant it is
@@LowByteProductions I guess. Seems very K&R to me. Why do you think developers use the idiom of things like: WriteReg(IER, 0x00), where 0 is written as hex with two digits? It bothers me to see 0x1 or 0x01, as if hex 1 is different than decimal 1.
Great stuff sir, I hope you like this series and make more episodes, because I really love it. Really wonderful and approachable, exactly the kind of pace and gentle progression that makes it understandable.
It is definitely one of the most interesting educational IT series, I have seen in months. Pure knowledge ...
This video, in particular, has given me so much respect for the people that come up with all of this stuff, and shown me how much I take "simple" things like printf for granted
Really loving this series, it’s inspired me to go through the code myself. Thanks for the series!
Absolutely love these videos, the level of detail is perfect! Makes learning about the inner workings of unix like os alot less daunting. Thank you!!
Brilliant! I knew something like this was going on under the hood of a kernal but to see it in the flesh is very interesting. looking forward to the rest of this series.
Fantastic job explaining it. I can see how the original creators reduced stack allocations by moving as many arguments to the calls.
Can't wait for part 4, this is becoming more and more interesting as time passes. Keep it up!
Damn, this is probably the only place that someone explains kernel code line by line. I'd LOVE it if you can go through the whole xv6 kernel, or maybe Linux 0.12/0.13 (0.95).
I like the fact that you are so enthusiastic about it too ! I really wish I had you as a teacher
This is great! Ive been reimplementing xv6 line by line so I can learn more about operating systems. This will help me get a better understanding as well
I LOVE this series! Please make more!
Unreal. Just right level of detail. Please please continue!!
Really enjoying these, the length and detail are great, keep them coming!
The series is a great idea and I think you're really nailing it. I think the level of detail and explanation is a great balance. I'm far from a C expert but it's enough info for me to get by with a google here and there. I only wish you were making them faster :D but I know it takes time to make high quality content. I really appreciate the decision to use risc-v and xv6.
this has to be the coolest channel I've ever seen imwith regards to comsci. super thanks sir!
Love this series, thanks for making it. Level of detail seems great. If you know a little more some of it might already be obvious but I haven't found that distracting because I know it helps people who know less.
I really appreciate this comment because it gets at the essential balance I'm trying to strike. I hope that theres enough in it for experienced folks, while keeping the door open for people who are just getting into this kind of stuff 🙏
i believe the details are amazing, hoping for more of this series
Oh I liked the printf building an array and then reading it backwards, really clever trick to not have to deal with counting the number of characters, or having to ignore empty values in the array
I have a horrible idea to write my own printf that recurses on %s. Oh no what have you done lol. Really enjoying this series so far, thank you!
I love your videos on operating system, please keep going. It's really interesting!
Congrats for the great video, real clear explanation and delivery. To me, it feels like this series it s complementing the equally great content from Ben Eater, like the 6502 computer, etc
Thanks a lot for sharing 👏
That comparison means a lot, thank you
I love it and as the other comments say this series is awesome, I'm enjoying the Bare Metal series as well, I really like your style.
Wonderful. Just the right level of details. Thank you very much
what an amazing series of videos!!
Thank you for the detailed explanations. Fantastic video!
this helps a lot! please please please don't stop posting
Not planning to 😉
I love your series! I would love to see some deep dives into something like FreeRTOS, smaller code bases such as the GNU coreutils, or maybe even the Doom source code would be interesting.
I am self taught on C, on but very limited as a programmer, I really enjoy your deep dive into the code.
Looking forward to the rest of the series. It should make the book "Modern Operating Systems" by Andrew S Tanenbaum a bit more approachable on second reading.
I hope so!
Just found this. Pretty cool stuff, please keep up the excellent work
This is fascinating. I guess one of the worst things you can do is print a lot of messages in kernel mode - this slows your system down to crawl (in this case 38.4kHz). I've seen this a lot; now I know why. Thank you very much!
What a good video and Amazing series
I really look forward to see more videos on this series. The level of detail is amazing.
As a mathematician, it's funny to see how you're amazed when the same algorithm worked for different bases. Let me tell you a little secret: The algorithm you've seen is the idea of numerical system itself. This algorithm itself is how you actually convert between the actual meaning of the number (imagining 15 apples in a plate) and its representation (by putting digits 1 and 5 next to each other).
ben eater actually has a video on that to, I find it especially interesting because the chip he does it on (6502) does not have modulus or division, just addition, subtraction, and bit-wise operations. It's cool how they were able to do that
@@barmetler Thanks for sharing
Awesome series, keep them coming!
Thank you for posting. I am following all of your videos.
Glad you're enjoying them!
Just wonderful. Thank you !
This series is sooooo good... 🔥
Loving your recent video 👏
Great series! Thank you!
wake up babe, low byte production dropped a new video
i enjoyed in every second very informative video
Level of detail is perfect imo, the code explanations are extra helpful because of the amount of "clever" code and variable naming ends up making things harder to understand
You're on a roll!
I was looking/waiting for this for two/three days, and lo here it is :p - I need continued source dives on this xv6 and others. Would you keep request in future on what source dive I want (and thus others also need that but cannot express or don't find the right person)?
You can ask here and I'll keep it in mind for the future 👍
I'm happy this is episode 003 and not 3
Loved this, please do more
I would love a 4 hour video lol, but you won't get many views with that. I thought you did great, it's always surprising how such complicated code is made of such simple parts. I program for the joy of it and have been for over a decade, what a fun time. Thanks.
love this series
Level of detail is really good. I am slightly annoyed that there is another level of abstraction that's hidden under the compiler's built in functions, but I get it. I would at some point like to get stupidly low level and compare a GCC built in for printf built for RISC-V vs Arm64 vs x86_64 (I know, I know, you are not a fan of x86_64). I'm mostly just trying to pull back all of the layers of the onion to get to its center. Fun thing about computers is that they are made by humans so EVERYTHING is understableable. A human mind built all of it, so all of it was understood at some point.
Honestly a great attitude to have when it comes to learning more about how things work!
Might be a fun challenge to look up the ABI details for variadic arguments in C on risc-v and implement the stdarg functions by hand.
Very interesting. Subscribed to your channel.
Can’t wait for the next video
More topics like this, please.
Great Video again. Detaillevel is correct.
I have enjoyed this video. Thx.
This is amazing btw, keep doing these
One thing you didn't really explain is what is on the other end of the UART, *how* does the character end up on the screen? There's no mention of a font, a screen printing routine etc
True - but all those details are kind of hidden from our view on the qemu virtual machine side!
That's true if you're emulating the hardware, sure, I assume qemu is emulating the UART and 'receiving' bytes and printing them in a fake terminal. However, this OS code was designed to run on real hardware, so what would real hardware have done with the UART data? @@LowByteProductions
Excellent. Thanks!
Yeah, love it. Fun stuff. So interesting (though totally understandable) that the kernel and userland have entirely different printfs... and this simple one is... so simple and elegant! Cool stuff. (Might be fun to look at the userland one sometime, though I imagine that gets much uglier...)
Also, I've used va_args stuff lots in the past... though you're right that it was mostly when writing library functions, especially wrappers for (vsn)printf. :)
Very interesting, more please.
Wonderful, thank you
Awesome, thank you 😍
A source dive into the batman-adv kernel module I would find very interesting. It is a routing protocol operating on layer 2.
Great series. Keep doing exactly what you’re doing.
I have a few questions. Who is on the receiving end of the UART? How are these characters actually getting displayed on the monitor? Or is qemu doing that all for us under the hood? Feels like there is a bunch of UART initialization missing.
Also if all cores share the UART and specific terminal instances only want to read/write their specific content, how do they filter appropriately? Do terminals just build their own protocol on top of UART to take care of this? Printf seems to just be writing chars directly though without any coordination. Is it because xv6 is intended to be single terminal?
To answer your first question- yes, that's all on the qemu side. Fundamentally, the OS is just sending out data to be interpreted by whatever kind of terminal device is on the other side of the uart.
Xv6 is indeed a one terminal system. You could of course do some kind of terminal multiplexing to emulate multiple terminals (this is exactly what computers used to do back in the Unix days!).
As for all cpus sharing the same interface - yes, you're right, they do. The uart interface is protected with a spinlock, so at the very least the cpus will not directly clobber each others messages, but if multiple processes are trying to output to the console, their output will be interlaced.
Thanks....the level was fine,,,,,,,,,☑
Goated.
All episodes are spots on, the peace is perfect, as well as the depth. I just would like to have a question answered: Is UART (or any other hardware component) the one that deals with the weirdness of VT-100 like escape sequences to allow for colors in the terminal, or is it usually the operating system and xv-6 does not support it?
Nope. The UART implements just a standard in-out serial interface, not really a specific useful device. There's no concept of displaying anything, you just send and receive bits (and the necessary configuration only for this). This uart implements rs-232 as far as I can tell (en.wikipedia.org/wiki/RS-232). It can talk with any devices that implements the interface.
The terminal is one layer above the serial interface. The terminal itself(either a hardware device or emulation) implements character sets, cursor, how it displays(colors, resolution, scrolling), and what the keyboard buttons do.
This code example sort of makes a mess of the separation, but it implements a text buffer + serial chip - which is a simplification of any terminal connected trough a serial interface, and that's what you need to implement c/unix's printf. It's kind of backwards, since microcontrollers and computers can implement their own display interfaces (or some part of memory mapped pixel buffer) and printf would work with emulation on top of that (not going trough a physical serial port). But for emulation and debugging, this is the standard way doing things - c prints and receives text trough a serial port.
@@PaulSpades So, when we are running this operating system in QEMU, how exactly does the virtual buffer that's getting written to get translated to terminal output on our machine? At what point is that data transferred from the buffer to our own operating system; what code makes that happen in the OS?
This is a cool series, I hope you find the strength to continue.
WRT your last question, for me at least it is just the perfect level of depth, otherwise it will turn out to be too complicated and focus more on some details and less on "how to write an OS".
thank you .
just right!
Hey!
Nice one :)
Will you still explain how an OS would implement and expose glibc to its programs?
Not glibc specifically since xv6 doesn't use it, but we will discuss the system call interface between the kernel and userspace, and how a glibc-like abstraction could be built on top of it.
Hopefully, the va_arg function has a call to the panic function on error, like when there is more % than arguments after fmt :) Thanks for the explanation!
I'm afraid not 😁 there's no way to connect the format string with the number of actual arguments provided - in fact at runtime, there is no way to know how many arguments the function was actually invoked with!
Try this out on a regular x64 machine, and you'll see that if you forget to add an argument for a format string, printf will just read random garbage.
nice one as always! could you share the vscode theme you are using? I really liked the color scheme
Dracula :)
@@LowByteProductions Thx!
Cool that you went through it all. I guess you can't print %4.4x or something similar and a shame floating point isn't included - kinda weak printf. But a great educational thing for sure!
There aren't any floats in the kernel code, so including it would be a waste. That said, I would recommend implementing %f as a challenge - the problem of turning a bit representation of a floating point number into a string is not trivial.
Great video!
I have one question: I think I missed how the address 0x10000000 is set to let the write end up in the virtual hardware and not just write to some random memory?
As far as I remember and understood - this is a starting point of standard output or stdout. Since everything in UNIX-like systems is a file, we can just write to that “file” using that address and get a string printed out in the console using system call. System will get string from that address and will print it out to the display using hardware specific instructions
The UART0 define is used to calculate the final memory mapped address!
@@LowByteProductions I think @LaSDetta is asking how 0x1... ends up addressing the correct memory when virtual memory mode is enabled. Does the UART code always run in real address mode or is the virtual table set up do a no-op translation on that address when addressing is in mapping mode?
Basically implement putc, this might print to screen, uart, leds or whatever else.
at 37:40, why do we even do an AND operation there? why not just assign the value to c?
Good question. The type of fmt[i] is char - which is essentially a *signed* 8-bit integer. The type of c, however, is int - or signed 32-bit integer. Since fmt[i] is being assigned to c, the int8 will be converted to an int32. If the value of c was "negative" (something >= 0x80), then it would be *sign-extended* when converted to an int32 (e.g. it would end up as 0xffffff80 instead of 0x80). Since we don't want that to happen, we force the interpretation of fmt[i] to be unsigned by anding it with 0xff. A more explicit way of doing this would be to say `c = (uint8_t)fmt[i]`, but thats confusing in a different way since the final type of c is still int32.
The printint function is really nice, tho I don't quite understand why the buffer is 16 long and not 11? From my point of view I try to think of the maximum number being inputted here which should be a signed 32-bit integer and thus 10 digits long.
You're correct and I don't know why it's the case 😁
I have a question about the for loop in printf. you said that the assignment and comparison in one might be a code length optimization, but I raise you this: how else would you do it? If you were to just compare fmt[i] with 0, then you would have to read from memory a second time when you then actually assign it to c.
The other way to do it I guess, would be to not have a condition in the for loop, assign fmt[i] to c in the body of the loop, and then conditionally break out of the loop right after.
Tbh, I would've done this loop exactly like they did, it's the only way to actually do it in the loop condition, and to do it with only one memory read. But maybe I'm missing something.
You're assuming that the compiler wouldn't be able to pick up on the multiple reads and optimise them away.
I put my money where my mouth is, and recompiled the code with the assignment out of line, and a separate check for null in the format array:
for(i = 0; fmt[i] != 0; i++){
c = fmt[i] & 0xff;
vs
for(i = 0; (c = fmt[i] & 0xff) != 0; i++){
The assembly generated is exactly the same:
800005d0: 000a4503 lbu a0,0(s4)
800005d4: 14050f63 beqz a0,80000732
800005d8: 4981 li s3,0
In other words, in both cases, the memory read for fmt[i] is performed and stuffed into a0, and then compared. No need for two loads because the value is already in a0.
@@LowByteProductions thanks for the reply! Yeah that makes sense. I personally am rarely a fan of duplicated code like this, I think it's [the assignment-expression] one of those things that look confusing at first, but the more you see it the more normal it becomes. At least to me it's relatively obvious what it does
In addition, you see this pattern not only often in code in general, but especially when iterating over null terminated strings. At some point you will just instantly recognize it as a very common pattern, that because of its simplicity does not require more verbosity. But I guess that's a taste thing
Yeah it's a fair point, and indeed a matter of taste. Personally I try to avoid anything that does more than one thing at once, or that might be easily missed at a glance, though i wouldn't really even put this in those categories. And I certainly make use of prefix and postfix in assignment, which is essentially the same thing as well.
Still, probably not a line of code I would write myself haha
Hi, why is the printf buffer actually 16 bytes long? this leaves space for 15 digits, right?, but this is an 64bit system, so is int not ranging from -9.223.372.036.854.775.808 to 9.223.372.036.854.775.807? Overflowing the buffer?
It is a 64 bit system, but the sizeof(int) is actually 32 bits
So do you say that when writing to the console, the WHOLE system (not only the process) freezes and waits for the character to be sent? This is not how I experienced writing to the console ... what do I get wrong?
The system blocks when using printf in the *kernel*. In userspace, another set of console functions are used, which make use of the uart interrupt mechanism. Basically when a process writes, it claims the uart, and goes to sleep until the transfer is complete. Other processes can still run during this time.
@@LowByteProductions Ah okay. So these other console functions have the same name (is this a good idea?) but are from a different header to not get confused? Thanks for answering btw!
the only thing i didnt understand is on 7:40, the consoleread and consolewrite functions. what are they? if you explain them i dont remember 😅
I didn't explain them in detail yet, only as how they fit in to the idea of a device abstractly. We will cover them in depth at some point in the future
@@LowByteProductions awesome, thank you
Wouldn’t printint break for outputting a base 2 representation? I mean it’s not supported by printf but why not just reserve 32 bytes (or 33 for a -, not sure if that’s needed) for the buffer and not worry about it?
It would break in terms of buffer size. I suppose they didn't make the buffer 32 because it simply wasn't necessary. In C, there isn't a binary format string.
@@LowByteProductions makes sense. I think I’ll implement that for fun, as you suggested. Thanks for these Code dives, they are great! I’m a CS and engineering student and haven’t really written code in quite a long time. This is perfect to get into it again, especially the low level stuff that is needed for my studies :) Keep up the great work, I can’t stop watching!
With the uartputc_sync spinning the core on waiting to output to the UART interface ... Doesn't that kinda mean that the whole system could be slowed down significantly by the UART interface speed. That entire core can't do anything while it's waiting for the UART's LSR_TX_IDLE register to change to a ready state. The lower I go, the more I wonder how any of this ever works in a constant manner to the point where the illusion of real time is maintained.
To be fair, only kernel printf spins, and kernel printf is only there for debugging and panic messages, so there isn't a lot at stake. In userspace, all reads and writes to the console are interrupt driven, and a process can be put to sleep while waiting for state to change, with another process doing useful work in between
I would just ask if you could zoom just a little so that people like me (with glasses) can see the code while you're going through it
Maybe i'm wrong, but i think, the buffer in the printint should be defined as "char buf[11]".
It's enough for 32 bit integer number with additional sign and we don't need additional byte for null terminate zero at the end in this particular case.
For 64 bits integers, space for 16 bytes isn't enough anyway, should be "char buf[21]" in this case (or 20 i'm not sure). It doesn't make sense.
As far as I can tell I think you're right. I'm not sure why they made it 16 bytes!
The function isn't specific to base 10. Wouldn't the buffer be too small if you'd print a large number in base2?
@@biohazard5837 For base set to 2, the buffer is too small even now. Moreover base is hardcoded to 10 and 16 in the code. I suppose, author had a plan to support base equal 8 or he just copy paste this code (because it's really smart).
Could it just be for alignment purposes? Keeping the memory the smallest power of 2 which can satisfy the supported bases.
You might've just spotted a bug, it appears the code's setup to handle base 16 or less. If an integer on that system's 32 bit and printint is called with base 2, there would be problems, might need a longer character buffer, not less.
Nevermind, the function's marked static, it's only used inside their printf function.
you explain why 'volatile' is needed and give the example of calling that interupt off register write twice (pasting it in). in fact the code is already there that shows the need. At exit the same reg is written to reenable registers. A compiler would see x= 0, then later x = 42 (or whatever) and seeing that x is not used in between it would simply ignore the first write. BTW , excellent series, for me its slightly too slow. Gonna take about 3 months to get to a shell prompt :-). You didnt explain the boot process, ie how we get from a powered on processor to something thats running our code, you first episode starts with 'ok we have this code at 0x80000000'
Thanks for your job. If I may, I would prefer a little more high level approach. I.e. give an overview of the topic in the episode that dig deep into details. I tend to get lost and watch some part of the video again. I hope this may help ...
I'll try to take it into the process!
Why do they use two different methods of printing hex? %x and %p are handled differently.
You need a different algorithm for base 10, so I guess they just made it generic for hex as well. Printing pointers is always something you do in base 16, so that could be optimised, plus the fact that it's for 64 bits rather than 32.
Not entirely sure of the reasoning, just venturing guesses 😁
Content's 'ok', but I find the delivery annoying, perhaps it's my age, but the USE of AQI (Google that with Stephen Fry), and the constant 'like' usage. Grrrr
One of the most interesting things about printf(), in term of its implementation, and now that we have the varargs macros, is in that it should return the number of characters output; yours is a void function, and as you pointed out, it's cut-down; but it's trickier than it sounds. I also think that to explain variadic functions, one should walk the format string and the stack manually, in code, i.e., . . . . imho is best explained and coded first *without using* stdarg. Switched away before the end, but did you explain the different calling conventions, and how they effect variadics? For example, . . . with __fastcall?
Great video! Personally, I am less interested in the generic algorithm "guts" of something like printing integers as opposed to the earlier parts like how the UART is initialized. That is to say, I'm here to learn about the OS/kernel, not specific algorithms. So less printing algorithm and more OS detail would be my preference, thanks!
I'm certain there'll be less printing algorithms in future videos!
"void main()" :: get off my yard, kiddo!
Assuming this is referring to int main vs void main:
In userspace, the kernel loads programs, who can indicate their success or failure to the kernel with a integer value passed to the exit system call. This is typically done automatically by the C runtime using the value main returns.
In the kernel, there are two things to consider here. First, the main function doesn't return at all in the kernel, so a return value would be kind of moot in any case. But even if it did, who would this value be useful for?
@@LowByteProductions the wait functions might need it. see man wait4, for instance.