51:30 `mlock`sounds interesting, but from my understanding, it only prevents pages from being swapped out. That's what I gathered from a quick look at the manpages. Could you explain what you meant or provide a source for further clarification?
@@James-uw4it Pinning this because you are right, mlock() is only to keep the memory from being swapped out of main memory and not to protect your addresses... My mistake! Thank you for calling it out!
I hate that the internwt works this way. You get only shown stuff by people who know how seo works, and not by people who know what they are talking about
Zero Copy means the cpu doesn't cp the data from memory to memory trashing the data cache of the cpu. Not trashing the cache has a performance benefit. It's very hard to do because usually you do a read() write() construction from user space. This is even a double trash, because it goes from memory to kernel cpu cache to user cpu cache then back to kernel cache, and then to the other device.
Not only that, but copying data from one part to another (from one buffer to another), even if we had some cache skipping feature (they do exist but not everywhere), copying is actually pretty slow by it self, and depending on copy size, the CPU will often have to resort to doing it by itself (as accelerators usually have startup overhead, and if not in the CPU, uncoupled speed, needing synchronization), and we don't need that added latency in the kernel, for small things like packets usually are.
Probably the best video explaining low level kernel stuff on UA-cam, thar even non-coders like me could understand everything clearly! Amazing man! Congrats, subscribed! 👏🤝
After waking up I usually start the first video UA-cam recommends to me to have some background noise, but this really is a gem, will watch again, thank you!
Mutex for mutually exclusive. Meaning that only one process can run or modify something. If two processes try to increment a memory location at the same time, synchronization needs to occur to ensure proper results.
34:25 - I really like the developers’ approach - “Let’s do the important thing now, and leave everything else for the future” - that’s why they decided to implement IPv4 first, not IPv6))
Some more detail on TLB, it is cache that is set aside for holding a virtual page id, and the "translation" to its actual hardware counterpart. I'm sure there are sentient values like if the data is actually in swap or possibly a register, but in general it is a good old fashion look up table. The piece I wanted to add is that it is simply a cache for the virtual memory to hardware translations, and the fact that programs actually deal in virtual memory, and ask the OS for the real stuff, which necessitates things like the TLB for everything not to be terribly slow.
Also, TLBs are usually implemented to do the lookup _in_ _hardware_ - not microcode, but a truly parallel implemented in silicon; i.e. you put in a virtual address and the physical address pops out _immediately_ . Hence why the TLB has a very limited size, since it isn't just another (dumb) cache but requires quite a lot of circuitry; e.g. in Zen2 the TLB has 2k entries (i.e. it can hold addresses for just 8MB of memory with the standard page size). Now, if we use the example from the video with 250k pages, it should be clear that this would make the TLB completely useless, as basically any access to that memory region would be a TLB miss, which causes an interrupt, that then needs to be handled by the kernel to put the missing entry from the process's page table into the TLB - so we just turned an immediate lookup into: context switch, execute handler code, lookup some more memory, context switch... which is obviously several orders of magnitude slower.
Good video, though you're off by one at 9:56. The AES-XTS speed boost uses the VAES and VPCLMULQDQ extensions, first used by AMD in Zen 3, not Zen 2. (Intel introduced them in Ice Lake, during which time it was wrongly thought to specifically be a new part of AVX-512.)
I can not imagine how informative this video for all if tou kernel developers, that even me as a casual user enjoy watching it to the end. I love linux, I use it for gaming all the time, and i'm so exciting for what's coming next. About that NTSync, idk exactly what it is, but it sounds good for later gaming experience. Great video..
Feels kind of crazy that we're getting hacks to improve Wine's performance in games, but it all makes sense -- it's a bigger use case than ever, thanks in no small part to Valve's efforts. And the pipeline from games to regular desktop usage is very real. Heck, Linux VMs running Windows games for streaming services don't feel like a particularly crazy idea any more.
7:44 an architecture from 2013 is new, but it may possibly be dying out already. even 64-bit x86 is 20 years old and the various 32-bit architectures are usually over 30 years old
X32 is not necessarily a 64-bit system faking a 32-bit system. AFAIK it is x86_64, but with 32-bit pointers and long integers (in C). Because the pointers are 32-bit, an x32 program cannot access more then 4 GiB of memory (or was it 3 GiB?). If a program should never use that much memory anyway, then x32 wouldn't hurt.
Please, add timestamps to your videos. Video is an hour long, description is nonexistent... Im not gonna waste my time on 100500 explanations and 1% real info
@@WololoWololo2 yeah its a massive waste of time. Unless you're a bisector or maintainer how is stuff automatically enabling itself is any usefull being acknowledged. Ah right you gotto enable them by hand because loonix is maintained by bunch of hippies who thought of genius ideas like enforcing vsync due to their "philosophy". Although I have to give credit where its due because defaults were garbage way back
TLB, translation look aside buffer. Just content addressable memory (cache). Flushing it is fast. It is reloading it that is painful, as you actually have to get the data from memory.
ntsync isn't going to be functional in 6.10, it's missing patches that'll be in 6.11. 20:30 performance boost is good but guess I'm one of those people who otherwise don't know what you're talking about, existence of ntsync driver isn't going to make porting something to linux any easier. as a dev you can either use wine/proton like before or make a proper port that is still system specific, existence of ntsync driver doesn't change that (in windows the syscall is NtCreateSemaphore() no such thing will be provided by linux kernel, instead there's a char device that wine makes an ioctl call to when NtCreateSemaphore() is called in wine)
I really hate Hans Reiser for "stepping out of the community" the way he did. In the 2.4 series, reiserfs 3.6 was the best and fastest filesystem that we could get. Due to his "personal project" that made him land in prison, reiserfs 4 didn't get the care it needed and turned out to be unstable crap, like jfs and xfs turned out to be. It took a lot of time before ext4 had the same possibilities as reiser had. And as a side note: though you did not understand everything correctly, it was a nice change for me, to just put up a nice voice that tried to explain what the kernel changes are while I am working. So don't stop with that. It's brought nicely, and you will learn more details in time.
that's the common name pronunciation for most other people afaik. I vaguely remember him commenting on adopting the "lin us" pronunciation years and years ago but idk. anything is better than calling Linux "loonucks" haha
Using a phonegram, It's probably closer to Lee-Lus, his kennel is Lee-nux.... The ftp server admins that (re)named the kennel way back in the 1999s used the name Linux arbitrarily as a contraction of whatever Linus had named it... I forget the original name.
Great video but can you make it a little more techincally and less dumbed down next time? For example when you talked about the TLB I think it would have been much better to just say that it caches virtual addresses instead of being so vague. I think understanding the absolute basics like virtual memory should be a prerequisite for this kind of video
I'm admittedly unfamiliar with the kernel under the hood, but zero-copy operations and io_uring sound mutually exclusive. Aren't ring buffers handled by the kernel and by extension the CPU? How can using them constitute a zero-copy CPU-less operation? Am I missing something?
anybody knows about John L. Gustafson's numbers format: Posits, or Unums, better than Reals!! Could Linux do it, or does it need advanced Hardware? Can do better than 64bit Floats with only 32bit_width Unums/Posits. Symetrical mapping from 0 to infinite. Needs only one: Not_a_Num_exception!
I agree that Posits and Unoms give more precision for less bit, but they are exponentially harder to compute thus useless in the linux kernel. Thoses implementations have their place in specialized software (;
Variables in Rust are immutable by default. Would it be possible for the rust compiler to use mseal on Linux to seal those variables? The kernel could prevent anyone modify the immutable (and thus sealed) variables.
i don't think they apply immutability at the same level... how your compiler is supposed to use data and how you os will manage its memory are probably 2 worlds away from each other. i have no idea how both work under the hood soooo who i'm i to speak on it (;
How NTSYNC faster than userspace? If mutex stays in userspace, you don't need to switch context to kernelspace all the time. As I've heard, switching to kernel context is slow.
I don’t know the details, but it’s likely that WINE has to call into kernel space already to emulate NT mutexes. It still needs to synchronise threads, but without NT style synchronisation primitives it would have had to wrap linuxes own ones in user space code to correctly implement NTs behaviour, likely resulting in multiple context switches for an action that should only take one
how would you go about porting a linux 5.11 code for a pecific device (open source and arm 32) to kernel 6.10? i've tried it but after dealing with compiler errors (by disabling affected modules, the problems were mostly spinlock -> raw_spinlock and spi_master -> spi_controller stuff) it compiles but doesnt boot. The device is a 3ds btw
I have the feeling the 3ds probably need some non-standard module... but I would try compiling and replacing the 5.11 by another 5.11 that you compiled. Once you know the right option to make it work, you can work your way up the versions.
@@Maple-Circuit i dont tbink you understand this/i sucked at explaining. Somone made a functional 3ds port of 5.11,and its open source with all driver and stuff, but i cant upgrade it because after resolbing some compiler errors it just blackscreens and also dont have a way if debugging it (yet)
@@ItsCOMMANDer_ maybe i still don't understand but the reason for compiling with the original version is to see if the commands and config you are using are good. You are dealing with a 3ds which is both really cool and insane at the same time (; being that i have never even compile the kernel for an arm64 device, i'm probably less knowledgeable than you on this (;
29:46 I've worked on a few rust projects and I can think of one example, but it's a general issue that rust development is full of cases where missing features in the rust compiler lead to project""s"" needing 10,000 lines of code. For example async corutines... Rust has corutines, functions that return more than once, and having a stream of data generated by an async block happens all the time and the number of lines the planet has to deal with because this feature missing is massive. I object to the Linux Kernel team getting priority!!! It may seem like they are more important, but that's fallacious. How popular a project is has no effect on how crucial it is for support from the rust compiler team.
51:30 `mlock`sounds interesting, but from my understanding, it only prevents pages from being swapped out. That's what I gathered from a quick look at the manpages. Could you explain what you meant or provide a source for further clarification?
@@James-uw4it Pinning this because you are right, mlock() is only to keep the memory from being swapped out of main memory and not to protect your addresses... My mistake! Thank you for calling it out!
Finally youtube is recommending me good channels. this is a hidden gem
@@AGentooUser thanks(;
Oh, another Gentoo user in the wild
A gentoo user!
gemtoo albeit
I too use Gentoo btw
in video tittle add "linux" keyword. example "Linux Kernel 6.10 | Locked & Optimized" for better seo
I hate that the internwt works this way.
You get only shown stuff by people who know how seo works, and not by people who know what they are talking about
Honestly not even seo itd just be a bit more clear
@@vappyreon1176 i agree on that, i was really tired and just wanted to rant lol
Locked In
RIP RiserFS, and RIP Han’s dead wife
Lol the your user image fits XD
UA-cam recommendations, finally being good for once! Awesome content.
You explain things very easely, your videos are really nice
cannot believe i am finally leveling up as geek and am watching linux kernel update video xD
LVL UP++
i don't even use linux and i still watched it
Zero Copy means the cpu doesn't cp the data from memory to memory trashing the data cache of the cpu. Not trashing the cache has a performance benefit. It's very hard to do because usually you do a read() write() construction from user space. This is even a double trash, because it goes from memory to kernel cpu cache to user cpu cache then back to kernel cache, and then to the other device.
Not only that, but copying data from one part to another (from one buffer to another), even if we had some cache skipping feature (they do exist but not everywhere), copying is actually pretty slow by it self, and depending on copy size, the CPU will often have to resort to doing it by itself (as accelerators usually have startup overhead, and if not in the CPU, uncoupled speed, needing synchronization), and we don't need that added latency in the kernel, for small things like packets usually are.
Thanks to both of you, now I know(;
its insecure though.
@@Maple-Circuit Think the original code, back in the days copied that data 3+ times.
wow, it's clear that you put a lot of effort into this, tyvm for the information rich video :d
Another gem channel. Congrats on getting onto the algorithm. Subbed.
Probably the best video explaining low level kernel stuff on UA-cam, thar even non-coders like me could understand everything clearly! Amazing man! Congrats, subscribed! 👏🤝
2 minutes in, this is the most useful channel I have ever found
Incredible production quality. Subbed!
After waking up I usually start the first video UA-cam recommends to me to have some background noise, but this really is a gem, will watch again, thank you!
I subbed based on this video. This is really great and I hope you continue to make this kinda stuff
Intel MIPI fix... Finally! This has been the bane of my existence at work recently 😭
as a developer I am glad for the mutex lock change. Something like that really is awesome for compatibility
Insane effort, thanks for the great long-format video
Finaly youtube algorithm showing some interesting video. Great work
Thx!
god sent this man directly from the algoheavens to inform us about this beautiful place called the kernel. what a gem!
just an fyi risc-v is pronounced "risk five", and clang is pronounced "klang"
Yes, my brain died when I said clang.. but thanks for risc-v, I didn't know
Genuinely, thanks - I've always said riskv 😂
you're right, but imo it sounds way cooler as "risk-vi" and will probably remain like that in my head while reading things.
@@Maple-Circuit Do we call С programming language K?
this content is great, please make more of this!
This is amazing! So glad I ran across your channel!
The shadow stack is just a way to split your stack into a data stack and a call stack.
Wow, a highly technical video that is actually interesting to watch! You're a genius, instantly subscribed
Mutex for mutually exclusive. Meaning that only one process can run or modify something. If two processes try to increment a memory location at the same time, synchronization needs to occur to ensure proper results.
34:25 - I really like the developers’ approach - “Let’s do the important thing now, and leave everything else for the future” - that’s why they decided to implement IPv4 first, not IPv6))
100% accurate, pragmatism is the best policy
Some more detail on TLB, it is cache that is set aside for holding a virtual page id, and the "translation" to its actual hardware counterpart. I'm sure there are sentient values like if the data is actually in swap or possibly a register, but in general it is a good old fashion look up table.
The piece I wanted to add is that it is simply a cache for the virtual memory to hardware translations, and the fact that programs actually deal in virtual memory, and ask the OS for the real stuff, which necessitates things like the TLB for everything not to be terribly slow.
Also, TLBs are usually implemented to do the lookup _in_ _hardware_ - not microcode, but a truly parallel implemented in silicon; i.e. you put in a virtual address and the physical address pops out _immediately_ .
Hence why the TLB has a very limited size, since it isn't just another (dumb) cache but requires quite a lot of circuitry; e.g. in Zen2 the TLB has 2k entries (i.e. it can hold addresses for just 8MB of memory with the standard page size).
Now, if we use the example from the video with 250k pages, it should be clear that this would make the TLB completely useless, as basically any access to that memory region would be a TLB miss, which causes an interrupt, that then needs to be handled by the kernel to put the missing entry from the process's page table into the TLB - so we just turned an immediate lookup into: context switch, execute handler code, lookup some more memory, context switch... which is obviously several orders of magnitude slower.
Commited Linux from prison, lol
I am still waiting for the Kernel 6.66 so I can rip and tear in style.
awesome video! would love some timestamps in the comments/description so i can skip around to what i find most interesting pls :)
Nice video. Thanks. Would love benchmark comparisons tho. 😇😁
i dunno how did i subscribe, but im not going to complain
Good video, though you're off by one at 9:56. The AES-XTS speed boost uses the VAES and VPCLMULQDQ extensions, first used by AMD in Zen 3, not Zen 2. (Intel introduced them in Ice Lake, during which time it was wrongly thought to specifically be a new part of AVX-512.)
Awesome rundown. Subscribed
I can not imagine how informative this video for all if tou kernel developers, that even me as a casual user enjoy watching it to the end. I love linux, I use it for gaming all the time, and i'm so exciting for what's coming next. About that NTSync, idk exactly what it is, but it sounds good for later gaming experience. Great video..
Feels kind of crazy that we're getting hacks to improve Wine's performance in games, but it all makes sense -- it's a bigger use case than ever, thanks in no small part to Valve's efforts.
And the pipeline from games to regular desktop usage is very real. Heck, Linux VMs running Windows games for streaming services don't feel like a particularly crazy idea any more.
okay how is this channel so good an why am I seeing it just now?
I was wondering why the CC said linux konel when I hover my mouse over the thumbnail, so I clicked. Well played sir
Lol, my accent sells me well even when I'm not trying XD
7:44 an architecture from 2013 is new, but it may possibly be dying out already. even 64-bit x86 is 20 years old and the various 32-bit architectures are usually over 30 years old
Well said!
X32 is not necessarily a 64-bit system faking a 32-bit system. AFAIK it is x86_64, but with 32-bit pointers and long integers (in C). Because the pointers are 32-bit, an x32 program cannot access more then 4 GiB of memory (or was it 3 GiB?). If a program should never use that much memory anyway, then x32 wouldn't hurt.
True! Thank you for the correction!
Thank you. Everyone seems to get x32 wrong. Which is sad because I think it is pretty cool for cache performance
Wine is now an emulator.
Great video!
amazing presentation , really interesting
Please, add timestamps to your videos. Video is an hour long, description is nonexistent... Im not gonna waste my time on 100500 explanations and 1% real info
Why are you watching Linux videos then?
Read the changelog then genius. If that's too verbose for you, try the commit messages rofl.
@@WololoWololo2 yeah its a massive waste of time. Unless you're a bisector or maintainer how is stuff automatically enabling itself is any usefull being acknowledged. Ah right you gotto enable them by hand because loonix is maintained by bunch of hippies who thought of genius ideas like enforcing vsync due to their "philosophy".
Although I have to give credit where its due because defaults were garbage way back
I'm reminded that watching loonix videos are a massive waste of time
Uuoh amazing video! Such enthusiasm! Amazing :3
Awesome video, thanks
TLB, translation look aside buffer. Just content addressable memory (cache). Flushing it is fast. It is reloading it that is painful, as you actually have to get the data from memory.
TLB shootdown is slow (at least on x86), I'm unaware for how it works for ARM but would imagine it to be similar.
You should make this into a searies ahha, nice work
Will do!
9:00 I have been reading RISC-V as RISC-FIVE all this time.
I had a few comments telling me that I'm wrong on that, Risc-5 makes a lot of sense too (;
Nice video. Im gonna check more of your videos.
It would be nice if you add timestamps to easily skip stuff we dont care about
lol brutal..
but I agree.
Oh, thank you for explaining everything!
great job. I approve.
ntsync isn't going to be functional in 6.10, it's missing patches that'll be in 6.11. 20:30 performance boost is good but guess I'm one of those people who otherwise don't know what you're talking about, existence of ntsync driver isn't going to make porting something to linux any easier.
as a dev you can either use wine/proton like before or make a proper port that is still system specific, existence of ntsync driver doesn't change that (in windows the syscall is NtCreateSemaphore() no such thing will be provided by linux kernel, instead there's a char device that wine makes an ioctl call to when NtCreateSemaphore() is called in wine)
I really hate Hans Reiser for "stepping out of the community" the way he did. In the 2.4 series, reiserfs 3.6 was the best and fastest filesystem that we could get. Due to his "personal project" that made him land in prison, reiserfs 4 didn't get the care it needed and turned out to be unstable crap, like jfs and xfs turned out to be. It took a lot of time before ext4 had the same possibilities as reiser had.
And as a side note: though you did not understand everything correctly, it was a nice change for me, to just put up a nice voice that tried to explain what the kernel changes are while I am working. So don't stop with that. It's brought nicely, and you will learn more details in time.
Thank you for the feedback, I won't stop!
Fantastic video, Google really knows my niche eh
MurderFS still the best
damn, this channel is a good find.
Nice vdo
informational, thank you, keep up with it
Why do people say line-us when it's linus by that logic linux would be line-ux.
That's exactly how people in my country pronounce Linux. Drives me crazy. Line ux 😂
Linus made it, so he gets to call it whatever he wants 😁
that's the common name pronunciation for most other people afaik. I vaguely remember him commenting on adopting the "lin us" pronunciation years and years ago but idk. anything is better than calling Linux "loonucks" haha
Originally it is LeeNux and I would like to interject a moment, it is called Gahnew/ LeeNox or now Gahnooh Plluss LeeNox
Using a phonegram, It's probably closer to Lee-Lus, his kennel is Lee-nux.... The ftp server admins that (re)named the kennel way back in the 1999s used the name Linux arbitrarily as a contraction of whatever Linus had named it... I forget the original name.
Duplicating the performances of arm so great
Very well explained!
can you add timestamps?
Nfsdctl looks cool
Amazing info
Hi i like Linux i hope Linux grow
Subbed, great stuuf.
THE GOATTT
and this is great content
Yay, more free stuff!
I wonder if shadow stacking could be used to help anti cheats
Great video but can you make it a little more techincally and less dumbed down next time? For example when you talked about the TLB I think it would have been much better to just say that it caches virtual addresses instead of being so vague. I think understanding the absolute basics like virtual memory should be a prerequisite for this kind of video
Time stamp would be great
very gud 👍
Gentoo MasterRace BTW!
I installed this and went from 12 threads to 8 threads. I'm not sure what config caused it
09:00 When will the kernel have Zig support? I personally think Zig would have been a better fit for the kernel than Rust.
Soon as you add it. the community can add stuff so go on.. 😆
i'm asking before i watch the video so i remember,
is drm kernel panic implemented?
Lol
I'm admittedly unfamiliar with the kernel under the hood, but zero-copy operations and io_uring sound mutually exclusive. Aren't ring buffers handled by the kernel and by extension the CPU? How can using them constitute a zero-copy CPU-less operation? Am I missing something?
Ring buffers contain the pointer to the data (the buffer), not the data itself (;
anybody knows about John L. Gustafson's numbers format: Posits, or Unums, better than Reals!! Could Linux do it, or does it need advanced Hardware?
Can do better than 64bit Floats with only 32bit_width Unums/Posits. Symetrical mapping from 0 to infinite. Needs only one: Not_a_Num_exception!
I agree that Posits and Unoms give more precision for less bit, but they are exponentially harder to compute thus useless in the linux kernel. Thoses implementations have their place in specialized software (;
Timestamp would be helpful
Noice!
algo comment
Variables in Rust are immutable by default. Would it be possible for the rust compiler to use mseal on Linux to seal those variables? The kernel could prevent anyone modify the immutable (and thus sealed) variables.
i don't think they apply immutability at the same level... how your compiler is supposed to use data and how you os will manage its memory are probably 2 worlds away from each other. i have no idea how both work under the hood soooo who i'm i to speak on it (;
💖💖💖💖
You're the first person I hear pronouncing clang as c-lang
@CamaradaArdi lol french canadian brain strikes again (; I normally say clang but ... why not XD
c-lang makes more sense than clang
When i say clang, i say klang klang like im tapping a pipe
Prison, ReiserFS all over again hahahahkek3krke😢😮😅😅😊😊😊😂 0:46
How NTSYNC faster than userspace? If mutex stays in userspace, you don't need to switch context to kernelspace all the time. As I've heard, switching to kernel context is slow.
I don’t know the details, but it’s likely that WINE has to call into kernel space already to emulate NT mutexes. It still needs to synchronise threads, but without NT style synchronisation primitives it would have had to wrap linuxes own ones in user space code to correctly implement NTs behaviour, likely resulting in multiple context switches for an action that should only take one
How does cfi deal with ebpf tracing well probes? Arent these functions not in thr jump table maybe?!? 6:56
Good question, but I couldn't answer. CFI is like magic to me (;
how would you go about porting a linux 5.11 code for a pecific device (open source and arm 32) to kernel 6.10? i've tried it but after dealing with compiler errors (by disabling affected modules, the problems were mostly spinlock -> raw_spinlock and spi_master -> spi_controller stuff) it compiles but doesnt boot.
The device is a 3ds btw
I have the feeling the 3ds probably need some non-standard module... but I would try compiling and replacing the 5.11 by another 5.11 that you compiled. Once you know the right option to make it work, you can work your way up the versions.
@@Maple-Circuit i dont tbink you understand this/i sucked at explaining. Somone made a functional 3ds port of 5.11,and its open source with all driver and stuff, but i cant upgrade it because after resolbing some compiler errors it just blackscreens and also dont have a way if debugging it (yet)
@@ItsCOMMANDer_ maybe i still don't understand but the reason for compiling with the original version is to see if the commands and config you are using are good. You are dealing with a 3ds which is both really cool and insane at the same time (; being that i have never even compile the kernel for an arm64 device, i'm probably less knowledgeable than you on this (;
timestamps?
29:46 I've worked on a few rust projects and I can think of one example, but it's a general issue that rust development is full of cases where missing features in the rust compiler lead to project""s"" needing 10,000 lines of code. For example async corutines... Rust has corutines, functions that return more than once, and having a stream of data generated by an async block happens all the time and the number of lines the planet has to deal with because this feature missing is massive. I object to the Linux Kernel team getting priority!!! It may seem like they are more important, but that's fallacious. How popular a project is has no effect on how crucial it is for support from the rust compiler team.
Rust is only really used on a case by case basis. Mostly it's used to fix memory safety issues with C bindings or for abandoned projects.
So.. is it faster? most thinks are over my head
yes
Lol yes
cute accent
French Canadian (;
@@Maple-CircuitJe me disais bien
hahah
rust language ever sucks
Amazing info