Insane Vulnerability In OpenSSH Discovered
Вставка
- Опубліковано 1 жов 2024
- Recorded live on twitch, GET IN
Article
www.qualys.com...
My Stream
/ theprimeagen
Best Way To Support Me
Become a backend engineer. Its my favorite site
boot.dev/?prom...
This is also the best way to support me is to support yourself becoming a better backend engineer.
MY MAIN YT CHANNEL: Has well edited engineering videos
/ theprimeagen
Discord
/ discord
Have something for me to read or react to?: / theprimeagenreact
Kinesis Advantage 360: bit.ly/Prime-K...
Get production ready SQLite with Turso: turso.tech/dee...
So this is the backdoor that 10x dev was talking about?
no that was ssl
@@orhansenglish shit so it was… fuckit im leaving this comment up
@@pearshaped9116would have been so cool
@@pearshaped9116 honestly this was my first thought too
It's definitly refering to jiatan's xz backdoor,
;) but the video is shot before people found about it...
My phone started blowing up last night right as I was about to take a shower. I was sitting there checking our infra butt naked. Jokes on me, the version of OpenSSH on our servers was old enough to not be impacted.
Cant fall victim to a 0day if your codebase is like a thousand days old 🗣
@@skyr3x Prevent 0day exploit by staying -1day!
@@skyr3x Yeah, fall victim to known exploits instead 😂. Also, this one wasn't a 0 day
@@XxZeldaxXXxLinkxX you talk like someone who thinks that reverse cowgirl is the best position
@@skyr3xyou talk like a virgin who doesn't know what 0 day means.
A very rare and unfortunate combo, what a shame for you...
this dyslexic man is doing his best to read for us and you're laughing
Well acting like a clown doesn't really help him although it's understandable since the twitch audience is notoriously brain-dead and hungry for low effort entertainment
I'm not laughing. I've got bad eye sight, so I appreciate him reading articles. It is convenient.
Dyslexia: where all numbers are x+1
Always someone that needs to be offended on behalf of others.
@technolung It is like having multiple do while loops running at the same time in your mind, but they keep returning even when the condition is no longer true. As a person who is and has a kid who is dyslexic, it turns into an asset when you learn how to deal with it. You have to practice selective attention, paying more attention in your weak spots, and know that you can't always trust your eyes, which is a plus in engineering.
Curious that they mention that the code that fixed this was "accidentally removed" again and again. Knowing what we know about agents introducing backdoors intentionally, how are they so sure that this was an accident? Not saying we should immediately start pointing fingers for sure, but going to the other extreme and emphasizing it was an "accident" without knowing it to be so also seems like a dangerous assumption.
ikr
we don't. safe to presume it's deliberate until proven otherwise
@@seccentral I'm pretty sure this like the opposite of how things are expected to be handled. Like "innocent until proven guilty" and all
Hanlon’s Razor - Never attribute to malice that which is adequately explained by stupidity
should add comment saying "don't remove this line before you read CVEXXX and CVEXXX fully"
Rare case when hacking/it security really looks like it's imagined to be: reading source code for hidden vulnerabilities that can be exploited with incomprehensible dark magic.
Im just a dumb web developer and don't understand any of this
sudo apt update && sudo apt upgrade -y
@@tenten8401and reboot the machine.
@@tenten8401Wouldn't it be wild to work at a place where you could just do that?
@@kricku Sounds like managing SSH security vulnerabilities is outside of job scope then, dumb web developer doesn't have to worry about it because it's sysadmin problem :)
@@krickuand doing that while not breaking critical production systems that dont follow proper update cycles. What a fucking dream…
I just came here to say that this doesn't affect OpenBSD, the project that created and maintains OpenSSH.
OpenBSD backdooring GNU + Linux confirmed
@@firen777 I ROFLd 🤣
lol
@@firen777 based
It does affect OpenBSD but only the free version (I saw the project maintainer mention it). Do you have a source that says otherwise?
0:33 we should -thank- *PAY* OpenSSH devs for their work.
Agreed! You can also use OpenBSD to thank them as well.
The name.... is 128KB long
i'm tired boss.
6-8 hours sounds long, but if you target the attack to start overnight or on on the weekend, that's incredibly serious.
Fail2ban correctly configure should limit the risk as they would burn through a lot of ip adresse (not a raison to not upgrade)
They publish vulns they can't use.
My conspiracy theory is that vulns like these are published purely as a marketing strategy for the hacking group. It makes them visible to clients interested in buying actually useful vulns. These clients are incentivized to keep said vulns secret (cause they can keep using them and get their money's worth).
If I'm right, then vulns published in CVEs are mostly theoretical or of such poor quality that they can't be sold in the black market.
I'm a normie corporate guy who manages some websites. This is such a great channel just for late-breaking security news.
Thank you!
Yayayayayaya! I love reading the write ups
@@ThePrimeTimeagen he replied! 🤩
This is big brain territory here
'So we started reading glibc malloc code' LOOOOOOL
Malloc internals (and the internal locking mechanisms) are some advanced voodoo. My traumatic encounter with malloc internals:
A few years back I was troubleshooting what we thought was a memory leak. Turned out it wasn't a leak per se; what had happened was that a new "optimization" had been added to glibc's malloc implementation, which attempted to mitigate lock contention by creating new heap arenas whenever two threads collided on a lock. The idea being that threads which did a lot of malloc/free calls would effectively get their own dedicated heap arenas (eventually), thereby minimizing future lock contention.
Problem was, over time this would cause the number of heap arenas to asymptotically approach the number of threads. And since heap arenas were created with a certain minimum size (64MB IIRC), in a long-running application with hundreds of threads you could eventually chew up ridiculous amounts of RAM.
Mitigation involved setting an environment variable to cap the maximum number of heap arenas, and living with the (tiny) performance hit from heap lock contention.
If you don't mind me asking, how did you even go about debugging that ?
@@filip0x0a98 Reading the glibc source code and looking at the heap data structures in the debugger.
Glibc moment
I hope to someday be able to read these high arcane runes.
@@filip0x0a98 +1
I wonder how long it took too. My simple brain would've never figured that shit out unless I've been keeping up with every library's updates which my code uses (I don't keep up with jackshit)
19:40 - actually the point is that is packet (with a final byte) is very tiny, it doesn't get segmented and then re-assembled at any point over the internet, hence delivering it is way more reliable from timing perspective, than sending a large chunky boy.
Gotta get LowLevelLearning in on this, this stuff is his bread and butter
“ssh is a joke, I know the guy who created the back door”
I’ve never felt so dumb in my entire life. This is too hard for quice-eater devs like me.
Just something to note, just checking the package version is not enough to assert the package is vulnerable, debian and ubuntu often backport patches for CVEs from later software versions, so even if you are using a supposed "vulnerable" version, if you check the package notes (and the package per se) you will see a lot of patches, especially in LTS versions.
This was what I did for one day every month in one of my first developer jobs. We would get a notice from our pci/dss audit scan that we were vulnerable to a whole load of things and I would (re-)investigate each one and send a response with links to patch notes showing we weren't actually vulnerable.
@@anewbimproves5622I've also had the task of documenting backports to address false positives in vulnerability audits. Tedious work that kept me away from actually developing code for the project.
I have no idea what any of this means. Sounds bad though
i use vim btw
What a chad
nano>
ed
too much bloat.. just use > and redirect 1s and 0s to file already
Emacs > Vim
Change my mind.
Such a shame prime doesn't pronounce ssh as "sssh"
disliked, unsubbed, reported, lost all respect
@@ChaosturnMusic Reasonable.
At least he mispronounced 'Char' as 'Char'.
@@DieDona shhh...
29:00 - they send authentication KEY - which is memcopied from packet into the memory for auth checking - this is why it's important to cause sigalarm while it's being checke, because they KEY is the malicious payload that when executed right jumps the execution pointer to "yes this dude is valid and give him shell"
On Unbuntu servers 1 line patch is: pro fix CVE-2024-6387
Google notified me of this yesterday (bc they host my VM). Went in and checked if my OpenSSH version was affected but luckily I use ancient Debian that's stable literally forever so the OpenSSH version was _older_ than the exploit. Which I believe is like more than a decade old
It just has to be more than 4 years old to not have this vulnerability. The article mentions the issue was (re)introduced in 2020.
when I connect to the VM using browser (instead cmd or putty), openSSH is used?
@@vitvitvitvitvitvitvitvit yes it should be. The VM doesn't know you're connecting from a browser, putty, or whatever, it just knows that a connection has been made.
don't you have other CVEs to worry about in older Debian versions though? Or are you actively backport patching them?
Debian Stable is literally stable.
I’m proud how well I managed to keep up with the text, yet horrified because of the implications of this. 1 day of SSH logins is nothing its not like I actually collect the logs properly most of the time…
Meh, the amd64 version hasn't been exploited yet, and the attack relies on precise timing. It should be fixed quickly, but the vulnerability isn't a drop everything levels of bad.
This should scare you to switch to OpenBSD LMAO.
webdev doesnt know how to read C... the quality of the Netflix Staff right here...
this. is. insane. ... just wow ... the effort and analysis they must have put into this! well well well but eventually did they try turning it on and off? :>
I'm just thrilled to see so many references to one of my favorite modern ska bands! :D
Hmmm nothing like an open public port 22 🤤
At the very least, please put a IP address whitelist!
Today is the day I realized, you look like Dr. Disrespect, but without goggles
I’ve always thought he was his kid lmao
Not enough likes here
actually this is his holy twin Dr. Respect.
Dr. Not sexting minors
Wonder if something like the delay symptom they spotted in that xz backoor could be used to nail this window more consistently?
Like a minor issue in one oss giving better odds at a basically probabilistic attack on another...
isn't this only 32bit? and can be mitigated with some config? or did i miss a bunch
Sending all but the last byte of the DSA packet isn't about timing due to packet coalescence. It's about not having to wait for the network to transfer all that data in one go. If you have to transfer 4K of data, that's going to take time. Transferring everything except the last byte will take the same amount of time (more or less) for the first part, but then as you approach your window to win the race, only having to send a single byte will be a lot faster, and therefore easier to guesstimate when it should be sent.
Dang quiche eaters...
so... me running ubuntu 24.04 as my SSH gateway while all the rest of my servers run debian 12 potentially saved me ?
nice ! :>
Chuck norris reads emails through heap overflows!
i could feel my head smoking cartoonishly throughout this
Maybe more exploitable if you are already on the box unprivileged and doing an ssh back to the same box to then get root.
You sound like me reading my college philosophy text out loud.
_laughs in musl_
Musl is also possibly affected.
@@lucyinchat it was shown not to be.
common alpine W
@@lucyinchat 0:03 it says on the top "on glibc-based Linux systems" though
@@d3stinYwOw Same with the xz backdoor that effected ssh. *_musl putting backdoors for glibc confirmed_*
...in October 2020 by commit 752250c, which **"accidentally"** removed...
It could be some adjacent code was modified and an older branch got merged without the fix
@limesta I guess that's what SOC is all about? Also... no test suite?
happy little accidents 🎉
@@McZsh automation testing typically isn't designed to intentionally break things, it's to make sure things work. So for a 20 step process requiring multiple devices with very narrow timing intervals for interrupts for a single bug on some systems that you're certain you have fixed isn't viable for any company to do. What would have been viable is someone vetting line by line changes instead of just hitting a merge button all willy. I'm not going to assume malice where there has been this exact same scenario of accidentally reverting code in every company ever, all it takes is one working file that gets patched in late, and you get a regression that can go undetected
@@McZsh and for the aforementioned bug that takes researches 400 hours to verify is even real? No shot anyone will validate that every patch or set up a system to check if it's possible. This is a very convoluted process that the best of security analysts will miss, but after being fixed the first time it shouldnt have regressed and hopefully the company maintaining it is verifying their process
the fact that they are interrupting the code withing free and using quotes from "the interrupters" is funny
Not sure why but your voice at high peaks is hurting my ear
Wow, rhese new vulnerabilities making me sus if stuff online. Gotta be careful
Just use Wireguard and SSH only to wg0
TL;DR: ssh was supposed to use single-threaded but was executed as effectively multi-threaded thanks to SIGALARM being implemented incorrectly (single-threaded program should not cause any non-volatile changes to program state from SIGALARM handler).
Had all of ssh been written as multi-threaded code the SIGALARM handler would have worked as expected because it would have had to use proper locking to access shared memory structures. Of course, that would have been true only if somebody had been able to write *correct* multi-threaded code in C - that is, without any security vulnerabilities. Even Linux kernel fails this every now and then.
Human programmers are not careful enough to write security sensitive code in C except for random happy mistakes.
Update: 41:05 Yes, in other words it's re-entrant bug. Shouldn't happen in single-threaded code in theory but incorrectly written signal handlers can break those assumptions.
Not me consuming an entire bag of taki's like it's popcorn at the movies...
this is a real thinker, and ngl this vuln is alot like one I theorized about and then may have found being exploited in the wild, first on windows then a few months later also on a few devices running a few different android releases.
All the android devices observed had outdated linux kernels (from 2017-2018 yet in phones made in 2022-2024)
I dont even wanna ask why some oems do it, but just please stop using old linux kernels with deprecated or known unsafe features!!!
CAN WE AS A COMMUNITY AGREE ON THAT
anyways... here's something interesting for us all to ponder upon and also wonder how TF together
(btw just think ocr style grid-array encoding but used on streamed-in frames and you''ll get where im going in this comment)
A Short Essay on Unsafe Decoding and Parsing Algorithms
"Why we Need More Intelligent Memory Filtering to Combat Address Space Grooming"
I found a really cool (i.e. SCARY) way to hide a header chunk, where hexadecimal will get read out as a series of blank space, when read as any plaint text file, even though the series of two byte values each individually do show up when read by a hex text reader app. You wanna know how stuff is getting smuggled into being used for supply chain attacks even after an org has done their job and securely restored their data, there ya go. You need to be inspecting headers and footers for all the things, and start logging possible autonomously triggered instances of 'head' and 'tail' commands.
I'd share my theory in full but some of the concepts are fringe at best, so idk who really would take it totally seriously. But suffice to say, if anyone has seen the distorted psychedelic coloring on some youtube videos, while using an ARM based android device (v6 or v7) then you at least can reproduce this bug if it affects your config, and maybe you might be able to confirm.
Android versions I can confirm have the behavior I referenced above: Android 11, 12
Android versions I cant confirm or have not witnessed the referenced behavior: Android 9, 10, 13
My hypothesis and/or ideas going forward:
Yall I think something is straight up introduced in Android 11, and fixed/changed at Android 13 release that either knowingly or inadvertently blocked the yt video coloring issue, but I think maybe the effects of whatever those patterns are, may be still there and just better hidden maybe even completely on accident.
I dont get nearly the amount of views that would necessitate making a video on this but I have been steadily gathering info about the issue for quite a while, ever since I found a memory leak issue which seemed to only plague devices connecting to a certain Sagemcom router. I have a hunch that its something similar to the recent Windstream Isp issued router vulns, which may have been the initial vector at which the issue started at least in my personal observations. (my router was NOT a windstream product, but it WAS a router from one of the US's big three carriers)
If you are a programmer or another researcher, and see this comment... PLEASE look into it if you have the ability.
*Chuckles* I'm in danger!
I think the fix they mention does not solve some root causes for these user-injected code. For example, first thing you do when you receive that username should be checking whether it is valid UTF-8 (in your unprivileged child), the only "names" that make sense. This removes lots of possibilities to include binary code because they are usually not UTF-8.
This is one way Rust "could have" prevented these issues, btw, but more in a "common practice" way than a "C cannot do this" way. Of course you can do these in C.
You're bad at reading c because they give the best variable names :).
I tried to update on Windows 11 to the latest but then I can not connect with HeidiSQL any more. So, I had to return back to OpenSSH_for_Windows_8.6p1, LibreSSL 3.4.3. I am using only OpenSSH client and I think it is not so dangerous to keep this version.
I think youtubers are hyping this vulnerability up. One would be vulnerable if one would do stupid things such as exposing ssh without rate limiting the connections or allow only trusted IP. Rate limiting is already a common practice because of Hail Mary attack on ssh ports. If you enable rate limit and use latest stable version you should be OK.
The calloc vs malloc in the PAM handler. Gottem.
"Hit by an STD" I thought you were trying to spin something awkward
48:07 "I like how they use word 'easily'" - I agree. It would be interesting to hear what kind of task the authors would call "barely controllable" or "nearly impossible to control".
I think you mentioned solar designer without realizing who it is. The guy is an absolute beast and author of John the Ripper, a famous password brute forcer.
I like the part where you go into the statistics of how to hit the race condition. Thats the boring math parts most will just ignore ^^.
it's because most people erroneously think they hate math because they had horrible math teachers at school
52:32 So, when you get to 52nd minute, you understand that this wasn't that bad at all.
This condition is probably present in many softwares.
Debian 13: dpkg -l 'openssh*'
ii openssh-client 1:9.7p1-7 amd64
What exactly is this video about?
Just rewrite it in rust...
my windows 11 has open ssh 8.6p1 💀💀💀 though 9.6p1 on ubuntu 24.04
if this was introduced by removing a define, why did it take several months and collaboration for a fix?
When someone comes to me saying they want to become a "Hacker", This will be the video I point them to. 😄
FWIW the "Malleus Maleficarum" was a book written in the 1400s and served as the Catholic Church's justification for witch burning/killing. That's where the "Malloc Maleficarum" is coming from.
I'm under the impression that we should replace all mallocs with callocs, just to be safe. It'll be slower, but a lot harder to exploit
I am a C/C++ programmer myself, but now I have just been heavily brainfckd!
sudo apt upgrade openssh-server "I get back" OpenSSH CVE-2024-6387 has been fixed for 22.04 LTS, 23.10 and 24.04 LTS. Now I am like do I need to go to all the other servers in my stack or are we good. I guess I will just to err on the side of caution.
I love that we talk about avoiding race conditions, but they talk about /winning/ them
openssh-{client,server} 8.9p1-3ubuntu0.10, 9.3p1-1ubuntu3.6, and 9.6p1-3ubuntu13.3 fix this vulnerability in Ubuntu 22.04, 23.10, and 24.04 respectively.
With respect to the one vs two packet delay time, in TCP, if you enable nagle on the server and delay ack is always enabled on Windows then the delay will be about 200ms longer in the 2 packet scenario. God help you if you get this wrong because an entire team of engineers couldn't figure this out for over a decade until I showed up and caught it.
59:40 if they just wrote the thing in JavaScript then garbage collection would prevent exploit... :D so to make C code safe we can add garbage collection at random moments LOL or make fake garbage collection!
someone in chat suggesting rewrite ssh in JS XD I furst thought too, but we should not rewrite everything in JS right? :D
I just wanna add that you generally should not report security vunerabilities directly to whoever made the software, unless you have a written contract to do so. You will get raided by FBI and have all your devices confiscated.
I sell it to the Russians
The Interrupters mentioned! "She's Kerosene" randomly plays in my head constantly. I had to focus on the music because I'm too dumb to understand the tech parts of this 😂
I was going to make a remark about OpenBSD reducing their vulnerability clock but apparently OpenBSD survived this.
Does not affect our 1000+ systems. My own laptop was patched a few days ago.
oh Jia Tan again?
thread safe and async signal safe are not the same thing. your mutex won't stop a signal
£ me too, Prime!
Should have used rust
It affects 32 bit only... most people are on 64bit... for years now..
Dang Woody was unstable when I installed debian for the first time
woody appears to be a stable build - looks like version 3.x & releases around 2002-2006
"when you're reading malloc, you're getting DEEP" 😂❤❤❤
The time you publish your video, it's already obsolete.
Always use port knocking in conjunction with SSH.
Spreading rumor for publicity wasting ppl time.
Interrupters mentioned, never thought I'd see this day
This is too much for my washed up php dev brain 😅
"I guarrantee there's at least like... 7 zero days in the linux kernel"
When have I ever heard that before? Well anyways, it's now painfully obvious it's much more than 7.
That's fine imo. There's probably much more on windows, but it's hard for us to know since we don't have the code.
This is one of those vulnerabilities where the more I _think_ I know, the more I know I know nothing.
If this came up in 2006, and freebsd wasn't vulnerable because it had thread safe code for syslog, it makes me wonder why linux/glibc didn't get thread safe syslog
because GNU and linux is full of backdoors.
Love this kind of videos about vulnerabilities. Keep up, Prime! o7
What did I get from this? Magical magic is magically magic. HURRAY!
Linux and C are not having a good time these days.
The title of this one should have been "Prime learns about the craziness it InfoSec exploits".
And he does it superbly!
My Ubuntu VPS doesn't have the patched version available so I just enabled fail2ban. It should mitigate the issue.
or lock the SSH port down. Then OpenVPN or WireGuard
Now I really want to see Prime and Casey do some exploit development.
Seems like this is only for DSA keys? Does any one use those in 2024?