revealing the features of the XZ backdoor

Поділитися
Вставка
  • Опубліковано 24 гру 2024

КОМЕНТАРІ • 964

  • @LowLevelTV
    @LowLevelTV  8 місяців тому +346

    THANKS FOR WATCHING THIS WEEK HAS BEEN A ROLLERCOASTER ( come learn C at lowlevel.academy 🥺)

    • @EStartive
      @EStartive 8 місяців тому +5

      tycoon

    • @Alfred-Neuman
      @Alfred-Neuman 8 місяців тому +1

      If I understand correctly, this hack wasn't discovered by auditing any source code, right? I heard the Microsoft guy just had a weird feeling about SSH executing his command for a few too many CPU cycles and was curious enough to investigate. Does anyone knows if it could've been easy to spot this by reading the source code or if it's hidden very well?

    • @manobendro
      @manobendro 8 місяців тому

      @LowLevelLearning Please revisit 2:36

    • @joshkeegan3009
      @joshkeegan3009 8 місяців тому +1

      Are there more videos to come on the topic?

    • @moe3060
      @moe3060 8 місяців тому

      is there a specific version of liblzma.so.5 or liblzma5.4.1 ?

  • @Patashu
    @Patashu 8 місяців тому +2074

    'Binaries must be verifiably created using only the code in source control' would be a great place to start.

    • @mitchellmnr
      @mitchellmnr 8 місяців тому +226

      The problem is this attack was done during the build process.... so you need the source to build it to verify it ... but its tainted ... so .... its valid?
      yea... this one was a wild one

    • @khai96x
      @khai96x 8 місяців тому +179

      @@mitchellmnr This is what the comment is talking about. The source of the build script must be part of the git repo.

    • @mitchellmnr
      @mitchellmnr 8 місяців тому +106

      @@khai96x they are ....
      The way this attack works is when it runs tests on the source...

    • @Zettymaster
      @Zettymaster 8 місяців тому +26

      at that point, you should generate the binary on the fly. not that that would help when the build process itself is infected

    • @jsrodman
      @jsrodman 8 місяців тому +24

      It's a decent idea but what do you mean in practice?
      Do you run your build in a way that refuses to give it access to any of the test data? That's not too crazy. Is it enough?
      Do you verify that others can produce your build without any of the test data?
      How do you ensure that the build process doesn't have a side channel running in both cases introducing other logic?
      What i'm saying is this idea has to be mapped to concrete practices and tools.

  • @DavidRomigJr
    @DavidRomigJr 8 місяців тому +421

    Jeez. Virtually every part of this is done in a way to be practically undetectable at every stage and effectively roots every bleeding edge version of the biggest distros and would have seeped into stable versions of them with time. These people are diabolical, and it’s practically a miracle it was found this early before it got worse. To think, detection started with someone noticing ssh connection speeds were a hair slower. Major kudos to them for being so prudent.
    And thanks for the updates on this.

    • @draken5379
      @draken5379 8 місяців тому +36

      Pretty much. I dont think it could of even been hidden any better.
      The main reason anything was looked into, is because someone noticed their SSH login taking half a second longer lol.

    • @lucyinchat
      @lucyinchat 8 місяців тому +10

      Except arch (due to how arch doesn’t patch sshd to connect to systemd-notificationd.)

    • @skilz8098
      @skilz8098 8 місяців тому

      @@draken5379 Oh but it could be... What if your billion, no trillion $ chip - board makers are embedding hidden algorithms as such but as undetectable hardware roms that the OS, the user, etc. doesn't know about. Then with that only a handful of elite personnel coming from these giant corps as well as limited personnel within some government agencies have direct remote access into these hidden systems.
      Think of it this way; some of these newer systems you can load the entire Original Doom Source Code and Assets directly into Modern Cache and never have to go back out to RAM or to the Hard Drive. If you go back to the days of the sneakernet with the good ole floppy disks, people used to sneak malicious code into their 512 byte boot sectors.
      They can literally miniaturize a 6502, and 8086, or even a basic Pentium these days into a chip so small it would look like a little controller on the motherboard or even onto some peripheral card that one would have no idea what it is or was truly designed, intended for. This doesn't even account for modern UEFI systems compared to obsolete CMOS Bios type systems.
      And now with the advancement of A.I. and designated A.I. chips showing up... The sky's the limit. Well, skynet is the limit according to some... They can always hide it somewhere that is least expected. How about within a button on your polo shirt? You have to realize that we aren't just in the age of digital we are also in the early stages of Quantum Processing, A.I. Systems sure, but we are also on the stage of Nanotechnology. They can literally make programmable machines that are 1,000th the width of one strand of your hair. You can't even see them with your eyes.
      So if they truly want to hide something they surely can! And they can hide it just about almost anywhere they want. Even in the foods you eat and the liquids you drink!
      And this is not science fiction either!

    • @Manavine
      @Manavine 8 місяців тому

      ...that feels like a paranoia syndrome waiting to happen because their fears have been proven so much more horrifically right than they thought. That's going to mark them, considering you already have to be pretty paranoid to check code over half a second. ​@@draken5379

    • @gabereiser
      @gabereiser 8 місяців тому +6

      This time… what about the others?

  • @window.location
    @window.location 8 місяців тому +1200

    Not enough people aware that maintainer/og auther (Lesse) is victim of social engineering,
    I read those messages in mailing list and it hurts me. Jia basically took advantage of his health condition.

    • @LowLevelTV
      @LowLevelTV  8 місяців тому +383

      yeah its really sad :(

    • @channelgogrvk
      @channelgogrvk 8 місяців тому +170

      @@mayshackwe don't know that Jia is one person or an identity being used by a group of people coordinating the attack

    • @eight-double-three
      @eight-double-three 8 місяців тому +21

      ​@@channelgogrvk I'd guess, it's the latter...

    • @franzlyonheart4362
      @franzlyonheart4362 8 місяців тому +31

      @@eight-double-three NSA, Google, CIA … ?

    • @imperiodosgameplays321
      @imperiodosgameplays321 8 місяців тому

      @@franzlyonheart4362 Probably some asian/russian threat actor. My guesses are North Korea, China or Russia. Maybe Lazarous Group? Killnet is out of question since they do mostly DDoS.

  • @dj1ch.
    @dj1ch. 8 місяців тому +911

    we definitely need to see the investigation behind the contributor who created the backdoor. definitely very good at social engineering, being subtle, etc. if it weren't for Andres Freund we would have never been talking about this.

    • @zirize
      @zirize 8 місяців тому +27

      We all know who it is, we just can't say who it is.

    • @ViciOuSKiddo
      @ViciOuSKiddo 8 місяців тому +65

      @@zirize Three letter agency funded by a big brother?

    • @saddish2816
      @saddish2816 8 місяців тому +83

      ​@@zirizeit's not the NSA they have Intel management engine and AMD PSP

    • @Superchunk-k2h
      @Superchunk-k2h 8 місяців тому +65

      @@saddish2816 Why would they stop at a hardware backdoor? Ideally you want to control every step of the computer's stack from bare metal to the user

    • @narrativeless404
      @narrativeless404 8 місяців тому

      ​@@Superchunk-k2hNSA isn't gonna do that
      FIB or IAA might tho

  • @alexnoman1498
    @alexnoman1498 8 місяців тому +106

    If anyone's takeaway here is "I don't happen to use those versions of of that library, I'm good", you did not understand the issue.
    That maintainer is still struggeling.
    Other maintainers are still struggeling.
    Other libraries are still compromised.
    The Perp and their Patron are still out there doing this exact thing and more for the next 30 years together with lots of direct and indirect colleagues.
    The people introducing vulns are payed exorbitant amounts of money, because buying vulns is more expensive, less exclusive, while just as bad for security.
    Getting the maintainers of our critical infradtructure payed and having them audited should be our priority world-wide. Every company in every country uses FOSS, but vanishingly few pay for it. Ask your manager whether your company pays for the software they are using!

    • @TotesCray
      @TotesCray 8 місяців тому +15

      While I agree with you in spirit, if you're running RHEL and paying for it, IBM isn't giving the maintainers of xz a cut of it, so just paying for FOSS in a corporate setting doesn't exactly solve the problem either.

    • @Rudxain
      @Rudxain 8 місяців тому

      I totally agree. Corps generate hundreds of dollars per tool used, but they don't even donate a dollar. Then we get a core-js or faker.js situation and people lose their minds

  • @johnfitzgerald2339
    @johnfitzgerald2339 8 місяців тому +242

    8:56 Agreed that there are certainly vulnerabilities with open-source, but this also highlights the strengths of open-source: having a huge cohort of noble, intelligent, and vigilant coders to sniff-out this type of espionage.

    • @arthurmoore9488
      @arthurmoore9488 8 місяців тому

      It's frustrating because I deal with people every day who don't like Open Source, but have no problems using SolarWinds!

    • @dascandy
      @dascandy 8 місяців тому +40

      Imagine this happening with a closed source company. Somebody applies, silently checks this in with the binary files, and there's nobody outside of the company that even has a chance at catching these backdoors.
      How many of these backdoors already exist in commercial software? How many does the company owning the code know of?

    • @justacat3639
      @justacat3639 8 місяців тому +26

      @@dascandy THIS.
      Every time i see somebody talking about how this is "an issue with open source", i think this exact thing. What if this happened in a closed source software? AT LEAST with open-source you're able to find out. With closed-source there's a good possibility that piece of code will never be looked at again because "it just works".

    • @dascandy
      @dascandy 8 місяців тому

      @@justacat3639Thinking back to some part of Windows NT4 shipping with a thing called "NSAKEY" (en.wikipedia.org/wiki/NSAKEY) and nobody knowing what it does or how it got there.

    • @stokdam
      @stokdam 8 місяців тому +2

      @@justacat3639It is the well
      known selection bias.

  • @wChris_
    @wChris_ 8 місяців тому +523

    whats scary is that this was only discovered, because the backdoor was slowing down ssh as well as generating a lot of errors in valgrind.

    • @scottschiller
      @scottschiller 8 місяців тому +38

      I'm sure I'm not the only one who was reminded of Clifford Stoll's "The Cuckoo's Egg" (1989), where a 75-cent computer usage accounting error ultimately revealed a remote attacker with superuser access.

    • @Mallchad
      @Mallchad 8 місяців тому +9

      to be fair the attacker had to disable and ignore several address sanitization features and test checks. Hence why stuff like valgrind was already setup to catch stuff out for this particular project.

    • @KingKoncorde
      @KingKoncorde 8 місяців тому +36

      so it does make me wonder how many of these backdoors are already out there, just not discovered?

    • @fellipec
      @fellipec 8 місяців тому +18

      And the attacker was working with one distro contributor to help fix the valgrind errors. And the said contributor, AFAIK, was helping in good faith.

    • @ZombieLincoln666
      @ZombieLincoln666 8 місяців тому +1

      @@AeduoI don’t really know jack about software dev and I was still able to use valgrind to find a memory leak in some important software and report it

  • @FireStormOOO_
    @FireStormOOO_ 8 місяців тому +203

    I still can't fathom how incredibly lucky we are this got caught as quickly as it did. I still haven't found a system on stable that actually shipped the compromised versions. We don't always get a warning shot like this.

    • @VonCarlsson
      @VonCarlsson 8 місяців тому +30

      The real question now is: how many times have we not gotten so lucky?

    • @Noname23489
      @Noname23489 8 місяців тому +2

      just curious.... why would large organizations be using operating systems that get updates from open source git repositories? maybe im understanding this attack wrong just curious if you dont mind educating me

    • @joshuavillwo
      @joshuavillwo 8 місяців тому

      @@Noname23489 A sizable portion of servers on the internet run some version of Linux. There's no good data, but I've seen stats that are well over 50%.
      And everything based on Linux is at least partially relying on open-source code.

    • @brunomachtchaos7876
      @brunomachtchaos7876 8 місяців тому +6

      /cries/ I use arch BTW :(

    • @Acorn_Anomaly
      @Acorn_Anomaly 8 місяців тому +9

      @@Noname23489 that's basically asking "why would large organizations run linux", and there's lots of reasons to do so.

  • @MikeC1
    @MikeC1 8 місяців тому +177

    As much as I love the technical aspects of this, its not what we should be focusing on. The "nature of [this vulnerability] of open source software" was primarily and overwhelmingly social. The project had a solo maintainer that was hopelessly overwhelmed. He was conned by one or more anonymous individuals into accepting a malicious co-maintainer by both attacking his weaknesses as the project owner, and offering him the answer he was desperately looking for to save his labor of love project. He was pressured into trusting code and a developer without sufficiently scrutinizing either.

    • @Dogeek
      @Dogeek 8 місяців тому +31

      My takeaway is that open source is only as strong as its weakest link. So many projects go to the graveyard because of the lack of maintainers, and there's a lack of maintainers because people need to eat, need a roof over their heads, so they work their 9-5 as software engineers, which is draining in of itself. So when a crucial piece of software has a solo maintainer, it's a prime target for attacks. If there was more compensations for open source maintainers, maybe some of them could afford to maintain the software full time, and as a result would not burn out and be less prone to social engineering attacks.
      Or we could just create a foundation for those crucial projects, so that there are always developers able to maintain the software, and need to reach a consensus before anything gets merged.

    • @ZakBellinger
      @ZakBellinger 8 місяців тому +23

      @@Dogeek I think all the companies that used open source software in their commercial projects on a massive scale that command massive profits should be obligated to divert a fraction of a percentage towards maintaining these things. Or better yet when there's a massive fork and that fork goes on to make billions of dollars, leaving the original project in the dust and penniless there should be something that allows maintainers to continue doing what they love to do.
      I love open source but the (un)ethics of business undermines such a great ethos it really becomes disgusting. Sucks to see such openness taken advantage of again and again, and even more now since technology companies rule as the market leaders in today's world.

    • @ichigo_nyanko
      @ichigo_nyanko 8 місяців тому +38

      Not even that, he knew the attacker for YEARS before he let him into the project. nearly 4 years. And the guy was nothing but nice and helpful until a month ago.
      Nobody, not a single person on this planet, would have not trusted him by that point.

    • @SlinkyD
      @SlinkyD 8 місяців тому +3

      ​@@ichigo_nyankoI wouldn't trust him until I saw how he handled LE or drugs/alcohol. Being internet based, I don't trust nobody.

    • @stxnw
      @stxnw 8 місяців тому +3

      Many projects do not need active contributors. Just stop adding more features.

  • @eructationlyrique
    @eructationlyrique 8 місяців тому +86

    What's interesting is the timing. We see the attacker working hard to make Ubuntu include the compromised versions in their 24.04 beta, but also trying to convince fedora to include it in fedora 40. This makes me believe they might've been constrained by time at the end be cause these two releases are very important: The next Ubuntu is LTS and the next Fedora is likely what the next RHEL is going to be based on a year from now

  • @sirflimflam
    @sirflimflam 8 місяців тому +149

    I feel really bad for Lasse. Jia probably looked like an angel sent from heaven to help with project maintenance at a time when they weren't doing so great. Then all this comes out, and they have to confront this nightmare with suspicion cast even on them. It's like a dad finally handing his keys to his daughter for the first time to drive on their own and within 15 minutes she drives through a crowd of pedestrians.

    • @JamesTK
      @JamesTK 8 місяців тому +27

      Yeah even going through their commits the majority of them are benign improvements... And then this, with the seeds having been planted in the test files ages ago

    • @nuke_parasitine
      @nuke_parasitine 8 місяців тому +7

      More like a snake you sheltered finally showing its true color and biting you to the ground

    • @ichigo_nyanko
      @ichigo_nyanko 8 місяців тому +40

      @@nuke_parasitine nah, the contributer, before the backdoor, was only ever helpful and nice.
      If you knew someone for years and they were only ever lovely to you - and it turned out they were a serial killer - it would be a horrible thing for people to imply it was only ever obvious (they were a snake, you let them in) when it wasn't.

    • @atticus2274
      @atticus2274 8 місяців тому +4

      @@ichigo_nyankothe shit you nerds say lol.

    • @sfall616
      @sfall616 8 місяців тому +14

      ​@@atticus2274 Nerd calling a Nerd a Nerd is peak irony.

  • @HolyGarbage
    @HolyGarbage 8 місяців тому +51

    I would not call this a vulnerability in open source, rather it's a strength. This could, and probably has, happen in closed source software too. They can just get some person to apply to a job and do the same thing. The only difference is that in closed source it's far fewer people that could potentially find it. I think one of the reasons it was found relatively quickly is precisely because it was open source.

    • @antelope6826
      @antelope6826 8 місяців тому +2

      Agree 100% with this, the fact that open source code is evaluated and run by so many independent people is a strength that leads to things like this being found. If this change was to a closed-source tool it could sit in a repo for a decade and no one would notice :/

    • @leexgx
      @leexgx 8 місяців тому +1

      It was more luck and that the new version caused half a second delay that a person was doing performance tests at the time most other people probably wouldn't have noticed unless they was doing performance testing, this was pure luck as it wasn't that Mutiple people picked up on this it was just 1 person
      if they fixed the delay part it wouldn't have been noticed until a couple of years later on when some high profile targets got compromised and don't know how

  • @ZeroUm_
    @ZeroUm_ 8 місяців тому +71

    The program being built being able hook back to the build process and change the linker behavior, worse -during the test phase-, is also super crazy.

    • @lonelybiscuit243
      @lonelybiscuit243 8 місяців тому +11

      It does some really complicated/shady things like altering the Global offset table and character recognition, I wonder if we ever get to find out who was behind this

  • @fellypsantos_
    @fellypsantos_ 8 місяців тому +327

    the scale of this attack is absolutely insane

    • @adibemaxwell6111
      @adibemaxwell6111 8 місяців тому +10

      Oh? You mean how this backdoor didn't affect any LTS distributions used in servers or production environments but only affected rolling release and testing distributions? That scale? In other words, only desktop users who are stupid and naive were affected.

    • @jasonhamm3370
      @jasonhamm3370 8 місяців тому +23

      @@adibemaxwell6111 My dude, what do you think 90% < of the internet runs on?

    • @Aomicplane
      @Aomicplane 8 місяців тому +45

      @@adibemaxwell6111 It targeted all Debian based distributions. The only reason only rolling distro's were affected is because that Freund dude just so happen to be benchmarking his PostgreSQL databases and therefore end up noticing the backdoor. If he didn't it could have been planted in LTS distro's as well after some time. Also, the bad actor has been committing to the repository for a few years and it's still being investigated if there's more malicious or at least sketchy code in there, which if there is it could mean that even older versions of the libraries could be compromised in some way, which would also mean compromised LTS releases. You have to be short sighted to think these sort of attacks is solely an issue for people running rolling release distro's...

    • @LaughingOrange
      @LaughingOrange 8 місяців тому

      ​@@adibemaxwell6111 If it hadn't been discovered, it would be on like 80% of servers within 6 months. The only reason it didn't is because some Microsoft employee noticed high CPU usage from sshd and looked into it.

    • @fellypsantos_
      @fellypsantos_ 8 місяців тому +5

      @@adibemaxwell6111 nice comment to generate engagement, go ahead dude.

  • @ellielikesmath
    @ellielikesmath 8 місяців тому +23

    the main issue is the main issue that has plagued open source recently: business expects services and support from the open source community, but is unwilling to pay for them. in this case, we have a security critical library being maintained by a single person. in no sane world would i expect that to be secure. system critical libraries either need to be fully audited and then quarantined with no more updates possible, or they need well paid, dedicated, redundant staff and regular security audits. the new layoff culture and over investment in ai (and underinvestment in mental healthcare) is going to do massive, yet untold damage to computer security.

  • @johnmoore8599
    @johnmoore8599 8 місяців тому +48

    This is a variation on the theme. Open Source projects may only have one or two maintainers. We had a wakeup call with OpenSSL a decade ago. This is just a sneaky supply chain attack via compromise of liblzma and oss-fuzzer. Trust was assumed, but trust was broken. How will single maintainer OSS projects audit all commits to their project if they are hobbyists or part-timers? The payload was to be included in deb or rpm packages during build via checks. That was another way to ensure it got into the main distros, but was low key.

    • @MrSmokinDragon
      @MrSmokinDragon 8 місяців тому +16

      Large developers that depends on smaller OSS projects needs to contribute to maintain all the projects they depend on. Either man-power or money.

  • @m4rt_
    @m4rt_ 8 місяців тому +34

    one way you could prevent something like this would be disallowing binary blobs, and if you need it for testing, etc, it should be generated by code that is readable. For example a shell script that generates a large amount of the letter a compressed with xz and then intentionally corrupts it.
    Though this doesn't prevent people from doing bad stuff like this, but it makes it harder to hide. (people can still intentionally introduce bugs in the code that can lead to something like heartbleed for example)

    • @ville_syrjala
      @ville_syrjala 8 місяців тому +11

      You could also mandate that test data must live in a separate repo, and must only be used while running the tests (in an isolated sandbox). Also the build system should be less flexible so that any weird stuff really stands out. But all of that is moot if it's the maintainer doing bad things and no one else actually reviews any of it.

    • @markusklyver6277
      @markusklyver6277 8 місяців тому

      @@ville_syrjalaAlso inspect binary blobs before they are uploaded to a repo.

  • @ayanami-rei-san
    @ayanami-rei-san 8 місяців тому +154

    That level of obfuscation though, the backdoor glows very bright

    • @lukarikid9001
      @lukarikid9001 8 місяців тому +31

      phosphorescent, even

    • @GelatinousSSnake
      @GelatinousSSnake 8 місяців тому +11

      Stop being antisemitic

    • @lionbnjmn
      @lionbnjmn 8 місяців тому +46

      @@GelatinousSSnake i thought glowing is a fed reference, what does it have to do with semitism

    • @nou712
      @nou712 8 місяців тому +30

      @@GelatinousSSnake Glow = government agent, not anything relating to ethnicity.

    • @Terszel
      @Terszel 8 місяців тому +4

      @@GelatinousSSnake wtf 😂

  • @MikkoRantalainen
    @MikkoRantalainen 8 місяців тому +110

    I think this is the modern variant of "Reflections on Trusting Trust" by Ken Thompson.

    • @smnomad9276
      @smnomad9276 8 місяців тому +2

      What's that and how is it relevant to the topic?

    • @gabereiser
      @gabereiser 8 місяців тому

      @@smnomad9276you should read it. It’s about “how can you trust your own trust chain” and what can happen if that trust chain is broken. “You can not trust code you did not compile yourself”.

    • @levonschaftin3676
      @levonschaftin3676 8 місяців тому

      @@smnomad9276 go read about it, you have the entirety of the internet at your hands.

    • @Acorn_Anomaly
      @Acorn_Anomaly 8 місяців тому

      @@smnomad9276 A famous speech/presentation he gave as part of an award acceptance speech.
      It's basic premise is "what if I maliciously modified the compiler included with the UNIX system to do two things - 1) if compiling a system component, insert a backdoor that allows me full access, and 2) if compiling a compiler, modify the output compiler to perform these two steps."
      Basically saying, "how far do you trust what's been given to you?"

    • @timewave02012
      @timewave02012 8 місяців тому +7

      Trusting Trust is a much deeper concept, and the reaction to it was the practice of Diverse Double Compiling. This trust gap is way easier to deal with, and more analogous to the use of nothing-up-my-sleeve numbers in cryptography. Distributing binary blobs alongside source code should be treated with great suspicion. Let test data be programmatically constructed, and therefore auditable the same as code, or by some other means be provably incapable of hiding machine code (e.g. pi).

  • @4.0.4
    @4.0.4 8 місяців тому +35

    Remember kids: writing a backdoor is no excuse to write poorly performant code that wastes CPU cycles so much it alerts the authorities.

  • @a_random_lemurian
    @a_random_lemurian 8 місяців тому +146

    one of the games i play (openttd) uses xz for savegame compression, and now the latest release is delayed.. because of the backdoor

    • @LowLevelTV
      @LowLevelTV  8 місяців тому +59

      I LOVE openttd. didnt realize it uses xz D:

    • @eight-double-three
      @eight-double-three 8 місяців тому

      S**t just got serious - no-one is touching my OpenTTD! 🤪

    • @wroomwroomboy123
      @wroomwroomboy123 8 місяців тому +5

      That's it!! I freaking knew I recognised xz from some ancient memory!

    • @DVSoftware
      @DVSoftware 8 місяців тому +1

      Not sure how OpenTTD would ever be affected, even if using xz with a backdoor, since it replaces RSA decryption function, and gets the payload from RSA key supplied to SSH.

    • @a_random_lemurian
      @a_random_lemurian 8 місяців тому

      @@DVSoftware OpenTTD may not be vulnerable to xz itself, but it certainly is vulnerable to GitHub taking down xz's repo and users getting pissed about potential malware being included with the game.

  • @meredithweston46399
    @meredithweston46399 8 місяців тому +17

    cs undergrad here. A lot of this is still over my head, but it is so interesting. Honestly just inspires me even more to learn. I appreciate your breakdowns that are very clearly presented and approachable.

  • @GEfromNJ
    @GEfromNJ 8 місяців тому +141

    To me, the scariest part of software these days are all of those 3rd party libraries. You ever install something with "sudo apt-get install X", and you'll see a stream of things being downloaded. Even just one part of one of those libraries could be infected and who would know? MD5 hashes or whatever would do nothing here because this was compiled into the release itself. Frightening as hell

    • @jsrodman
      @jsrodman 8 місяців тому +12

      Dependency graphs give us the tools to fix problems once instead of over and over and over. They also allow us to be more effective and efficient in our work.
      They also let risks and failures cascade through our ecosystems in astonishing ways. Not all of the problems are even malicious. left-pad in nodejs was just someone unpublishing his work and it took down tons of sites.
      What's the answer? i have no idea.
      But if we focus on practical situations we can perhaps come up with practical mitigations that reduce harm, and improve.

    • @christianbarnay2499
      @christianbarnay2499 8 місяців тому +36

      @@_devilfish303 The point is Linux in its current state doesn't provide better protection than Windows. The slogan "you can always recompile everything locally to make sure the binary matches the source code" doesn't work anymore. Because every single Linux library is dependent on dozens of others. It's physically impossible to recompile everything on your own. And even if it was possible it would still give no guarantee as this attack was built directly with a hook to inject the binary malware during the compilation process without altering the source files.
      Yes Microsoft is not better.
      But it's not worse either. And this is the true issue here. Why bother with all the hassle of securing your linux if anyone can trash all your efforts with a single commit in some obscure source repo you never heard of?

    • @goraxe01
      @goraxe01 8 місяців тому +13

      curl | sudo is the one that terrifies me. Literally random code from internet run as root. Multiple attack vectors there from ip / DNS hijacking through to just straight up blobs in the install script... Plus potentially no evidence of what you just downloaded

    • @askeladden450
      @askeladden450 8 місяців тому

      ​@@christianbarnay2499microsoft is not worse? If such a backdoor was installed in windows, it wouldnt get discovered EVER. Even a simple babckdoor wouldnt ever get discovered, which wouldnt even get merged if it was open source. Security isnt black and white. No system is ever completely secure. Its about reducing the attack vector. And the attack vector in open source or linux is extremely small compared to proprietary.

    • @paulstelian97
      @paulstelian97 8 місяців тому

      @@christianbarnay2499 Gentoo is from what I understand a distro where yes, recompiling everything is practical.

  • @natam1616
    @natam1616 8 місяців тому +12

    THATS IT! im done
    im building my own O.S. and all the libraries it need myself, and running on a custom board with a custom riscv processor

    • @mineton1293
      @mineton1293 8 місяців тому +16

      Little do you know that riscv could also have a backdoor. mwahahahaha. You must create a completely custom architecture and custom silicon to be sure.

    • @v0ldy54
      @v0ldy54 8 місяців тому

      and also blackjack and hookers

    • @natsuzkan
      @natsuzkan 8 місяців тому

      @@mineton1293oh but then there's backdoors built into your circuit design tools as well, so have fun arranging logic gates entirely by hand

    • @Arekadiusz
      @Arekadiusz 8 місяців тому

      ​@@mineton1293 Easy AF, I'm taking my soldering iron and going to work :P

  • @manzenshaaegis8783
    @manzenshaaegis8783 8 місяців тому +7

    One key correction here is that the hosted git repo does not contain the necessary code to build the backdoor. Only the private repo for the library maintainers contains the necessary code. Your highest upvoted comment already starts off on the wrong track. I spent a lot of time sorting through this confusion today.

  • @ingemarastrom4623
    @ingemarastrom4623 8 місяців тому +3

    Really thankful for Andres Freund's agile mind; finding a needle in the 'stack is hard enough; he found a bd even though he wasn't looking for one, anticipating one.

  • @whamer100
    @whamer100 8 місяців тому +96

    what's really terrifying to me was how it was legitimately almost not noticed in the first place

    • @Leonhart_93
      @Leonhart_93 8 місяців тому +22

      That's because everyone is under the impression that these kinds of things are rigurously checked by someone else, therefore no one checks them. This should be a wake up call.

    • @TheOzumat
      @TheOzumat 8 місяців тому +60

      Who would win: A multibillion dollar government agency, or one giganerd noticing his ssh takes 0.5 sec longer to log in?
      That's a W for the nerds, boys.

    • @wroomwroomboy123
      @wroomwroomboy123 8 місяців тому +17

      @@TheOzumat Obsessive optimisation is the root of all evil - and that is why it also naturally finds it.
      Dude wanted to optimise performance and probably spent days untangling the mess he ran into - just because he wanted a near unnoticeable improvemet.

    • @Superchunk-k2h
      @Superchunk-k2h 8 місяців тому +14

      Everyone's calling it a W, but imagine all the projects that aren't being autistically benchmarked by someone with a unique problem.

  • @jfftck
    @jfftck 8 місяців тому +23

    Isn’t the fact that this was caught proof that more people are watching what is going on in open source? This was the best case scenario for a malicious attack and the openness of the code allowed having all the evidence available. I agree that this should be the start of regular audits of crucial open source software. But the log4j exploit a few years ago might have had some influence on the Microsoft employee’s idea to dig deeper.
    I see this as a large opportunity for more secure code practices in open source.

    • @ratchet1freak
      @ratchet1freak 8 місяців тому +11

      this was found by chance more than anything, it's only because someone tried to get a good baseline for their system that they noticed that sshd was doing more work than it should and dug into it
      If that didn't happen then this would never have been found

    • @Cavi587
      @Cavi587 8 місяців тому +2

      Well yeah. But a lot of bad actors are now fear mongering so that people get scared of open source projects. We should doubt every project, for sure, but we have the ability to look at code and the whole build process and catch such exploits. But of course, as this example showed, there are simply too many projects for people to realistically screen with every commit. But we have to start being more cautious, starting with the most critical packages would be a good beginning.

    • @helipad_huck1543
      @helipad_huck1543 8 місяців тому +2

      ⁠​⁠​⁠@@ratchet1freakBut on the other hand, if the library hadn't been open source, he wouldn't have been about to look into it as deeply as he did. He likely would've had to conclude that the developer introduced a bug, and at most reach out to them, which would've only served to tip the bad actor off that their backdoor was still too detectable. So overall, open source did "win" here.

    • @Sandromatic
      @Sandromatic 8 місяців тому

      ​@@helipad_huck1543as it happens, someone else actually did exactly that from my understanding, and that was the patches that the second version of the backdoored version of xz fixed, some valgrind issues that were found by a maintainer of a different distro, so that's xz 5.6.1 :P

  • @Pentium100MHz
    @Pentium100MHz 8 місяців тому +6

    So, as I understand it, if the compromised version made it to the server distributions, the only way to guard against it would be to limit who can connect to ssh using iptables (or some other firewall), since the attacker would still need to connect to the ssh port to send the commands.

    • @Sandromatic
      @Sandromatic 8 місяців тому

      That, or install an uncompromised xz on that server instead.

  • @Sparks621
    @Sparks621 8 місяців тому +25

    It's cool that this vuln has been found, but the scariest thing to me personally is the very real possibility of similar backdoors that may already be present in other widely used OSS projects. The fact that even this one was found by total accident is genuinely terrifying.

    • @benjaminpatrickpatrickgarr9291
      @benjaminpatrickpatrickgarr9291 8 місяців тому +8

      Small OSS projects like this usually have a very high bar for performance to meet, and they usually meet them no problem, so it makes sense that the valgrind performance anomalies were the reason why this one got discovered.
      Now imagine trying to discover "performance anomalies" in commonly used proprietary software...

  • @jhirn2957
    @jhirn2957 8 місяців тому +24

    Plot twist. Andres has the private key. He did this all to rise to stardom in the security world. Being discovered only strengthens his reputation of supreme social engineer. He’s brilliant either way.

  • @benfinch4969
    @benfinch4969 8 місяців тому +73

    Imagine how pissed off the exploiter(s) must be right now... "Noooo, it was perfect!"

    • @WinterAyars
      @WinterAyars 8 місяців тому +20

      They spent years setting all this up, too.

    • @Uerdue
      @Uerdue 8 місяців тому +27

      And yet they managed to calmly disappear once it was discovered, instead of yelling insults at the xz mailing list. True professionals!

    • @WinterAyars
      @WinterAyars 8 місяців тому +8

      @@Uerdue As opposed to when they were setting this up where they (presumably this was them) were quite rude.

    • @Cavi587
      @Cavi587 8 місяців тому +10

      I have a feeling that it's not just a simple uber haxor kid, this is too big and the setup is too perfect in the way it was being set up slowly. I don't want to sound like a conspiracy theorist but I just have a feeling that the guy who we all saw as a co-maintainer is not the sole brain behind this attack. It had to be a group who would greatly benefit from having this backdoor available.

    • @Uerdue
      @Uerdue 8 місяців тому +8

      @@Cavi587I think most people consider a state-level actor (like some intelligence agency) the most likely behind this.
      It's not just the amount of time/work spent on it, it's also the likelihood of failing which was too high for the average cyber criminal without government funding to risk so many resources on it. What if systemd drops its xz dependency 2 years into the project? Or another contributor shows up who likes to do his own improvements to the tests or build system (that touches your manipulated files, so you have to turn down all of their PRs for made-up reasons)? Or the original maintainer is just stubborn and says "no binary blobs of unknown source as test files, period"? Or some person simply finds the backdoor before it makes its way to stable distros?
      There are so many ways for such an operation to go wrong. You either have to be able to scale it, working on a bunch of these in parallel and hoping for one to succeed - or you have to just be ok with the operation failing and the only thing gained from it being the insight that it's quite a hard thing to pull off successfully.

  • @oliverhuth5662
    @oliverhuth5662 8 місяців тому +40

    Wow this guy is some sort of evil genius. Appreciate the constant video updates, it's great to have someone summarise these really complex topics but still keep the essence of whats happening.

    • @franzlyonheart4362
      @franzlyonheart4362 8 місяців тому +9

      Guy? Or guys, plural? Or … state sponsored? NSA, CIA, ((Washington)), etc.?

    • @MarBL23563
      @MarBL23563 8 місяців тому +4

      @@franzlyonheart4362 its always a government conspiracy, eh?

    • @MechazoidApocalypse
      @MechazoidApocalypse 8 місяців тому +10

      @@MarBL23563 It does make sense in this case though

    • @ancientaccount7982
      @ancientaccount7982 8 місяців тому

      @franzlyonheart4362 why is everyone in these comment sections assuming it's got to be an American agency? My money would be on China, Russia, or Mossad. Hell, Mossad's bread and butter is this kind of diabolical social-engineering into unnoticeable attack vector.

    • @tsubaki-kouryuu
      @tsubaki-kouryuu 8 місяців тому

      ​@@franzlyonheart4362 No reason to assume it has to be the US government. Could be state-sponsored hackers from China, Russia, or North Korea, too! But with so little information at this point in time, I don't think it's worth baselessly speculating. It could totally be non-state sponsored, as well-there's a lot of money to be made from having this sort of backdoor access.

  • @acemany
    @acemany 8 місяців тому +48

    this backdoor is just... brilliant

    • @rawallon
      @rawallon 8 місяців тому +16

      @7DuRd3n In the shower

    • @rian0xFFF
      @rian0xFFF 8 місяців тому +3

      I've heard something like that, I thought it was common

    • @percy9228
      @percy9228 8 місяців тому

      @7DuRd3n geniuses that have specific domain knowledge sharing ideas with others in their domain knowledge. this most likely was a state operation.
      you think this is wild, imagine the concepts and ideas in modern mathematics where you are not constrained by physical limitation. everything is abstracted and abstracted.

  • @Zuriki09
    @Zuriki09 8 місяців тому +13

    Binary blobs are basically immune from standard audit, I don't see much of a way to prevent this kind of attack except to ban blobs from being included in source control. Which is obviously impractical in certain scenarios.

  • @ebin2222
    @ebin2222 8 місяців тому +18

    But how would the attacker know which server got infected? Does it phone home in any way, or was the attacker hoping to just attempt connecting to random open ssh servers accessible through the internet by scanning IP ranges?

    • @aeghohloechu5022
      @aeghohloechu5022 8 місяців тому +15

      the latter is very viable, given that there are botnets pinging every single public IP address right now in hopes of hitting something

    • @gie4830
      @gie4830 8 місяців тому +9

      Attacker prob patiently waiting for their code to be published with stable release then do crazy stuff. Remember it got exposed just because of 1 dude noticed ~500ms delay on his ssh connection.

    • @gie4830
      @gie4830 8 місяців тому +6

      After the hard part is done, the attacker just hopes the biggest targets are infected.

    • @oleksandr8482
      @oleksandr8482 8 місяців тому

      @@gie4830 500ms is crazy delay, no way this backdoor will come to stable

    • @ebin2222
      @ebin2222 8 місяців тому +3

      @@gie4830
      I don't know. I'm pretty sure actual big actors don't have SSH servers directly addressable through the public internet, you most likely need to jump through some enterprise VPNs to get access. So the targets were mid-sized.
      At least now he's vulnerable to getting honeypot'd. Wonder if someone will attempt to catch him like that. Though it would have been easier to not alert everyone if he really was targeting only a few actors.

  • @johnathanstephenson8107
    @johnathanstephenson8107 8 місяців тому

    Can we agree that whoever made this backdoor was pretty skilled
    Also the guy who found it is a hero for being diligent and trying to optimize and finding it

  • @ScottHess
    @ScottHess 8 місяців тому +3

    An issue with this entire thing is that we keep trying to invoke a technological “Web of Trust” to let us bind dozens of dependencies into a single binary, but instead we should perhaps be encouraging larger teams to form. Many projects have one or two or three primary authors who carry the entire load alone, which is intrinsically broken over the long term.

  • @MyCodingDiary
    @MyCodingDiary 8 місяців тому +2

    I've been struggling with this topic, but your video cleared it up for me. Thanks a ton!

  • @Fudmottin
    @Fudmottin 8 місяців тому +12

    I don't think the brilliance of this evil attack can be understated. Not only is it a backdoor, it is a backdoor where only one person or group has the key to open!

  • @elkvis
    @elkvis 8 місяців тому +2

    A couple thoughts on prevention. Move the test environment to a temp folder, completely separate from the source, and only run the tests on a stripped binary, so that the actual source cannot be found by the tests.

  • @danielbaulig
    @danielbaulig 8 місяців тому +7

    I've been following this as it was unfolding over the weekend. Just watched the first 3 minutes of your video. The fact that this backdoor is very effectively secured against being used by anyone else than the attacker is a super strong signal for me that this is not some random cyber criminal but a state actor. The sophistication and attention to detail in this whole exploit reminds me a lot of stuxnet. Honestly, it's a whole notch up from stuxnet in a lot of ways.

    • @somejoe7777
      @somejoe7777 8 місяців тому

      Without question. This was a well-funded state-level attack, likely one of the 1st-world governments.

  • @LarsFL
    @LarsFL 8 місяців тому +3

    I'd love a video that summorizes the entire thing in one, to be able to send that to people and inform them

  • @AnalyticMinded
    @AnalyticMinded 8 місяців тому +12

    I saw the twitch livestream where he followed the command cmd with a shutdown... and then his vm went dark. That was so awesome.

  • @leshommesdupilly
    @leshommesdupilly 8 місяців тому +29

    The perfect, most complexe and most elaborate hack was avoided because one random guy found that one random program was a few milliseconds slower than usual 😂😂😂
    Remember kids, always track the bloat lmao

    • @ZombieLincoln666
      @ZombieLincoln666 8 місяців тому +3

      It wasn’t a few milliseconds, iirc it was 500ms

    • @percy9228
      @percy9228 8 місяців тому

      he didn't investigate because of slow down, "FWIW, I didn't actually start looking due to the 500ms - I started looking when I saw failing ssh logins (by the usual automated attempts trying random user/password combinations) using a substantial amount of CPU. Only after that I noticed the slower logins."

  • @outofbound28
    @outofbound28 8 місяців тому +8

    So, are you using a virtual machine setup on an air gap?
    I usually use a set of virtualbox machines in their own virtual network on an old notebook for stuff like this. Would love to see how your setup looks for cases like these.

  • @mustafa_el-rashied
    @mustafa_el-rashied 8 місяців тому +1

    a great father who spend time with his children AND keep up with youtube, keep up the great work!

  • @Uerdue
    @Uerdue 8 місяців тому +13

    In cryptography, there is this concept of using a "nothing-up-my-sleeve-number" for (technically artbitrary) seeds / initial values within a cryptographic algorithm. For example, the Blowfish cipher uses digits of Pi as initial values in its key scheduling algorithm. This is done to prove that whoever invented the algorithm did not backdoor it by picking constants that are vulnerable to some sort of cryptographic attack only known to him.
    Maybe the same concept should be established for binary test files. Use known, publicly available images that rarely change (such as the Wikipedia logo). Or, even better, use a well-known, deterministic pseudo-random number generator seeded with a fix "nothing-up-my-sleeve-number" to generate your test files. If your test file needs a specific structure (that is e. g. known to not suit your compression algorithm well), understand what is needed for the test file to behave the way it should, formalize that and then again generate files of the needed structure in an accountable way.
    This would make it practically impossible to hide stuff in these files. (And additionally make the repo smaller in most cases.)

    • @alexnoman1498
      @alexnoman1498 8 місяців тому +4

      You can find any arbitrary sequence of digits inside of Pi if you search long enough. Meaning you can find a sequence you like because it's vulnerable, then pretend like you rolled a dice.

    • @thewhitefalcon8539
      @thewhitefalcon8539 8 місяців тому

      That's just silly. They're test files.

    • @Uerdue
      @Uerdue 8 місяців тому +4

      @@alexnoman1498 Indeed, which is why you would want to use digits starting from the very beginning. (I think the Wikipedia article on "nothing-up-my-sleeve-numbers" also notes this in one of the first paragraphs.) I just left that part out in the initial comment because it was already long enough.
      Anyway, my point is: Binary (test) files of which you cannot explain how you created them are inherently suspicious. They may contain a rickroll link stored in low order bits of some pixel values, the mighty flag if you're playing a CTF challenge, or a secret backdoor to-be-extracted by the build script, as in this case.

    • @scaredyfish
      @scaredyfish 8 місяців тому

      The starting point would be its own magic number which should be a 'nothing-up-my-sleeve-number'@@alexnoman1498

    • @Turalcar
      @Turalcar 8 місяців тому

      That concept exists because of stuff like Dual_EC_DRBG (although Blowfish predates it).

  • @fritzw
    @fritzw 8 місяців тому +59

    I don‘t understand how people can frame this as „a problem of open source / linux“. It is only BECAUSE the whole thing is open source, that this backdoor could be found AT ALL. Think about how it would be infinitely easier to do this in a closed source software!

    • @MrSmokinDragon
      @MrSmokinDragon 8 місяців тому +6

      not necessarily. Many OSS projects have only single maintainers, which makes this kind of attack easier. Large software developers have extensive code reviews with many eyes looking over changes before commit, meaning that a malicious actor (other than the company itself) would have to fool or corrupt many developers to sneak in the backdoor. I am not saying that Closed-Source is in any ways better, what I am saying is please don't try and make it seem that OSS is always better able to withstand these attackers. OSS is much better on average than closed source, but lets not get complacent, but try and figure out processes to prevent something like this in OSS in the future.

    • @fritzw
      @fritzw 8 місяців тому +3

      @@MrSmokinDragon I agree, that this wakeup call should encourage everyone to be more on the lookout for such things. My point is, that you can ONLY do that in open source software. With closed source software, you never know. Sure, large companies also do code review, but not all widely used software is written by large companies with good practices in place. And good practice sometimes get thrown under the bus when time or money is short. If you can't see the git repo, the build scripts, the test files, then you have no way find out why your your program is suddenly slower by a factor of 4, or if some malicious employee committed some questionable binary blobs some time in the past. *shrugs*

    • @bryede
      @bryede 8 місяців тому

      @@MrSmokinDragon Yes, but the makers of closed software can put these things in on purpose with no one knowing. We know governments are the biggest cartels in the world and can get your cooperation.

    • @goraxe01
      @goraxe01 8 місяців тому

      Heard about the nist recommend crypto standards.... Quite a few have mysterious issues. Check out en.m.wikipedia.org/wiki/Nothing-up-my-sleeve_number (the counter examples section). Why subvert the code when you can subvert the committee defining the crypto algorithms everyone has to implement.
      Espionage predates computers.

    • @draken5379
      @draken5379 8 місяців тому +6

      No lol ?
      How exactly are you going to get your code, anywhere near, lets say, UA-cam or Photoshop`s source code ?
      I mean sure, you could try get a job at said place, work your way up, and etc etc.
      But that is vastly more work/time and secure than letting people who created a github account a week ago to contribute to key aspects of the worlds infrastructure. Like jesus.

  • @TweakMDS
    @TweakMDS 8 місяців тому +17

    Great demo. From a cybersecurity point of view, this should teach you to think bigger, implement zero trust and secure your platforms beyond just the front door. Treat every server/service as if it's already been compromised, and limit access from that perspective.

    • @kapytanhook
      @kapytanhook 8 місяців тому +5

      Just so impractical :( I just want to use my system and write things that jave some power to alter things. A big stack of sandboxes and passwords is so frustrating

    • @hellowill
      @hellowill 8 місяців тому

      @@kapytanhook then we need to make that easier. I hated the idea of 100s of passwords (for example), but a password manager has made that easy.

    • @kapytanhook
      @kapytanhook 8 місяців тому

      @@hellowill yeah, honestly. But this is deeper, it's that you can't trust your own machine or things running on it. People put backdoors in everything. We need well funded open source hardware and software but we are moving away from that

  • @0dayCTF
    @0dayCTF 8 місяців тому +1

    This is so wild, excited to see what else is found!

  • @dylanoonk6456
    @dylanoonk6456 8 місяців тому +52

    the NSA must be pissed that they lost their best backdoor

    • @outtakontroll3334
      @outtakontroll3334 8 місяців тому +18

      more likely the chinese

    • @anatolydyatlov963
      @anatolydyatlov963 8 місяців тому +17

      As if that was the only attempt - it's the only one that failed, but numerous others have succeeded

    • @bryede
      @bryede 8 місяців тому +8

      Yeah, this isn't a lone actor. It shows you how much more complicated it is to compromise an open source project.

    • @poisonouspotato1
      @poisonouspotato1 8 місяців тому +2

      @@outtakontroll3334 both

    • @anirudhg5629
      @anirudhg5629 8 місяців тому +3

      Would the NSA really be silly enough to try and backdoor an open-source project?

  • @bloeckmoep
    @bloeckmoep 8 місяців тому +2

    Everyone knows the solution to this question. NO non disclosed dependencies in the build routine. If compiling program a, it must not (under no circumstances) (never ever) ever use pre compiled files, be it executables, libraries or anything else. Building from source must mean just that, FROM SOURCE. If a program needs dependencies, fine, but let the build script point to that repo and compile that that too. Not impossible, just a hassle to properly include, setup and maintain.
    Also we don't know what that Jia Tan fake account did bad with its other commits. There is still bad stuff out there.

  • @misarthim6538
    @misarthim6538 8 місяців тому +3

    This is crazy. But I think this just shows the power of the open source. This would be completely undetectable in closed source software. Also, mass audit needs to be conducted, the level of sophistication of this attack makes it almost certain this wasn't the first time they did this. God only knows what has already been compromised.

    • @BillAnt
      @BillAnt 8 місяців тому +1

      All thanks to a bored and curious Microsoft engineer who was running a timing test on the SSH login process. The time difference between a normal and infected login was less than 500 milliseconds , and that's what tipped him off. It's pretty remarkable.

    • @misarthim6538
      @misarthim6538 8 місяців тому +1

      @@BillAnt That's a story of human evolution haha.

  • @mohamedmonem2645
    @mohamedmonem2645 8 місяців тому +4

    This whole thing glows brighter than the sun

  • @wallyhall
    @wallyhall 8 місяців тому +18

    The unix philosophy of “many things which each do one thing really well” could make attacks like these harder. SSHD does a lot - and it’s inevitably network facing. Breaking it down to have a separate (unprivileged) process which handles everything up and unto authentication and setuid… wouldn’t necessarily have *prevented* this, but would have made it a lot harder to pull off.
    I’ve always been a little surprised SSHD has been this way for so long - given most other services make every effort to run unprivileged, then offload switching user to some other process.

    • @JMurph2015
      @JMurph2015 8 місяців тому +6

      I mean sshd can already hand off to libpam for authorization iirc, and realistically, authenticating requests and launching a shell is kinda the whole point of sshd.

    • @nurmr
      @nurmr 8 місяців тому +1

      sshd _does_ support privilege separation though. I assume the backdoor activates on the secure side of it though (or disables it entirely).

  • @t0rg3
    @t0rg3 8 місяців тому +3

    People should be sifting through all kinds of archived conversations involving project maintainers looking for signs of social engineering.

  • @thegame4027
    @thegame4027 8 місяців тому +2

    Everyone will forgett about this in a month and we will be back to pressuring maintainers for open source projects to work harder and faster in no time.

  • @guitarbuddha74
    @guitarbuddha74 8 місяців тому +1

    wow I was thinking it still would have ran as an unprivileged user since I figured they were just getting login but it makes sense it would be able to execute as root since they are using the daemon process to execute. This was a really informative video.

  • @ANONAAAAAAAAA
    @ANONAAAAAAAAA 8 місяців тому +6

    We should value the simplicity of software a lot more to reduce the attack surface.
    I would like to appreciate simple software even if they are a little bit underperforming and feature poor.

    • @ANONAAAAAAAAA
      @ANONAAAAAAAAA 8 місяців тому +3

      Also, code freezing or "don't fix what isn't broken" was not so bad idea after all.

    • @DaveBucklin
      @DaveBucklin 8 місяців тому +1

      Perhaps the popularity of BSD is about to increase!

    • @diadetediotedio6918
      @diadetediotedio6918 8 місяців тому

      This has nothing to do with "simplicity of software" but whatever

    • @diadetediotedio6918
      @diadetediotedio6918 8 місяців тому

      @@DaveBucklin
      I hope, BSD is such a great system

  • @andreujuanc
    @andreujuanc 8 місяців тому +1

    Public/Private key explanation at 2:45 ish is the other way around. Public key encrypts, and private key decrypts.

    • @LykaiosFlux
      @LykaiosFlux 8 місяців тому

      I don't know enough about crypto to know whether "encryption" and "decryption" are the correct terms to use for signing, but it is the private key that "encrypts" (generates) the signature and the public key that "decrypts" (verifies) it.
      Which is the opposite of how public/private keys are used for encryption purposes.

    • @andreujuanc
      @andreujuanc 8 місяців тому

      @@LykaiosFlux you can use public key to encrypt

    • @LykaiosFlux
      @LykaiosFlux 8 місяців тому

      @@andreujuanc You can, but he's talking about signing, not encryption.

  • @ABHISHEKJAIN-wv7bh
    @ABHISHEKJAIN-wv7bh 8 місяців тому +20

    This is absolutely wild social engineering attack, jia tan enters fix a non problematic bug which decreases efficiency of system (which has negligible contribution before and after)and then jigar kumar (which has no activity before and after) pressurized developer Lasse Collins to maintian the code , they found the weak exploitable link of the chain Lasse Collins, then Dennis Ens enters and pressurizes even more led Lasse collins to forcefully pass the baton to jia tan and the actors played well and jigar kumar forces jia tan now and then they collectively triggered a hard to trace a backdoor but putting a corrupted code,
    this is insane because i studied cryptography and algorithms to encrypt which are generally very hard to detect
    i fear that they can be successfull in what they want to do ,they are really smart guys targeted and 15 year old weak link ,imagine how much backdoors does open source and god knows how much in close source software
    there is a urgent need for "V for Vendetta "for developers, we need to protect developers like Lasse Collins these guys are pillars of human evolution literally.

  • @drewnorthup9809
    @drewnorthup9809 8 місяців тому

    I mentioned this over on Sam's gist..., but it looks to me like there was also an attack planned for things built with CMAKE, not just the GNU gnulib build tools... Urk.
    (Oh, and please don't spend too much time reading M4 if you don't have to, but also consider learning it because this attack leveraged people not understanding it.)

  • @TechnopolisDotTV
    @TechnopolisDotTV 8 місяців тому +35

    7:20 too late buddy. I will WinNuke95 you! 😎

    • @LowLevelTV
      @LowLevelTV  8 місяців тому +15

      LOIC INCOMING

    • @TotesCray
      @TotesCray 8 місяців тому +4

      Enjoy your new cupholder *ejects CDROM drive*

  • @DominicDW8
    @DominicDW8 8 місяців тому +2

    Terrifyingly well executed 🥶

  • @MrVecheater
    @MrVecheater 8 місяців тому +4

    1:54 "thank god the researcher at Microsoft found it"
    Andreas Freund: 🙂

  • @CH4NNELZERO
    @CH4NNELZERO 8 місяців тому +1

    But what was detected by Andres ? His benchmarking showed ssh connections were 4x slower when sshd was backdoored.
    What was taking so long ? Was it just the decryption of the signing key ?

  • @x-user3462
    @x-user3462 8 місяців тому +23

    "turn yourself in" xD

  • @nomadicsynth
    @nomadicsynth 8 місяців тому +1

    Where else has this already happened? How many libraries out there are already compromised with this or something like it? Who is behind it and why? How do we go about finding out?

  • @_mmarciano
    @_mmarciano 8 місяців тому +4

    Super valuable work!
    Outside the tech community no one really cares, but this stuff is HUGE. Thank you for what you are doing

  • @Repligon
    @Repligon 8 місяців тому +1

    Maybe the good approach to catch things like this would be a kind of "enhanced" coverage test. Mark every accessed address in the binary, what was left not marked is either not tested or not needed at best or malicious at worst.

  • @J.erem.y
    @J.erem.y 8 місяців тому +4

    Why do you say it like this is just a problem with open source software? I would say the bigger problem is that we don't know how much closed source software does this, and we probably never will because of the nature of closed source.

    • @halfsourlizard9319
      @halfsourlizard9319 8 місяців тому

      Yeah, I don't understand this 'argument' ... You have absolutely no way to verify anything about closed code other than nebulous claims made by whoever released the software ... and blackbox testing. It's not like code review is any different for closed-source code than for open -- a trusted contributor who violates that trust could just as well have sneaked this backdoor into a closed project ... and it would be nearly impossible to detect beyond seeing some sus network traffic.

  • @mailoisback
    @mailoisback 8 місяців тому +2

    Any courses/books recommentations where someone can learn all this stuff to better understand what is being said in the video?

    • @DaveBucklin
      @DaveBucklin 8 місяців тому

      There's an Encryption course from UC San Diego on Coursera that gets into the implementation details of RSA.

  • @foobarf8766
    @foobarf8766 8 місяців тому +3

    Sheer luck it didn't get into the next Ubuntu, attacker missed the change freeze by hours, this could have been massive 😳

  • @fazlerabbi2666
    @fazlerabbi2666 8 місяців тому +2

    i don't know enough about the actual impact
    but this feels like the most of the internet could've gone dark due to this

  • @trudyandgeorge
    @trudyandgeorge 8 місяців тому +8

    ARE YOU KIDDING ME?! This was discovered more or less by accident. How unlucky is the perp. He would have had a golden ticket to some serious wealth if he got away with it. And how lucky are we?! This vuln blows my mind.

  • @quas-r
    @quas-r 8 місяців тому +2

    2:40 isn’t the private key used to decrypt and public to encrypt as opposed to what was said? Am I trippin?

    • @LowLevelTV
      @LowLevelTV  8 місяців тому +4

      signing

    • @BlackM3sh
      @BlackM3sh 8 місяців тому +2

      Not when signing, then you encrypt using the private key to prove to others you own the public key.

    • @quas-r
      @quas-r 8 місяців тому +1

      Oh I see thank you!

    • @filipmarkovic8638
      @filipmarkovic8638 8 місяців тому +1

      You can use them in whatever way you want, if one key is encrypting, second will always decrypt. Because of that, you can use the private key to prove identity of yourself, and anyone with your public key can decrypt the message and check if it's correct.

  • @russianmadman322
    @russianmadman322 8 місяців тому +4

    Signature algorithm does not always equal an encryption algorithm. What's been said in this video applies to RSA, but doesn't apply to ECDSA, for example.

  • @ExaltedVanguard
    @ExaltedVanguard 8 місяців тому

    This slipped through the cracks because it utilized binary files used as "known good" (or bad) values in tests. A fairly straightforward way of reducing/eliminating this attack surface is to instead use a hash of the generated data instead of comparing it to a "known good" binary.

  • @gareth2021
    @gareth2021 8 місяців тому +7

    andres freund is legit a hero

    • @IDJEGOI
      @IDJEGOI 8 місяців тому +7

      You could say he is a freund to us all

  • @bbrandonh
    @bbrandonh 8 місяців тому

    What about the xz changes caused the half second delay that eventually caused it to be discovered? Was it just the large amount of uncompressing that couldnt be avoided or could the attackers have done it in a way where it was faster and wouldn't have raised suspicions?

  • @Veilzlol
    @Veilzlol 8 місяців тому +3

    And now you have to wonder what is out there that has not been discovered by 500ms noticers

  • @techyguyaditya
    @techyguyaditya 8 місяців тому +1

    Why can't distros use automated build tools and host their generated tarballs in mirror servers instead of pulling from releases tarball?

  • @sydneytcp
    @sydneytcp 8 місяців тому +3

    well anticipated video

  • @GirLikesFarming
    @GirLikesFarming 8 місяців тому

    2:45 the public key is used to ENcrypt, the private key is used to DEcrypt. It would essentially be pointless to encrypt something if your decryption key was publicly available.

    • @brandyballoon
      @brandyballoon 8 місяців тому +7

      He's talking about a digital signature, which is achieved by encrypting the message hash with the private key. The idea is anyone with the public key can decrypt the hash, and if it matches their own hash of the message, then it proves two things: 1. Whoever/whatever encrypted the hash (signed the message) has the private key, which serves as authentication, and 2. The message has not been altered since it was signed.

    • @GirLikesFarming
      @GirLikesFarming 8 місяців тому

      @@brandyballoon Alright, thanks for clearing that up :)

  • @hqcart1
    @hqcart1 8 місяців тому +8

    a reminder to always protect your server to be accessed by 1 IP only, then even if you got infected, you are safe.

  • @ChrisTheCringe
    @ChrisTheCringe 8 місяців тому

    2:40 There is a small mistake from this point. Isn't a public key used encryption and a private key used for decryption? Am I missing something here, or is it a small mistake from @LowLevelLearning?

    • @tablettablete186
      @tablettablete186 8 місяців тому

      No it's actually right, there is no mistake. Signatures work "in reverse".
      Essentially, you want people to verify that you is you, so only you can encrypt the content (private key) and you use the public key (that you know it belongs to the person you want to verify). If it decrypts, we know to combination works. (I skipped tha hash verification for simplicity).
      -> Only the person that has the private key can encrypt the data correctly (so you know that the public key corresponds to the right person)
      When you want to hide information, you encrypt with the public key and decrypt with the private one
      -> Only the person that holds the private key can decrypt the message (which was encrypted with the corresponding public key).
      And don't worry, I also used to mix the two.

    • @ChrisTheCringe
      @ChrisTheCringe 8 місяців тому +1

      ​@@tablettablete186, This is actually quite confusing and interesting. Wow. I didn't know signatures acted that way!

    • @tablettablete186
      @tablettablete186 8 місяців тому

      @@ChrisTheCringe Tbh, I think I did a poor text, but I hope it helped.

  • @highdefinist9697
    @highdefinist9697 8 місяців тому +16

    Yeah... we need something like KYC for important open-source projects.
    Because, there is this one country to the east of Eastern Europe which will keep doing stuff like that.

    • @hyoenmadan
      @hyoenmadan 8 місяців тому +2

      Yeah.... In this way you just kill all the appeal of being Open Source. You may as well just rollback to being a closed source walled garden. ReactOS has done this for years (yes, they require KYC for commit access) and check how well goes to them.
      At least I would never contribute anything or support in any way any project which forces on me any sort of KYC scheme. Embrace, extend, extinguish.

    • @eight-double-three
      @eight-double-three 8 місяців тому +1

      Historically many countries have done many unethical things many times...

    • @highdefinist9697
      @highdefinist9697 8 місяців тому

      ​ @hyoenmadan No, the entire issue was only discovered thanks to open source. Closed source projects are much easier to infiltrate for organized adversarial actors.
      ​@@eight-double-three Even for a whataboutism, this one is kind of lazy.

    • @goraxe01
      @goraxe01 8 місяців тому +1

      ​@@hyoenmadanI think that just increases the chances of stuff like this being missed. This was caught early enough damage potential is low, solar winds from a couple of years back had much bigger impact

  • @rmod8
    @rmod8 8 місяців тому +1

    What's concerning is what other software could have been compromised using this method and if it already has been.

  • @Mudflap1110
    @Mudflap1110 8 місяців тому +11

    Starting to sound like a government 3 letter agency at work in the night 😮

    • @Uerdue
      @Uerdue 8 місяців тому +4

      *GCHQ hackers sighing with relief*

  • @manobendro
    @manobendro 8 місяців тому +1

    2:36 Is it really a public key for decrypting?

    • @brandyballoon
      @brandyballoon 8 місяців тому +2

      For a digital signature, yes.

  • @loganhodgsn
    @loganhodgsn 8 місяців тому +3

    7:21 lol, that's my ip address too!

  • @klauspsch
    @klauspsch 8 місяців тому +1

    The public key is used to ENCRYPT and the private to DECRYPT.

    • @CheCheDaWaff
      @CheCheDaWaff 8 місяців тому +1

      He was talking about signing not regular encryption. He got it the right way round!

    • @tablettablete186
      @tablettablete186 8 місяців тому

      Not for signatures, he is right.

  • @phoneix24886
    @phoneix24886 8 місяців тому +3

    This is MojoJojo level Evil.

  • @markosdelaportas3089
    @markosdelaportas3089 8 місяців тому

    Had libzma not been open source we ought to forget about ever discovering such implement!

  • @St0rMsk
    @St0rMsk 8 місяців тому +5

    Somewhere down in the basement of my hearth, I still believe that this was just an college / university student working on his thesis...

  • @louiskwan4970
    @louiskwan4970 8 місяців тому +1

    If they were smarter, maybe they could have encrypted all the extra logic on the sshd / xz side with their key? So the reverse engineering would be even harder? I suppose cpus have no execution bit to prevent writing to code segment but if it is running as root surely it should be possible..

    • @BillAnt
      @BillAnt 8 місяців тому +1

      True, but if they did that it would have been suspicious by a quick glance of the code looking encrypted which normally is not.

  • @n1t_
    @n1t_ 8 місяців тому +4

    Finally i just finished your video with Prime

  • @____-ps2qf
    @____-ps2qf 8 місяців тому

    > 5:48 ssh by default does not depend on lzma without running as a systemd service
    So if sshd from my distro repos isn't linked with liblzma, but i run as a systemd service, im still screwed?