I think the context that is missing here is "we at fly io are running other people's untrusted code on our servers". This is a pretty rare situation for a SaaS. In most other cases - containers work fine.
@@TheNefastor well, we have a bug bounty program. A lot of services have that - it is actually "challenging hackers to hack your system", and a lot of big names think it's a great idea😄
Containers solve multiple issues and not just isolation. You can still use micro VMs for isolation and containers inside those VMs to handle application deployments/dependencies.
Yes, I fully agree. Also, Containers are used to split your huge, monolithic system into smaller, maintainable parts. I wonder how much people hear, "oh, now I can go back to run my stuff into a single VM that has the same performance as a container and even higher security".
Exactly. For me, and my small business and personal use, containers are not about isolation. They are about dependency management. They completely solve the, "it works on my computer" problem. Edit: they say they have a kubernetes option. Interesting! That made just do the trick...
Good video but i really hope new devs dont walk away thinking containerization is bad. Containers like docker and docker compose mainly solve the, "well it works on my machine issue". It's not specifically made for security, per say. Although it's difficult and there are limitations. Docker container can be ran on a linux server with firejail for additional security that prevents cross contamination mentioned. That being said, firecracker is probably much easy and effective.
Often people think of "well it works in my machine" as being reproducible builds, which Docker does not and trying to use it as a tool to achieve this goal is counterproductive.
Having a consistent setup and decent documentation is also an answer to the "will it work on my machine" issue. Docker is just a shortcut that lets people get lazy with administration and development.
@@tbarnum6315 chroot isn’t like jail though. Chroot isn’t meant to be a security control. It’s just specifying an arbitrary / to start on. Jails were the first to build a security implementation with chroot as a component
@@classicrockonlyyes, as you noted, chroot is one beginning of a jail as it offers path isolation, as opposed to all the other parts like processes and so on. And all these were being explored back in the 70s on mainframes .
I must say that containerization has completely solved botched deployments for us. We haven't had to do a single rollback in 4 years, and it's because our entire service runs locally and deploys as-is. It took at long time to not be surprised that it just works in production.
Brilliant! Is really interesting to know how these hosting services run behind the scenes, and even more interesting to know they are not just AWS wrappers 😂
I'm convinced your product offering is much better thought out than most modernized hosting solutions. I really like the idea of micro VM's over containers.
they arent doing anything really different to aws lamda. having this automated vm spin up is just seemingly what firecracker allows for, although cool ig that amazon open sourced it
@gamagama69 yes but I'll be damned if I'll ever host with the company that screws small businesses. No way. I show love to the companies that at least pretend to love me back.
Yes, virtualization isolates better. However, Podman and CRI-O configured correctly completely get rid of the complaints you have with containers. Your issues as presented are with Docker, not containers.
containers still are chroot glorified i don't see how podman is diff or sandboxed ? sure you can use gVisor or some form of sandbox but how is that really helping much ?
you people are all insane from the perspective of someone who knows that VMs are very very slightly slower than containers and doesn't do web development. this mentality does not work in embedded system development where there are not system resources for VMs because you only have memory space for 1 kernel. but from the perspective of a web development SaaS provider i am sure these VM solutions offer maximum security and performance isn't a concern because it's all "in the cloud" on a load balanced cluster on AWS.
VMs really do eliminate more ambient-authority than cgroups do. That's only half of the battle, of course: how you manage the interfaces you expose to the tenants makes a huge impact to the overall security of the platform and the tenants on it, too!
This is my gripe with Docker. People just assume that containers = Docker. There are much better alternatives like Podman/CRI-O which can be configured so that all the security concerns mentioned here are addressed.
This is based on the same disregard for reality as the video. All container technologies (nowadays) are using the same Linux kernel features (unless they are VMs in disguise). You end up with processes running in confined environments resembling but not being isolated operating systems. I believe that such environments are reasonably secure, but necessarily less so than virtual machines. The real question is how much you reduce the likelihood to be affected by vulnerabilities by choosing Podman vs. Docker vs. KVM vs. firecracker? Quantifying that is incredibly difficult. I would guess that this decision is unlikely to have a huge impact on any particular deployment. But it is likely to have a noticeable impact on the monthly hosting bills. There are so many other aspects that would have a huge impact on the security of a system that are being ignored. Just looking at the causes of leaks and pwns reveals that the main reason for them is blatant disregard for security.
@@yanushkowalsky1402 The main difference between docker and podman is that podman does not run a (privileged) service on the host. Both end up running a process in a restricted environment using the same kernel features. The greatest problem I personally have with docker is that the company prioritizes its own agendas so much higher than those of their users. You can see that in how they handled dockerhub, how they annex the hosts firewall, how they package their software. If you use docker and are sensitive to security, you just don't feel comfortable.
@@yanushkowalsky1402 From my understanding it is "same" in terms of the API, but not implementation, making it more secure. I think daemonless and rootless are two big things for them, and they offload a lot of process management to systemd instead of implementing everything themselves.
Insightful! Thank you for making the video. The audio is really quiet... had to turn up my system volume and open a different video to make sure I wasn't going crazy and now all the other videos are shouting lol
Thank you for a good video that explains the approach to choose appropriate technology for appropriate usecase. Yes, Container was born for many thing but not multi-tenant. So in this case, choosing a MicroVM is the best balance for security and performance. Great choice! Developers should watch this video closely and learn the analysis, not to copy the entire decision. Container is still the greatest invention in software industry in our last decades.
There are roughly 4.5 types of virtualization along a spectrum with some hard cut offs. type 1 (hypervisor on hardware), type 2 (hypervisor within host os), type 2 + para (the .5), language VMs (e.g. java), and os virtualization (containers). There are significant trade offs between these in security, performance, and manageability. Devs need to understand this space so they can choose the appropriate technology for their applications. The tight separation concerns you had here that drove you towards what is almost a type 1.5 (type 1 sitting alongside a supporting OS?) might not be the same ones that drives somebody towards using containers. You provide a really interesting case that people can test their own needs against.
You're point of view is interesting. If I may add, thinking that containers are only here for security concerns is missing the point, they are a very handy way of distributing microservices over the whole architecture. VMs are here for the strong isolation, containers are here for the orchestration and we use both in combinations to deploy our clusters
Had a talk last week with a large enterprise and they are moving back from VMs to bare metal. Now I hear moving back from containers to VMs and cloud repatriation will be a big item in 2025. We all love the nostalgia of what we were used to use it seems!
I don't think security is the main problem with containers; clearly this has been figured out. (Sure, nothing except air-gapping gives 100% guarantee, but that's the same with hypervisors.) The actual problem with containers is that they're inflexible and awkward to administrate. It would be much more convenient to have something like NixOS running on those VMs and submit a configuration instead of a Docker image.
@@NitroNilz I mean, sure... in doubt, there's always the bribe-the-guards-to-let-you-in-attack. My point wasn't really about air-gapped system but non-airgapped ones, and for sure those aren't 100%. But the question shouldn't be whether a system is 100% safe but 99.how-many-nines-9% it's safe.
@@leftaroundabout I understand Ur point. Mine was to correct the 100% statement - and I was not thinking of "bribe the guards" even. Just recently there was yet ANOTHER air-gap attack disclosed. Not social engineering, but technological atacks. Use a search engine if interested. There are plenty!
@@NitroNilz yet, all else being equal, an air-gapped system will always be much safer than one connected to the internet, let alone one running somewhere in the cloud.
I have been debating with a lot of professionals that VM better than Containers..... finally. :) And this is the first time I hear chroot is pronounced literally, rather than "change root". Cool to know.
CloudLinux also made CageFS which essentially is the deluxe edition of a BSD jail where it will emulate a lot of the normal structure and enforce separation in a kernel module. They also have LVE for respurce limits, but that's basically just cgroups with some better accounting at this point.
I love containerization, but you're right. Running arbitrary user code in anything but a vm is reckless. I think those disagreeing with your points don't understand what your end goal is. Otherwise, in literally any other case besides running arbitrary user code I'd use containers instead.
Regarding your point at 1:56 - the same argument could be made for a single breach of the hypervisor as well. I feel the perspective in this video basically boils down to "hypervisors get more security attention than kernels" but I could be interpreting wrong. If I'm interpreting correctly though, I'm not sure I agree with this video. but then again I don't run a cloud company so.....
Containers solve a reproducibility issue first, then isolated security second. Its main benefit over VMs its that it provides a declarative way to define a setup that can be pushed to version control that will always work regardless of your machine (for the most part, even different ISA servers can work with the correct multi-arch base container images). While yes, automated tooling like ansible and terraform exist for setting up infrastructure, but they are a lot harder to get started compared to containers and solve a different problem. Containers are the perfect balance between consistent deployment and ease of use allowing you to start from a base image, add dependencies and then deploy that image locally or on the cloud.
As a senior SRE, I disagree with the concerns you’ve raised about containers. In a well-architected Kubernetes environment, it’s possible to achieve everything you’ve mentioned using containers, and this can be done without the overhead of VMs.
I'm team qemu and avoid containers for the reasons that you listed. I pronounce, chroot as "chuh'root" as it comes from "change root"; it is interesting to hear someone pronounce it differently.
Good information for sure. I am not familiar with Firecracker, but if I know what’s running in other containers, then I can create a named system semaphore that will await and then block threads in those containers. However, as I said, I may not familiar with Firecracker, and since named semaphores are an OS wide resource, if FC does truly isolate process into a separate OS, then it would prevent this. I can think of several OS wide resources I can create inside of a container which can and will affect the entire host OS, given the proper convergence of asset allocated/named entry vectors, but this is highly unlikely to be unnoticed by a properly run SECOPS team (a rarity among organizations today, sadly). If you have any doubt, I would certainly recommend complete process isolation via a dedicated OS, else someone like me (I am a white hat, someone like me whose a black hat) is, given the motivation, likely to become your organizations last cybersecurity incident.
While this video is accurate, It dismisses that a type one hypervisor is still running a kernel that all the VMs talk to. The difference here is that the user applications don’t talk directly to that kernel. Each VM kernel sits between the user code and the hypervisor kernel. It’s a little misleading to act as if though there’s still not some kernel bridging everything together. Just a typical OS kernel allows multiple processes to run independently of each other, a hypervisor kernel lets multiple OS kernels run independently of each other. The tradeoff being the overhead of running n+1 kernels or the concern that the Linux kernel process isolation is more easily exploitable than the firecracker one.
If you use containers for isolation, you do not understand containers at all. Containers are not for isolation, containers are for giving an application everything it needs to run on a system without having to rely on or even touch the system itself, making it easy to move the container from one system to another without having to worry about dependencies or system configuration. Containers are not a security feature; they do increase security compared to systems that don't have containers or virtualization at all, but that's a byproduct, not the primary reason why containers were created in the first place.
Yea containers are just apps in the era of microservices. You still have to actually secure the apps themselves 💀 they aren’t going to be secure just by the nature of being in Docker or Kubernetes
This is not an accurate description of what is going on. Thinking of containers as "chroot on steroids" is valid, with all that implies. But to say "containers are not a security feature" is the same as Virtual Memory is not a security feature, when it, absolutely, is. All process isolation is a form of security. If you have a hole in your kernel, you're in trouble before you start... Plus, you're assuming that there are no security issues with the kernel/hypervisor you're VM is running on. If you know anything, you would know that that is a very foolish assumption indeed...
@@edwardcullen1739 VM is not a security feature in the sense that it was designed to prevent malicious hackers from taking over your computer. It was designed to make sure that a process that goes haywire doesn't kill your entire system. When VM was invented, processes stealing data from other processes wasn't really a problem, because in most cases a process couldn't do anything with the stolen data because there was no communication it could use to send it anywhere outside the local machine. Computers that were always on networks may have implemented VM to increase security, but VM is older than those computers. Just because you can build a steel door and it's more secure than a wooden one doesn't mean that steel is a security feature or was invented to create more secure doors. That you can use it to make more secure doors is a side effect, not a core property of steel.
@@xcoder1122 In the context of computer security; what does the "A" in the acronym "CIA" stand for? Virtual Memory is, absolutely, a security feature, you just have an incorrect definition of what "security" means. I'm not splitting hairs or "being smart"; this is fundamental to the whole discussion of security. The most secure system is one buried in concrete in the middle of nowhere, where everyone who knows where it is - including the guys who dug the whole for an unknown reason - is dead. It's not a very _useful_ system, but it is "secure".
I love all the comments with "Yes, BUT ...." describing scenarios that are completely different than the one described on the video. No one is saying containers are bad, however, a lesson from this is that you need to understand that no software product is the best tool for everything. No silver bullets exists. Yes, sometimes using containers is not the best idea
0:22 "They provide better isolation" that is very dangerous generalisation! Also for me, all CPU exploits discovered to date push me away from any PAAS offering
I actualy tried to play with kata on k3s with fc to test it but I couldnt get it to work, but this video inspired me to try it again, BTW you are probably runing fc on k8s BTS for scaling, am I right?
But how exactly VMs provide better isolation? Would love to hear more about that. Edit: nevermind, it was explained, I just assumed from the tone that you were going to just casually mention it and not come back to elaborate, my bad 😸
Correct! We basically just unpack the layers of your image into a Firecracker VM. I made another video talking all about this, look for "Docker without Docker".
It's not really clear to me how applications sharing the same kernel being able to compromise other containers differs from applications sharing the same hardware being able to compromise other VMs
I love the explanation about microVMs, but what about real-world scenarios? Could you provide examples of actual workloads, not just “state-of-the-art architectures solving imaginary problems”?
That's how the Docker started, but now the Docker containers can run different kernels, like Linux-kernel in container on Windows machine, but that fact would make the alternatives to Docker look not so shiny anymore.
What if the Hypervisor layer is compromised through a vulnerability? I don't know if it's possible or not but I imagine everything is possible in the hacking space if they put enough effort
The irony is, people say VMs are more secure, but last time someone actually sat down a did a check on all the security patch releases, turns out, containers had less security issues than VMs. But was was most disappointing of the video was no use of the pun: that ain't gonna fly
Let's make a poll, how do you pronounce "chroot"? 1) ch-root (change root) 2) churoot 3) cheroot 4) c-h-root On another note, Firecracker is great, it provides a lot of isolation, but still things like Medusa and Spectre are dangerous and can have a really great performance impact on performance (if mitigated), still Firecracker is possibly the most secure way to run multi-tenant apps that are almost as fast as a container, so I guess it is a really good trade-off.
if you need good isolation (multi tenant, untrusted code) then yes that's a valid point. If you just multiple trusted services on one host, virtual machines are kind of overkill. So as always in IT.. it depends!
I agree with security concerns about containers, they are great yes, but not for every application and type of workloads, for example I'd not expose a linux server with docker engine on top in a DMZ
Do what you must but no technology can negate the golden rule: If it touches a network, it’s vulnerable. And since completely isolated systems are useless in most cases, do your due diligence, in any case.
I hope new devs don't take what you say too much to heart, as it depends on a lot of factors. Also, VM's and containers solve different problems beyond isolation, and they are not at all mutually exclusive. To phrase it as using VM's over containers is a false comparison, and shows a lack of maturity in development architectures and options. For example, not all containers are rootful, and you are generalizing that Docker == all container tech. In any case, knowing what is most widely used is generally a worthwhile investment. As such, if you want to work in AI or related fields with a range of great companies, you better at the very least know Docker, Conda, and Python/PyTorch, or whatever serves your vertical. I can't tell you the number of young people coming in who have massive gaps for AI and related dev and wind-up starting at much lower salaries because of this, which could have been easily addressed.
Containerization for me it’s not so much about the application isolation, but more so about consistent development environments and easier deployments.
Don’t mislead people. Just say that you are running untrusted docker images from other’s people, and that’s why you need more security. This shouldn’t be an issue for ppl who runs their own containers
You kinda just reinvented the wheel. You can have (and most do) both containers and virtualization. Make clusters based on trust level and deploy apps according to level of trust. The platform runs on VMs, the apps run on containers. Haven't seen a PaaS run on bare metal for more than 6 years now.
We use containers in production and it’s so much better than using virtual machines. Also both virtualization and containerization are built right into the Linux kernel and are both secure. Your argument is a non argument. Also the tooling around containers is much better. And you can use serverless containers e.g. on AWS easily, with ECS.
mudi2000a Container tooling is definitely easier to manage. My point is that they are not "both secure" (or equally secure) when it comes to executing arbitrary code. There is a reason why AWS runs containers under a VM and not directly in bare metal.
@@tablettablete186 yes that makes complete sense. I fully agree with that. If you are however self hosting your own app the question is if that is needed.
mudo2000a Yeah, if you are self hosting, you are not runnig arbitrary code. Agree, a VM might not be too benefitial. I still think you might get some protection if you host a internet facing service (situation: your service gets compromised and now runs arbitrary code).
Not true, Docker can be protected via namespace which allow to run applications in non root setup and then you have AppArmor or SELinux as additional layer of security.
Docker uses lightweight _containers_ not lightweight VMs. Containers share the host kernel, which provides less isolation than VMs which each have their own kernel. This was literally the point of the video.
@@phobos.anomaly running Docker barebones on a system is foolish if security is that important. You have a VM layer running the stuff that has outside access - that's how proper security is done. Simply running VMs instead of docker takes more resources, and isn't inherently more secure than a normal secure setup. The gap in reasoning seems to be based on how security works, which, to keep it simple, run everything in a VM but don't tout it as less secure to do the exact same thing just with docker for convenience.
Everything sounds great and all but your VMs are not scaling quick enough for big payloads like image binaries. On the other hand containers are smooth like butter on GCP. Note: I have load tested your systems for two weeks / created a ticket on support site/ email to support specialist etc.. before switching to GCP.
I think the context that is missing here is "we at fly io are running other people's untrusted code on our servers". This is a pretty rare situation for a SaaS. In most other cases - containers work fine.
You mean other services think they can trust user code ???
@@TheNefastor ha. In my system, "user code" is a bunch of XML. You are welcome to try a container escape with that.
@@guss77 challenging hackers to hack your system has never been a good idea, but you do you...
@@TheNefastor well, we have a bug bounty program. A lot of services have that - it is actually "challenging hackers to hack your system", and a lot of big names think it's a great idea😄
@@guss77 fly is a PaaS
Containers solve multiple issues and not just isolation. You can still use micro VMs for isolation and containers inside those VMs to handle application deployments/dependencies.
Yes, I fully agree. Also, Containers are used to split your huge, monolithic system into smaller, maintainable parts. I wonder how much people hear, "oh, now I can go back to run my stuff into a single VM that has the same performance as a container and even higher security".
Exactly is not a question of everything VMs or everything containers, we can mix them
Exactly. For me, and my small business and personal use, containers are not about isolation. They are about dependency management. They completely solve the, "it works on my computer" problem.
Edit: they say they have a kubernetes option. Interesting! That made just do the trick...
There are no right answers, only trade-offs!
is not "just" a VM, she is talking about microVM so security like a conventional VM but lightweight as a container.
Good video but i really hope new devs dont walk away thinking containerization is bad. Containers like docker and docker compose mainly solve the, "well it works on my machine issue". It's not specifically made for security, per say.
Although it's difficult and there are limitations. Docker container can be ran on a linux server with firejail for additional security that prevents cross contamination mentioned.
That being said, firecracker is probably much easy and effective.
Often people think of "well it works in my machine" as being reproducible builds, which Docker does not and trying to use it as a tool to achieve this goal is counterproductive.
well it works on my docker
Having a consistent setup and decent documentation is also an answer to the "will it work on my machine" issue. Docker is just a shortcut that lets people get lazy with administration and development.
Another option is to fucking learn how software works and how to fucking install your fucking shit without using docker.
@greyshopleskin2315 I mean you could also learn emotional discipline when commenting. Dear God, imagine how bad your commits are...
Jailing is a real term. Jails, the original containers, originated from FreeBSD in the 90s, before Solaris zones and Linux containers existed
Look up history of chroot command, it's before then and early work on mainframes predates that
@@tbarnum6315 chroot isn’t like jail though. Chroot isn’t meant to be a security control. It’s just specifying an arbitrary / to start on. Jails were the first to build a security implementation with chroot as a component
@@classicrockonlyyes, as you noted, chroot is one beginning of a jail as it offers path isolation, as opposed to all the other parts like processes and so on. And all these were being explored back in the 70s on mainframes .
@@tbarnum6315 it’s cool stuff. I’ve been particularly interested in Solaris Zones as of late
Maybe AIX's LPARs pre-dated Jails and Zones...
I must say that containerization has completely solved botched deployments for us. We haven't had to do a single rollback in 4 years, and it's because our entire service runs locally and deploys as-is. It took at long time to not be surprised that it just works in production.
Brilliant! Is really interesting to know how these hosting services run behind the scenes, and even more interesting to know they are not just AWS wrappers 😂
I wish I had a mentor who could explain things as simple and clear as she does.
Aww thank you!!
Well here’s your mentor buddy
@@vikingthedude 😂
Simple does not mean correct. Videos like this are misleading to young devs
@@JohnArroyo1 What's misleading here?
I'm convinced your product offering is much better thought out than most modernized hosting solutions. I really like the idea of micro VM's over containers.
We do too!!
they arent doing anything really different to aws lamda. having this automated vm spin up is just seemingly what firecracker allows for, although cool ig that amazon open sourced it
@gamagama69 yes but I'll be damned if I'll ever host with the company that screws small businesses. No way.
I show love to the companies that at least pretend to love me back.
Yes, virtualization isolates better. However, Podman and CRI-O configured correctly completely get rid of the complaints you have with containers. Your issues as presented are with Docker, not containers.
Do you have a recommendation for a good resource to learn about the "correct" config for Podman or CRI-O?
containers still are chroot glorified i don't see how podman is diff or sandboxed ?
sure you can use gVisor or some form of sandbox but how is that really helping much ?
you people are all insane from the perspective of someone who knows that VMs are very very slightly slower than containers and doesn't do web development. this mentality does not work in embedded system development where there are not system resources for VMs because you only have memory space for 1 kernel.
but from the perspective of a web development SaaS provider i am sure these VM solutions offer maximum security and performance isn't a concern because it's all "in the cloud" on a load balanced cluster on AWS.
@@GreatTaiwanWell gsivor is essentially a VM that fowards syscalls to the host
THIS TBH. She's assuming running containers = running rootful containers
VMs really do eliminate more ambient-authority than cgroups do. That's only half of the battle, of course: how you manage the interfaces you expose to the tenants makes a huge impact to the overall security of the platform and the tenants on it, too!
This is my gripe with Docker. People just assume that containers = Docker. There are much better alternatives like Podman/CRI-O which can be configured so that all the security concerns mentioned here are addressed.
This is based on the same disregard for reality as the video. All container technologies (nowadays) are using the same Linux kernel features (unless they are VMs in disguise). You end up with processes running in confined environments resembling but not being isolated operating systems. I believe that such environments are reasonably secure, but necessarily less so than virtual machines.
The real question is how much you reduce the likelihood to be affected by vulnerabilities by choosing Podman vs. Docker vs. KVM vs. firecracker? Quantifying that is incredibly difficult. I would guess that this decision is unlikely to have a huge impact on any particular deployment. But it is likely to have a noticeable impact on the monthly hosting bills.
There are so many other aspects that would have a huge impact on the security of a system that are being ignored. Just looking at the causes of leaks and pwns reveals that the main reason for them is blatant disregard for security.
Isn't podman pretty much the same as docker though?
@@yanushkowalsky1402 The main difference between docker and podman is that podman does not run a (privileged) service on the host. Both end up running a process in a restricted environment using the same kernel features.
The greatest problem I personally have with docker is that the company prioritizes its own agendas so much higher than those of their users. You can see that in how they handled dockerhub, how they annex the hosts firewall, how they package their software. If you use docker and are sensitive to security, you just don't feel comfortable.
@@yanushkowalsky1402 From my understanding it is "same" in terms of the API, but not implementation, making it more secure. I think daemonless and rootless are two big things for them, and they offload a lot of process management to systemd instead of implementing everything themselves.
Docker needs to run as Windows admin, not Podman.
Insightful! Thank you for making the video. The audio is really quiet... had to turn up my system volume and open a different video to make sure I wasn't going crazy and now all the other videos are shouting lol
I only noticed that once I hit publish, oops! I'll fix the audio next time.
Concise explanation with zero filler. Great video.
It's not true that you can't escape the hypervisor it's just harder
Just ask Tron.
Much harder than to escape container anyways
@@brylozketrzyn The issue is that it is still possible but only for small percent of people.
@@et2931 what makes that percentage small? i understand it should have something to do with how you escape the vm but why "fewer people"?
Few people are that level
All you said it was valid for containers around 7-8 years ago
Today, it is very different.
Why is it different today? Do containers share a kernel? Yes!
Not really
@@ankopainting I think he's talking about rootless containers, as podman does.
The video problem is a docker thing, not a container thing afian
@@ankopainting gVisor my friend ;)
I wouldn't expect this video to have 2 million views
Thank you for a good video that explains the approach to choose appropriate technology for appropriate usecase. Yes, Container was born for many thing but not multi-tenant. So in this case, choosing a MicroVM is the best balance for security and performance. Great choice! Developers should watch this video closely and learn the analysis, not to copy the entire decision. Container is still the greatest invention in software industry in our last decades.
There are roughly 4.5 types of virtualization along a spectrum with some hard cut offs. type 1 (hypervisor on hardware), type 2 (hypervisor within host os), type 2 + para (the .5), language VMs (e.g. java), and os virtualization (containers). There are significant trade offs between these in security, performance, and manageability. Devs need to understand this space so they can choose the appropriate technology for their applications. The tight separation concerns you had here that drove you towards what is almost a type 1.5 (type 1 sitting alongside a supporting OS?) might not be the same ones that drives somebody towards using containers. You provide a really interesting case that people can test their own needs against.
I'm surprised no one in the comments has mentioned podman yet. It resolves each issue she brought up with containers
How so? It still shares the kernel, just like docker.
You're point of view is interesting. If I may add, thinking that containers are only here for security concerns is missing the point, they are a very handy way of distributing microservices over the whole architecture. VMs are here for the strong isolation, containers are here for the orchestration and we use both in combinations to deploy our clusters
Had a talk last week with a large enterprise and they are moving back from VMs to bare metal. Now I hear moving back from containers to VMs and cloud repatriation will be a big item in 2025. We all love the nostalgia of what we were used to use it seems!
I don't think security is the main problem with containers; clearly this has been figured out. (Sure, nothing except air-gapping gives 100% guarantee, but that's the same with hypervisors.)
The actual problem with containers is that they're inflexible and awkward to administrate. It would be much more convenient to have something like NixOS running on those VMs and submit a configuration instead of a Docker image.
Airgapping is not a 100% guarante. Many techniques circumvent(pun intended) air gaps.
@@NitroNilz I mean, sure... in doubt, there's always the bribe-the-guards-to-let-you-in-attack. My point wasn't really about air-gapped system but non-airgapped ones, and for sure those aren't 100%. But the question shouldn't be whether a system is 100% safe but 99.how-many-nines-9% it's safe.
@@leftaroundabout
I understand Ur point. Mine was to correct the 100% statement - and I was not thinking of "bribe the guards" even. Just recently there was yet ANOTHER air-gap attack disclosed. Not social engineering, but technological atacks. Use a search engine if interested. There are plenty!
@@NitroNilz yet, all else being equal, an air-gapped system will always be much safer than one connected to the internet, let alone one running somewhere in the cloud.
This is why the visor project, which we started at Google in 2014, works the way it does -- very lightweight VM.
I have been debating with a lot of professionals that VM better than Containers..... finally. :)
And this is the first time I hear chroot is pronounced literally, rather than "change root". Cool to know.
love your content. fly io did very well by letting you do these stuff.
Realy great delivery!
CloudLinux also made CageFS which essentially is the deluxe edition of a BSD jail where it will emulate a lot of the normal structure and enforce separation in a kernel module. They also have LVE for respurce limits, but that's basically just cgroups with some better accounting at this point.
I love containerization, but you're right. Running arbitrary user code in anything but a vm is reckless. I think those disagreeing with your points don't understand what your end goal is.
Otherwise, in literally any other case besides running arbitrary user code I'd use containers instead.
What an awesomely explained video. Makes me want to use Fly
This is so well explained... it took just 4min not 27min dang
Containers are not about security, they are about managing dependencies.
And idempotent
Love the bloopers 😀
Regarding your point at 1:56 - the same argument could be made for a single breach of the hypervisor as well. I feel the perspective in this video basically boils down to "hypervisors get more security attention than kernels" but I could be interpreting wrong. If I'm interpreting correctly though, I'm not sure I agree with this video. but then again I don't run a cloud company so.....
Containers solve a reproducibility issue first, then isolated security second. Its main benefit over VMs its that it provides a declarative way to define a setup that can be pushed to version control that will always work regardless of your machine (for the most part, even different ISA servers can work with the correct multi-arch base container images).
While yes, automated tooling like ansible and terraform exist for setting up infrastructure, but they are a lot harder to get started compared to containers and solve a different problem. Containers are the perfect balance between consistent deployment and ease of use allowing you to start from a base image, add dependencies and then deploy that image locally or on the cloud.
That outro, well done.
I just love this. Docker cultists vs Rusted cultists. Bring me my popcorn 🍿
As a senior SRE, I disagree with the concerns you’ve raised about containers. In a well-architected Kubernetes environment, it’s possible to achieve everything you’ve mentioned using containers, and this can be done without the overhead of VMs.
I'm team qemu and avoid containers for the reasons that you listed. I pronounce, chroot as "chuh'root" as it comes from "change root"; it is interesting to hear someone pronounce it differently.
QEMU is great, but it looks like Firecracker is faster and more secure out of the box
Great vid... would have been nice so see a set up demo tho!
Good information for sure. I am not familiar with Firecracker, but if I know what’s running in other containers, then I can create a named system semaphore that will await and then block threads in those containers. However, as I said, I may not familiar with Firecracker, and since named semaphores are an OS wide resource, if FC does truly isolate process into a separate OS, then it would prevent this. I can think of several OS wide resources I can create inside of a container which can and will affect the entire host OS, given the proper convergence of asset allocated/named entry vectors, but this is highly unlikely to be unnoticed by a properly run SECOPS team (a rarity among organizations today, sadly). If you have any doubt, I would certainly recommend complete process isolation via a dedicated OS, else someone like me (I am a white hat, someone like me whose a black hat) is, given the motivation, likely to become your organizations last cybersecurity incident.
While this video is accurate, It dismisses that a type one hypervisor is still running a kernel that all the VMs talk to. The difference here is that the user applications don’t talk directly to that kernel. Each VM kernel sits between the user code and the hypervisor kernel. It’s a little misleading to act as if though there’s still not some kernel bridging everything together. Just a typical OS kernel allows multiple processes to run independently of each other, a hypervisor kernel lets multiple OS kernels run independently of each other. The tradeoff being the overhead of running n+1 kernels or the concern that the Linux kernel process isolation is more easily exploitable than the firecracker one.
There is better knowledge base in comments than in the video. I love it. 😅
If you use containers for isolation, you do not understand containers at all. Containers are not for isolation, containers are for giving an application everything it needs to run on a system without having to rely on or even touch the system itself, making it easy to move the container from one system to another without having to worry about dependencies or system configuration. Containers are not a security feature; they do increase security compared to systems that don't have containers or virtualization at all, but that's a byproduct, not the primary reason why containers were created in the first place.
Yea containers are just apps in the era of microservices. You still have to actually secure the apps themselves 💀 they aren’t going to be secure just by the nature of being in Docker or Kubernetes
This is not an accurate description of what is going on.
Thinking of containers as "chroot on steroids" is valid, with all that implies.
But to say "containers are not a security feature" is the same as Virtual Memory is not a security feature, when it, absolutely, is.
All process isolation is a form of security.
If you have a hole in your kernel, you're in trouble before you start...
Plus, you're assuming that there are no security issues with the kernel/hypervisor you're VM is running on. If you know anything, you would know that that is a very foolish assumption indeed...
@@edwardcullen1739 VM is not a security feature in the sense that it was designed to prevent malicious hackers from taking over your computer. It was designed to make sure that a process that goes haywire doesn't kill your entire system. When VM was invented, processes stealing data from other processes wasn't really a problem, because in most cases a process couldn't do anything with the stolen data because there was no communication it could use to send it anywhere outside the local machine. Computers that were always on networks may have implemented VM to increase security, but VM is older than those computers. Just because you can build a steel door and it's more secure than a wooden one doesn't mean that steel is a security feature or was invented to create more secure doors. That you can use it to make more secure doors is a side effect, not a core property of steel.
@@xcoder1122 In the context of computer security; what does the "A" in the acronym "CIA" stand for?
Virtual Memory is, absolutely, a security feature, you just have an incorrect definition of what "security" means.
I'm not splitting hairs or "being smart"; this is fundamental to the whole discussion of security.
The most secure system is one buried in concrete in the middle of nowhere, where everyone who knows where it is - including the guys who dug the whole for an unknown reason - is dead.
It's not a very _useful_ system, but it is "secure".
Firecracker is great, finally someone shares this
I love all the comments with "Yes, BUT ...." describing scenarios that are completely different than the one described on the video. No one is saying containers are bad, however, a lesson from this is that you need to understand that no software product is the best tool for everything. No silver bullets exists. Yes, sometimes using containers is not the best idea
Interesting video!
very enlightning, thank you very much
0:22 "They provide better isolation" that is very dangerous generalisation!
Also for me, all CPU exploits discovered to date push me away from any PAAS offering
I actualy tried to play with kata on k3s with fc to test it but I couldnt get it to work, but this video inspired me to try it again, BTW you are probably runing fc on k8s BTS for scaling, am I right?
My first thought when seeing the title was "what, no dictionaries or arrays?" before remembering that Docker exists.
But how exactly VMs provide better isolation? Would love to hear more about that.
Edit: nevermind, it was explained, I just assumed from the tone that you were going to just casually mention it and not come back to elaborate, my bad 😸
This would have been a good video if it was on a personal channel. I don't think this should have been uploaded to an official company account.
If you rent your compute powers to third party, dedicated VMs make sense
I have been using fly and it is the best for indie dev like me best product.
How hard is it to install firecracker? Do I need to do some skullduggery in my bios, like with Docket Desktop?
When I clicked on the video I thought you were going to try to convince me to throw away my Tupperware.
So educational. Please more videos
4:09 Beta kubernetes? For managing container or what?
You can manage firecracker vms with some kind of k8s setup
So is it correct to say that you convert your users's container images to whatever is necessary to run a Firecracker VM?
Correct! We basically just unpack the layers of your image into a Firecracker VM. I made another video talking all about this, look for "Docker without Docker".
Some context of what you're using your lightweight VMs for would be useful
Kata Containers does the same thing. Kata containers is open source so that you can set it up yourself.
Have you guys tried exploring Podman?
It's not really clear to me how applications sharing the same kernel being able to compromise other containers differs from applications sharing the same hardware being able to compromise other VMs
this video was in my feed and I thought it was going to be about why this woman doesn’t use food containers
I love the explanation about microVMs, but what about real-world scenarios? Could you provide examples of actual workloads, not just “state-of-the-art architectures solving imaginary problems”?
That's how the Docker started, but now the Docker containers can run different kernels, like Linux-kernel in container on Windows machine, but that fact would make the alternatives to Docker look not so shiny anymore.
What if the Hypervisor layer is compromised through a vulnerability? I don't know if it's possible or not but I imagine everything is possible in the hacking space if they put enough effort
Have you used OpenShift and its security ?
The irony is, people say VMs are more secure, but last time someone actually sat down a did a check on all the security patch releases, turns out, containers had less security issues than VMs.
But was was most disappointing of the video was no use of the pun: that ain't gonna fly
Liked just for the wordplay!
Missed opportunity to call your VM, Dragonfly 😅
Let's make a poll, how do you pronounce "chroot"?
1) ch-root (change root)
2) churoot
3) cheroot
4) c-h-root
On another note, Firecracker is great, it provides a lot of isolation, but still things like Medusa and Spectre are dangerous and can have a really great performance impact on performance (if mitigated), still Firecracker is possibly the most secure way to run multi-tenant apps that are almost as fast as a container, so I guess it is a really good trade-off.
if you need good isolation (multi tenant, untrusted code) then yes that's a valid point.
If you just multiple trusted services on one host, virtual machines are kind of overkill.
So as always in IT.. it depends!
I agree with security concerns about containers, they are great yes, but not for every application and type of workloads, for example I'd not expose a linux server with docker engine on top in a DMZ
All these people being using containers and VMS...
Meanwhile, I just run everything barebone on a VPS.
Thank you so much for sharing this. is the code that starts/stops firecracker code open source as well ?
clear and to the point!
Thanks!
Do what you must but no technology can negate the golden rule:
If it touches a network, it’s vulnerable.
And since completely isolated systems are useless in most cases, do your due diligence, in any case.
With Docker actually i have always freedom of choice, i can run it everywhere , don't forget this!
Hypervisor escapes exist, they are just owned by the nation states, so they are much more rare and exclusive. You're swapping your risk.
Containers for security is a bad idea, but they are a fine solution in other areas.
I hope new devs don't take what you say too much to heart, as it depends on a lot of factors. Also, VM's and containers solve different problems beyond isolation, and they are not at all mutually exclusive. To phrase it as using VM's over containers is a false comparison, and shows a lack of maturity in development architectures and options. For example, not all containers are rootful, and you are generalizing that Docker == all container tech.
In any case, knowing what is most widely used is generally a worthwhile investment. As such, if you want to work in AI or related fields with a range of great companies, you better at the very least know Docker, Conda, and Python/PyTorch, or whatever serves your vertical. I can't tell you the number of young people coming in who have massive gaps for AI and related dev and wind-up starting at much lower salaries because of this, which could have been easily addressed.
What about performance in VM's? Or are we just going to talk about perf in interviews?
Running separate VM aint cheap
unless a particular requirement is needed , containers are fine
Containerization for me it’s not so much about the application isolation, but more so about consistent development environments and easier deployments.
Don’t mislead people. Just say that you are running untrusted docker images from other’s people, and that’s why you need more security. This shouldn’t be an issue for ppl who runs their own containers
You kinda just reinvented the wheel. You can have (and most do) both containers and virtualization. Make clusters based on trust level and deploy apps according to level of trust. The platform runs on VMs, the apps run on containers. Haven't seen a PaaS run on bare metal for more than 6 years now.
great video
We use containers in production and it’s so much better than using virtual machines. Also both virtualization and containerization are built right into the Linux kernel and are both secure. Your argument is a non argument.
Also the tooling around containers is much better. And you can use serverless containers e.g. on AWS easily, with ECS.
The best part is that the serverless containers you mentioned run on Firecraker on AWS lol
@@tablettablete186 I know how it works the important part is the management and building of the containers. That is much easier than dealing with VMs.
mudi2000a Container tooling is definitely easier to manage.
My point is that they are not "both secure" (or equally secure) when it comes to executing arbitrary code. There is a reason why AWS runs containers under a VM and not directly in bare metal.
@@tablettablete186 yes that makes complete sense. I fully agree with that.
If you are however self hosting your own app the question is if that is needed.
mudo2000a Yeah, if you are self hosting, you are not runnig arbitrary code. Agree, a VM might not be too benefitial.
I still think you might get some protection if you host a internet facing service (situation: your service gets compromised and now runs arbitrary code).
Not true, Docker can be protected via namespace which allow to run applications in non root setup and then you have AppArmor or SELinux as additional layer of security.
Oh so you're using lightweight VMs with extra functionality... Kind of like exactly what docker is?
Docker uses lightweight _containers_ not lightweight VMs. Containers share the host kernel, which provides less isolation than VMs which each have their own kernel. This was literally the point of the video.
@@phobos.anomaly running Docker barebones on a system is foolish if security is that important. You have a VM layer running the stuff that has outside access - that's how proper security is done.
Simply running VMs instead of docker takes more resources, and isn't inherently more secure than a normal secure setup. The gap in reasoning seems to be based on how security works, which, to keep it simple, run everything in a VM but don't tout it as less secure to do the exact same thing just with docker for convenience.
@@phobos.anomaly besides, escaping a VM isn't trivial but it isn't impossible either. It's security through obscurity. Look it up.
So is each Kubernetes pod it's own Firecracker VM?
People if you use proprietary vm you limit yourself to only one provider , that's just for notice , think about that.
what are you going to do about your TLD
We don't use containers, we use harder containers.
It is possible to escape the hypervisor, just waay harder than going around the container.
I get it vms are more secured but it no where near docker ecosystem so that a bummer.
Containers are often favoured because they are the new buzz. Virtualisation for the most part is a battle hardened platform for all to consume.
Everything sounds great and all but your VMs are not scaling quick enough for big payloads like image binaries. On the other hand containers are smooth like butter on GCP.
Note: I have load tested your systems for two weeks / created a ticket on support site/ email to support specialist etc.. before switching to GCP.
What? VM have been scaling quick enough for decades what do you think ec2 is? The strategy they follow is different but you can scale quick.
Actually it *is* a real term but only in several Unix systems (Solaris, HPUX, DragonflyBSD, FreeBSD, etc) , not in Linux
Considering containers for security is same considering VMs for flexibility and performance.