How fast can I create 10,000 kubernetes pods?!

I managed to squeeze over 11k pods into a cluster with just 44 x 1vCPU worker nodes. I had to deploy my own control plane, increase the maximum number of pods/node, and use a tiny container that could run with just couple thousandths of a CPU!
This was a fun challenge to celebrate reaching 10,000 subscribers! In case you are wondering, the cost to spin this cluster up for the time spent recording was $1.67.
Here is Jeff's original video: • 10,000 Kubernetes Pods...
---
Join the Community:
💬 Discord: / discord
💻 GitHub: github.com/sidpalas/devops-di...
🐥 Twitter: / sidpalas
👨‍💼 LinkedIn: / sid-palas
🌐 Website: devopsdirective.com
---
Timestamps:
00:00 - Intro
00:19 - Jeff Geerling's 10k pod video
00:47 - Google Kubernetes Engine (GKE) Autopilot
01:35 - Quota increases
01:48 - New tactics!
02:30 - Kops (Kubernetes Operations)
02:51 - Cluster modifications
03:06 - Defining deployment config
03:28 - Attempt footage!
05:00 - Halfway
05:38 - Success!
05:55 - Turning it up to 11
06:50 - Outro
---
Community size at time of posting:
- Subscribers: 10209
- Channel Views: 245054

[@JeffGeerling]

Heh, getting the quota increases is probably the part that took me the longest!
I wish it were easier to increase max pods per node, since there are use cases where you want more than the default and know the potential pitfalls. With AWS's default networking it's even harder though, since their networking stack can't assign more than a pretty tiny number of IP addresses per node!
Good job getting it working, and here's to the next 10,000 subs-and beyond!
I think next up, someone does need to partner up with a cloud platform and try to get 100,000k pods :D

[@DevOpsDirective]

Getting the quota increase was the hardest part!
To get the vCPU quota up to 48/region I actually had to get on a phone call with someone from the GCP team to convince them I knew what I was doing and wasn't going to accidentally incur a massive bill and come begging for forgiveness!

[@madhank93]

Congratulations on your 10k subs 🥳

[@DevOpsDirective]

Thank you @Madhan! 🙏

[@Henoknowz]

Just stumbled across this video, super interesting work! Do you happen to still have that python script that you used to monitor the pods and map it out to that neat graph? I'm starting to learn python myself and am curious about how you did that.

[@DevOpsDirective]

Here ya go! (most code samples from the channel should be in that repo!)
github.com/sidpalas/devops-directive/tree/master/2021-04-19-10000-pods/visualizations

[@DevOpsDirective]

Also... dont @ me for writing to the csv file with kubectl and then reading it repeatedly from python 🫣
The "proper" way to do this would be to use the python kubernetes client github.com/kubernetes-client/python

[@Henoknowz]

@@DevOpsDirective haha I definitely will not. Thanks so much! :)

[@uploadvoice]

Great job! Btw do u know how many pods I can run on a 8cores cpu with 32GB of ram ? (.. my laptop :) )

[@DevOpsDirective]

Good question -- give it a shot! Probably a few thousand if you use similar tricks I did! 😀

[@sampath853]

Just 73 likes for this awesome video. it is not fare...

[@DevOpsDirective]

The UA-cam algorithm is a fickle friend (or foe) 🤷‍♂️
Thank you for the kind words!

[@eleah2665]

Cool. I saw my pod being created at 5:11. Hope you haven't taken it down already.

[@DevOpsDirective]

😀-- Since you commented you get a bonus pod from the surplus too!

[@omkar219jcb]

haha lol

[@DevOpsDirective]

Lol indeed. 😅