Wonderful video, Hynek! You did a great job of capturing how tumultuous Python packaging has been over the years. It was really exciting to see the speed up UV granted Nox when reviewing the PR. :)
Such a great video, Hynek! I am adding this to my "Understanding Python virtual environments" Gist that I share with my students. Thanks and best wishes on the channel!
Not enough mic punching 😂. Excellent video, love the deep dive on resolving. Do you think that Python tools not written in Python will attract a similar level of contributor support?
yes, not punching, BUT: check out how it’s wobbling around especially by the end! (background: the mic is on a boom, but the boom is connected to my table so when i touch the table…) and no I don’t think they will, and I do think it’s a problem, but packaging has currently bigger problems so I’m still bullish on it in general!
Dude way cool! Subbed (which is rare for me) I've been staff ML Platform engineer. I was caught completely off guard by how mature your take is. I love that you could speak to an announcement back in the day when setuptools maintenance was handed over to the community. Also had no idea about the pip conf file or that PDM attempts to do x-platform lockfiles.
Fantastic overview. I love these nuggets of knowledge. Packaging is such a hard problem that it's incredibly difficult for a group of volunteers to solve without any VC backing. I'm glad Astral is addressing this issue.
@@The_Hynek The former. My brain reports: "aha, it is some trickery, the man just glitched into another dimension for a fraction of a second, we shall follow him to the comments section".
Right, in fact I think FCP added a few that I didn’t want added (between real cuts) that added some uncanny valley. I’ll try to catch those better next time!
Small addition to the video around 6:55, the ~/.pip/pip.conf location is the legacy location. It nowadays follows XDG and prefers ~/.config/pip/pip.conf.
Thanks. Really great video. I'm wondering. Your final take away on you being sceptical about Astral being able to deliver on "Cargo for Python", is that fundamentally rooted in the cross platform lock files problem or something more fundamental?
To be clear: I'm not skeptical it's possible, I'm just skeptical it's gonna happen anytime soon. As I've outlined, it's a much more difficult problem to implement a Cargo for Python that to implement a Cargo for Rust. But it's currently our only hope.
Great content, as always. About your format - it is definitely better to have more shorter videos rather than fewer longer videos. A 10-20 minutes video is easy to pick up impulsively over a coffee break even in the middle of the week, instead of all those 30+ minutes videos languishing in my reading list waiting for me to get a large enough block of time to view them.
Great overview of Python packaging and why it's a hard problem. That mic looked a lot happier this time around. 8pt font - 'and one week to rip off' LOL
Great content Hynek! I learned a fair bit about the low-level packaging tools in the Python ecosystem. I've been using Poetry for long enough that I almost forgot that virtualenvwrapper and friends still exist. RE ideas to improve the video, captions would be a big plus. It can be difficult to understand technical terms in an unfamilar accent, and automatic captioning doesn't help when such terms are involved. I realize it's probably a fair bit of work to add manual captions, but maybe there are some tools to semi-automate the process these days, especially with the rise of GenAI.
thanks! yeah I would LOVE to provide captions (I use them myself a lot) but it's so much work :-/ I'm trying to put important terms on the screen in writing as a compromise. Not sure GenAI can work with my Czech-German accent. 😂
Yesss! 💪 I would also be lying if I said that listening to CGP Grey for years didn't get me into UA-cam. Or that his tight, concise style didn't have an influence on me. :)
I can't wait for "native" lockfiles - managing requirements.txt myself always felt annoying at best. Hope it also makes containerization feel like less of a chore.
Sounds like the remaining "big unsolved problems" are: cross-platform dependencies are hard to spec out and a pain to compile (upcoming PEP may help), the speed of packaging code in hot loops should be faster, the fact that solving dependencies is NP complete sucks, the reliance of scientific packages on programs in other languages creates additional finnicky dependencies, compile dependencies require lots of troubleshooting and yaml soup, support for virtual environments that doesn't overwhelm new users, and creating a simple interface that is easy to use. Does that sound right?
Depends what you mean big unsolved for what. The PEP will define a format, but not make it easier to resolve. All resolvers in Python currently trip over certain technically solvable situations. The locking format is HOPEFULLY just a formality and once uv's resolver improves, I don't think implementing the lock file will be a big deal. So yeah, the only big unresolved problem is the NP-complete dependency resolution that's made more difficult by Python's diverse landscape - combined with Python's single namespace (e.g. in Rust, resolution is simpler since you can add a dependency several times - they don't create a constraint that complicates the solution).
Great video both on the content side and on the production one. While I usually get very bored with python packaging drama (as you pointed out it's now mostly ill informed reheated old soup) I'm very curious about what uv can deliver
It's kinda hard to list all virtual environments when they aren't stored in a central place. It's just a different philosophy that is preferred by most people (myself NOT included) - for better and for worse.
The probem with python packaging is, that there are too many tools and you do not know which one to use, or whether you need it all. Adding more packaging tools will not solve this.
That wouldn’t be a problem if all the tools did everything necessary which is emphatically not the case. Whether one of the existing tools will grow to a 100% solution or whether we will need a new one is kinda unrelated. The actual problem is the vast problem space, as I’ve tried to illustrate, which is the reason why we have many 90% solutions with varying missing 10%.
@@The_Hynek Yeah maybe I should rephrase this: *My* problem with python packaging is that there are too many tools and *I* do not know whicht to use. Should I use a solution that appears easy to use (like poetry) or use the low level tools venv, pip, setuptools. As a non-proffesional developer I solve this stuff by not packing stuff at all, sometimes I write my stuff in C so I can just distribute an EXE easily and not have to tell Python users to install the dependencies manually.
Six months later, I've recorded an answer to this video! Check it out: ua-cam.com/video/8UuW8o4bHbw/v-deo.html
Wonderful video, Hynek! You did a great job of capturing how tumultuous Python packaging has been over the years. It was really exciting to see the speed up UV granted Nox when reviewing the PR. :)
Thank you Thea! 💜💜💜
Can't beat informed takes on Python packaging!
As a build master in my day job, this is the best content ever
This breakdown of Python packaging is terrific!
Thanks Hynek sharing your insights and perspectives!
Such a great video, Hynek! I am adding this to my "Understanding Python virtual environments" Gist that I share with my students. Thanks and best wishes on the channel!
Not enough mic punching 😂. Excellent video, love the deep dive on resolving. Do you think that Python tools not written in Python will attract a similar level of contributor support?
yes, not punching, BUT: check out how it’s wobbling around especially by the end! (background: the mic is on a boom, but the boom is connected to my table so when i touch the table…)
and no I don’t think they will, and I do think it’s a problem, but packaging has currently bigger problems so I’m still bullish on it in general!
Dude way cool! Subbed (which is rare for me)
I've been staff ML Platform engineer. I was caught completely off guard by how mature your take is.
I love that you could speak to an announcement back in the day when setuptools maintenance was handed over to the community.
Also had no idea about the pip conf file or that PDM attempts to do x-platform lockfiles.
Thank you so much! Secret teaser: I'm writing the successor to this video right now!
Fantastic overview. I love these nuggets of knowledge. Packaging is such a hard problem that it's incredibly difficult for a group of volunteers to solve without any VC backing. I'm glad Astral is addressing this issue.
Well-balanced and funny as always. (I would omit the morphing transitions, though)
Thanks!
Interesting… do you mean the fill-ins to soften cuts or the ~3 real transitions (before/after the paper and at the very end IIRC)?
@@The_Hynek The former. My brain reports: "aha, it is some trickery, the man just glitched into another dimension for a fraction of a second, we shall follow him to the comments section".
Right, in fact I think FCP added a few that I didn’t want added (between real cuts) that added some uncanny valley. I’ll try to catch those better next time!
Glad I wasn't the only one that noticed. 😂
i thought i was watching AI for a second when i saw it 🤣
Small addition to the video around 6:55, the ~/.pip/pip.conf location is the legacy location. It nowadays follows XDG and prefers ~/.config/pip/pip.conf.
This was a great video Hynek! The "consistent goal-post moving" take hits home, Python is building on a lot of history :)
Those little octopi behind you look like the Octopus energy mascots!
Finally someone noticed!
I got them from PyCon UK 2023 where their tech arm had a recruiting booth.
Amazing video, Hynek! Just learn more about history of python packaging, than in a few years working with python as a primary tool
Great video! Appreciate you sharing your hard-earned opinions with all of us.
Thanks. Really great video. I'm wondering. Your final take away on you being sceptical about Astral being able to deliver on "Cargo for Python", is that fundamentally rooted in the cross platform lock files problem or something more fundamental?
To be clear: I'm not skeptical it's possible, I'm just skeptical it's gonna happen anytime soon. As I've outlined, it's a much more difficult problem to implement a Cargo for Python that to implement a Cargo for Rust. But it's currently our only hope.
@@The_Hynek I understand, thanks
Great content, as always. About your format - it is definitely better to have more shorter videos rather than fewer longer videos. A 10-20 minutes video is easy to pick up impulsively over a coffee break even in the middle of the week, instead of all those 30+ minutes videos languishing in my reading list waiting for me to get a large enough block of time to view them.
I SWEAR I'm trying to make JUST ONE video under 10 minutes. I'M TRYING. 😫
Yes, the Beaver logo is awesome.
Good video. I was undecided on uv vs pdm. You shed some nice light on the differences
Yeah currently the answer for me is both: hynek.me/articles/python-virtualenv-redux/
Nice. I like the direnv approach for activating venv
I used to use pyenv for managing version, now I’ve switched to mise
Thank you for explaining the Python packaging world. I am fully with you.
Great overview of Python packaging and why it's a hard problem. That mic looked a lot happier this time around. 8pt font - 'and one week to rip off' LOL
It was so happy it wobbled the whole time. 😅 (see my reply to Mike why)
Very nice, very well-informed overview!
Great content Hynek! I learned a fair bit about the low-level packaging tools in the Python ecosystem. I've been using Poetry for long enough that I almost forgot that virtualenvwrapper and friends still exist.
RE ideas to improve the video, captions would be a big plus. It can be difficult to understand technical terms in an unfamilar accent, and automatic captioning doesn't help when such terms are involved. I realize it's probably a fair bit of work to add manual captions, but maybe there are some tools to semi-automate the process these days, especially with the rise of GenAI.
thanks!
yeah I would LOVE to provide captions (I use them myself a lot) but it's so much work :-/ I'm trying to put important terms on the screen in writing as a compromise. Not sure GenAI can work with my Czech-German accent. 😂
Keep it up Hynek! Thanks for tampering my uv excitement just a tad. lol
Nice Cortex T-shirt! 😀 Fellow Cortexan here from Sweden.
Yesss! 💪
I would also be lying if I said that listening to CGP Grey for years didn't get me into UA-cam. Or that his tight, concise style didn't have an influence on me. :)
I can't wait for "native" lockfiles - managing requirements.txt myself always felt annoying at best. Hope it also makes containerization feel like less of a chore.
You and everybody else my friend. 😅
Sounds like the remaining "big unsolved problems" are: cross-platform dependencies are hard to spec out and a pain to compile (upcoming PEP may help), the speed of packaging code in hot loops should be faster, the fact that solving dependencies is NP complete sucks, the reliance of scientific packages on programs in other languages creates additional finnicky dependencies, compile dependencies require lots of troubleshooting and yaml soup, support for virtual environments that doesn't overwhelm new users, and creating a simple interface that is easy to use. Does that sound right?
Depends what you mean big unsolved for what. The PEP will define a format, but not make it easier to resolve. All resolvers in Python currently trip over certain technically solvable situations. The locking format is HOPEFULLY just a formality and once uv's resolver improves, I don't think implementing the lock file will be a big deal. So yeah, the only big unresolved problem is the NP-complete dependency resolution that's made more difficult by Python's diverse landscape - combined with Python's single namespace (e.g. in Rust, resolution is simpler since you can add a dependency several times - they don't create a constraint that complicates the solution).
Great video both on the content side and on the production one. While I usually get very bored with python packaging drama (as you pointed out it's now mostly ill informed reheated old soup) I'm very curious about what uv can deliver
Thank you, me too! 😅
Wonderful video, Thank you Hynek
im still sad were all moving to toml. a format abandoned by its own creator on account of what a mistake they felt it was.
No one seems to talk about hatch :( Oh well, another one then.
Hatch added support for uv: hatch.pypa.io/1.10/blog/2024/05/02/hatch-v1100/ 🤓
> Open Source maintainers spend a lot of time waiting for CI
Literally watching this while I wait for Azure DevOps jobs 😄
the pain is real 😅
uv does not have a built-in command to list all virtual environments. Unlike Conda that provide commands like 'conda env list'.
It's kinda hard to list all virtual environments when they aren't stored in a central place. It's just a different philosophy that is preferred by most people (myself NOT included) - for better and for worse.
Whenever a tool claims to solve all our problems it solves them but introduces worse ones
Love the small rip off comment
🤐
@@The_Hynek won't bug you about that ever again - appreciate you, regardless of the medium!
@@matmair1915 all good, I’m surprised/amused ppl noticed immediately. :)
so.... pipx install uv ?
For example! github.com/astral-sh/uv?tab=readme-ov-file#getting-started I use pipx myself too.
Enjoyed 🎉👍
Just use ant.
based takes
The probem with python packaging is, that there are too many tools and you do not know which one to use, or whether you need it all. Adding more packaging tools will not solve this.
That wouldn’t be a problem if all the tools did everything necessary which is emphatically not the case. Whether one of the existing tools will grow to a 100% solution or whether we will need a new one is kinda unrelated. The actual problem is the vast problem space, as I’ve tried to illustrate, which is the reason why we have many 90% solutions with varying missing 10%.
@@The_Hynek Yeah maybe I should rephrase this: *My* problem with python packaging is that there are too many tools and *I* do not know whicht to use. Should I use a solution that appears easy to use (like poetry) or use the low level tools venv, pip, setuptools. As a non-proffesional developer I solve this stuff by not packing stuff at all, sometimes I write my stuff in C so I can just distribute an EXE easily and not have to tell Python users to install the dependencies manually.
I have no idea. I prefer pip.