Cloud Imports in Python
Вставка
- Опубліковано 25 чер 2024
- Imagine and it is possible!
Installing dependencies is such a pain! What if we could just point to a GitHub repo and just use whatever code is there? Python is a dynamic language after all, why do I need to install before running my code? Well with cloud imports you can eliminate the need for installing! Create modules on the fly without ever installing a thing! No need to mess with pesky virtual environments! What could go wrong...?
CONTEST CURRENTLY CLOSED!
OFFICIAL CONTEST RULES:
1. All entries must comply with the UA-cam community guidelines ( ua-cam.com/users/tcommunity_gu...) and UA-cam Terms of Service (ua-cam.com/users/static?gl=US&t.... Entries that violate UA-cam guidelines are automatically disqualified.
2. UA-cam is not a sponsor of the contest and viewers are required to release UA-cam from any liability related to the contest.
3. Privacy notice: no personal data will be collected for this contest.
4. In order to enter, you must (a) be one of my subscribers, AND (b) make a top-level comment to the video including #mcoding somewhere in the comment.
5. The contest is free, there is no fee required to enter.
6. Winners will be chosen randomly 1 week after the date the video went live from all users who have entered and not been disqualified.
7. Each winner will be notified via a comment reply from me that details what prize was won (e.g. "Congratulations! You have won XYZ. Please email me."). I will ask the winner to contact me by email, and I will reply through email with a random token which must be posted as another reply to the winning comment from the winning account in order to verify account ownership and prevent fraud.
8. Each winner will have 72 hours to respond AND prove account ownership or their prize is automatically forfeited and another winner will be chosen.
9. A winner can only win 1 prize per contest.
10. The prize pool for this contest is: 2 licenses ("Free 1-Year Personal Subscription") to any of these JetBrains IDEs: AppCode, CLion, DataGrip, DataSpell, GoLand, IntelliJ IDEA Ultimate, PhpStorm, PyCharm, ReSharper, ReSharper C++, Rider, RubyMine, WebStorm, or dotUltimate. A prize consists of 1 license, which will be delivered in the form of a redeemable code that can be redeemed at www.jetbrains.com/store/redeem/ before January 17, 2024.
11. You may not enter the contest if doing so would be a violation of any relevant federal, state, and local laws, rules, and regulations, including U.S. sanctions.
― mCoding with James Murphy (mcoding.io)
Source code: github.com/mCodingLLC/VideosS...
importlib docs: docs.python.org/3/library/imp...
SUPPORT ME ⭐
---------------------------------------------------
Sign up on Patreon to get your donor role and early access to videos!
/ mcoding
Feeling generous but don't have a Patreon? Donate via PayPal! (No sign up needed.)
www.paypal.com/donate/?hosted...
Want to donate crypto? Check out the rest of my supported donations on my website!
mcoding.io/donate
Top patrons and donors: Jameson, Laura M, Dragos C, Vahnekie, Neel R, Matt R, Johan A, Casey G, Mark M, Mutual Information, Pi
BE ACTIVE IN MY COMMUNITY 😄
---------------------------------------------------
Discord: / discord
Github: github.com/mCodingLLC/
Reddit: / mcoding
Facebook: / james.mcoding
CHAPTERS
---------------------------------------------------
0:00 Intro
1:00 Wait, it's real?
1:34 The import system
2:16 The meta path
3:03 Finders and Loaders
4:52 Thx - Наука та технологія
Ok, deep breath. You absolutely got me. I was genuinely about to rage. Caps locks were stewing.
Hats off. You're a gentleman. Great surname too.
same 😂😂😂. its like my thermometer hits the roof when i see bad practice
Wait, so I could make an import handler that you pass a string to search on stackoverflow, and it just starts grabbing code from every result, wraps it in an object, and presents it?
Oh! Or better! A string to send to ChatGPT, telling it the functions you want.
In theory, yes. :D in practice I guess we both know how bad of an idea that would be.
Look up "python cataclysm" which just autoimports fynctions that weren't defined when you use them, with implementation or just direct computation by chatgpt. It just uses the name of function and "works".
@@sehaless why? I love using ChatGPT for security critical stuff
Something like this would be safe if you specified the hash of the entire module so that if the repo suddenly gets modified to contain malware the hash/checksum would fail and not run the module, although you could argue if you have to do stuff like this you might as well use the effort to install the library locally.
Exactly how golang does it.
At that point might as well just install from pip so you can keep version control
I remember tinkering with the import system a while back. It was un-intuitive even before I found the API was changed in Python 3.8.10. This video helps to make sense out of all that mess. #mcoding
I already knew about custom importers when porting Python (full desktop version) to the Nintendo 3DS, as I had to add a custom import path so frozen modules would get loaded from ROMFS instead of from SD, but it has never clicked in my head until many years later you showed us that you can wreak havoc with this feature.
Great stuff!
I might actually end up making my own importer based on yours, so my *local* projects would get used instead of having to make a copypaste in the folder of the projects, causing many desynchronization issues by fixing a bug in the copypasted version, but forgetting to backport it to the master copy on the disk.
Dude, have you heard about PYTHONPATH environment variable?
@@KompleksnyeChisla Well, the Nintendo 3DS OS doesn't have a concept of an environment variable, so it's quite difficult to set it...
@@SonoSooS I think they meant for your local projects lol
@@izzikora6751 Oh, I can't believe I could be this hard-headed to notice that! Thanks for pointing it out 😅
Definitely got me at first! I was so confused 😂
Awesome showcase! #mcoding
Man, as much as I agree this is not a great practice, I might use it for a proprietary package we use internally. There's so much freakin' handholding where I work that I'm expected to provide and manage a the Python environments for the hundreds of users. Apparently, asking users to create a venv and install our package using pip is "too much" for some.
When I was doing some of this, I built an installer in tkinter and compiled with pyinstaller to handle creating the venvs and installing the requirements instead of having a lengthy tutorial on getting that all setup (used in education on teaching python and specific packages). It wasn't pretty but it got the job done and was significantly safer than online loaders haha
A problem I've run into at work is that python's dependency management is such a frigging mess that even venv and pip don't always work because apparently whatever version of pip I have refuses to download specified versions of the dependencies I need because they use some old dependency format or something. What's even the point of being able to download specific versions if they're going to break? I love python for some things but dependency management is a real problem to the point where I refuse to use Python if I have any other choice now despite having sunk years of my life into it
Just write a shell script, name it install.exe or setup.exe (doesn't even have to be on windows, these names will make newbies click on them - chmod +x, put a shebang at the start) and have it check then create the virtualenv, run poetry install or whatever equivalent you use (if any). Seriously, runtime imports are a completely different level of crazy.
@@danielgysi5729 Poetry might solve your package dependency issues. Look up python-poetry.
docker is not an option for you?
Looks sick af, gotta use it in my projects
Watch further in the video
Thank you, this helps to better understand how python import system works.
Great opportunity looking forward to results
Really interesting to see a video on importlib
Would the jetbrains license be a perpetual license or a monthly/yearly license? #mcoding
This is like piping curl to sh
Exactly, and there are legitimate usecases for that too, IF you verify the contents with a checksum first. Nothing truly wrong with it.
I can see this being useful If there was a way to guarantee the security of the server and the quality of the code on it.
The future is now, old man!
Yes, it would be nice to have a system similar to Golang where you download "local" packages you can then import, as part of the language.
This is not terribly different from using venv, but being part of the language (and not the OS) would help
this maybe quite useful for some extension system that does not require high security
bruh i needed this video
The devil's advocate: most of your modules are used after retrieving them from the internet without any sanity check, after all, aren't they?
Yeah, I'm confused. I pip install from pypi all the time. Heck one time I had to pip install from a github repo. Am I not supposed to be doing that?
Yeah I don't understand why he keeps saying it's so "unsafe" - he never explains his reasoning
@@irnehhenri Probably because its fetching it at runtime, so if your repo ever gets compromised in the future, you are now going to be downloading malicious code. Normal imports you have to manually update so you can be confident that they wont just break in the future.
@@nathansnail I suppose there is a bit more risk in that scenario - but I feel the main use of something like this would be continuous deployment, where the alternative is always automatically updating the library without requiring user intervention. Then the risks from a potentially compromised repo would be the same in both cases.
#mcoding
On one hand it is amazing how customizable Python is. You can literally change anything. On the other hand it is really scary. I mean, you can literally change everything. Sometimes it feels like cutting the branch you are sitting on. Could be a good idea though, if you overwrote gravity before... Which in Python is possible probably...
Be careful when you over at gravity, because you might just accidentally make the strong nuclear force stronger. It is python after all and if you're overriding gravity you're probably doing it with some horrifying medical class.
that actually comes built in, simply import antigravity (no really, try it)
@@clay_geo lol I totally forgot this one.
Also try
"from __future__ import braces"
Nice xkcd reference
@@BrianWoodruff-Jr really? was not on purpose.
I can actually see this being somewhat useful in very specific scenarios. #mcoding
I was asking that myself. How about company SW importing modules from company git?
Wait... this could work where you fork all the dependencies as subrepos into your own github for that script. You can essentially control all the dependencies and their versions. If someone WANTS the bleeding edge, they could just pip install it into the virtual environment...
This would make it so that it works out of the box without installing requirements.txt dependencies. You just clone and run the project entirely.
JavaScript imports work like this, it's considered good
Its also why cross site scripting exists...
Honestly, I'd like to see this functionality in some lightweight python distribution. Currently I'm using Deno for any bootstrapping tasks within portable environments, since it it a single binary that can be downloaded with cmd/shell and has cloud imports
yet another great idea to experiment with. #MCODING
not a novell idea though
Thanks for deep dive into python .
#MCODING
This can also help debugging import errors when you know the module exists but the interpreter refuses to find it or imports it but it behaves weirdly when used/is completely empty (has no methods/classes). #mcoding
As an additional feature you'll get lots of angry mail from customers, because SW magically changes its behaviour or stops working just because someone dared to reboot the system
Bruhhhh the way I was about to snap when you it wasn’t a security risk 😅😅😅😂😂😂
Was a great start... I really did think someone tore another massive hole into Python security practices. Well, it still could be coming.
Thanks for the video, I think there could be an interesting set of videos for Python security pitfalls.
#mcoding
I don't think this is as horrible of an idea in a corporate proprietary github setting, where it's possible that different departments with different packages on the company github can optimize and improve their source code on set packages, and then other departments can use these readily available on a certain corporate repo.
For example, let's say there's an automation department in the company that handles leveled access to the company's customer database. The automation team creates a script that fetches data based on a query where you run it in the cli with a command called "query," for example. A team that accesses this query function for whatever reason can just import the most recent version of it from the production environment every time the script runs (or automate importing the updated version on a production commit).
#mCoding
How would you do this with .pyd files instead of .py files?
I hate and love the idea. I'd really like to try even though its bad practice.
#mcoding
00:50 why does deno (javascript runner) support https import?
i see tons of usages for this already
Thanks for the great explanation. #mcoding
I wasn't really skeptical because of Deno lol...
Muy bien amigo
thank you #mcoding : )
This is just how Deno for js/ts handles all imports, to eliminate the need for a package manager. And your browser does this every time you visit a webpage. Deno has some security features to get access to filesystem etc unless you allow it though.
True, and there have been dozens of cases where js modules were compromised
But also pretty much all of web security is this going horribly wrong
Importing code directly from the internet is what my country's complete electronic voting system does.
fancyyyy!
i like that JetBrains gives back to the community with giveaways #mcoding
Perhaps this might be a good idea if the imported code is hosted in a private network, like a NAS or something?
#MCODING
#mcoding Hey, thanks for the nice video! I appreciate the mentions of some important python pieces.
Great content as always! #mcoding
Is it possible to make mypy work in this scenario?
Very cool #mcoding
Which HetBrains IDE license would be up to win if I included #mcoding? And more importantly, can it be included from the cloud? 😅
Great video. Thanks! #mcoding
so you can have fun with making import statement do crazy stuff, like printing
#mcoding
What about imports that, if the module isn't found, tries to install using pip? Would that be less of a risk?
Then common spelling errors would make it so you could import a repository that's meant to capitalize on that misspelling and be malicious
Great stuff #mcoding
#mcoding lol, that was cool :D
What's the point of being able to configure the way imports work? #MCODING
I toyed with the idea of loading scripts directly from the internet a long time ago purely as a fun exercise and rather quickly abandoned it because why would I even want to raw run code dynamically from the internet that's just begging to be pwned
#mcoding Importing from the cloud…. The convenience surely trumps all the possible issues that might arise 😂
I don't see anything wrong with cloud imports at all :) #mcoding
With pinning of hashes of cloud imported files, this might be doable in a less dangerous manner. #mcoding
#mcoding knows things
I love learning stuff through the lens of an awful idea
#mcoding That's a crazy thing to know on how simple it is although I won't use it
#mCoding rules. I would like the license please 😄
Oh yea
Did you heard about micropip, from Pyodide?))
#MCODING Thanks for the video
#mcoding ty
If modules are hosted on an internal server or something, could this potentially be a good idea to deploy and change code remotely for multiple devices at once? For example, if it was all embedded devices or something. #mcoding
I was thinking the same. As long as you have full control over the imported files, there should not be any security issues. A threat actor would have to modify the including code (the main program so to speak) to include something malicious from somewhere else - and that option exists even with local imports.
However, I am not sure if it is a good idea performance wise. And in case the network connection fails, your program won't run.
I don't think it would necessarily be a bad idea, but it would go against Zero-Trust policy that large applications require and that is recommended for everything else. If a threat actor manages to modify the file on the other machine, it can then transmit modifications to your new one.
Maybe if you stored the hash of the fileon the Code-User, using a hashing function (or salting) that is unknown to the Code-Host, it might be better ?
@@Colaholiker You could maybe fetch and store a local copy, and only actually redownload if its got changes. This resolves the no internet issue because it'll just be an outdated version, and it also means at runtime it effectively becomes a normal cost import if you are up to date.
The 39.99 a month was what got me
#mCoding
You win! Email me to redeem your JetBrains license!
@@mCoding Thank you! Will def write a lot of new projects with this. Hoping to hit 2306 by the end of next year.
Honestly, this is one of the best features I've heard of from Python. I think I'll make it an undocumented feature in my own language.
#mcoding let’s give it a try.
great video!!
This is absolutely cursed. Good job. #mcoding
> cloud imports are a mind bogglingly bad idea
> HTML has joined the chat
I agree of course though
Isn't half of web security. Basically this going horribly wrong
@@LiEnby Ehhh not half but probably a good third yeah.
That actually seems useful for CI/CD pipelines and other ephemeral environments
no
@@gishee18 why
@@FunBotan it's good to have dependency and test/check/build steps separate for easier caching
@@gishee18 ok makes sense
#mcoding this was great
I was about to write an angry comment around 0:40. 😁
Including #MCODING
#mCofing 🔥
It’s like deno imports but scary! 😅
#mCoding
Schrödinger's joke
Looks very interesting #mcoding
Iʼm thinking of *so many* cursed ways to do this. A json finder loader so you can say `import blah` and it will load blah.json and the `blah` module will have the json data. A chatgpt-based loader that has to be the only thing more dangerous than your example. An importer that does something completely unrelated to importing because you are *running arbitrary python code.* #mcoding
`import paper` now makes an order for paper on some foreign website to be delivered to the user, based on geoip. That is, it will literally import it into the country.
#mCoding Would love a license to PyCharm!
just 39.99!? SOLD! #mcoding
could you .add_repo pipy?
#MCODING
#encoding, i'll take Clion
This matters a lot when u get into lazy import which is used in a lot of ml libraries.
I uave no idea how it works I just know its there for some reason
mb because you don't want to import gigabytes of network or training data if you won't be using it
@unflexian no its not the data libararies its the preformance packages. My guess its a combination if there is a lot of code and cach optimization likes having small code
@@nevokrien95 very reasonable :)
have a good day nevo95
Deno can do this right? Doesn't seem like an awfully bad idea if you just have a requirements.py file instead with everything you want listed using this lib
Right, and you can import from a release tag instead of just taking whatever master branch points to!
#mcoding
Also, this is terrifying.
Go na try to do this myself, #mcoding
😱 #mcoding
#mcoding
`import whatever_chat_gpt_generates` would be a good follow up
Everyone laughs at cloud imports until they take a look at deno :)
I really want to try this but shouldn't #mCoding
Not a bot :) #mcoding
Oh, this is so cursed #mcoding
This needs to be in base python before its any use really.
No that is risky how have you checked it is safe..