Use Shelve For EFFECTIVE Persistent Data Storage In Python
Вставка
- Опубліковано 29 гру 2024
- In this tutorial we're going to be looking at the shelve module, which is an in-built module provided by Python. It is backed by the pick module, so it allows us to store all sorts of crazy objects. Check it out!
Pickle Tutorial: • WHAT Is "Pickle" In Py...
▶ Become job-ready with Python:
www.indently.io
▶ Follow me on Instagram:
/ indentlyreels
For someone who self taught with python and has built loads of stuff using barebones knowledge, these tutorials are so amazing. Ive spent so long writing messy code wishing i knee a better cleaner method, and this guy teaches it so well
I wasn't aware of this, and think it is both elegant and useful.
7:34 I am aware that when you shelve a class instance, you do not have to have the class definition within the same python scrip where you will open the shelve.
Is it impossible to use shelve to write concurrently to a database across multiple threads? Even if I make sure that each time I write to file I use a unique key?
Oh God. I've wasted so much time in my daily work which involves editing and loading large amounts of data. I've spent tens of minutes almost daily to reload things that could have been stored this way. Thanks a lot!
Watch it from Russia, thank you good person!!!
Does it support search like TinyDB?
Or do you have to use filters or loops to look for specific keys or values using structural pattern matching?
what editor are you using?
In what occasion it will be benefit to use shelve? Is it better using this to create a kind of database than using sqlite3 to create a database? Is it better than storing data in JSON format?
You can think of it as the sqlite3 of mongodb.
It implements dbm, which is one of the earliest NoSQL databases created.
However since it stores pickles it is a bit more dangerous, as it opens your code up to arbitrary code execution, and that's about it. But that's also it's selling point in a way. You're not limited by the type limitations of json, or the rigid schema and normalisation horrors of SQL.
Not sure if I want to use this in many cases. If I want to load a set I can you load the file, and I can still modify the file as a text file in some editor.
Is the database file portable? Particularly across architectures?
Is creating foreign keys, indices, unique constraints, multiple tables in one DB etc. possible using this library? If yes, could you make another video covering these?
It's not SQL, it's just pickle dumps.
i was going to make a video on this, but this is a great video. Well done...
the only question is, when would one use persistent data storage as a use case ?
Logging of raw data would be very helpful, to be used later for a replay simulation. Especially useful if that db file is portable so can be used on one particular hardware then replayed on another, something I do a lot at work.
I love your videos so much 🙂❤
Thank you!
Wait, would it be possible to store a keras.Sequential this way? Just theoretically?
Could you shelf the Fruit class itself and then retreive it ?
You say that pickle can only pickle one object, but you can simply call pickle.dump multiple times for unlimited number of objects. The only advantage I see for shelve is that you don't have to remember the order the multiple objects are pickled. So simply to me, shelve is like a dictionary whereas pickle is like a list.
Essentially that is what it is. "A shelf of pickled objects"
Amazing 😍
Thanks for your videos. For me the key is: is it safe in a multithreading scenario?
The docs say no concurrent read/write but concurrent reads are ok. So I guess multiprocessing is fine for reads, but for writes you would need to let each process have it’s own file and then combine the shelve dictionaries in the main thread.
@@thatsunpossible312 thanks
Under Windows, with cygwin, gdbm is available, but not under anaconda, so it seems.
Does it improve the performance?
Nice
What extensions do you use in vscode?
This is PyCharm, and I don't use extensions.
When I do use VSCode, I use life refresh for my JavaScript websites.
hey man that intelisense type hint dropdown is sick! it clearly distinguishes between classes and methods and stuff.
id really love to install it. does anyone know what it is?
I believe it's part of the experimental NewUI they have for PyCharm
why your pycharm looks different its cool
Could this be used instead of an .ini file for a configparser?
It can be used for arbitrary code execution, so just be careful
@@aeghohloechu5022 yeah, I would only load my own pickled files
We’ll now I feel like an ass inheriting UserDict and implementing this myself.
Going through uneccesary effort to create our own implementations of a feature will probably never be avoided 😂 BUT, at least you get a much higher appreciation for the built-in modules.
@@Indently to be fair I looked into shelve after watching your video and the inability to support concurrent writes limits my use case. It’s the reason I used on disk storage for my pickles instead of a database. But thanks for this video, this might be my go-to now when running into memory issues. I’ll benchmark it against my solution.
That is fair
Nice.
Nice
Can i store a lot of data in one: db[a] = 1, 2, 3 etc
Try it 😡
I think so: the 1,2,3 will be turned into a tuple (1,2,3) just as if you did
x = 1,2,3
db[a]=x
@@Chalisque Yeah, you can pickle a tuple, just like any built-in or user-defined object in Python.
👍🙏👍
Another profile picture update??
And not the last
I think it is garbage. There is no reason for Shelve to exist, except causing confusing. I don't want to know this.