Вставка
- Опубліковано 8 лют 2025
- SD cards in Raspberry Pi computers wear out and die after a while because they are not made for computer usage, and many write cycles. In addition, the SD cards can become corrupted if the Pi loses power without a proper shutdown. We have several possibilities to avoid this disaster. Only one is simple and cheap. But, unfortunately, it has its caveats. So let’s get an overview and use a nearly unknown Raspbian feature to help us solve the problem.
Links:
Raspberry Pi 4: amzn.to/3kBF5nv
Barefoot method 1: / make-your-raspberry-pi...
Barefoot Method 2: www.raspberry-...
Adafruit's method: learn.adafruit...
The links above usually are affiliate links that support the channel (no additional cost for you).
Supporting Material and Blog Page: www.sensorsiot.org
Github: www.github.com...
My Patreon Page: / andreasspiess
Discord: / discord
If you want to support the channel, please use the links below to start your shopping. No additional charges for you, but I get a commission of your purchases to buy new stuff for the channel
My Amazon.com shop: www.amazon.com...
For Banggood bit.ly/2jAQEf4
For AliExpress: bit.ly/2B0yTLL
For Amazon.de: amzn.to/2r0ZCYI
For Amazon UK: amzn.to/2mxBaJf
For ebay.com: ebay.to/2DuYXBp
www.facebook.c...
/ spiessa
www.instructab...
Please do not try to email me. This communication channel is reserved for my primary job
As an Amazon Associate, I earn from qualifying purchases
#no#midroll#ads
there is another option using write-behind caching on EXT4 so it caches all the writes to ram and then using the COMMIT=xx seconds mount option it will write the changes from ram to disk every so often, the best of both worlds
Good point! And it seems to be easy. Similar to log2RAM, I think.
It doesn't ensure that writes will be cached to ram though. Just that it will cache up to "commit" seconds of writes before automatically syncing the filesystem so if your system suddenly crashes, you know that you have lost at worst this amount of writes. Applications can request their writes to be immediately recorded to the disk/SD Card. Applications who use databases (such as SQLite) or unbuffered logs will do that frequently.
@@_gawen i forgot to mention, that its a setting to enable journal_data_writeback using "tune2fs" on ext4 filesystems, as well as "nobarrier". this will run it all to ram first at the cost of data security and reliability, obviously
I have a pi3 running on the same sd card I started with, probably 6 years of running 24-7 with not a single issue. Pi4, has already burned 3 sd cards up in less than 2 years.
Can't for the life of me remember what commands I did to limit the card writes, or whatever I read at the time, but it's worked. Wondering if this is it...
I was able to reduce my sd card consumption drastically by logging to my remote homeserver. (almost) No local logs anymore. There is another advantage in remote logs: you can find suspects, even after the device in question died. Since there is a homeserver up and running 24/7, it was just a matter of some simple configuration.
Remote logging seems to be an interesting alternative (or extension) to this concept. Maybe I have to try it too...
@@mixxx2005 For anyone considering this: make sure to put security in your network config. Unless you want a random device in the wild to give an attacker access to the files on your home servers/computers.
Have any details on how you did this? I'd love to do the same.. have an NFS share or similar on a pre-existing NAS that would support logging from all of my various PIs, which could then run completely read-only.
@@SteveJones172pilot I enabled one of my Linux hosts to accept remote syslog messages, and enabled all clients to log to this remote logging facility. This is quite straight forward. How you have to do i depends on your software, of course.
This came just as I needed it!
Releasing 24 dashboards driven by raspberry pi's next week and was worried about configs being changed and other (python) code config that needs to write to the memory! Much appreciated Andreas!
Indeed this should prevent bad things from happening.
YES! Finally I can cut the power to my rasppi whenever I want without the fear of destroying the SD-Card!
True. I like this, too!
I tried it with hdd. I've had to re-set it up twice. 😒
I also tend to install ZRAM on every pi I own. It removes the need for swap space on the sdcard and it allows you to fit about 50% more stuff into the physical available RAM.
I never used it because I only have 2GB and bigger Pi4. So memory is no more a big problem for me. Probably good for a Pi zero if it does not consume a lot of CPU.
@@AndreasSpiess I even use it on my rpi 8gb model. Sometimes I notice raspian swapping stuff out even with minimal applications running and plenty of main mem free.
@@misteragony For some reason I think the default "swappiness" value is a bit too high in raspbian (how likely the os is to use swap). I adjusted it to 10 from its default 60 and have only noticed minimal swap usage since then, but it is still there for emergencies when you are running close to the limit (I have also done this on my laptop with similar results).
It is also very useful to mount the filesystems with noatime,nodiratime options in order to avoid updating access time in the file and directory attributes each time the file or directory is accessed, no matter it was just read.
Another viewer also mentioned noatime. I am not aware of this feature. So I have some reading ahead ;-)
@@AndreasSpiess This is how I use these mount options, on each and every extX filesystem which resides on SD, SSD or CF on my machines.
root@raspberrypi:~# cat /etc/fstab
proc /proc proc defaults 0 0
PARTUUID=60650d97-01 /boot vfat defaults 0 2
PARTUUID=60650d97-02 / ext4 noatime,nodiratime 0 1
@@AndreasSpiess I just noticed on the mount(8) man page that nodiatime is implied if noatime option is used. There is always something new to learn. From the mount(8) man page:
noatime
Do not update inode access times on this filesystem (e.g. for
faster access on the news spool to speed up news servers).
This works for all inode types (directories too), so it
implies nodiratime.
SD cards are not by definition unreliable. They come in different flavors for different use cases. If you pick the right one, an SD card can be more reliable than a big cheap SSD.
The most important criteria are the cell technology used and the Bill of materials.
Cell technology:
* Most consumer cards use TLC or QLC (3D) cell technology. A single cell can typically be rewritten approx 200 to 300 times.
* Some consumer cards advertise with 'long endurance' or 'industrial' terms. Often such cards use MLC cell technology (but this is not guaranteed!). In this case a single cell van typically be rewritten approx 2000 to 3000 times. This type is also often used for EMMC memory and modules.
* For more demanding environments (automotive, medical, extreme conditions,..), often (p)SLC cards are used. For these type of cards, a single cell can be typically rewritten up to 30000 times. That's >100 times more than a typical consumer card!. You will not find these cards in a regular (online) shop. Cards utilizing this technology often have more advanced flash controllers onboard providing additional security and features compared to consumer grade cards (e.g. power failure protection, better wear leveling...).
Bill Of Materials:
* Most consumer cards do not have a fixed Bill Of Materials (BOM). This means the internal components can change without notification and the manufacturer will choose whatever is available and the cheapest that complies more or less with the specifications. So for one batch of cards, the cells can be TLC, and a few months later, QLC becomes cheaper and the manufacturer switches to QLC components, or uses another flash controller... This is also the reason why almost no manufacturer will give the complete specifications of the used components for these cards.
* Most manufacturers also offer fixed BOM cards. These are cards where the used components are fixed and will not change without notice. This guaranties that if you buy the same card next year, it will behave exactly the same as the ones you already have.
An example of a 'cheap' (p)SLC card with fixed BOM is the SDSDQED-016G-XI model by Western Digial/Sandisk.
It is a 16GB SLC card with an extended temperature range (-40+85C) and an endurance of 480TBW. It costs about 30 euro's at Mouser. This is typically cheaper than a usb->sata/M2 adapter + decent SSD.
For comparison: a Kingston A2000 SSD of 500GB has an endurance of 360TBW. And if you only look at smaller very low cost ssd's, then the endurance is often 40 TBW)
So for projects that do not depend on very high speed disk access or require a lot of storage, such SD card is unrivaled. Both for price and reliability.
(The explanation above is a bit simplified... there are other factors playing also a role, but this comment is already long enough for now :-)
Thank you for bringing light into the dark of the SD cards! So far I have only used SD cards from online suppliers. Good to know other variants exist.
Concerning SSDs: Even my home automation server only writes MB and not GB because I store sensor data and not videos. So with my Raspis, I can live with very low TBW values (other than with my UA-cam production machine ;-) ) I also only use the smallest SSDs I can get because the volume is small (
Best comment ever! Thanks.
I see that there is also a SDSDQEC with up to 3200 TBW. Search for "brochure-removable-flash-storage.pdf" for a good summary of current offerings.
With the newer Raspberry Pi 4 with the higher RAM this overlay makes a lot more sense now. I guess you could use read/write Boot drive and periodically copy the overlay to the SD or to an external network drive for data that may need to be kept longer term. Thanks for the great information.
You are right, many combinations are possible if you include network disks in your setup.
I use a different approach, and I find it much simpler and scalable: I use a NAS, so shared directories to save logs and data. So I get automatically backups (I need to set up only once for the base directory in the NAS), fast disk (but it depends also on network), cheap disks. And I'm able to split or merge services and rpi much easier.
Storing the files on a NAS is for sure a good thing for many applications (if you own one).
When a Raspberry is booted without connection to the NAS (e.g. NAS is down, or no network connection), the Raspberry runs without NAS connection forever! No retries to connect later.
Solution: autofs mounts the NAS directories when needed, and releases the connection after 5 minutes when not in use, so this also costs less memory. Thus a retry is provided after connection-loss.
When speed is not needed, you can make a NAS from a Raspberry and 2 USB disks (even USB-thumbs) as RAID-1 using mdadm.
@@AndreasSpiess It is super easy to make your own NAS, just install Linux on a machine (could be another raspberry pi) and configure a RAID 5 (preferably not on USB, which is too unreliable for it, as it can disconnect randomly from time to time, which breaks the RAID and makes it go into degraded mode) and then share certain folders via NFS, very easy. The advantage is that you don't have a proprietary NAS like Synology that is both inflexible, hard to fix when it breaks and expensive and often uses a non standard proprietarily modified Linux.
I have a NAS as well and I prefer using it as well. I just need to get 1 that's a few teraflops my problem is the $.
@@PvGeens Sounds like I will need to be careful with doing that.
Thanks Andreas, I was worrying about my sd card in. my headless Raspberry Pi 4 recoding NOAA weather images 24/7 so I made images of the card if it fails, I haven't had any card fail yet but you saved the day and me worrying, thanks for this tip will implement it : )
As said: A backup is always good! But your application probably is a good use case to reduce the chance for failure.
IIRC, OpenWRT uses OverlayFS, but their implementation is pretty static without lots of hackery. Really neat the Pi Foundation was able to simplify it down to a toggle switch! 👍
As a Linux noob I was not aware of it until it was simplified ;-)
Very useful video, Andreas👍- this setup is indeed ideal for 24/7 applications that do "something" and send the results to a remote server or such. Or another idea: Run Minikube on the Pi4 and push containers to it from remote. That way the Pi would become application-agnostic?
So far I never used Kubernetes. Maybe a project for winter ;-)
I've usually mitigate writes to the sdcard by making sure the noatime and nodiratime option is on the sdcard in the fstab and mounting a small flash drive to the /var directory, and using a larger flashdrive for the home directory (you could put both partitions on one flashdrive). This should eliminate all writes to the sdcard except when installing and updating software. Not on PIs, but on PCs, I frequently use a virtual disk and keep my work on flashdrives, and have scripts with clickable icons to setup, load, and when my session is done, save and destroy the virtual disk.
Thank you for sharing how you do it. Virtual environments are also a good thing if you want to protect the original.
Sometimes, at good points, I hit the save work icon, and it saves only the files that have changed, using rsync, and I continue on.
I use RAMFS and mount it over the top of the log directories. Then, when a program writes to the log files in the log directory they are intercepted by RAMFS instead. When the max size is reached they are discarded. They are never written to the SD card. My SD card is in its 4th year of operation (as a LoRa gateway) with no problems so far.
I never used RAMFS, only log2ram which seems to have a similar strategy. This is another valid way to solve this problem.
I have a bunch of raspi 3's running on the field and they have been working nice for the past 4 years with this tmpfs trick. A couple of sd-cards have became corrupt during this time, but I suspect its due to power outages, since after dd'ing they seem to work ok again.
I would love to see any scripts or configs you have to accomplish this.. Sounds cool.. Also sorry you're "not" mark Knopfler.. it would be cool to be Mark Knopfler! :-)
Can you write the command for setting this up?
I have found this:
sudo nano /etc/fstab
And added
tmpfs /var/log tmpfs defaults,noatime,nosuid,mode=0755,size=20m 0 0
Is this correct or are there other directories?
Thx.
Thanks Andreas for the tip. What I do is to use a high endurance SD card, specially designed for high number of writing cycles ( they are used in security cameras). Is a very lazy solution though.
I hope this works (because I ordered a few of them after comments of other viewers...)
@@AndreasSpiess I personally have been lucky using Samsung pro endurance in Rpi for a while after testing them 1+ year on dashcam. Only 1 out of 10 have made one file corruption (as far I have noticed). Before that the Sd cards was not saving all the files or saving them empty. Just a note, that bigger sizes have bigger large ratios before failure.
Other option is to add some USB drive, which comes in micro size, so fairly extends out of devices itself
@@AndreasSpiess Nothing is failure-free. I used to work with MDVR (mobile video recorder) and moved from a Sandisk Ultra type SD to a SanDisk high endurance. From a batch of 20 units, we used to have one SD dead every 2 weeks; after the change the first failure happened 6 months after. Using Transcend High Endurance SDs in three Raspi working as LoRa Gateways for one year, no problem so far.
@@jdsan6009 Thanks for the info about Transcend SD card. I will try that one next time. Currently using SanDisk 128GB High Endurance in my 4K dashcam. Working fine so far. My old 64 gig SanDisk SD card worked fine for 2 years before I replaced it. I always get the biggest SD card that the device can support to extend the life out of it.
@Andreas Spiess. You will like to know Tesla uses industrial type SD Cards to save logs. Here is the reference ua-cam.com/video/ODdIRr5RzI8/v-deo.html
Add commit=300 as a mount option in /etc/fstab. EXT4 default is to commit all writes every 5 seconds. If you mount with commit=300 it will cache writes in memory then write them to disk once every 5 minutes. You can reduce your SD wear by 60x by adding that line.
Good point. Thank you. I will try it
In addition to these tips I would strongly recommend getting an SD card that's more suitable for this kind of usage. I've personally had good results with the WD purple SD cards, I've got 3 pi's that have been running 24/7 for about 3 years now on an unmodified raspbian installation with full logging enabled (and one of them is constantly writing due to a pihole installation). So far I haven't had any issues. But... I would recommend partitioning only a small part of the SD card (i.e. 25% or so) so the wear leveling algorithms can do their thing and make sure your SD card won't run out of working blocks.
Thank you for sharing your experience. Because other viewers also recommended these WD purple cards a few are in the mail by now. I hope they implemented wear levelling. The descriptions I found were not clear. Just that they monitor "bad cells".
@@AndreasSpiess I've read on reddit that they are supposed to have wear leveling according to WD suport. Anecdotally however, I've had many SD cards fail before I switched. Even high quality ones
I do not believe having a small partition would make any difference to the wear leveling since it uses all the cells to write data regardless what file system you use. The idea is get the biggest card the device can support to extend the life out of it.
@@Darkk6969 You're right, assuming you have TRIM enabled it won't make any difference in that aspect. But if you only partition a small part of the disk you leave the SD card with enough space to keep wear-leveling without any worries and you can't accidently fill up the disk with logs or other files. Given how cheap memory cards are and how much space you actually need, this has been a good tradeoff for me.
Sandisk do a range of rugged IoT targeted SD cards using more robust flash. Makes sense.
Thank you! Extremely useful, extremely inexpensive, reversable and can be done in seconds. Love it! The overlay wins for me.
Glad it was helpful!
After 2 SD cards busted, I put an SSD for my Home Assistant RPi4 and it is running since then very reliable, hope I don't jinx it. But very good tip to make the SD card last longer.
I also use an SSD on my server. Also for speed...
Instead of a SSD, you can also use a network filesystem and redirect things there through symbolic links. Handy if you have a NAS on the same network. Beware that SAMBA/CIFS does not work well for that and some services will randomly fail if their data are put on a SAMBA mount. NFS on the other hand works fine with everything.
You can also use the iotop command for instance 'iotop -o -b -a' to log the cumulative writes of the different processes so you can see who is writing and how much.
Thank you for your input. Remote logging indeed would be a possibility. Maybe even with these switches (if you want to keep your logs)
Great tips, nice video!
To be a bit pedantic, overlayFS itself doesn't mean writes are put in RAM. Rather, overlayFS is one piece of the puzzle - it allows redirecting writes to somewhere else, and when combined with tmpfs (RAM-based filesystem) it achieves the reboot-to-discard-temporary-changes feature.
You are right, of course!
Thanks Andreas. I watch your videos and like them every time but this is the first time I comment. I did not know about this feature and it is very useful for me because my children use Pi 400 for school. They love to mess with the OS and I have to reimage their systems often. Well, not any more. Thanks again
Perfect use case, I think...
So, why isn't there an option to write the ram files back to sd on close? Thus dramatically reducing the writes
My guess is it’s too hard to guarantee that. If people expect their data to be saved when the pi reboots then they will be angry if 100’s of megs get erased when it looses power or crashes. If the system expects files to be written on shutdown then the file system would be corrupted on ever single unexpected reboot. It’s much safer to just assume all data is temporary in the RAM disk.
@@CodeMonkeX Yes I see if the rewrites were automatic, there could be a problem. My thoughts were for a Manual/programmatic option - a simple 'do you want to save' or cmdline save
I am sure a lot of things can be done. But, as a Linux noob I am happy to have a simple way. All my use cases can be appointed to one or the other scenario: No longterm storage needed --> SD card or longterm storage needed --> SSD
Interesting question. That would be a nice option. I'm all about those SSD drives because of the speed. I only use SD cards for the portable builds like my hotspots.
In some way a similar functionality can be achieved with "apt install log2ram". If I'm not mistaken "log2ram" is installed by default in armbian-os, on raspberry-os we can install it ourselves.
I saw this option before in dietPI, I think it's a cool feature, glad the ported it on raspbian too
Another viewer asked if it is available on dietPi too. So here we have the answer...
@@AndreasSpiess Not sure if it is exactly the same. I know that in DietPI logs are not saved directly to SD and when this functionality was introduced, it was not yet in Raspbian. It is a really interesting concept to preserve SD life expectancy
I've been using raspberry pi for quite some time It didn't realize that was there. I really enjoyed this video. I really enjoyed listening to you on the ham radio workbench podcast as well. Thanks
There were a lot of discussions in the past about this problem.
It was also a pleasure to be guest in the workbench podcast.
Clever ways to save on writes :) Phones uses F2FS file system to save on writes. F2FS supports ZSTD compression. Image with bear minimum of applications would write less compared to stock Rasbian.
There are MLC SD cards that last ~10x at same capacity compared to most micro SD cards in the wild. Jet MLC cards cost ~4x (on SSDs the price difference is more reasonable ~2x). Higher capacity cards will last longer (grows lineary with the size). As for disconnecting the drive unsafely - I had luck with usb sata bridge and microSD readers, they usually go before the storage.
Thank you for the additional info!
Nice Video, learned something new. Now it would be great if specific log operations could be made to go to the SD card and the majority just to ram...
You an du that by using RAMFS or log2RAM to only store logging in RAM. But it needs a little more Linux knowhow. Or you mount a network drive for data as other viewers suggested.
great features
Thanks for sharing your experience with all of us 👍😀
Thanks for watching!
Thanks Andreas,
I believe a short lifecycle is typical for low-cost SD.
I use several SD cards in raspberries for 3 years already, incl my Majordomo home automation (decades of mqtt sensors, etc.) and had issues while using sandisk-like SDs).
After changing to less cheap I've forgotten about it.
Thank you for sharing your experience. It seems you were lucky. At least I found mentioning Sandisk and other branded SD cards in the forums complaining about failures. Maybe the operating systems also got more robust against corruption of SD cards.
Please let us know which are the BAD and the GOOD SD cards you used.
@@paulcohen1555 Sandisk were the worst, now Samsung, no issues yet.
I can't remember particular models, sorry.
@@paulcohen1555 I don't think it would be a healthy data because of the chip shortage. Even good SSD brands like samsung started to use less expensive less durable components. It's safe to assume that same tactics are deployed in the SDcard production.
Day of installation:
# ls -ld /lost+found
drwx------ 2 root root 16384 Nov 29 2017 /lost+found
thats a neat trick, plus is more secure, becasue if you loose your rapsberry pi whilst transporting it, you wont have to worry about still being logged into google etc and saved passwords. just enable it on a new install after you have setup wifi and customized the dektop
Good point!
Good video, no mention of emmc SD card adapters?
I would classify them as SSDs (price, form factor)
@@AndreasSpiess eMMC SD keys or adapters are small and directly replace the micro SD card, much smaller than an SSD, although I expect the SSD will beat on performance
I did not know they are so small! I have to look at them
Not Pi related but Linux related. In the early days of SATA SSDs I had one at 16GB from Imation. I had the most 'write intensive' volumes mounted on an LSI RAID volume (e.g. /tmp /var/ and SWAP). It lasted over 6 years in 24/7 use before it was replaced by a new computer. I did pull the drive before sending the computer to freegeek. Those are the 'high write cycle' mounts for a system drive. Data drives are not addressed.
Thank you for sharing your experience. So far I also had no problems with SSDs, also not on the PC. Even if I do some video editing. For the Pi I do not expect any issues because the data volumes are very small (mostly sensor data and logging).
Add autofs against an NFS box with your homedir, we did diskless workstations that way for years back in the 90’s.
You are right. Remote file systems can be a good solution instead of an SSD if you need to keep the changing data.
This was a great video.. I had been wondering why there wasn't an easy way to do this.. Next step would be to map a network drive and use it for logging and other data you dont want to lose, and it would be the best of both worlds! Thanks!
Agreed. Other viewers also suggested that method. So far I never used it.
Very educational content. Just wanted to point out that I'm personally using High Endurance SD by SandDisk which meant for CCTV application (run 24/7 + consistent read & write application). So far works well (more over 8 month with multiple time of trips by thunder storm - in turn will be a sudden power loss) for my simple LAMP project without overlay file which is running 24/7 when ever possible.
Maybe can give those a shoot for extensive test?
Other viewers also commented about those SD cards with different results. I do not plan such a test because I do no more use SD cards for heavy usage. I changed to SSDs, also because they are faster.
Indeed interesting. Guess I should make work of adapting my RPi asap.
If you have one which does not have to keep data it is a good thing.
Thank you Adreas for the easy to understand ex[;amation of the Overlay File System. It's just what I needed for my weather station which uses an ePaper display controled by a Pi Zero W.
That seems to be a perfect fit!
Is it a perfect fit, though? You mentioned in the video that 2G of RAM is recommended, and the Pi Zero W only has 512M.
I don't know how the Overlay File System decides how much RAM to use: is it dynamically allocating memory based on the amount of data written, possibly running out of memory over time? Unfortunately, Googling doesn't return any tips on reducing its memory usage, and I couldn't find confirmation that anyone is using it with a Pi Zero W successfully.
I guess the only way to be sure is to try it.
Thanks for pointing out the option, though: I didn't know it existed.
Another astonishingly simple option is to use a bigger SD cad. Let's say that your card is 80% full, then there are 20% storage/repair capacity left. If you chose an SD card with twice as much capacity, you'll get 120% storage/repair capacity left relative to your original one. This leads to a 6 times as much lifetime of the SD card as before.
I agree with Jan. But bigger cards usually also are faster (at least in the past).
@@AndreasSpiess
Wear-Leveling If Not done by Hardware is a Problem a OS should handle... But Looks Like they dont.
Maybe writte a Script to do monthly a copy of the Files that have IO writtes and rename the old File to dead Disk space +timestamp
If i build a System i Always keep more then 50% Disk free
And If the Disk get to 20% free i search for Upgrade Options.
Have a nice Day
And greatz from Germany
opo
Excellent information, I already put UPS boards on all my Pi’s but this will is a great trick.
UPS is also a good think, But not cheap ;-)
@@AndreasSpiess No they are not, but I wont run a PI without one, I use the PiSuger2 on all of mine, they come with a remote web interface for configuration and status, totally worth it.
Looks like a good solution!
Great hint, Andreas! Thank you!
You are welcome!
Another thing to do is to enable periodic fstrim operations to discard deleted blocks. This applies to all flash based storage: SSDs as well as SD cards.
Would this not increase the wear?
@@AndreasSpiess It spreads the wear more evenly as it allows the flash controller to know which blocks are free and can be reused instead of having to reuse (erase, reprogram) the same flash block when it runs out of spare unused blocks. Wear levelling.
Are you sure that this is true for SD cards. I heard that their main problem is that they have no wear levelling implemented.
@@AndreasSpiess It really depends on how the card is made. The better ones do implement the TRIM operation. Alternately have the operating system use a more flash friendly filesystem for logs, eg. f2fs
after the loss of 2 SDcards, ive switched to using a USB Stick in my Pi, worked sofar without issues.
Thank you for sharing your experience.
I'm testing the overlay file system on 11 Raspberry Pi, running camera applications.
Users were instructed to just unplug the power without powering off the system, so I could see if this was a viable solution.
So far no issues with sd card corruption.
2 of the Raspberry Pi were running without interruption all day (9-18) for 3 month and they still work perfectly fine.
I don't know if i'm lucky or if it works well, so take it with a grain of salt.
Thank you for sharing your test data. For me it looks like it works as designed!
The overlay system looks like a cool feature but I almost never use raspbian. Tbh, I've never had a problem with tweaking fstab but this is handy to know to recommend to others as it's simpler to explain.
If you use different distros I assume you are an experienced Linux user and do not need such a simple solution. For a Linux noob like me it is different ;-)
@@AndreasSpiess not sure I’d call myself experienced but I am comfortable with *nix. Like I said, didn’t know about this feature and I will recommend it to those that don’t really care about what’s under the bonnet. Thanks for sharing.
Great video, as always. Also, a great tip I will try both ways. Overlay and with a small SSD drive.
I use both ways, too. For different use cases.
I still prefer to use an ssd. You can buy a 128 GB M.2 disk for 20-30 € and a small enclosure for a similar amount. On the other hand I don’t have Scottish ancestry….
For me it is mainly about the form factor. As you write, the prices are ok these days.
excellent... i wondered what that setting was for but never looked into it!...thanks Andreas
You are welcome!
Very very good video and here I was wondering why my card would die after a few months of usage. Thanks a lot for this video. Now to the terminal window!
I hope this will solve your problem. Other viewers commented also that they were successful using "endurance SD cards"
Thank you for these useful tips. Much easier that I had imagined
Great comments on this video too. :)
I agree, I have a lot of knowledgeable viewers and commenters!
Thank you very much! It was useful and interesting 🙂
Glad to hear that!
Useful AND interesting! I was unaware of the read only setting.
Glad you like it!
@@AndreasSpiess I realized I was unknowingly using the read only feature on my pistar hotspot. 🤣🤣 I thought it was part of the project and didn't realize that was a feature of the OS.
AFAIK Pistar does not use Raspbian and therefore probably implemented a different concept. But it seems to work ;-)
Years ago, my daughter had been using my core 2 duo laptop running ubuntu 12LTS booting(with pre-loaded apps) from the dvd drive loading it on ram for several years till the internal cooling fan conked out. You could still save on the hard drive but everything else resets after shutting down.
Others also suggested that the overlay FS was used for USB stick booting which was probably similar to your concept. So nothing new, but still useful and now integrated into Raspbian (good for noobs like me)
thanks to the guy with the Swiss accent, another great tip and information
Happy to help!
You can enable iscsi drive from network for writing.
Unfortunately, I do not understand :-(
@@AndreasSpiess As far as I understand, Raspberry Pi 4B supports iSCSI in the boot-firmware, so it might be able to just boot off of an iSCSI disk.
Alternatively, one could OverlayFS mount an iSCSI volume on top of the SD-card.
(and if you want to make things complex, which is off-topic, then you could include docker and mount network-shares only inside containers).
The overlay technique is actually pretty old. They used it for bootable Linux CD-ROMs back in the day.
Good to know. And now they made it simple to use it.
This is a discussion about two topics:
1. How flash works.
2. BC/DR.
Lets solve 'two' first. If your data/system is critical make sure you take/keep a remote copy and keep it off site. End of story. 'Local resiliency' doesn't help you if your data center - or in this case raspberry pi - goes up in smoke.
For 'one' let's talk about flash's 'peculiarities.' Flash devices - and by the way both memory cards and SSDs work the same way - have the following characteristics:
1. Fixed block size for writes. Unlike spinning disks flash can has a fixed write size. i.e. If a device writes at 32kb and your write is 4kb. The device writes 4kb of data and 28kb of zeros. If your write is 128kb the same device writes 4 × 32kb writes.
2. SLC, MLC, TLC. Denotes the number of bits in a cell. 1, 2 or 3 respectively. Additionally, it indicates 'burn rate' how quickly cells those cells die. The greater the number of bits in a cell the faster they wear out. Most consumer grade SSDs and flash cards are MLC.
3. Overprovisioning. Every consumer flash device SSDs and flash cards are overprovisioned to take wear and tear into consideration. SSDs have wayyyyyy more than Flash cards as they have to deal with the writes an OS throws at them. So don't buy an SSD buy a bigger memory card. It will last a lot longer, be easier to deal with and work out cheaper.
4. Fade. As magnetic media is hit by 'bit rot' flash gets hit by 'fade.' Over time the bits flip or the cells fail and the files become unreadable. Magnetic interference and heat accelerate this process so keep your pi cool.
4. Device firmware. Not all flash devices are created 'equal' and - no disrespect to the engineers who do this - the firmware on the a given generation of devices often gets written in a hurry. So the lifespans of two different cards with the same size advertised size and identical workloads may differ drastically in terms of lifespan.
5. Reads are free, writes are poison. High levels of random writes are absolute cancer for flash. Everything slows down if your device is bangoed with high write traffic. So if you are logging tons of data? My advice send it to a NAS backed by spinning disk DON'T store it locally on your memory card it will kill your flash device quickly.
So in summary:
1. Keep a back up.
2. Buy a bigger memory card than you think you need.
3. Keep your PI cool.
4. For all random write traffic send it to spinning disk.
5. Check the product reviews for angry consumers complaining about 'failures within weeks of purchase.'
How do I know this? I was employee 186 for the best enterprise flash array vendor on the market. As an aside - but related to this - 'middle out' from the 'Silicon Valley' series is based on a real algorithm that optimised wear and tear on MLC using meta data references to avoid writes if possible.
Thank you for your additional information. Everything is true. Do you have information about the wear-levelling of SD cards? I read a lot that they do not have such an algorithm and manufacturers explicitly forbid to use them in file system scenarios (and refuse warranty as one viewer wrote).
Just a few additional thoughts about the use cases of the video:
I talk about Raspberry Pi computers which are not often used in data centers nor do they have to deal with lots of data, because its performance is weak compared to servers. On the other hand its data often is not very valuable for the long run or does not change a lot. Many of our Pis just do some data transfers (sensors, gateways). or are used to drive displays, etc. and are sometimes operated in harsh situations.
Fortunately, we will hardly hit any of the specified transfer volumes of SSDs. However, we all experienced crashes of SD cards. Sometimes also because the power was switched off during a bad time which corrupted the FS.
Wow..I didn't know about this feature..?...thanks!
You are welcome!
I feel you should look at using a network drive so you do not need to use so much RAM. One simple way to make a network drive is to use an old desktop as a Linux box and have it share the hard drive to the raspberry pi.
You are right. Other viewers also mentioned remote logging as an opportunity. I assume it would need at least some Linux knowledge.
Write files to USB Flash & update as needed if data logging with minimal data loss. ...There exists a "HAT" that goes under the Pi and is screwed on, supports an M.2 SSD via a small USB 3.0 adapter. The bonus: you get SSD speeds with low power draw.
Do you know if USB sticks use wear-levelling? Otherwise we probably have the same issues.
Good that the "hat" goes below the Pi4. I still see too many "top hats" which create heat issues.
I have the same SD-card running since my Raspi 3B+ was new. 2018? 19? Never got corrupted (headlesss lite raspian doing DNS and Cups). I still did that, thanks.
So you were lucky and got a good SD card, I think.
Great. I just needed this.
Very good!
You can even get rid of any local storage by booting them over the network and have the rootfs on a nfs share
You are right. I never tried it so far.
What would be the difference when leaving the boot partition on read/write while having overlay enabled?
I never thought about that :-(
Nice!, this will also improve the performance. I am thinking about to create a minimum raspbian version to improbe the boot speed and memory usage on a Raspberry Pi Zero.This will help me for sure.
And it is needed for the Pi zero. I do not like it because it is very slow. Maybe with your changes it is ok...
Hi Andreas,
Last week I wanted to update a 2017 installed Raspbian Jessie on its 8 GB µSD card. Of course, I dumped the card beforehand ... or at least I planned to. Because dd reported many defective blocks. The dump ran for 48 hours because of the many retrys.
Out of interest, I checked the file systems in the dump: were both okay. Obviously the mechanism that hides defective cells in the µSD card works very well. But I'm also sure that the card was on the verge of total failure ;-)
vy 73 de Micha, DD0UL
Good that you were able to get the content. This was not always the case here...
I stopped to use my 8GB cards when I saw how slow they are compared with the bigger cards, not because of unreliability. So I spent a few bucks.
The medium article instructions, while time consuming are better when it comes to usability
With overlay fs, you need to reboot every time you need to make any changes and then reboot again to return
Yes, you need to reboot twice for changes.
More complex than the other methods, but wired Ethernet allows diskless PXE (network) boot. Unfortunately wifi doesn’t support that.
Others also mentioned network boot. Also a good idea.
Useful and interesting as usual.
Glad you think so!
Great find!
I'm going to have to check to see if it works on my OctoPi server 😃
I never tried so far. But it should not need to keep a lot of data after reboot.
Ooooo DMR radio hot spot. What radio chat room # do you haunt? usually 93 or scanning is what i'm doing.
I am on our Swiss TGs plus the Redit TG.
@@AndreasSpiess So 228 and TG950 .
228 is only used by non-Swiss guys calling Switzerland. Very quiet TG. 2280 / 2284 are the ones used. Reddit is now on 98003
great video. happy christmas
Thank you! Happy Holidays, too.
Nice hint I was not aware of! It's there a known command to write down all memory changes to disk aka SD? That would be great if you are experimenting and at the end think, ah I should have saved this. As I understood it, writing or not ALWAYS requires a reboot first, correct?
Log2RAM does something like that. It stores the data in memory and saves them every hour or so as well as before a proper shutdown.
@@AndreasSpiess awesome, that is what I'm interested in, I will look into it. Thank you!
Puppy & other distro's can be setup to load into RAM thus SD card can be removed or just to store personal files
Thank you for the tip. As a Linux noob I only use Raspbian :-(
Yes, good and useful video.
However, I disagree slightly on the SSD; they do wear out.
Harddisks are less likely to wear out (choose CMR, *not* SMR; choose WD-RED, not WD-BLUE).
But ... There is another option which isn't extremely complicated: Use NFS for storing your log-files. You don't have to set up netboot (which I just attempted yesterday and failed). You can make your fstab auto-mount your NFS shares and then have your 53 Raspberry Pi's (including Raspi Zero) log to one server.
You'll then have the convenience of looking on one computer rather than 53. ;)
-That is if you have your RasPi's connected to your network (they don't have to have internet access, though).
Remote logging is a good alternative if you really are interested in logs. I rarely use them (for finding errors) on such simple installations I use SD cards.
SSDs usually specify a max amount of data stored before they die. The data on my Raspberries are very, very small compared to these numbers because they consist of sensor readings and alike, not videos (different to my UA-cam production machine)
@@AndreasSpiess What I'm concerned about is writing. Writing kills SD-cards, MicroSD-cards, USB-sticks and SSD drives - not the amount of data stored.
But as you're indicating; if writing a lot of data, the card / SSD will die quickly compared to if you write only a little data.
But repeatedly writing small files and deleting them (eg. compiling) is bad for all such devices. My brother's two SSD drives just died recently (mounted in a Mac Pro). Those drives didn't last more than a few years. Even WD-GREEN harddisks last longer (they're no longer available; WD-RED is much better anyway).
NFS can also be good for data-logging and that 'family' (including analyzing) - this also opens up doors for off-loading analyzing to the server instead of the device. :)
Now you mention video; I know you can actually attach cameras to the RasPi, which means you could make smart surveillance via NFS shares too. ;)
Have you tried using high endurance SDCards sold for devices like dashcams that repeatedly overwrite the storage?
Not yet. But a few viewers suggested to use WD purple. So they are in the mail now ;-)
@@AndreasSpiess They should be a lot better, but they will also wear out at some point. This also depends a bit on the file system you're using. Unfortunately I don't think LittleFS can be used seriously with Linux yet.
I know I stated that SSD would wear out and that harddisks would last longer. This also depends on the use case. If for instance your harddisk is placed in a -20° shed, then the SSD will probably last longer. Harddisks may die if turned off and put on the shelf for 15 years (SSD likewise). -So it all depends on things we know and things we don't know. :/
I usually use a μSD card only to boot from and I have the storage on network with Ceph.
Also a good possibility if you know what you do...
Just a quick question about your description, why the anti-docker statement? Is it against docker corp (understandable) or the concept of containerization itself? Thanks for your wonderful work towards better IoT and wireless communication!
No. It is only for UA-cam. I use Docker in my videos…
Have you tried using USB sticks instead of the SD cards? I am mostly using small (memory and physical size) USB sticks now, because they are easier to handle and faster than SD cards, if you use USB3. I haven't had any corruption of SD cards or USB sticks in a long while, but i may use the overlay option for my pan and tilt Pi cameras in the future. For a more expensive SSD solution, depending on the CM4 module, one could always use a CM4 board with M.2 SSD connector like the CM4-IO-BASE-A, which starts at about 25€ for the board and 30€ for the CM4.
No, I never used USB sticks. Maybe they become an alternative to small SSDs if they are fast enough. I would not use them instead of SD cards because I like that the SD cards are so small and fit into the case.
WD line Purple of microsd cards for video security has a write cycle of up to 16tbw for a capacity of 32gb.
Thanks for the tip. I ordered a few of them ;-)
Very useful for making changes and checking they work correctly. One question, once you make some changes can you then commit the changes to SD so the changes take effect on reboot?
Not with this method, AFAIK
• 2:15 - Another disadvantage of SSDs is that as flash-media, they wear out like a memory-card does, so it's no better.
• 3:19 - The write-protect switch on SD cards is just a piece of plastic, the write-protect function is only in the card-reader/writer, and not all card-readers (*cough*Chinese*cough*) honor the switch, so it doesn't guarantee to protect a card from being written (this is the same as with floppy disks).
• 4:22 - This makes the system even more vulnerable to data-loss than ever. 🤔 You assume reboots are intentional.
1. SSDs use wear leveling which was not the case for SD cards. This is why they were not specified for computer usage.
2. You are right
3. I do not agree because I only use it for relatively stable applications. So reboots for configuration changes are very rare.
a pxe boot with a tftp server hosting the image of the operating system ?
or even , clients on a vnc environment that runs elsewhere and maps the hardware of the pi to that user profile .
Using network boot can also be a good possibility if you trust the network connection. I never used it.
Very useful and simple, many thanks.
Glad it was helpful!
If you want to store data and you have a NAS available, maybe ist worth to create a network share, where you can store your variable data.
Also a good idea!
@@AndreasSpiess next step with my raspberries: booting from network ...
For logs and things I want to keep I would create separate partition. This way reducing system writes and keeping the data I chose.
I do not get the point if the partition is on the same SD card :-(
@@AndreasSpiess Well, guess it depends how much logging are you doing. If it is a constant dense stream then SD cards is not good enough.
Just to be sure I meant to use it with Overlay System so the only writes would be from logs.
What I was thinking of doing is setting up the overlay for most of the file system, so that all the system logs etc went to RAM. Then I’d have a small partition with my Python code and the small amount of data I intentionally write that was RW.
I’d forgotten about USB SSDs, though. I think I will get one of those because I’d like to log sensor data locally (battery charging and power consumption and temperature and so on. Just saw some potentially anomalous behavior from the 72A battery charger and realized that a 1+ Hz data log would be nice…)
Thank you this was a simple but effective trick!
You are welcome!
thanks! this helps me a lot!
You are welcome!
It was interesting. I do have an original Pi 1 that has the operating system on an attached powered Hard Drive. Just uses SD card to boot.
I only started with the Pi3.... I tried once to attach a HDD but had to solve the power issue...
@@AndreasSpiess yes, definitely works. Actually, I use nfs mode to develop o/s image updates, then use squashfs to compress it and send it over the network and into ram from a server at boot time. Runs as normal, no sd card required. Needs 2gb ram minimum. I have 5 pi4s on my network doing the home automation (node red and home assistant) - I can't have part of my house dropping out because of a worn out sd card!
Disable file date updates. This will beat the heck out of sd cards. It will also speed up access times.move logs to ramdisk and dump and rotate logs to external storage.
Agreed. I assume that the instructions I mentioned do such things.
I have installed "remote-iot" to my pi3B/3B+ and am monitoring total writes -the only tool I found that would read microSD stats. I added "/dev/root / ext4 rw,noatime,commit=60 0 0" to mtab" to minimize writes. I have computed writes over many days using the "total writes" count, and it is about 10/sec. I am trying to figure how "many" this actually is...... Doesn't seem like much. Hard to translate this. Thanks for your video! Wish stretch/jessie had overlay FS.
I did not know of remote-iot. Its SD monitoring function is very interesting! The only problem: Nobody seem to know how many write cycles a typical SD card supports before it dies...
Hi. Can I do this with twister OS? This OS is so beautiful. I ve seen a review but I want to follow your advice. If Twister OS has this set up for overlay, would be great.
I only use Raspbian:-(
Hey there,
i had the same problem.
I solved it by just using a MicroSD to SD Adapter in a SD to MicroSD. ^^
The Different is that y can switch the writeprotection on at the MicroSD to SD Adapter.
Old SDs had a small switch at one side. The smaler MicroSD did not have one.
Sometimes its simple like that.
Regards Richard
Good idea! However, the OS has to support it...
True, I've lost a few SD cards due to unexpected power off
Thank you for sharing your experience!
strange i dont see this Performance tab in my OS of desktop raspian
Strange. Maybe it changed place in newer releases...
thanks for the knowledge
You are welcome!
On aRaspi 4 8GB I disabled swapping to reduce disk operation
This is also a good thing. I recommended it also if past videos. But with the bigger memories it does not help a lot because they usually do not use the swap file a lot.
Does this also help in the event of a sudden power loss? I imagine as long as there's no writes occuring at the time of loss it could be okay?
Yes. Because nothing is written directly to the disk (or SD card). So, your Pi always starts with a "fresh" image
What about buying one of those cheap mini USB sticks and put the /home directory on that? Maybe even those directories with most logs.
I do not know about the wear-levelling of USB sticks. They are also made for a few large file transfers and not for small and frequent write cycles. So they might suffer the same issues as SD cards.
@@AndreasSpiess I'm using an old 16GB usb stick in my cctv system (RPi Zero W + OTG cable for stick + Rpi Camera V2) since 6 month without any issues. With SD cards, the system is not surviving more then 2 months.
The system uses raspbian lite without any special configurations.
@@AndreasSpiess Yeah, i guess someone needs to investigate :-)
I have few application that move files from 1 folder to another in my Pi. I assume that the overlay file system will revert everything back on restart?
I getting a SSD for my 3B+ soon, which is a better solution for my situation (i think)
1. Yes, it will revert to exactly the initial state
2. An SSD is a good thing. However it will not be very fast on the USB2 of a Pi3B+
I have a ton of raspberry pi computers (mostly 3b & zero W models) and I've done the 'barefoot' method before. I haven't tried the built-in raspbian feature yet but I wonder, does the feature disable write to other storage? If it doesn't, then you could redirect or save 'persistent' files and such to another device like a simple thumbdrive.
I assume it only blocks the SD card from writing, but I did not try. Other viewers also suggested to use network drives for valuable data.