Great video. Thanks for this! Just wanted to add that at 10:12 instead of adding VM-Drives as a directory storage you can add it as ZFS storage. That way you'll be able to snapshot your VMs and containers.
I missed that. Thank you for pointing this out to me ! Still, i'll use TrueNAS Scale VM for my ZFS stuff as it gives me more flexibility with zfs, snapshots, smb and other goodies. Once again, thank you!
This was so helpful. The explanation of not adding storage straight away and adding directories with the type of your choice was the bit of knowledge I was missing. This video helped unblock the next stage of setting up my Proxmox server. Thank you!
Stumbled on these videos and going through them one by one. Absolutely great to get an understanding on Homelabs and all these great opensource tools and in the sametime building a proper environment up at home. Great work, it helps me a lot in getting things actually done! Keep doing this, it's much appreciated!
This is great for beginners like me. For instance I liked how when you where creating the directory you took a second to show the viewer the location of the directory you were creating in the shell to show them how you got that information
Most of the things I learned since started my home lab journey is through trial and error. Set everything up, try something, mess up, break OS, reinstall and try again :)
I like that you are showing an economical solution for homelab projects. I’ve seen so many presentations with inaccessible or prohibitively expensive equipment. I also heard that while it is advantageous to have the disks from the same vendor of the same size for mirrored arrays. That they shouldn’t be from the same factory or product run as that increases the probability of a systemic failure from production of the hardware. There is less of a chance flaw if they are from different manufacturing line.
Yes, i completely agree. For my home lab i wasn't that worried. For my friend you wanted a simple RAID nas i told him to buy same brand drives but from different online vendors.
Excellent ! on a laptop ! this guy has achieved hero-status; a shining example of "the under-staffed, accomplishing the impossible, equipped with a nothing(laptop)."
This was such a great video. I appreciated how you just walked through the process at a steady pace so that I could follow. You demonstrated why not to do something and you didnt get stuck through any of it. Keep up the great work!
Exactly the guide I needed! I was trying to set up ZFS with two 2TB NVME drives and your tips on using “zfs create” and “move storage” made it so easy. I was afraid I would have to remake my VMs! 😅 Fantastic! 👍
Thank you for taking the time to put this together. Clear, simple to understand, feel a lot better about the entire ZFS config in my Proxmox box! Huzzah!
excellent video. You made my life much easier :-) Only thing I wish you mention or explain is why to create and use ZFS as I'm learning everything about Proxmox. Thank you for sharing.
ZFS gives a lot of benefits, for example: - compression. If you set ZFS compression to ON or LZ4 (Proxmox default compression is LZ4, doesn't matter which one you pick). Compression will save you some space inside ZFS pool by compressing files. - Snapshots. Before you do major changes to data that is stored inside ZFS i suggest to do a snapshot. If things will go wrong with that data - you can restore snapshot to a different place and this way retrieve files. - Redundancy (When you are using 2 or more drives). If one drive fails your data is still their (depending on RAID configuration)., - ZFS is go to for servers. It is a good way to learn things about ZFS. You never know, maybe in a future you will need to do something with ZFS and using ZFS inside your home lab is a good way to learn, practice.
Brilliant presentation! I was glued to this video as if it was a suspense movie. Also took my time to take proper notes! It is seldom that you find someone that goes into so much depth of a topic and present it so well. Liked and subscribed. Will definitely check out your other videos. I have definitely learnt such a lot already. Kudos
I got one of those mini desktops with i5-6500T and I love it. It runs like minimum 9W's and even when streaming ripped dvd's from Jellfin I see power consumption from wall being about 13-17W's. Absolute max was something like 45W's when hitting it fully with services and Handbrake. Computer I built years ago takes like at least 10 times more power idling. Considering adding my old "NAS" one of the first celeron NUC's to add as node if I need. I also takes like whopping 7W's idling and maxes like 16W's. There's absolutely sense to run HL in laptop or these mini PC form factors. 24/7 usage just adds up to electricity quite fast these days.
wonderful explanation of local ZFS. Many thanks for showing the "content" box which blocked me forever to figure out the issue with limited directory options.
Really helpful! I will definitely have a few comments/questions after I complete the initial setup for my PVE host # 1, looking forward to more enterprise proxmox videos in 2023, all the very best!
Good video! I have another reused server and it is based on the Internals of a 2003 HP d530 SFF. It contains a Pentium 4 HT (1C2T; 3.0GHz); 2x 512MB DDR (400MHz); 2 IDE HDDs 3.5" (250 + 320GB) and 2 SATA HDDs 2.5" (320 + 320GB). It runs FreeBSD 14.0 on OpenZFS 2.2.0, released November 2023 :) I have 2 striped datapools; zroot on the 2.5" HDDs and apool on the 3.5" HDDs. The system just can store 2 snapshots of my desktop and has ~20% free space in each datapool :) The snapshot transfer reaches ~30MB/s (240Mbps) on a 1Gbps link. The limit is caused by a 90% load on 1 CPU thread. I like this setup of a 20 year old PC with the latest OS of 3 month ago.
Want to thank you!! I am new no this, the part that a look for apart is the fdisk to delete partitions because they dont apear me in Créate ZFS, later of that everything was perfect!!
Thank you for explaining how to use the ZFS pools, this is something that was kinda confusing when installing Proxmox. My first intention was to use the BIOS RAID option, but it seems that Proxmox hates that fake RADI solution, which left me with ZFS for the redundancy :D
thank you for the concise guide! i just installed pve on 2 drives yesterday and it had some unallocated space. i was unable to create a zpool with the gui since the OS already used all the drives so i used the sgdisk and zpool command in the shell to make them. i followed the video for the rest.
Thanks, great walk through. I had created a ZFS pool but it didn't create any storage on it for me (which is normally how it does it) and I was confused why. Now I see there is a "Add Storage" check box that I must have unchecked somehow.
Thanks nice video just one thing to mention if you add a zfs as directory you cannot perform snapshot for containers you must add a zfspool type in order for that to work
This was the 1st video I followed! I have the same setup almost! this guy also does a file server video don't follow anyone elses video when it comes to the file server, you download one thing and it's done
5 bay enclosure is exactly what I plan to buy for my NAS and populate with 5x8TB HDDs. What would you prefer, one Raidz2 on all 5 drives or one Raidz1 on 4 drives and one HDD as additional backup? How would you configure it? It is important for me to make right decision now, because it is not easy to add a drive to the pool later.
Looking forward for the next tutorial! I have been following these and I will keep thanking you for saving me hours of research by putting those commands in the descriptions and explaining what they are for. In the next video would you please explain if this or the new configuration you will show is suitable to use with TrueNas? I watched some tutorials but few cover how it is with HDDs that are already ZFS in proxmox. Thanks a lot
In my TrueNAS Scale video i used the same drive but completely formatted them. There is a way to migrate ZFS from proxmox in to TrueNAS and i done that twice. But on the day of recording i just could not get that to work. Spent a lot of time trying to migrate ZFS pool from Proxmox to Truenas and after almost half a day of multiple reinstalls i just gave up and went for complete format.
You explain how to use ZFS for the disks vm's are installed on, but what if I just want an extra drive to show up inside my VM to store data on? Specifically, I want to run Nextcloud on my Ubuntu VM but store the data on the large ZFS drive I configured in Proxmox so I can have snapshots. But, I still want that storage access to be performant from my VM. Should I be going a different route?
I do have a couple questions. Apologies in advance for the long post but wanted to be thorough. I noticed that you don't have "local-lvm (pve)" in you storage list. When I installed Proxmox 8, it created 'local-lvm (pve)", an LVM Thin volume and "local (pve)", a directory pointing to /var/lib/vz out of my 1TB install disk; (2) 1TB SSDs mirrored with Startech ASM1062R SATA hardware RAID1 controller. The installer allocated 100GB for "local (pve)" directory and 853GB for "local-lvm (pve)". Did you destroy that volume? Is it safe to remove the "local (pve)" directory if I'm not using it? My research mentioned that VMDisks have less overhead when stored .raw on ZFS Pool (block level storage, zvol) as opposed to .qcow2 on Directory (file level storage). Comments? Are there any snapshot concerns? I created a ZFS RAIDZ pool of (4) 4TB NAS drives name "Depot", created data sets as you explained, "ISOs", "VMs", "Templates", "Backups" and created Directory storage for each. I also created the data set "VMs" under "Depot" and in Proxmox as ZFS to /Depot/VMs. Are these considered separate pools even thought they're nested? Thoughts. I've really enjoyed your videos and have subscribed. Many Thanks
I took the GPU out of my Laptop Dell M6600 and it now runs with screen off at 17watts pretty cool, will possibly use MSata converters so can have 2x Msata SSD per Sata Port.
@@MRPtech its been driving me a bit crazy trying to figure out how to get ZFS setup well amd each option sounds like.money lol. I mean the Msata are not cheap and would like to get 4x 2TB drives in there. People are saying not to use ZFS on USB. Other options are get a Mini PCIE to PCIE converter and get a HBA card in the laptop lol but thats crazy. Ps I realised that thr laptop os drawing around 9 watts idle the rest is the router!
Hey - love your videos. I somewhat disagree with the concerns for the enterprise server. I currently have 2 Xeon Gold 6154s and like 192GB ram. *Even* with a draw of like 300w at all times, your power bill shouldn't be more than 50 dollars USD or more extra - but perhaps I guess it depends on the country your in. Additionally, you can do power management via the bios with like C-step options and just turn off the system when your not using it. However, its possible that its not economical to have it run 24/7 in the long run depending on what your goals are. For me its to show off during SysAdmin interviews lol. Love your stuff keep up the good work.
Enterprise grade stuff are much more powerful then laptop or NUC style device. I totally agree/ In UK if i run server with 300w power usage going 24/7 - each month i will have to pay £120+ just to keep that thing on with no load. With what i have now (3x NUC proxmox cluster + synology NAS) total usage ~110w, monthly that costs me ~£45. Yes, Server grade stuff is great. but when you look at this: £45 < £120. My setup is almost 3x more power efficient.
Yep I see you mentioned running it 24/7 -only way to do this at home is running it at most 12 hours at a time and for like a very specific purpose.@@MRPtech
@@dan91121 I am so neck deep in to all this thing - I need to run a lot of stuff 24/7 :) Otherwise my how network will fall apart - no DNS resolving, home automation, how security, minotoring ect.
@@MRPtech Gotcha - I run proxmox mostly and am virtualizing a lot of this on the box. DNS resolving etc still done by via gateway ISP IP for the rest of the house. For my lab I let Pfsense handle the DNS Resolving.
I'm pretty sure that it's also possible to add the storage, so that it can be seen in the datacenter tree, then add a new dataset within it and finally to create a directory within this dataset, where any kind of data can be stored as well. It is not critical to uncheck the "Add storage" marker...
Great video - thank you so much! Just one question about the compression (which is done by zfs automatically if I'm not mistake). Why did you enable compression of the backups as well? Isn't that kind of double?
i don't think it doubles. if i understand correctly - by default if ZFS fails to compress a file it just gives up and skips it which is what happens with backup compressions.
Hey MRP, I have a question: If I create directories like you showed us and I add one of those dir paths to my container I cannot make use of zfs snapshots. I can only do the snapshots if the ZFS pool is directly added to the container. What is better now? To utilize the paths like you did here or just create ZFS pools and then add those pools to the containers/VMs so that we can also use snapshots? Do you know? Thanks a lot for sharing this
17:30 ZFS is more for reliability than anything else, so I would instead recommend to try to use different type of disks since the same ones would probably fault in the same manner (ps for same reasons I'm usually using two or even three way mirrors)
Greetings Sir, Really nice concise video and I enjoyed the lesson on creating additional datasets and then adding directories to them. You gave me a good jump start on learning Proxmox ZFS. One question on the window that creates a zfs pool. In the window ant the end of each listed available drive is a column called 'Order'. I have not found an explanation of exactly what that is. I know zfs doesn't care about the location of where the drives are connected to the computer. Does assigning a number to the 'Order' over ride this in zfs?
I dont think u have to create all those folders for ISOs VMs etc, if you just select multiple in the drop down it will create all the sub dirs automatically in that main dir u entered as your manually created ZFS dataset.
White this is true - you select all in drop-down option and all sub-folders will be created, it is not practical. ZFS does snapshot on dataset not sub-folders. If i set snapshot to be twice a day and keep it for 2 weeks - why would i need to keep ISO snapshots for 2 weeks? When you have datasets created per data you planning to store: dataset1 = ISO dataset2 = backups dataset3 = Data dataset4 = ..... You can set snapshots as you want without taking all the valuable space for unnecessary snapshots. ISO can have one snapshot a week, DATA - twice a day and keep for 2 weeks. Backups snapshot can run straight after backups are done and be kept for 1 week. In additional to all of that, separate datasets allows you to have separate compression options. You maybe wants compression set on DATA dataset but not ISO dataset. You can't do that per sub-folder bases - only per Dataset.
@@MRPtech I guess it depends on what changes in the folders; the snapshot will not take disk space if nothing changes. For example, if it is an ISO folder then the snapshot will only take up more space if you add or change items.
Hey MRP! Thanks a lot for the video, extremely helpful! What if i have 2 x 2TB drives and one is nvme and the other is mechanical. Do I create 2 separate zfs pools (tanks) for each drive and i use the faster nvme for vms and the slower mechanical for backups? Would you do it this way? Best regards.
Hi, Thank you for your comment. If i had 2 drives, different types. i would do 1 drive as single zfs for VM (that would be NVME) and another drive as single zfs for backups, ISO ect. If you combine them into one mirror zfs raid, speed of that raid will be maxed out at the slowest drive in that raid which in this case will be standard HDD.
i bought 2 proliant gen8 mlp350 for around 230€ :D they have around 150gb ddr3 ram, 20 cores and als backpane for 8 nas drives xD. sure its loud, but i can hardmod it. i dont like those small nas bays for that price
How do I store other types of files on the ZFS pool? I am running an Emby server on Proxmox and want to store holiday videos and music etc on the RAID storage. Thanks!
let's say you have mirrored ZFS with 500GB drive each. and you want to increase that size, first you replace one drive with bigger size, get resilvering going and once that is done replace 2nd drive - wait for resilvering. Once all that is done zfs will still use old size (500gb). You will need to expand size of each drive to give ZFS all storage of those drives.
@@MRPtech thanks! I've removed the ZFS option and now looking into the TrueNAS way. So have the 500GB there - same question- can i somehow upgrade or add other drives without causing any hassle on the current config? Is that stripe mode the right for one drive?
Do we need to setup a ZFS scrub to run every X days and how do we do it in terminal/proxmox? I currently have TrueNAS but Im gonna switch everything to proxmox. Currently I have ZFS scrub setup every 2 weeks, so I'm wondering how to set it up on Proxmox/shell
I setup my ZFS Pool like this using a directory called BACKUPS. I've since reinstalled 8 and now I need to re-add this to the new server to restore my vm's. Can you provide a link that shows how?
I am looking for software for backup. So i have Proxmox with 2 HDD for multimedia. So i want to use 1 HDD (for multimedia) and the HDD 2 should stay off, and only once a week(for example) to be turned on for a backupp to whole hdd1 or specific folder. i dont need hdd2 to be running 24/7. any recomendations?
How do you create a SMB Share with this data set? I'd like to not only have a way to run VMs, example a Windows Stream Server with GPU passthrough, and a way to have mass data storage for my local network for work station access.
@@MRPtech The drives were being used as Truenas storage in a separate system. I moved it to my old z820 on SAS drive bays, imported zfs pool in proxmox and attached it to truenas VM in z820.
I created a RaidZ1 pool of 4x 4TB drives. This means I should be able to use 12TB/16TB. But when I create a drive for my VM I can only use upto 7TB. It then shows I'm using 11.5TB/12TB. Why is that?
@@MRPtech Apparently it was because of the blocksize. Zraid1 + 4 disks + 4kb blocksize (default) there is a lot of inefficiency, I could only use 7TB. With 16kb I could use 10TB
Woups... So it was so easy this crash my proxmox install... I connected an older sata disk via an USB-Sata cable and create two partitions (yes, it's a home lab...). I then create a zfs mirror on these two partitions and some datasets. One of the datasets I mounted to a smb sharing container and it all executed fine. But then... I decided to see what would happen if I disconnected the usb cable (just to see how robust it was) and then everything stopped working. Rebooting just takes me to the emergency mode. Error message: Failed to start zfs-imort@test\x2mirror.service - Import ZFS pool test\x2dmirror
With zfs destroy and zpool destry i finaly managed to get back on track again, but I must say that it was kind of surprising that there was so hard dependencies on the disk layer for the proxmox environment to start
@@MRPtech Because I have one and found that based on SMART temperature reporting it couldn't keep 3 drives (8TB WD Red) under 55C while under light load at 28C ambient temp. The airflow is not adequate for three drives spaced out in the cages let alone five. There is only ~7mm between each drive. The Inverse Square Law always applies and there isn't enough airflow to overcome it. Look at the ridiculous exhaust grill. And the drives are oriented horizontally which only helps roast the top drive the most. I ignored these facts when selecting the Yottamaster enclosure.
Hmmm. i haven't notices anything bad with cooling when i was using Yottamaster. Long story short i end up with Synology DS423+ and this is how hot drives are at this moment: prnt.sc/rCgx_wOWg5Ct
Guys like these are the secret legends of youtube. Providing real value that lasts years, just waiting to be discovered by the next n00b like me :)
Great video. Thanks for this! Just wanted to add that at 10:12 instead of adding VM-Drives as a directory storage you can add it as ZFS storage. That way you'll be able to snapshot your VMs and containers.
I missed that. Thank you for pointing this out to me !
Still, i'll use TrueNAS Scale VM for my ZFS stuff as it gives me more flexibility with zfs, snapshots, smb and other goodies.
Once again, thank you!
Thanks a lot for that notice!
I think you may have answered my question I just posted. Thanks
This was so helpful. The explanation of not adding storage straight away and adding directories with the type of your choice was the bit of knowledge I was missing. This video helped unblock the next stage of setting up my Proxmox server. Thank you!
Your welcome. I am glad my video helped you 😊
I was stuck here as well
@@MRPtech Sorry which content ZFS type (diskimage, container...) I should use for media/document storage like movies, music, pdf?
Stumbled on these videos and going through them one by one. Absolutely great to get an understanding on Homelabs and all these great opensource tools and in the sametime building a proper environment up at home. Great work, it helps me a lot in getting things actually done! Keep doing this, it's much appreciated!
This is great for beginners like me. For instance I liked how when you where creating the directory you took a second to show the viewer the location of the directory you were creating in the shell to show them how you got that information
finally the correct explanation about why i couldnt setup directories the way I wanted - even chatgpt was no help 🙂
Most of the things I learned since started my home lab journey is through trial and error.
Set everything up, try something, mess up, break OS, reinstall and try again :)
I like that you are showing an economical solution for homelab projects. I’ve seen so many presentations with inaccessible or prohibitively expensive equipment. I also heard that while it is advantageous to have the disks from the same vendor of the same size for mirrored arrays. That they shouldn’t be from the same factory or product run as that increases the probability of a systemic failure from production of the hardware. There is less of a chance flaw if they are from different manufacturing line.
Yes, i completely agree.
For my home lab i wasn't that worried. For my friend you wanted a simple RAID nas i told him to buy same brand drives but from different online vendors.
Excellent ! on a laptop ! this guy has achieved hero-status; a shining example of "the under-staffed, accomplishing the impossible, equipped with a nothing(laptop)."
This was such a great video. I appreciated how you just walked through the process at a steady pace so that I could follow. You demonstrated why not to do something and you didnt get stuck through any of it. Keep up the great work!
I saw some clips and notice the same, pretty cool explanaition
I like the fact that you are doing this in a low power. do with what you have way. exactly what I am interested in.. Very good.
Super video, helped me immensely! Love how you walked through it so clearly, and edited the video to minimize waiting. Thanks!
Exactly the guide I needed! I was trying to set up ZFS with two 2TB NVME drives and your tips on using “zfs create” and “move storage” made it so easy. I was afraid I would have to remake my VMs! 😅 Fantastic! 👍
Thank you for taking the time to put this together. Clear, simple to understand, feel a lot better about the entire ZFS config in my Proxmox box! Huzzah!
Beginner friendly! Much appreciated, thanx!
excellent video. You made my life much easier :-) Only thing I wish you mention or explain is why to create and use ZFS as I'm learning everything about Proxmox. Thank you for sharing.
ZFS gives a lot of benefits, for example:
- compression. If you set ZFS compression to ON or LZ4 (Proxmox default compression is LZ4, doesn't matter which one you pick). Compression will save you some space inside ZFS pool by compressing files.
- Snapshots. Before you do major changes to data that is stored inside ZFS i suggest to do a snapshot. If things will go wrong with that data - you can restore snapshot to a different place and this way retrieve files.
- Redundancy (When you are using 2 or more drives). If one drive fails your data is still their (depending on RAID configuration).,
- ZFS is go to for servers. It is a good way to learn things about ZFS. You never know, maybe in a future you will need to do something with ZFS and using ZFS inside your home lab is a good way to learn, practice.
Rarely seen a video so clean and on point while matching the video title.
Thank you very much! :)
Very helpful as a new proxmox adopter. I was stuck here and your video corrected that.
I was going crazy trying add ISO images in the list of storage using a zfs RAIDZ pool, this video made it happen.
This is incredibly helpful; thank you!
Impossible someone to dislike your effort! Great job! Thanks
Brilliant. Useful, to the point and well explained. Thank you.
Thank you from Russia!
Brilliant presentation! I was glued to this video as if it was a suspense movie. Also took my time to take proper notes! It is seldom that you find someone that goes into so much depth of a topic and present it so well. Liked and subscribed. Will definitely check out your other videos. I have definitely learnt such a lot already. Kudos
Thank you for the great tutorial. I have successfully setup my new iso storage with this video!!
I got one of those mini desktops with i5-6500T and I love it. It runs like minimum 9W's and even when streaming ripped dvd's from Jellfin I see power consumption from wall being about 13-17W's. Absolute max was something like 45W's when hitting it fully with services and Handbrake. Computer I built years ago takes like at least 10 times more power idling. Considering adding my old "NAS" one of the first celeron NUC's to add as node if I need. I also takes like whopping 7W's idling and maxes like 16W's. There's absolutely sense to run HL in laptop or these mini PC form factors. 24/7 usage just adds up to electricity quite fast these days.
+1 on all supportive comments. This one helped me alot. Thx @MRP
wonderful explanation of local ZFS. Many thanks for showing the "content" box which blocked me forever to figure out the issue with limited directory options.
Really helpful! I will definitely have a few comments/questions after I complete the initial setup for my PVE host # 1, looking forward to more enterprise proxmox videos in 2023, all the very best!
This was a great tutorial - thanks heaps. I learned how to make the directories in ZFS, which was historically limiting my use of ZFS
I am so happy that my content helped you out.
This is a great video and explained many questions I had about storage in Proxmox. Other great tips I did not know also in video. 2 THUMBS up!!
Good video! I have another reused server and it is based on the Internals of a 2003 HP d530 SFF. It contains a Pentium 4 HT (1C2T; 3.0GHz); 2x 512MB DDR (400MHz); 2 IDE HDDs 3.5" (250 + 320GB) and 2 SATA HDDs 2.5" (320 + 320GB). It runs FreeBSD 14.0 on OpenZFS 2.2.0, released November 2023 :) I have 2 striped datapools; zroot on the 2.5" HDDs and apool on the 3.5" HDDs. The system just can store 2 snapshots of my desktop and has ~20% free space in each datapool :) The snapshot transfer reaches ~30MB/s (240Mbps) on a 1Gbps link. The limit is caused by a 90% load on 1 CPU thread.
I like this setup of a 20 year old PC with the latest OS of 3 month ago.
Hell yeah. This is why I like your channel
I really appreciate that.
Want to thank you!! I am new no this, the part that a look for apart is the fdisk to delete partitions because they dont apear me in Créate ZFS, later of that everything was perfect!!
Thank you so much, im searching this for a whole day.
Thank you for explaining how to use the ZFS pools, this is something that was kinda confusing when installing Proxmox. My first intention was to use the BIOS RAID option, but it seems that Proxmox hates that fake RADI solution, which left me with ZFS for the redundancy :D
newbie here and you made it quite understandable. thanks!
Perfect, exactly what I needed! Thank you.
thank you for the concise guide! i just installed pve on 2 drives yesterday and it had some unallocated space. i was unable to create a zpool with the gui since the OS already used all the drives so i used the sgdisk and zpool command in the shell to make them. i followed the video for the rest.
Thanks, great walk through. I had created a ZFS pool but it didn't create any storage on it for me (which is normally how it does it) and I was confused why. Now I see there is a "Add Storage" check box that I must have unchecked somehow.
nice explanation really helpful =) i just got 4 HDDs and planning on using it with proxmox
Thanks nice video just one thing to mention if you add a zfs as directory you cannot perform snapshot for containers you must add a zfspool type in order for that to work
Great video - very helpful and I liked the pace of the video as well. Cheers.
This was the 1st video I followed! I have the same setup almost! this guy also does a file server video don't follow anyone elses video when it comes to the file server, you download one thing and it's done
You save my life, real hero bro !!
man thanks for the info I'm starting with proxmox and this help me a lot
5 bay enclosure is exactly what I plan to buy for my NAS and populate with 5x8TB HDDs. What would you prefer, one Raidz2 on all 5 drives or one Raidz1 on 4 drives and one HDD as additional backup? How would you configure it? It is important for me to make right decision now, because it is not easy to add a drive to the pool later.
Looking forward for the next tutorial! I have been following these and I will keep thanking you for saving me hours of research by putting those commands in the descriptions and explaining what they are for.
In the next video would you please explain if this or the new configuration you will show is suitable to use with TrueNas? I watched some tutorials but few cover how it is with HDDs that are already ZFS in proxmox.
Thanks a lot
In my TrueNAS Scale video i used the same drive but completely formatted them.
There is a way to migrate ZFS from proxmox in to TrueNAS and i done that twice. But on the day of recording i just could not get that to work. Spent a lot of time trying to migrate ZFS pool from Proxmox to Truenas and after almost half a day of multiple reinstalls i just gave up and went for complete format.
yeah.. nice video. helped a lot and saved me hours of testing out!
Super helpful!!! Thanks mate!!!
You explain how to use ZFS for the disks vm's are installed on, but what if I just want an extra drive to show up inside my VM to store data on?
Specifically, I want to run Nextcloud on my Ubuntu VM but store the data on the large ZFS drive I configured in Proxmox so I can have snapshots. But, I still want that storage access to be performant from my VM. Should I be going a different route?
I do have a couple questions. Apologies in advance for the long post but wanted to be thorough. I noticed that you don't have "local-lvm (pve)" in you storage list. When I installed Proxmox 8, it created 'local-lvm (pve)", an LVM Thin volume and "local (pve)", a directory pointing to /var/lib/vz out of my 1TB install disk; (2) 1TB SSDs mirrored with Startech ASM1062R SATA hardware RAID1 controller. The installer allocated 100GB for "local (pve)" directory and 853GB for "local-lvm (pve)". Did you destroy that volume? Is it safe to remove the "local (pve)" directory if I'm not using it? My research mentioned that VMDisks have less overhead when stored .raw on ZFS Pool (block level storage, zvol) as opposed to .qcow2 on Directory (file level storage). Comments? Are there any snapshot concerns? I created a ZFS RAIDZ pool of (4) 4TB NAS drives name "Depot", created data sets as you explained, "ISOs", "VMs", "Templates", "Backups" and created Directory storage for each. I also created the data set "VMs" under "Depot" and in Proxmox as ZFS to /Depot/VMs. Are these considered separate pools even thought they're nested? Thoughts. I've really enjoyed your videos and have subscribed. Many Thanks
Funny Mr P I was watching Patrice O'Neal right before this. His fans need zfs pools too :) 0:04
This video is so amazing.
Great vid. So easy to understand and follow.
Just found your channel. Excellent Content - Another sub for you sir!
Very informative. Thank you.
I took the GPU out of my Laptop Dell M6600 and it now runs with screen off at 17watts pretty cool, will possibly use MSata converters so can have 2x Msata SSD per Sata Port.
17W o.O
@@MRPtech its been driving me a bit crazy trying to figure out how to get ZFS setup well amd each option sounds like.money lol. I mean the Msata are not cheap and would like to get 4x 2TB drives in there. People are saying not to use ZFS on USB. Other options are get a Mini PCIE to PCIE converter and get a HBA card in the laptop lol but thats crazy. Ps I realised that thr laptop os drawing around 9 watts idle the rest is the router!
Hey - love your videos. I somewhat disagree with the concerns for the enterprise server.
I currently have 2 Xeon Gold 6154s and like 192GB ram. *Even* with a draw of like 300w at all times, your power bill shouldn't be more than 50 dollars USD or more extra - but perhaps I guess it depends on the country your in. Additionally, you can do power management via the bios with like C-step options and just turn off the system when your not using it. However, its possible that its not economical to have it run 24/7 in the long run depending on what your goals are. For me its to show off during SysAdmin interviews lol.
Love your stuff keep up the good work.
Enterprise grade stuff are much more powerful then laptop or NUC style device. I totally agree/
In UK
if i run server with 300w power usage going 24/7 - each month i will have to pay £120+ just to keep that thing on with no load.
With what i have now (3x NUC proxmox cluster + synology NAS) total usage ~110w, monthly that costs me ~£45.
Yes, Server grade stuff is great.
but when you look at this: £45 < £120. My setup is almost 3x more power efficient.
Yep I see you mentioned running it 24/7 -only way to do this at home is running it at most 12 hours at a time and for like a very specific purpose.@@MRPtech
@@dan91121 I am so neck deep in to all this thing - I need to run a lot of stuff 24/7 :)
Otherwise my how network will fall apart - no DNS resolving, home automation, how security, minotoring ect.
@@MRPtech Gotcha - I run proxmox mostly and am virtualizing a lot of this on the box. DNS resolving etc still done by via gateway ISP IP for the rest of the house. For my lab I let Pfsense handle the DNS Resolving.
Excelent information. Thank you
Thank you very much!! You help me a lot!
I'm pretty sure that it's also possible to add the storage, so that it can be seen in the datacenter tree, then add a new dataset within it and finally to create a directory within this dataset, where any kind of data can be stored as well. It is not critical to uncheck the "Add storage" marker...
Great video - thank you so much! Just one question about the compression (which is done by zfs automatically if I'm not mistake). Why did you enable compression of the backups as well? Isn't that kind of double?
i don't think it doubles. if i understand correctly - by default if ZFS fails to compress a file it just gives up and skips it which is what happens with backup compressions.
Great tutorial !! Thank you for that
excellent stuff
Did you made a video on how to restore zfs pool after a disk failure or replacement?
Thank you very much!
thanks mrp
This definitely helped.
I like the GUI and the taskbar. What OS are you running?
That is Samsung DeX running of my Fold 3 Galaxy Phone.
Thanks for this tutorial..great and useful content.
Thank you!
Hey MRP, I have a question: If I create directories like you showed us and I add one of those dir paths to my container I cannot make use of zfs snapshots. I can only do the snapshots if the ZFS pool is directly added to the container. What is better now? To utilize the paths like you did here or just create ZFS pools and then add those pools to the containers/VMs so that we can also use snapshots? Do you know? Thanks a lot for sharing this
Thank you
17:30 ZFS is more for reliability than anything else, so I would instead recommend to try to use different type of disks since the same ones would probably fault in the same manner (ps for same reasons I'm usually using two or even three way mirrors)
Thanks for this!
Greetings Sir, Really nice concise video and I enjoyed the lesson on creating additional datasets and then adding directories to them. You gave me a good jump start on learning Proxmox ZFS. One question on the window that creates a zfs pool. In the window ant the end of each listed available drive is a column called 'Order'. I have not found an explanation of exactly what that is. I know zfs doesn't care about the location of where the drives are connected to the computer. Does assigning a number to the 'Order' over ride this in zfs?
I dont think u have to create all those folders for ISOs VMs etc, if you just select multiple in the drop down it will create all the sub dirs automatically in that main dir u entered as your manually created ZFS dataset.
White this is true - you select all in drop-down option and all sub-folders will be created, it is not practical.
ZFS does snapshot on dataset not sub-folders.
If i set snapshot to be twice a day and keep it for 2 weeks - why would i need to keep ISO snapshots for 2 weeks?
When you have datasets created per data you planning to store:
dataset1 = ISO
dataset2 = backups
dataset3 = Data
dataset4 = .....
You can set snapshots as you want without taking all the valuable space for unnecessary snapshots.
ISO can have one snapshot a week, DATA - twice a day and keep for 2 weeks. Backups snapshot can run straight after backups are done and be kept for 1 week.
In additional to all of that, separate datasets allows you to have separate compression options. You maybe wants compression set on DATA dataset but not ISO dataset. You can't do that per sub-folder bases - only per Dataset.
@@MRPtech I guess it depends on what changes in the folders; the snapshot will not take disk space if nothing changes. For example, if it is an ISO folder then the snapshot will only take up more space if you add or change items.
@@MRPtech Very informative. And having everything organized helps my OCD lol.
Hey MRP! Thanks a lot for the video, extremely helpful! What if i have 2 x 2TB drives and one is nvme and the other is mechanical. Do I create 2 separate zfs pools (tanks) for each drive and i use the faster nvme for vms and the slower mechanical for backups? Would you do it this way? Best regards.
Hi,
Thank you for your comment.
If i had 2 drives, different types. i would do 1 drive as single zfs for VM (that would be NVME) and another drive as single zfs for backups, ISO ect.
If you combine them into one mirror zfs raid, speed of that raid will be maxed out at the slowest drive in that raid which in this case will be standard HDD.
A quick question if we can move these two hard drives to another Proxmox installation if the PC is broken or if you decide to upgrade the system.
really nice help you save my day:)
i bought 2 proliant gen8 mlp350 for around 230€ :D they have around 150gb ddr3 ram, 20 cores and als backpane for 8 nas drives xD. sure its loud, but i can hardmod it. i dont like those small nas bays for that price
can ZFS be a shared storage and implemented on multiple nodes??
Very helpfull video.
How do I store other types of files on the ZFS pool? I am running an Emby server on Proxmox and want to store holiday videos and music etc on the RAID storage. Thanks!
Followed everything cool! Managed to get one external 500GB (not SSD) - Is there option to get a larger storage added on top of this ZFS?
let's say you have mirrored ZFS with 500GB drive each. and you want to increase that size, first you replace one drive with bigger size, get resilvering going and once that is done replace 2nd drive - wait for resilvering. Once all that is done zfs will still use old size (500gb). You will need to expand size of each drive to give ZFS all storage of those drives.
@@MRPtech thanks! I've removed the ZFS option and now looking into the TrueNAS way. So have the 500GB there - same question- can i somehow upgrade or add other drives without causing any hassle on the current config? Is that stripe mode the right for one drive?
Do we need to setup a ZFS scrub to run every X days and how do we do it in terminal/proxmox? I currently have TrueNAS but Im gonna switch everything to proxmox. Currently I have ZFS scrub setup every 2 weeks, so I'm wondering how to set it up on Proxmox/shell
I setup my ZFS Pool like this using a directory called BACKUPS. I've since reinstalled 8 and now I need to re-add this to the new server to restore my vm's. Can you provide a link that shows how?
I am looking for software for backup. So i have Proxmox with 2 HDD for multimedia. So i want to use 1 HDD (for multimedia) and the HDD 2 should stay off, and only once a week(for example) to be turned on for a backupp to whole hdd1 or specific folder. i dont need hdd2 to be running 24/7.
any recomendations?
How do you create a SMB Share with this data set? I'd like to not only have a way to run VMs, example a Windows Stream Server with GPU passthrough, and a way to have mass data storage for my local network for work station access.
Niice! good job
how do i boot via pxe now into the tank?
How do I share folders across lxc or VM?
Subscribed!
I have a problem with this method that whenever system reboots, I have to run zpool import command otherwise it is not available
How your drives are connected to the system ?
@@MRPtech The drives were being used as Truenas storage in a separate system. I moved it to my old z820 on SAS drive bays, imported zfs pool in proxmox and attached it to truenas VM in z820.
i click create ZFS and then it tells me none of my disks are unused so I can't make one. What gives?
Check if disks gets detected and if they do l, are they wyped? Formatted?
I created a RaidZ1 pool of 4x 4TB drives. This means I should be able to use 12TB/16TB.
But when I create a drive for my VM I can only use upto 7TB. It then shows I'm using 11.5TB/12TB. Why is that?
Hmm. If i am not mistaken, during 4x4TB drives in RaidZ1 will give you approx 9 - 10TB of space. Not sure where you got 12/16TB from.
@@MRPtech Apparently it was because of the blocksize. Zraid1 + 4 disks + 4kb blocksize (default) there is a lot of inefficiency, I could only use 7TB. With 16kb I could use 10TB
Woups... So it was so easy this crash my proxmox install...
I connected an older sata disk via an USB-Sata cable and create two partitions (yes, it's a home lab...). I then create a zfs mirror on these two partitions and some datasets.
One of the datasets I mounted to a smb sharing container and it all executed fine.
But then... I decided to see what would happen if I disconnected the usb cable (just to see how robust it was) and then everything stopped working. Rebooting just takes me to the emergency mode. Error message:
Failed to start zfs-imort@test\x2mirror.service - Import ZFS pool test\x2dmirror
With zfs destroy and zpool destry i finaly managed to get back on track again, but I must say that it was kind of surprising that there was so hard dependencies on the disk layer for the proxmox environment to start
The Yottamaster enclosure will cook hard drives.
Why do you think that ?
@@MRPtech Because I have one and found that based on SMART temperature reporting it couldn't keep 3 drives (8TB WD Red) under 55C while under light load at 28C ambient temp. The airflow is not adequate for three drives spaced out in the cages let alone five. There is only ~7mm between each drive. The Inverse Square Law always applies and there isn't enough airflow to overcome it. Look at the ridiculous exhaust grill. And the drives are oriented horizontally which only helps roast the top drive the most. I ignored these facts when selecting the Yottamaster enclosure.
Hmmm. i haven't notices anything bad with cooling when i was using Yottamaster.
Long story short i end up with Synology DS423+ and this is how hot drives are at this moment:
prnt.sc/rCgx_wOWg5Ct
@@MRPtech Ah, that's a nice box. And look at those beautiful twin 90mm exhaust fans on the rear. 😉
PS- Thank you for the great content. Subscribed.
Schmidt Cliff
Jones Karen Thompson Charles Young Nancy
Many Thanks🙏