This mirrors what I have seen in terms of performance difference. Thanks, Tom. When will IX-Systems actually address this? Forums are live with these issues and radio silence so far.
Scale may actually be broken. We've been trying to deploy it for a few months, and are running into (and finding others online as well) issues with quick drive resets or halts that often won't destroy the pool but will absolutely destroy net performance. The more number of drives and the larger the drives, even on proper LSI HBAs, shows the issue even more. A heavy correlation of the issue showing up more or in higher severity is also throwing concurrent workloads at drives, especially random block read/writing, or larger transfers to the entire pool while trying to have it fit in a SMART extended test. Your testing, being limited to 3 drives, being SSD as well, may not show much of this because of cache differences and such, and may only show small symptoms like decreased performance in certain test scenarios. Until TrueNAS Scale gets this addressed, the comparisons are only really valid for letting viewers know that something is actually wrong/broken with Scale.
@@johndroyson7921I'd recommend it. I have replacement both LSI mini-SAS to backplane and breakout cables for an older HBA (9305-16i) coming in, as well at 4 x HC 550 DC hard drives to add to the mix (I'm currently using 8 x 18TB Seagate Exos X18 18TB CMR drives) both onside and outside the Supermicro backplane to see if I can pinpoint the issue, but I'm suspecting is a MPR/MPS driver issue for Linux causing this. Drives large like 18TB, on a LSI HBA that's 9400 series, are certainly having an issue with not only throughout but dropping of some connections (drive will support as "reset"), in TrueNAS Scale or at least ZFS v(2.1).
Thanks for the data! I tried Scale first as my very first NAS project and was dismayed at the SMB transfer numbers. Did a lot of troubleshooting, and throwing parts at the problem, and found that even though iperf3 showed that my network connection wasn't the bottleneck and "get " tests within SMBclient yielded expected numbers for being on NVMe, there was something else causing me to use just half or less of my network connection bandwidth (be it 2.5G or Wi-Fi). I just switched to Core 13 and my experience mimicked your test numbers, so I guess Scale is still a little rough around the SMB edges indeed.
I tried Scale and the numbers on even a 16 SSD zero parity array over SMB were atrocious. Thought "Well that's the risk of new unrefined software" switched to Core and saw disappointingly middling-poor numbers but wayyyyyy better than SCALE. Gave up and went Windows Server 2022. Really hoping that ZFS 3.0 and upcoming Samba updates can replace WinServer
@@im.thatoneguy is your experience over 10GbE connection or lower? I found some TrueNAS Core tunables from 45 Drives' website and managed to get unsynced writes, either to NVMe or spinners, to saturate my 2.5GbE connection. NVMe reads also manage to saturate 2.5Gbe, while reads from spinning rust are understandably slower.
LAWRENCE SYSTEMS... If possible, I would really appreciate a video on how to setup openvpn and/or wireguard (a beginner's guide) on TrueNAS Scale. I just setup my first server w/TrueNAS scale and as a newbie I'm struggling to find a video with simple instructions for the above. Thanks.
I dont care that much about performance but I have to say my swap from Truenas Core to Truenas Scale changed allot. I now can easy setup VMs, configure the Apps as I have no clue about docker or kubernetes. I just now finished my migration project where I transfered everything to my IBM dx360 M3 with 11 bays filled. Yes its an old server but I dont have the finance to buy a newer one. Would like to but cant. Now everything is on there and running fine. Even now I have read cache for the first time. Now I can sell the other server and hope I get enough out of it to cover some of the costs. But in short, I am very happy they made Truenas Scale.
It would be interesting if you re-ran this on the latest BETA of SCALE Bluefin 22.12 that just came out and enabled the Blake3 hashing on the latest OpenZFS vs current SHA-256. Blake3 is a little bit faster on current hardware, and a lot faster on newer systems with AVX-512
Agree, reading around on TrueNAS community, many are touting that SCALE is almost on par for performance that CORE has...and if a new user is coming on board, instead of redoing from CORE to SCALE in a few months or a year, if they decide to say, kill off CORE?
I will tell you my experience with Truenas Scale: Went from happily using Truenas Core 12 (latest release) to Scale with high hopes, silly me. Installed it on mirrored sata ssds as the boot pool and recreated my storage pool (3 x 4x10TB raidz1) , so far so good. The gui is still a bit barren compared to Truenas Core, but it is still pretty new in its development, so i wont complain just now, right ? Transferring my data from a backup nas to the new zfs pool went surprisingly fast (5-600 MB/sec over 10G lan), no complaints there either. But then i noticed, the idle cpu usage was always around 5-10% and the cpu temp went up to around 50 degree Celsius, with no apps installed, no VMs running, just a single Samba share, no network traffic going on. With Truenas Core 12 (same hardware, same config, no jails) the cpu idles at 0% and 32 degrees Celsius ! The culprit seems to be the kubernetes implementation which is always doing something, whatever that might be and the seemingly complete lack of energy management. Then i noticed the mirrored zfs boot pool just decided to degrade, so i checked both boot ssds in a different pc. No smart or other errors, they were fine. But then i realized there was no way to replace a "failed" device in the boot pool via the web gui and i was in no mood for command line torture. At that point i was starting to get very annoyed and Truenas Core 13 was just released. Guess what i did next .... Truenas Core 13 runs perfectly fine on the exact same hardware with the data pool created with Scale. Maybe i'll revisit out Truenas Scale again in a year or two, but until then there is still a looong road ahead.
This is worth some digging I think. I'm very curious what the issue is, can it be reproduced on a vanilla linux distro? Is the issue nfs related or zfs? Testing io locally on the box would be a good idea to make sure your getting expected hardware speeds without network protocols involved.
My biggest issue with scale so far has been the poor network performance on same hardware. When testing torrent transfer on Core I would get some where in 85-95 MB down rates from a known source, now can't break over 30 MB down. Honestly contemplating swapping back to Core
I see PiHole as a plugin that you can install, yet on my Core 13.01 I do not see it in my list of community or iXSystems tab? How can I find this plugin or others that work?
When I see the performance drop that dramatically for 4K blocks that would indicate a difference in block sizes used for I/O. I'm guessing there's a way to adjust buffering to alleviate the differences.
I'd love to see how Openmediavault 6 performs as a comparison using the new OpenZFS driver. Makes me wonder whether this is an OpenZFS on Linux thing or a TrueNAS scale thing.
TrueNAS SCALE is visually just as fast as a tuned Ubuntu system and can run 10Gb if needed, it's on a 40Gb interface but all the other hardware is speed limited so I can't tell how far over 10Gb it can do. I did notice that without a log drive (maybe even with one) having sync set to default will hang everything connected to it and the array will struggle to do 10MB/s. Turning sync off at the pool makes everything come alive and all testing is showing it works quite well. Unfortunately, I can't run BSD on this hardware because the virtual network cards don't have BSD drivers written for them or I would have started on that path first.
Have you noticed any bugs when doing an in place upgrade? Would that be hampering your speeds? I know when I did an in place upgrade a few months ago there were buttons missing. I couldn't upgrade my pools for example. The button wasn't there.
What about the use-case of 4K video editing (SMB)? I’m just a solo editor, but would be good to get your perspective on a 5 person team as well. Would love to dive in to True NAS Scale at some point. Thanks!
Truenas core 12 is probably still the way to go for now. Getting 7200 rpm platters with a ZIL SSD or NVME will do you well. Though you can do without the zil if you are ok with disabling sync writes. I can not see an L2ARC being of much use though. Unless this NAS also gets used for other general purpose things. SMB works pretty good, and its performance in large block transfers will be efficient. Especially if you are on 10gb network. But what a friend of mine just did, since he doesn't like SMB and much prefered windows shares, was build his truenas box on bare metal, and used the virtual machine within truenas for a windows VM which had a virtual drive that took up the most of the raid. And used windows shares with a direct connected network port. And he saw really great performance with that. But he also had a machine with 64 gigs of ram for his NAS. He figured it gave him the best robustness of ZFS and the simplicity of windows sharing. Though I feel it goes through too many layers, and I think SMB right to the raid works just fine. And you can probably get away with a little less ram and CPU ect.
I've noticed that on my system all apps were performing suboptimal. Then I realized that by default it seems they are constrained to very limited cpu and ram usage. Was trying to run handbrake and it just wouldn't use any of my CPU. Changed the CPU and RAM assigned and worked great.
What’s the goto for backups these days? You mentioned backups before stating any migration but what/where to backup to these days, if we are talking large scale TB’s of data.
I was really excited for SCALE mostly for performance reasons. Looks like the performance isn't there though. I'm going to have to wait. I might play with SCALE at home where performance doesn't matter but for work rollouts, I'm definitely sticking with CORE.
Hey there, it's quite an insight but when such dramatic drop is observed maybe try peer a little deeper like do some isolated IO tests on the machine, with different protocol like SMB and same NFS but instead of SSDs a small Ramdrive? There are quite a few places the bottleneck could be coming from
I would have appreciated a little investigation what could be wrong with scale that it performs so bad in some write tests. I don’t think thats normal.
Side note. On TrueNAS Core 13 if you create a VM you are unable to connect to it using the bhyve vnc viewer without upgrading to a more recent beta build.
If you have the time it would be wonderful to see the performance when you build a cluster with them. I had so big hopes but it’s not good by any metric compared with Ubuntu or Rocky Linux. I didn’t do phoronics tests, I might do in the future and then I’ll share them
With the Ryzen G series CPU, you can free up some Ram by going into the Bios -> Video Memory option and reducing it to 512 (since you are not running graphics, on the console, the memory is not needed). This will free up Ram that the OS can use.
Scale looks better for people that want a NAS that can run a couple of VM's & some Docker images and aren't so concerned about raw network throughput, as the VM & Docker support I think is better on Scale than Core. Currently use UNRAID and tossing up whether to replace that aging hardware with a TrueNAS Scale box or keep using UNRAID but with the a ZFS array instead of the default XFS with parity drives.
I tried Core 13 in a VM but it would constantly kernel panic on shutdown so I was going to deploy SCALE instead, guess I won't use either, maybe I'll try Core 12.
I'm learning since I got 10gb that I could spend my whole life performance tuning truenas. All the little configurations that didn't really matter on my old network have big implications when ur trying to fill a bigger pipe. Random writes I guess would become an issue with a multi-year system? Im not sure that that's a metric thats really relevant to my use case. Thanks for the videos!
@@Felix-ve9hs makes sense. All my users have local storage and none are working directly off the nas storage. Would probably get really annoying for the users too if their os were waiting on small reads and writes all the time
Forna drop off like that seems like a bug or misconfiguration happening within Scale. Will be cool to see it mature.
Really would be interested in iSCSI & SMB benchmarks over the network
Great info as always, Tom!! Btw, I saw your interview with David Bombal and really enjoyed that as well. :)
Glad you enjoyed it!
This mirrors what I have seen in terms of performance difference. Thanks, Tom. When will IX-Systems actually address this? Forums are live with these issues and radio silence so far.
I know it's on their to do list
Thanks for sharing the test results!
I've been waiting for this!
Party time! Excellent! Thumbs up!
Scale may actually be broken. We've been trying to deploy it for a few months, and are running into (and finding others online as well) issues with quick drive resets or halts that often won't destroy the pool but will absolutely destroy net performance.
The more number of drives and the larger the drives, even on proper LSI HBAs, shows the issue even more. A heavy correlation of the issue showing up more or in higher severity is also throwing concurrent workloads at drives, especially random block read/writing, or larger transfers to the entire pool while trying to have it fit in a SMART extended test.
Your testing, being limited to 3 drives, being SSD as well, may not show much of this because of cache differences and such, and may only show small symptoms like decreased performance in certain test scenarios.
Until TrueNAS Scale gets this addressed, the comparisons are only really valid for letting viewers know that something is actually wrong/broken with Scale.
@@johndroyson7921I'd recommend it.
I have replacement both LSI mini-SAS to backplane and breakout cables for an older HBA (9305-16i) coming in, as well at 4 x HC 550 DC hard drives to add to the mix (I'm currently using 8 x 18TB Seagate Exos X18 18TB CMR drives) both onside and outside the Supermicro backplane to see if I can pinpoint the issue, but I'm suspecting is a MPR/MPS driver issue for Linux causing this.
Drives large like 18TB, on a LSI HBA that's 9400 series, are certainly having an issue with not only throughout but dropping of some connections (drive will support as "reset"), in TrueNAS Scale or at least ZFS v(2.1).
Thanks for the data!
I tried Scale first as my very first NAS project and was dismayed at the SMB transfer numbers. Did a lot of troubleshooting, and throwing parts at the problem, and found that even though iperf3 showed that my network connection wasn't the bottleneck and "get " tests within SMBclient yielded expected numbers for being on NVMe, there was something else causing me to use just half or less of my network connection bandwidth (be it 2.5G or Wi-Fi).
I just switched to Core 13 and my experience mimicked your test numbers, so I guess Scale is still a little rough around the SMB edges indeed.
I tried Scale and the numbers on even a 16 SSD zero parity array over SMB were atrocious. Thought "Well that's the risk of new unrefined software" switched to Core and saw disappointingly middling-poor numbers but wayyyyyy better than SCALE. Gave up and went Windows Server 2022.
Really hoping that ZFS 3.0 and upcoming Samba updates can replace WinServer
@@im.thatoneguy is your experience over 10GbE connection or lower? I found some TrueNAS Core tunables from 45 Drives' website and managed to get unsynced writes, either to NVMe or spinners, to saturate my 2.5GbE connection. NVMe reads also manage to saturate 2.5Gbe, while reads from spinning rust are understandably slower.
LAWRENCE SYSTEMS... If possible, I would really appreciate a video on how to setup openvpn and/or wireguard (a beginner's guide) on TrueNAS Scale. I just setup my first server w/TrueNAS scale and as a newbie I'm struggling to find a video with simple instructions for the above. Thanks.
whats the movies there are for truenas or freenas and read documentation. These things have not changed much.
I dont care that much about performance but I have to say my swap from Truenas Core to Truenas Scale changed allot. I now can easy setup VMs, configure the Apps as I have no clue about docker or kubernetes. I just now finished my migration project where I transfered everything to my IBM dx360 M3 with 11 bays filled. Yes its an old server but I dont have the finance to buy a newer one. Would like to but cant. Now everything is on there and running fine. Even now I have read cache for the first time. Now I can sell the other server and hope I get enough out of it to cover some of the costs. But in short, I am very happy they made Truenas Scale.
Looks like Core 12 is still the most uniformly performant system still. Probably wait out until the U1 of 13.
It would be interesting if you re-ran this on the latest BETA of SCALE Bluefin 22.12 that just came out and enabled the Blake3 hashing on the latest OpenZFS vs current SHA-256. Blake3 is a little bit faster on current hardware, and a lot faster on newer systems with AVX-512
Agree, reading around on TrueNAS community, many are touting that SCALE is almost on par for performance that CORE has...and if a new user is coming on board, instead of redoing from CORE to SCALE in a few months or a year, if they decide to say, kill off CORE?
I will tell you my experience with Truenas Scale:
Went from happily using Truenas Core 12 (latest release) to Scale with high hopes, silly me.
Installed it on mirrored sata ssds as the boot pool and recreated my storage pool (3 x 4x10TB raidz1) , so far so good.
The gui is still a bit barren compared to Truenas Core, but it is still pretty new in its development, so i wont complain just now, right ?
Transferring my data from a backup nas to the new zfs pool went surprisingly fast (5-600 MB/sec over 10G lan), no complaints there either.
But then i noticed, the idle cpu usage was always around 5-10% and the cpu temp went up to around 50 degree Celsius, with no apps installed, no VMs running, just a single Samba share, no network traffic going on. With Truenas Core 12 (same hardware, same config, no jails) the cpu idles at 0% and 32 degrees Celsius !
The culprit seems to be the kubernetes implementation which is always doing something, whatever that might be and the seemingly complete lack of energy management.
Then i noticed the mirrored zfs boot pool just decided to degrade, so i checked both boot ssds in a different pc. No smart or other errors, they were fine.
But then i realized there was no way to replace a "failed" device in the boot pool via the web gui and i was in no mood for command line torture.
At that point i was starting to get very annoyed and Truenas Core 13 was just released. Guess what i did next ....
Truenas Core 13 runs perfectly fine on the exact same hardware with the data pool created with Scale.
Maybe i'll revisit out Truenas Scale again in a year or two, but until then there is still a looong road ahead.
mmm I'm keeping my truenas core 12 ... thank you for your video.
This is worth some digging I think. I'm very curious what the issue is, can it be reproduced on a vanilla linux distro? Is the issue nfs related or zfs? Testing io locally on the box would be a good idea to make sure your getting expected hardware speeds without network protocols involved.
My biggest issue with scale so far has been the poor network performance on same hardware. When testing torrent transfer on Core I would get some where in 85-95 MB down rates from a known source, now can't break over 30 MB down. Honestly contemplating swapping back to Core
I see PiHole as a plugin that you can install, yet on my Core 13.01 I do not see it in my list of community or iXSystems tab? How can I find this plugin or others that work?
When I see the performance drop that dramatically for 4K blocks that would indicate a difference in block sizes used for I/O. I'm guessing there's a way to adjust buffering to alleviate the differences.
Thanks for the number harvesting. It helps a lot.
What do you think of Unraid?
I'd love to see how Openmediavault 6 performs as a comparison using the new OpenZFS driver. Makes me wonder whether this is an OpenZFS on Linux thing or a TrueNAS scale thing.
Openzfs has been based on Linux for a long time now. I find it far more likely that issues are being caused by scale if anything.
@@JustSomeGuy009 maybe nfs, this should be relatively easy to narrow down.
TrueNAS SCALE is visually just as fast as a tuned Ubuntu system and can run 10Gb if needed, it's on a 40Gb interface but all the other hardware is speed limited so I can't tell how far over 10Gb it can do. I did notice that without a log drive (maybe even with one) having sync set to default will hang everything connected to it and the array will struggle to do 10MB/s. Turning sync off at the pool makes everything come alive and all testing is showing it works quite well. Unfortunately, I can't run BSD on this hardware because the virtual network cards don't have BSD drivers written for them or I would have started on that path first.
Have you noticed any bugs when doing an in place upgrade? Would that be hampering your speeds? I know when I did an in place upgrade a few months ago there were buttons missing. I couldn't upgrade my pools for example. The button wasn't there.
I would be curious to see VMware ESXi benchmarks with Phoronix with the 4k blocks... it might surprised a lot of us.
So clearly... Something is wrong with the 4k block testing on Scale, obviously. 🙄
Thank you.
What about the use-case of 4K video editing (SMB)? I’m just a solo editor, but would be good to get your perspective on a 5 person team as well.
Would love to dive in to True NAS Scale at some point.
Thanks!
Using it as bulk storage? Or as the actual scratch drive that you work from?
@@wishusknight3009 Both/either.
Truenas core 12 is probably still the way to go for now. Getting 7200 rpm platters with a ZIL SSD or NVME will do you well. Though you can do without the zil if you are ok with disabling sync writes. I can not see an L2ARC being of much use though. Unless this NAS also gets used for other general purpose things.
SMB works pretty good, and its performance in large block transfers will be efficient. Especially if you are on 10gb network.
But what a friend of mine just did, since he doesn't like SMB and much prefered windows shares, was build his truenas box on bare metal, and used the virtual machine within truenas for a windows VM which had a virtual drive that took up the most of the raid. And used windows shares with a direct connected network port. And he saw really great performance with that. But he also had a machine with 64 gigs of ram for his NAS.
He figured it gave him the best robustness of ZFS and the simplicity of windows sharing. Though I feel it goes through too many layers, and I think SMB right to the raid works just fine. And you can probably get away with a little less ram and CPU ect.
Can I ask you a question concerning truenas and NFS ? Thx
My biggest issue with scale is the slow docker deployments. Works great when they are finally up, but the initial setup requires a lot of patience.
I've noticed that on my system all apps were performing suboptimal. Then I realized that by default it seems they are constrained to very limited cpu and ram usage. Was trying to run handbrake and it just wouldn't use any of my CPU. Changed the CPU and RAM assigned and worked great.
@@JustSomeGuy009 I'm a newbie, could You tell a little more. Where can I find it? What should I change?
@@JustSomeGuy009 I second that request. Could you point us in the direction of those settings, please?
What’s the goto for backups these days? You mentioned backups before stating any migration but what/where to backup to these days, if we are talking large scale TB’s of data.
Backblaze
I was really excited for SCALE mostly for performance reasons. Looks like the performance isn't there though. I'm going to have to wait. I might play with SCALE at home where performance doesn't matter but for work rollouts, I'm definitely sticking with CORE.
Hey there, it's quite an insight but when such dramatic drop is observed maybe try peer a little deeper like do some isolated IO tests on the machine, with different protocol like SMB and same NFS but instead of SSDs a small Ramdrive? There are quite a few places the bottleneck could be coming from
I would have appreciated a little investigation what could be wrong with scale that it performs so bad in some write tests. I don’t think thats normal.
I am not a coder, I just test and report the numbers.
Side note. On TrueNAS Core 13 if you create a VM you are unable to connect to it using the bhyve vnc viewer without upgrading to a more recent beta build.
I fixed it by changing the VNC port to 5900, and the resolution to 800x600
@@AwesomeDedrick Thanks
Do you know if/when are they going to address this?
It's on their to do list
If you have the time it would be wonderful to see the performance when you build a cluster with them. I had so big hopes but it’s not good by any metric compared with Ubuntu or Rocky Linux. I didn’t do phoronics tests, I might do in the future and then I’ll share them
With the Ryzen G series CPU, you can free up some Ram by going into the Bios -> Video Memory option and reducing it to 512 (since you are not running graphics, on the console, the memory is not needed).
This will free up Ram that the OS can use.
Scale looks better for people that want a NAS that can run a couple of VM's & some Docker images and aren't so concerned about raw network throughput, as the VM & Docker support I think is better on Scale than Core. Currently use UNRAID and tossing up whether to replace that aging hardware with a TrueNAS Scale box or keep using UNRAID but with the a ZFS array instead of the default XFS with parity drives.
Plugins based on packages are better. They just need some basic support. And jails are far simpler and easier to maintain.
LLPoolJ 🤣
Can Scale really be classed as "ready for most people" with this kind of performance?
Yes
If Scale was the only product then it could be considered even higher.
I tried Core 13 in a VM but it would constantly kernel panic on shutdown so I was going to deploy SCALE instead, guess I won't use either, maybe I'll try Core 12.
I tried Core 12 in a VM and it won't shut down either.
Keyword: you tried it in a VM. Try it on bare metal?
@@UntouchedWagons What hypervisor are you using? Im running truenas core 12u8 in esxi 6.7 and I haven't had any issues.
I'm learning since I got 10gb that I could spend my whole life performance tuning truenas. All the little configurations that didn't really matter on my old network have big implications when ur trying to fill a bigger pipe.
Random writes I guess would become an issue with a multi-year system? Im not sure that that's a metric thats really relevant to my use case.
Thanks for the videos!
If you work with multimedia files, no
it you work with virtual machines, yes
@@Felix-ve9hs makes sense. All my users have local storage and none are working directly off the nas storage. Would probably get really annoying for the users too if their os were waiting on small reads and writes all the time
I am seeing the same degraded performance @4K drive writes inside a VM using Crystal Disk Mark 7.
Does anybody know if you can go from scale to core 13?
No
@@LAWRENCESYSTEMS Thank you i already thought so, but i can do a clean install and import my pools right?
Only 8:23 of this video is proper content
Add Proxmox to the comparison chart.
Proxmox is a hypervisor and not really a NAS.
@@LAWRENCESYSTEMS Neither is TrueNAS Scale, but ya still checked it.. 😉
i thought this is linus tech tips logo lol
comment for algo
appreciated
I'm early :)
ooof
First