How to speed up formatting 520 formatted enterprise hard drives down to 512 format standard
Вставка
- Опубліковано 20 лис 2020
- So how to speed up formatting 520 formatted enterprise hard drives on a grand scale. As in doing more then one that is. I have posted videos that talk to changing the format of a 520 format NetApp enterprise hard disk into 512 format so normal PC can read and write to it. But if you boot off of a USB drive to CLI mode, you will able to format one disk at a time.
Yes this work but would like to do all of them with in a 2 hour window vs. 23 hours? You can! Here is how you do it...
First you need your PC and a SAS controller that can handle at least 24 drives. Second you need a disk array and cabling to allow you to do more than one disk at a time. Lastly you install Ubuntu on the PC hard drive and boot into the UGI interface desktop.
Now all you have to is setup your service so you can format a disk drive. Here are the steps below just in case. (You will need internet connectivity
How to build a system to test these SAS drives go to this link: • OK, so you asked so he...
open a terminal window:
step 1 ►sudo apt-get update
step 2 ►sudo apt-get upgrade
step 3 ►sudo apt-get install software-properties-common
step 4 ►sudo apt-get install sg3-utils
Now the tool is install and you can now format 520 disks by using this set of commands
Next you do a ►sudo sg_scan -i This helps you to ID all the disks you want to format. (Look for /dev/sg4 or /dev/sg5 for these are disks) Work with just one to learn how to ID them
and lastly once you ID the hard disk you want to format to 512 from 520. You use this command (Remember after sg value is the drive number)
►sudo sg_format --format --size=512 --fmtpinfo=0 /dev/sg5
So now you have a disk formatting to 512K format so copy the format command in the terminal window and than open a new terminal window and past
►sudo sg_format --format --size=512 --fmtpinfo=0 /dev/sg6
So now you have two disks formatting to 512K format so copy the format command in the terminal window and than open a new terminal window and past
►sudo sg_format --format --size=512 --fmtpinfo=0 /dev/sg6
So now you have three disks formatting to 512K format so copy the format command in the terminal window and than open a new terminal window and past and do this over and over until all of your 520 format disks are being reformatted.
I do about 15 to 24 at a time and it is done very fast for they are all being formatted at the same time...
So if you are looking at this then you already know how to do it the old way and want to be better / faster to get stuff done. Here go! Keep enjoying and the learning!
related sites: none
Enjoy! - Наука та технологія
Life saver dude would've taken me years to reformat the 70 drives I found haha
Thanks - I was sitting here watching mine count the percentage and decided to google to see if you can run mulitple terminal windows at once, and you answered my question! Thanks again, saved me a day!
Glad to help out!
Thanks for putting the info in the description! I did this to some NetApp disks years ago using CentOS, but I never thought of having multiple terminal windows open and working at the same time.
It me two days of searching to even find out why I couldn't initialize my 6 used seagate sas 2tb drives i got for $16 each on ebay. then I find out about the 520b sector thing in a STH forum but it said it has to be done with centos. So spend hours just figuring out how to freaking install centos and everything ending up being read-only. So I find a level 1 tech forum that uses ubuntu to format to 512b. Alright! Now been waiting about 6 hours for one drive to finish wondering if I can do more than one drive at a time. And BOOM! I find your video. Thank you so much man Im going to be to get some of that time back. What a nightmare this NAS project has been.
Glad to help out! Make sure you take a break away from the effort from time to time so it ends up being more fun instead of a pain in the butt. lol Have fun dude learning!
@@leadiususa7394 oh I do have fun. I've fed my mind as much as I basically could on consumer hardware and now im expanding to the enterprise. Which is like exploring another planet.
@@maynardcrow6447 very true...
Great video! Very detailed. Obviously there are several ways to accomplish this task, but using a server with SAS HBA and Linux is pretty straightforward.
Glad it was helpful!
u can use in console : screen -S disk1 ; screen -S disk2 ; screen -S disk3 . . . . . in the screen just copy past the command for sg_format with ctrl + a + d you leave the screen to show screen agen screen -r disk1 ; screen -r disk2 . . . . if disk is finish you can kill the screen mit ctrl +a + K and press Y
I wrote a script to do this 2 nights ago, you can find all the disks attached to the HBA at /dev/disks/by-path, then I basically just looped through them spawning a background process for sg_format and a control loop that checked I wasn't doing too many at once using $(ps -ef | grep sg_format | wc -l) and a sleep.
Took me an hour to figure out and can run as many as you like in parallel.
Can you share the script please?
@@ksaksa111111 Easy!
Create a text file with all the hard drives you want to format, like this (call this file hd_list.txt):
/dev/sg10: scsi1 channel=0 id=09 lun=0
/dev/sg11: scsi1 channel=0 id=10 lun=0
Then, create a shell script, i.e. call it hd_format.sh
#!/bin/bash
for i in $(cat list.txt | cut -d ':' -f1); do
sg_format --format --size=520 --fmtpinfo=0 $i --early
done
####################
Save your script, then make your script executable: chmod 777 hd_format.sh
Then run your script:
./hd_format.sh
Very important: Make sure you don't accidentally add any HD you did not mean to format to the list, ie. the boot drive.
@@SuperVertrix For anyone running this script, be aware that it shouldn't be "--size=520"; it should be "--size=512". The entire point is to change out 520 byte sectors with 512.
Thanks! Nice tutorial, I got same problem with recently bought sas drives in 520 byte format. One more greeting from Ukraine. Gonna fix this tonight with my SAS drives.
Very cool, make sure you check out my other video on how to do many drives at once. Speeds things up some...
saved my day! Thanks
Glad it helped!
You don't need a GUI to open multiple terminal windows. You can use tmux / screen to open as many consoles as your RAM can fit (probably hundreds, if not thousands).
Agreed and that is something a seasoned IT vet knows. But this video was to helps those that are trying to learn this stuff in a basic format. as are many of my videos. I also talk to many others ways that don't use any of this u=including your way,,, There are ten ways to do anything in IT environments. My goal was to help beginners to learn so they become people like you. Thanks for your input and have a great week!
I hope at some point you found screen, or find it now. You do not need gui for this but it is a great tutorial.
Yes and yes, here to help and this video is getting dated but people still use it to get started... /:>
Why not just use tmux or screen? You don't need a GUI. You can start multiple terminal sessions and run commands in parallel.
Please bring that up, there are many ways to format SAS disks in bulk, I personally don't like using any OS based formatted at all. Including Linux, I prefer Controller formatting. I can format all disks in a five minute kick off sessions. As in hundreds of disks. But that is to advance for most. SO I show a GUI version because yes there are those that just like doing things thru GUI for some reason. I showed them one way.. But please put more out there to share the know how! Enjoy dude!
Men i was gonna do this but tmux saves me a lot of pain on accidental terminal closing.
Thank you! you helped me. Greetings from Ukraine
No problen and please enjoy! Also be safe over there! God Bless
Hi, I'm trying to reformat the Hitachi HGST HUC101212CSS600, after preforming the sg_format --format --size=512 /dev/sgX
the drive changes the "Logical block size:512" but at the same time the "User Capacity:512 bytes [512 B]" changes.
If I revert to Logical block size:520" the disk capacity changes to normal disk capacity "User Capacity:1,200,243, 694,848 bytes [1.20 TB]"
Any ideas what is wrong with the HDD?
possible FAT issue going on or your have bad sectors and they surface in the restructuring of the File allocation table to 512. Might be a bad drive or close to failing.
“This format is only defined for disks (using SBC-2 or RBC) and MO media.
sg_format failed: malformed SCSI command
Sorry. That is the most common I ran into when buying bulk enterprise disks but you are correct. But no one is asking so I didn't go after that formatting process to research.
@@leadiususa7394 ok, that error was a error on my part. So when I list all the drives, attempting it on a drive called “promise”. That seems to be a place holder for the controller itself. Anyhow.. everything works.
Well I did plan on making one of my servers into a Debian server so this will be great (I have an extra Proliant I can use) but if I were to make my array in a 4U case using an ATX PC motherboard, would I need to find a drive enclosure w/ a backplane or one of those JBOD cages (to connect the drives to the HBA's connectors?
Or do they possibly sell connectors that connect the HBA directly to the SAS/SATA drives?
That's one part I am still a little unclear about (how the HBA/controller's cables connect to the drives if you do not have a backplane/server chassis to begin w/.
most use a SAS external cable to a SATA / SAS controller HBA device. That should work for your needs.
Can you tell me what cable you used for this exact configuration. I need to buy one that is just like this one.
Go with this cable set and make sure you also have a LSI SAS controller for your motherboard most likely will not support SAS drives. Most PC motherboards can support SAS drive. If you have a server level motherboard. Here is the link to the cable set. www.ebay.com/itm/284618234227?_trkparms=amclksrc%3DITM%26aid%3D777008%26algo%3DPERSONAL.TOPIC%26ao%3D1%26asc%3D243537%26meid%3Db21e8b3625ae4f42ab81e9df59a4364a%26pid%3D101524%26rk%3D1%26rkt%3D1%26itm%3D284618234227%26pmt%3D1%26noa%3D1%26pg%3D2380057%26algv%3DRecentlyViewedItemsV2%26brand%3DUnbranded&_trksid=p2380057.c101524.m146925&_trkparms=pageci%3Af58872f5-8c61-11ed-9a66-fa8e8af6dacb%7Cparentrq%3A7e275f6b1850a9b136a1ada5fffed772%7Ciid%3A1
3/4 finally finished to format. 2 are detected after reboot. 1 it's still not detected after reboot and 1 is stuck at 34% :( I also tried changing the firmware on all of them but that made no change. Any idea how to diagnose the 2 that are not detected? Because when I try to use fdisk -l is not getting listed. I am getting this constant error in log "device /dev/sdc has size zero"
May be a bad drive... That is why I buy them in bulk... /:> Sorry dude...
I just purchase some 15k 450gb sas drive for my hba and one of them is stuck at .99% when reformatting to 512 bytes. I’m using sg utils on windows and running the command sg_format -format -size=512 pd(drive number)
I’m new to this whole process so if anyone knows a solution let me know!
Try a basic DIAG check on the drives first. Go here to understand how to do this... www.fosslinux.com/1297/check-hard-disk-for-bad-sectors-and-errors-by-command-line-in-ubuntu-and-linux-mint.htm
Can this process be done with a raid controller or does it have to be vis HBA or direct connect sata?
Yes it can but first you need to setup the RAID on the controller. Then go for it...
In the RAID controller, make all drives a RAID0 (single) VD (virtual disk) and they will show up in Linux as /dev/sda, b, c, d, etc. Then you can follow this video on them. ALSO: depending on the RAID controller and firmware, you can just "boot into the RAID controller," meaning when you cold power-on the computer, the controller will show on the screen a dialog to get into it's firmware (ctrl-L a lot of them use) and you can go into their firmware and change the sector sizes (which is what this video does) of all drives connected at once. It's quicker then this method but you have to understand the controller firmware, AND be sure you don't have any drives on said controller with data you want to keep. LSI-based SAS2/3 RAID controllers do this operation well.
Hello, Great video ... I just received NETAPP 600GB SAS drives ... On my Dell R420 - I can see them in controller utility but cant use them... I did installed ubuntu on 1 virtual drive on 1st 2 sas dell 300gb sas ... and with sg utility I can only see that virtual drive and DVD rom ...Cant see any of NETAPP drives... What Im doing wrong ?
Start first by making sure your HBA or RAID Controller sees the single SAS 600 GB drive on boot up just to test the SAS setup. Then go (Only working with just one SAS drive) to the SU CLI command screen and see if you can see SAS drive first, if so then and do the reformat command on the single drive that I show in the video. If you do it right it will take some time to complete. Once done then see if the drive show as a new resource for your Linux OS to use or Windows. Just for fun do a normal reformat to make sure the 520 to 512 format is working. Keep in mind some drives will fail during this process. If so repeat the same steps with another drive. Good luck and have fun with this effort. Let me know if you need more help... /:> P.S. I did another video showing how you can do many 520 to 512 reformats a one time, greatly reducing the time to reformat disk drives. It is on my channel..
@@leadiususa7394 -- thanks for reply. Looks like that my H710 is not in IT mode. I can see drives in controler bios but not in Ubuntu. I think controler need to be in IT to see individual drivers in OS. Thinking of buying H310 that is flashed in IT and use it to pass thru those drives. Still researching if that is best option to go.
Received H310 in IT mode .... Works like charm - Formatting in progress
@@ivicanikolic6355 Great
I found a good program to check my drives using smart. It's cheap and on sale for $13.50 single computer registration. Hard Disk Sentinel. It says the 4 drives Im looking at have about 2,600,000 something's in power on time. I thought it was hours but it would mean the drives have ran for 5000 years. So I did a recalculation based on minutes and that turned to be about 5 years of power on time. Is my calculation correct?
Very cool. I will check it out for i have 6 drives that don',don't, want to initialize correctly
I believe so
@@leadiususa7394 It's super cheap for $13 right now. The link is on their main page. It has a good layout and identifies many attributes
@@anthonystrohmayer9191 Yea, it is a nice base line tool. I needed something more (deeper disk base recovery tools) so I went with
EaseUS Partition Master. That way I can even clear bad sectors to revive a disk with sector issues. Like two disk I have currently.
Hi i received some netapp drives instead of the HP i ordered and i was going a little nuts as i had a course on monday. Ive got them in a dell r720, is it possible to boot up ubuntu gui via a USB and do the above?
Nope. needs to be a SAS connection. I am sorry...
@@leadiususa7394 Ah sorry i meant i have the drives in my server via an HBA....can I use an Ubuntu live usb to do the formatting so i dont have to install the OS to a drive?
@@umirza I don't believe so for the USB drive format is to support USB mode. IT would have to be on a real SAS or SAT bus to the motherboard to be formatted correctly, sorry.
@@leadiususa7394 The drives are on SAS bus, what im saying is to run Ubuntu from a USB key so i dont need to install it....
@@umirza Oh yes you can boot off the USB Ubuntu memory stick and reformat the drives... yes.
SIR, we have to use this iso on usb ?
No, you could do this with a CD image as well...
You should have millions of views and lots of followers
Thanks for saying that...
@@leadiususa7394 Here is the story about the time to format a 520b hd to 512b. The bigger the drive the longer it takes. I'm formatting a 2tb sas drive and 3 900gb drives. The 900's are 4% currently and the 2tb one is 1.5% which is why it takes 9 hours to format a 2tb and 3 hours for a 900gb. And probably why your 600gb drives format in 1.5 hours.
Also is there a way to bench test these drives to see if they are in spec? I get the feeling there is a setting wrong in this t7610's bios. I installed a working 500gb sata drive already had windows 10 and it did the same as my sas drives. It was a snail to install a game and stopped halfway through.
@@anthonystrohmayer9191 Yea, you download this tool and you can get wither report matrix data on your drives or benchmark testing of your drive. Make sure you read about each. I use them both for my work. crystalmark.info/en/software/crystaldiskmark/
@@anthonystrohmayer9191 You may also like this disk tool. show your storage in color and workloads. I use it when I have a lot of storage to manage... Enjoywindirstat.sourceforge.net/
This is how I formatted 4 disks at a time on my t7610 workstation. And the interesting thing is it has an on-board sas controller that wasn't hba or it mode. It took all of the drives 9 hours each to format but I did them 4 at a time. How does your way speed up the formatting to 1.5 hours?
Because I started the formatting for all of the drives at once using this trick so it only took about 1.5 hours for all the disks. Not just one.
@@leadiususa7394 I don't understand how you got the drives to format in 1.5 hours when I apparently did the same as you and it took 9 hours
@@anthonystrohmayer9191 Well I just two rounds of nine disks of formatting and I did see that I had one drive (for it was lagging) that did act up but I remove it and all was good. When I see that I know that drive is going to die soon so I remove it (replace it) but yea my first group of 600 GB drive took about 1.2 hours and the second round tool 1.5 hours.. You know you can compare them side by side in the GUI display to see if a drive is taking to long. That is just part of the DIAG mode of buying in bulk enterprise disks... I just bought these and I want to test them within there 14 day warranty period so if I have any bad drives I can get replacement under warranty.. Are you seeing this issue maybe?
@@leadiususa7394 One of the 8 drives did lag or have difficulty formatting because it said it formatted the same percentage about 10 times. It finished formatting. All 9 drives took 9 hours each but I formatted them in lots of 4. Perhaps the limitation is the lsi2308i mustang raid controller. Or it could be an issue with sas drives and throughput. I read someone had a problem with writing and their drives didn't achieve 6 gbps. Who knows. I'm not disheartened it took 9 hours a set and the raid controller wasn't even IT mode. Just happy they formatted right out of the box.
@@leadiususa7394 This is how I formatted the drives. Maybe it's not how you formatted yours? mikeyurick.com/reformat-emc-hard-drives-to-use-in-other-systems-520-to-512-block-size-conversion-solved/
What is the advantage of reformatting a disk from 520 to 512 sectors?
Depends on your system but if takes a few hours. That is why I posted how to do many at once in my video...
Well, if you format back down to 512 KB formatting, any PC will be able to use the hard drive as a normal hard drive to boot from. That is the most key value of doing it. You are recycle very high-end 10k or 15k RPM hard drive for normal use. These hard drive out preform all basic 5.2K or 7.2K RPM hard drive found in PC today.
Uh, for god's sake. I spent 20 hours formatting 8 2tb sas drives and now you say it can be sped up. Plus you could make a video on formatting them, setting them as mbr or gpt depending on size, setting the bios to boot a gpt raid array, and installing an os properly
Hope it helps! Enjoy
No need to do this. I found an easyer way: just use the dd command in linux to overwrite the first few megabytes of the disk device and you can repartition and use like a new drive.
dd if=/dev/zero of=/dev/(device) bs=1M count=100
Takes 1 second
I will test this out. Thanks
@@counttoast2647 OK cool
HMM, I don't think this works.
# dd if=/dev/zero of=/dev/sdr bs=1M count=100
dd: error writing ‘/dev/sdj’: No space left on device
1+0 records in
0+0 records out
0 bytes (0 B) copied, 0.00137162 s, 0.0 kB/s
The log file shows this:
kernel: sd 1:0:17:0: [sdr] Unsupported sector size 520.
@@SuperVertrix yes, I got it wrong. The netapp drives I successfully treated like this were SATA drives, and on those netapp didn't use that format. So I was misled
I know this video is old but maybe this will help someone
#!/bin/bash
for i in {1..15} {17..31} {33..47} {49..63}
do
sg_format --format --size=512 /dev/sg$i & done
That would do all of the disks at one time
Yeap! I am getting requests to redo this video on a newer Linux OS releases and command line string and yes this looks good to me. Is it OK for me to include you and this set if CLI commands with the new video? First I need some 520 formated disks from EMC... /:>
@@leadiususa7394 sure is!
Nice video but why do you use --fmtpinfo=0 since i have seen tons of videos and most do not use it. So very interested what this is all about :)
So, I got a Netapp DS2246 enclosure and got some HITACHI 1.2TB Hard Drives. I was able to format the drives to 512B, but for some reason I"m unable to write to them. Not even a partition table.
When I tried to write the partition, these errors popup in dmesg:
[ 4763.317114] scsi_io_completion: 10 callbacks suppressed
[ 4763.317153] sd 1:0:18:0: [sds] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[ 4763.317164] sd 1:0:18:0: [sds] tag#0 Sense Key : Aborted Command [current]
[ 4763.317175] sd 1:0:18:0: [sds] tag#0 ASC=0x81 ASCQ=0x0
[ 4763.317184] sd 1:0:18:0: [sds] tag#0 CDB: Write(10) 2a 00 00 00 00 00 00 00 08 00
[ 4763.317189] blk_update_request: 10 callbacks suppressed
[ 4763.317194] blk_update_request: I/O error, dev sds, sector 0
[ 4763.318297] buffer_io_error: 2550 callbacks suppressed
[ 4763.318302] Buffer I/O error on dev sds, logical block 0, lost async page write
----------------
Of course, when checking the partition, the partition is not there.
Any ideas?
How long did it take to format each disk. Also what HBA or controller are you using? LSI HBA's are the best for they don't place pre-restrictions on disks. Some HP and Dell controllers do. If so take all but one drive out and reformat again. It should take about 30 on avg. for a drive this size. If very short or really long. also check the cable config to the disk array and go with only one path (SAS) cable for now. Just to make sure you are not routing dual SAS cable (HA) pathing to the drive causing a disk conflict. You could also just have a bad batch of disks.... DO you have a different disk drive you can use to make sure it the array or the disks? Just a thought! Sorry if this is to techi for you... But that is why we do this! To have some fun and learn...
@@leadiususa7394 Thanks for replying back.
I have an LSI SAS 2008 in IT mode which is working fine with another SAS enclosure and an LTO7 tape drive.
When I was playing with these hard drives, I made sure only the NetApp was connected to the LSI. I had it connected via a single SAS cable (if you connect two, Centos sees the same HD twice.)
The format took about 1hr to complete.
I ended up returning the hard disk, but I'm about to order others (same model, but different FW, they even have a NetApp label on it). The HDs I returned, I found out they came from a Hitachi VSP enclosure. Upon digging on Google, I found out that Hitachi disables writes to these HDs unless connected to a Hitachi ecosystem.
I even tried replacing the FW (I found the playing vanilla FW for these models), and even that did not take. I was unable to write the FW to them.
I'll report back when I get the new HDs.
BTW, I totally understand you, you're not been to techi.
I'm a Computer Systems Engineer by trade.
Cheers!
If you have a Netapp enclosure, you don't need to format the drives to 512b. Just leave them at 520b and you're all set. Reformat the drives back to 520b and have a nice day.