How to copy large amounts of files in Windows
Вставка
- Опубліковано 5 вер 2024
- Robocopy is one of those commands that is often known about, but mis-used. Let's go over all the common options and do a little speed test.
Website Guide: www.christitus... .
►► Digital Downloads ➜ www.cttstore.com
►► Reddit ➜ / christitustech
►► Titus Tech Talk ➜ / titustechtalk
►► Twitch ➜ / christitustech
It should be noted and I missed it in the video. If you are copying TONs of very small 1-100kb files you can completely turn off logging with /NFL /NDL /NJH /NJS - This will 10-100x the speed, but at the cost of logging.
NFL = No File List
NDL = No Directory List
NJH = No Job Header
NJS = No Job Summary.
You can copy 100,000s to millions of small files extremely fast, but will have no summary or log.
is syncToy using robocopy?
IS galaxies still a game???
@@TinS0lder SWG Legends EMU is alive and well they just released city in the clouds
@@ChrisTitusTech can we robocopy it to a local drive and play it there? 😁
hey Chris, how's the performance of this vs Rsync?
This was so helpful. Unfortunately 2 weeks late for me. I ran into issue doing a multi terrabyte copy for my NAS migration. I got around it by zipin gup the folders and unziping them to the new destination. Robocopy would have been much smarter and faster. This was awesome! Thanks so much for sharing.
With that much data you might want to look at something like imaging with Macrium Reflect free. That is a lot of data and if you don't have the switches right can be a difficult copy. Using Macrium Reflect might be a better option.
😂
Been using this tool, especially the mirror + copy all attributes to make backups and matching with logfiles to verify all the file copying ever since it was a downloadable tool for Windows XP. A really great commandline tool that really should have a GUI wrapper in the modern age for the less tech savvy.
This just reminds me of how we used to copy files from drive to drive back in the day using 3.11 in DOS :)
xcopy? By any chance.
No such problems. RS232 to another computer. Then reset connection, restart machines (for the IBM), copy or cp depending on the DOS or Unix machine you access. What I miss though are the cassetes in Amstrad. And the time to go out to flirt chicks.
I switched to robocopy long ago. xcopy is good, but inferior
You can double click the icon in the title bar to close a window.
Just thought I'd throw that out there since Chris was right clicking the title bar of a window with the controls off-screen, and choosing close.
Been using Windows for decades and never knew about robocopy, thanks.
Thank you!!! I had a NAS hard drive die - despite having multiple backups, the VMDK file on the drive was the ONLY corrupted spot on the disk! I loathed COPY and PASTE to another computer/drive, and this has saved my bacon!! Thank you!!!! 👏👏
Sounds like you also could have benefited from GRC Spin Rite, Roadkil's Unstoppable Copier, maybe even Test Disk/Photorec.
I feel like for normal day to day file copying tasks this would be kinda overkill. In the example you gave of that 5GB file you saved roughly 10-20 seconds. It would take that long just to type everything out for the command thusly not saving anytime at all. Now I can clearly see the huge professional level applications for such a thing for sure. Just not sure if this is worth it for the average user. Thank you for teaching me something about windows I didn't know though! Really enjoyed the video nonetheless
Yes he said it's for large files, and mostly for big servers and corporation not the average user
Let's say you copied 500GB of data to another drive and it was going to take 1 hour to complete. You copy/paste and walk away and go grab some lunch. You come back only to find that it did not finish copying because it quit for one of the many reasons Chris spoke of. So many different things can stop the Windows UI copying. With Robocopy and using the right switches that is never going to happen. It will always complete the copy process.
@@surfingsub5854 ok yeah I can see that being really useful. Peace of mind is always a good thing.
I work in migration projects and use robocopy a lot for copying more than 500g of data.
Ex: for copying 2gb folder which contains 2m files in it.
U don't know the pain to wait for copying 2gb of data without robocopy..
Robocopy saves the day!!
It has nothing to do with "the average user". if the aver user has 5gb of normal data, then duh, not worth it. if LOTS of data or like kiran said "2gb folder which contains 2m files", then duh, it is worth it.
The one thing I miss in Robocopy is what rsync have, the --backup-dir option so that "deleted" files are actually moved to another location.
I dunno I would rather stick to Teracopy since it does all of what you say but wit a gui. Most probably would want to avoid command line for something like this and anything more important there is backup software better suited.
Teracopy has always failed for me on large copy jobs. I've had it simply skip files and not tell me and has caused me problems. Either that, or it simply crashes. It's a POS.
@@justsaynotoboomers I dunno I've copied hundereds of gigabytes of files across multiple computers and not once did it fail and warned me on corruption even resumed after accidently disconnecting an ethernet once. Sounds to me nothing more than user error on your part
If anyone uses lazy slang like dunno and wit when conversing with others, especially when already writing more than a few words, then anything they have to say is immediately invalidated. Have you even looked at r/DataHoarder? Its chock full of people complaining about how badly Teracopy has been failing on them, only within the last few years, even after years of flawless use. Perhaps you should not be so quick to accuse of others of 'user error' when you obviously don't know the situation, and can't even communicate intelligently.
@@othername2428hey man, people can talk however they want, it doesn’t invalidate their point. I agree with you on Teracopy being issue prone, but you can be right without putting people down and judging their choices. I’m on your side I’m just trying to get you to communicate empathetically. r/DataHoarders is always welcome at my house 😂
@@pcmv6832whew! That’s good, maybe have a backup plan or some other tool to verify it worked as well as it said it did if you continue to use it in the future. A tool is only as good/bad as our expectations of the work we assume it did.
I've been using this since Server 2003R2 days. Once small caveat to the video is that there are times when you may want to turn off security all together. Like if you're copying user home folders from an old file server to a new one on a different domain prior to a user migration. I also might add that using robocopy in a scheduled task with a service account works well if you want a copy of something to go somewhere else to be backed up. No not as a backup itself, but a copy to be backed up there too. It's a powerful tool, and one that so many admins don't use or forget about using. DFS has gotten better, but it still has quirks (child/forest file copies are garbage), where robocopy will still succeed every time.
Indeed. Just remember it cannot copy any open locked files. Not geared for the same purpose but TSCOPY is an excellent tool that can copy any locked file including the hive files.
I was using shared drives to access my NAS and copy files around. I recently installed Synology Drive to synchronize my laptop to my NAS. So far, so good. The only thing missing is a way to synchronize my Android phone to my NAS. I currently have to dedicate space on my laptop and use Syncthing to sync my phone to the laptop, then the laptop to the NAS.
My "big" file structure is about 300 GB (so far) of music albums stored as FLAC. There are all sorts of challenges with long filenames for classical music. Some people have overloaded the filename field with too much information. I shorten filenames when I rip the CDs. There are alternate character sets used for some albums, such as Blue Oyster Cult. These played havoc with my old Debian-based NAS.
I wish there was a way to put all my music in a database AND be able to play it back. The file system Music > Artist > Albums > Tracks is cumbersome at best. I actually added two folders to the structure to keep software from failing. I use Music > (A-M and N-Z) > Artist > Albums > Tracks. I may have to spit things again.
This is so very much appreciated. I just want to copy everything on an older smaller drive to a new larger one... but NOT lose any of the dates in any way. Sounds simple... been hard. I will test my method first before I go "BIG".
I love this channel. I have learnt so many things from this channel ❤
Robocopy rocks. Have been using it since it was a part of the Windows 2000 Resource Kit. Was pleased to see it promoted into default install of Windows. One useful switch is /XO (exclude older files). Prevents an older file from overwriting a newer one.
Also, if you don't use the in-built logging function and just pipe the output into a file yourself with >, switch off the progress (NP). If you don't, you'll get loads of percentage counts in your log file and it looks a horrific mess!!
The GUI copy will not preserve security settings most of the time, usually just replacing the source ACL's with the target folders ACL instead. Very bad if you need those ACL's preserved intact.
I used robocopy extensively to migrate a file server with over 2 TB of roaming profiles years ago. Did one pass to get the bulk data mirrored to the new system (ran overnight, took hours), command saved for later use. Later we just ran it again and again to get the modified files (took minutes each time). The ability to save logs each run helped us sort out a number of access issues reliably in the end too (use /NDL and /NFL to only log errors and the summary).
You can also use it (in rare cases) to salvage data from unreliable systems or drives as it can resume or at least skip files already copied. Had to do this ONE TIME to get data off a system with failing raid drives (lesson here is to actually check your raid and hardware logs regularly and not wait to the alarms to sound, yellow and red drive lights don't always get noticed). We managed to save all but two files if I recall correctly but it was touch and go for sure.
I use fastcopy and it has a GUI and it syncs files.
Fastcopy rocks!
I also use Fastcopy. Most people have no idea what it's capable of. Can even shut down the pc when done, do wipe deletions, etc etc the list is long.
I use Teracopy because I can verify the files with different Hash functions. I usually pick SHA-256 or SHA-512
@@apreviousseagle836 Can almost do the same with Fastcopy except Terracopy has a couple more algorithms
Not gonna lie, I tought that I would never use this when I saw your video, until I had to backup my sister's external hdd.
This is a god sent, the only thing that I miss is a general progress bar to the whole process...
It sounds like a small thing, but I'm copying a bunch of stuff right now. It's been going on for hours and I wish I had at least a rough idea of where it's at in the process.
This looks more like a Linux way of doing things! It may be faster, but typing in all that stuff would actually take me a lot longer than just a couple of mouse clicks here and there, plus I'd mess up somewhere. That's why I can't use Linux. Give a GUI any day.
Great video! Does robocopy work with Linux server and Windows client? Is this essentially scp/rsync but for windows?
If your Linux server has samba, then yes. Synology is basically a linux server and that's what he used in this example.
Yes, rsync has some methods that robocopy does not and if you're copying from one network to another and can use rsync that would be better
The number of people that don’t get why this is used over other processes have not done copy paste or even terracopy in bulk. At my job we have around 100k files of average around 1 mb. That are required to be pasted into dedicated folders for each public launch from our system. And then they have to be moved into drives to be sent out as well. When robocopy is taking 5-10 minutes the normal copy paste would take 15-20 hours or even days so robocopy is a lifesaver for us
Is there a GUI overlay for RoboCopy? That might be more comfortable for many Windows users... Also, how does TeraCopy, and other third-party software to replace Windows default coping service stack up to Window default service, or against RoboCopy?
ChoEazyCopy exists its a gui
53.05 seconds vs. 52.07 seconds.
1.84% faster.
Interesting.
It would be interesting to see how much faster robocopy would be if you are saying using 10 GbE and passing data through a system in order to sync up two servers (or migrate the data) say from one NAS server to a bigger, newer, NAS server.
Exactly, the time is almost equal... The copy over the network is limited by the network speed and in both cases it was used to the max...
I can't find the source but someone was speculating that Win10 was using robocopy with default 8 or 16 threads so the performance is similar.
But so often I have one file failing in the end of the Windows way and then all I can do is click cancel and then you go to the folder and it moved everything back with that one file.
@@nlx78
I guess that the question is if you are using robocopy, and the same file fails -- would you know/would it tell you that the copy failed?
I would be surprised if it failed using the Windows File Explorer vs. but then it would be successful using robocopy. It wouldn't be unheard of, but I would be surprised (given that it's still Windows).
@Mister Robato
I don't think that rsync would really be all that much faster, especially since I don't think that it is native to Windows.
(Like you'd have to install something else (e.g. cygwin) to be able to get rsync. Or at least that's one of the ways to do it.)
Would be nice if they had a gui. A lot of the switches can be automated by detecting the type and size of files if the manufacture wanted to. Make it easy for those who doesnt use robocopy often because I'll forget all the command line switches.
There used to be, but MS ended support for it in 2016.
Thank you for explaining this tool. I will use it now. 👍
*"...can use all 16 cores.."*
?? Okay, but where's the BOTTLENECK usually?
I would assume the bottleneck is usually the read or write drive. And if you're reading from, or writing to an HDD wouldn't trying to access multiple files at the same time actually HURT performance because the read/write head is jumping around?
And if you're reading from or writing to an SSD which is RAM (as in Random Access Memory) then what's the point of trying to access files simultaneously?
If system crashes during robocopy is in progress, some time source folder gets corrupted. I guess it's due to robocopy using mft directly.
nice tutorial video...thanks chris. just share my experience, i use "fastcopy" written by japanese, it do able to handle long file name as well. hence, it have a log in notepad for each file copied.... and lots of great features...
I was going to suggest Fastcopy. Thanks.
YO keep these vids coming Chris!!
Liked the deeper dive into windows powershell
if you copy to one hard disk what's the point of using multiple threads? also in large files will it cause file fragmentation?
Great tut! Don't need anything fancier.
Whoa! You got Star Wars Galaxies? I played that religiously back in 2005-06. I still dream about it.
Check out SWG Legends. Fan run EMU and they have even entire zones like cloud city to the game.
@@ChrisTitusTech Thanks! I'll have a look.
one thing i don't like is, the /MT flag is only for file copyings. when ever i do server migrations there are multi millions of files and lots and lots of folders. It would be really cool if the multithread could handle directory transversal as well. it never fails the initial population to the destination takes weeks then one needs to run a delta to catch new/modified files. Robocopy is a huge waste of resources when its only 1 thread to traverse the files/folders.
Awesome tutorial - I am sure many will benefit, i will for sure.
Hi Chris.
Another handy switch for robocopy is /L
This will do a dry run of the command and list out what it would do or not do without actually touching any files.
Well done on put out such great content.
good vid. Thanks. Fan of your deep dive presentations vs the glossy overview.
I mean I am giving you a like before I even finish for Star Wars Galaxies alone!
Owman Chris, you can't stop yourself hu? In the end you give (even though you leave a lot out) so much options, I still don't know what to do.
Thanks anyway.
👍🏻Greetings from Netherland✌🏻
Thankx for breaking it down.. great tutorial for beginners
Robocopy cut/moving in same directory is very slow. It basically copies the files first and then deletes it. Very annoying.
I am going to try this, and you may have changed my life.
Ty for this helpful information; I always learn something new from your videos.
Large numbers of files, maybe a large quantity of files, even a large volume of files, but not a large amount of files. Sorry - that triggered me. Amount is so often incorrectly used these days, and often in professional media, that the distinctions are often forgotten. (Simplistically - Number for countable objects, Amount for non-countable objects)
Useful video too. Personally I would prefer a GUI, but I completely get that a multi-threaded solution has benefits of the standard copy tool in Explorer
Dude, this is insanely faster than normal method. Thank you so much :D
Nicely explained. Many thanks for sharing!
If robocopy so great why isn't it built into native Gui? Honest question...
It replaced xcopy, which was a command line utility. Robocopy is used in scripted tasks. I do agree Microsoft needs to reinvent the regular gui copy to use robocopy under the hood. Would be nice.
Folks need Eye candy to look at with a progress bar.
To be honest in end of day not all users are admins or feel comfortable in command line that's why Microsoft had to evolve into eye candy GUI. If I. Not mistaking Linux is also following with some nice and impressive GUI.
@@ChrisTitusTech It's not just eye candy. Informations are important.
Sometimes you need to have a good guess on how long a task will take.
Can you just wait, grap a coffee or maybe you should take your break now? You don't know if you only see the current file.
In your case in this video, open PS and typing the command takes longer then your saved time. Of course, if you copy milions of files or terabyte of data, you are faster.
The basic windows features are designed for users, not administrators. That's what PS / CMD is for.
Could the copy feature be better? Of course, there's plenty of software out there you could use instead. Same goes for robocopy.
A nice use case is moving game save files to dropbox for those games that Steam does not sync the saves for you. I've got a task set up for that.
How does this compare with teracopy? Anything I’m missing as the average user?
Ah yesm the wizard again. Thanks, your videos are Goodstuff!
The 255 path limit is a bad default setting of windows explorer. You can enhance that using a registry key. But yes i also forget it always…
Saw the title and instantly knew you were talking about robocopy. Use it all the time mainly for the MT option and especially with mapped drives at my enterprise. You mentioned UNC would be faster (we use SMB shares). How? UNC via file explorer is still single threaded mostly and via command line I just get UNC paths not supported.
I like your tutorials Very much
Robocopy runs out of steam at a point where rsync just keeps going.
Need a lot of data on NTFS volumes to be moved around? Boot up System Rescue and do it with Linux.
This video came in the right time
Excellent subject…… one week short lol. That explained all the duplicate files and why nothing could be found. Even when I located it that did not transfer to the very next file in same folder. Still going through all 7 ssd’s and finding whole directories 3 or 4x deep under user/name/etc. dupchecker helpful as long as you tell it where to look. I wish it would scan everything everywhere and show which are complete and working and which can be trashed. So far though I was able to get back 6Gb of space. All I wanted was keep some important program files and empty each drive start with reformatted and clean pc. That didn’t go as expected. Lol.
This is amazing I just had to sit and listen to
This video enough I wanted to take over 20,000 Files off my external hardrive and put it on my pc very fast if u sit down and follow these steps u will be fine use your note pad to write the codes first so u can copy in post in command prompt without feeling discouraged lol I didn’t need to put my network Ip and it worked
Copied 2M files from my HD to my SSD last week, it took days ... I wish I had known about the no-logging option mentioned here!
I think the corporate antivirus slowed things down a lot though.
Titus gonna Bitus.
Why not use Teracopy or other software like that ?
Thank you very helpful!
To clone user folders like documents, pictures, music, desktop: robocopy "C:\Users\USER\Documents" D:
obocopy\Dokuments /MIR /XJF /XJD
It does a fast and perfect copy e.g. onto an external drive for transferring to a new PC or just as exact clone. Even the „folder icons“ are copied.
Very informative! Thanks.
Great video. Makes it a lot clearer!
This works great. Small issue: when the transfer was complete, the only way I can access the folder is to type in the path in explorer, the folder I transfer everything to disappears for some reason once robocopy has completed the transfer. and it’s not hidden. I just created a shortcut to get to it. Just curious if there’s a way around this?
This was excellent. Thank you.
what if ive accidenntally put in too many threads when copying ive done 32 instead of the 16 i have on accident from just copy pasting got the directories correct but it appears to be copying fine still ? am i going to be bottlenecking or will it see i dont have those threads available ?
cool, informative, educational, thank you kind sir
thanks alot for your decent effort,
Nice video.. Thank you I learn new everyday from you..💕💕
I used to use xcopy in dos or command prompt.
Hi, what is the difference between xcopy and Robocopy? I use xcopy a lot, it's faster than in the GUI, and it controls the files?
Everything counts in large amounts... It´s a competetive world!
Depeche Mode yo
Thanks for the great video! I didnt find a switch that would skip over unmodified files already in the destination folder. Or does Robocopy skips over by default?
Which one of the following commands is preferable for copying shared files from an AD serv to another (Win servers)
Robocopy Source destination /DCOPY:T /COPYALL /E /R:0
or
Robocopy Source destination /E /ZB /copy:DATSOU /R:3 /W:3 /V
SMB Multichannel copying is amazing. my nas has 80gbit connection and I have 20gbit. Rocks. You need a NAS that's 10g, qnap has some that are decently affordable now and 10g switches. qnap 10g nic works really well in linux and windows (very plug in play) for $100. But 10gig is SOOO much better than old school gigabit.
Hi Chris, thanks for this video. I will be helping a relative transfer basically everything on her old laptop to a newer one. Is connecting them with a crossover Ethernet cable and then using robocopy the best way to approach this?
Her old laptop is win 7 and the new one is win 10.
Star Wars Galaxies!!! Are you a SWGEmu player as well?!? I unlocked my force sensitive character slot on the offical SOE servers one week before Sony dropped the NGE update and nerfed my years of work!
Point being, 99% of users will never open a command line. You have Total commander for secure copying.
Total commander is not the same as robocopy and it's also not free.
@@surfingsub5854 It is free and it is even better for general use.
@@J4ckCr0w Intersting. Have not looked at it in years but have seen cracks so I assumed it was not a free program. Thanks for the update.
Mirror is good if you think there is a probability that someone might modify your files on a shared drive.
Great video!
Thank you!
Why should multi threading be the factor that speeds the transfer up? Any modern single CPU thread can handle file transfer io streams.
is the a GUI version of this?
Copy in windows is compete trash. The fact that it doesn't start copying and scans all files first is a tremendous waste. There should at least be a registry entry to eliminate that
Moving an entire 3TB drive to another local drive.
Needing to keep ACLs, time stamps, etc
Would I need /ZB at all? In case my permissions are not sufficient?
Also, do I need /COPYALL and /DCOPY:DAT ?
So the reason my copy n paste keeps crashing my pc or aborts midway is because i go over some limit and it cant handle that? I have been frustrated as hell doing 500 latge files one by one because it keeps crashing and blue screen and what have you...
Is there a way to verify file integrity to make sure everything copied correctly and without corruption?
Thanks. Very helpful. I wish there was an easy way to detect hackers that have logged on.
What program do you use for screen casting and especially how do you insert yourself in the screencast with cutting everything behind you?
OBS for recording with a Green Screen w/ Alpha Layer
@@ChrisTitusTech Ok, thanks
Very good video. Thank you. Can I also access my internal phone storage and copy videos from tis location usin robocopy. It shows up as MTP-device and doesn't have any drive letter assigned? Thanks very much in advance!
Well, in principle nothing else what my Synchronisationsoftware does, but way uncomfortable. Working on Windows to accomplish task with the DOS way, makes no sense to me.
By the way, Total Commander uses this too, but way more comfortable.
But thanks for showing this, Chris!
Can you show how to do the same thing but with linux?
rsync
@@apreviousseagle836 Thank you
Trying to convince windows users to use command line is kinda futile. What are your thoughts on Teracopy? It doesn't have as many resources but it copies big files fast
funny the timing of this video came up, I am being forced to get my A+ via new contract regulation and I was wondering what the heck robocopy is. kind of neat, now how can this replace the old one on the gui lol.
I use it for copying only files of a certain type.
robocopy D:\Images R:\GIFs *.gif /s
SUPER CONFUSED: I am trying to do:
robocopy "D:\BACKUP FOR HDD REPAIR\DASHCAM\" "F:\DASHCAM\" /e /r:0 /w:0 /COPY:DAT /DCOPY:DAT /mt:12
but it keeps saying there is no destination specified????
fastcopy, robocopy is another copy of same program,. whatever you copy to same drive, other or usb stick or drive, it goes so much faster than windows or total commander. also no need to flush 16gb like total commander
thank you for this lesson on robocopy. I did a test, copying files from my C: drive (all of which have dates and times in the past - relative to 4/17/23) to an external USB drive that has bitlocker on it. INITIALLY, the dates and times on the destination drive matched the source drive, but after a minute or so, bitlocker must have done something to them because the dates and times on the destination drive all changed to today's date and time. Are you aware of this occurring with bitlocker? Is there a way around what bitlocker is doing? thanks
FreeFileSync works great for me now that MS Powertoys (Synctoy) is kaput. I prefer to use an intuitive GUI.