2 Ways To Find Duplicate Files On a Mac
Вставка
- Опубліковано 3 лип 2024
- macmost.com/e-2724 If you suspect that you have some large duplicate files on your Mac, you can find them without any special software. You can use the Finder to search for files and sort them so duplicates are together. You can also use the Terminal to find duplicates with a multi-part command.
FOLLOW MACMOST FOR NEW TUTORIALS EVERY DAY
▶︎ UA-cam - macmost.com/j-youtube (And hit the 🔔)
▶︎ UA-cam Email Notifications - macmost.com/ytn ⭐️
▶︎ MacMost Weekly Email Newsletter - macmost.com/newsletter ✉️
SUPPORT MACMOST AT PATREON
▶︎ macmost.com/patreon ❤️
00:00 Intro
00:45 Using the Finder
02:31 Terminal Command
05:26 How It Works
#macmost #mactutorial - Наука та технологія
Excellent info on every broadcast. And quick-moving without the fluff. Thanks.
Fantastic "one liner"! Just used it to find a ton of dups in my Documents folder - thank you!
Great video, as always, Gary! I'm just beginning to learn BASH scripting and terminal commands, and so I *understand* what you're doing, even though I could never have come up with this process on my own.
Thank you so much Gary! Very helpful!
Man, all of your videos are so good!!!! Thank you very much!!!! 👍⭐👍⭐👍⭐👍⭐👍⭐👍⭐👍⭐👍⭐👍
Just saved me so much time. Thank you!
outstanding! thanks Gary
Thank you for a very useful video, succinctly explained.
Yes very useful Gary !
You Sr. Have saved me some much time and space on my hard drive you are a life saver
Every person who owns a mac needs this as a pre-requisite. TY
You are awesome! Thank you so much for this video!
Great video 🌺
Hi Gary! Got my first mac 2 months ago after 20 years of using Windows. So now I'm blazing through your videos in search of useful tips and I really tell that you know a lot and you are a really smart guy!
Meanwhile, in this video you show up a terminal script that search for the duplicates but it is extremely inefficient to me. It works for some small amounts of files but will take a really long time if you run it for hundreds of gigabytes of data, because it will take a ton of processing power and time to hash all the data. It will also get WAAAAAAY slower if you run same task for the files that are stored on the slow HDD storage. I'm talking hours and even days slower. It also put unnecessary wear on the storage since you have read every single byte of data to hash everything.
You can optimize this process to be way more efficient by searching for the files of the same size first, then calculate the hashes of the same sized files and then compare the hashes. This way you will be hashing just the files that are the same size (files have to be the same size to be duplicates) saving just absurdly huge amount of time and resources.
I'm not good enough with terminal commands yet, so I can't offer such command myself but you probably can remake it in the described way and make another video about it :)
Thank you for your videos!
Neat, neat, neat! Thanks
Thanks!
Unbeliveabe easy, and very useful.
Brilliant!
impressive. i have a demo copy of macpaw gemini. it will find the dupes but wants an activation code to finish the job so I was just using it to 'find' the duplicates. Yours is much better. Back in the 90s I would use tax software to do all the calculations but you had to pay in order to print the tax return. So i just copied all the numbers by hand to the paper return.
Thanks bunches
Excellent video, as always. I would not stop on that, though. I would (and I will) make an application with this code using Automator.
Or, Shortcuts...
I found it thank You
This guy has a gift of teaching things, he makes things people like mnyself have no knowledge on and making the topic extremely clear and understanable. Many thanks @macmostvideo
Most duplicates are going to be found in Downloads caused by people clicking on the same online document to view it. The filename of the duplicates will be followed by a number, like -1, -2, etc., or like (1), (2), etc.
Great video, as always. BTW, on partitions formatted with APFS, duplicates don't take up extra space.
Clones don't take up space. If you take a file and make a duplicate of it, it is a clone and doesn't use space. But say if you download a file once, and then later download the exact same file again, it will be there twice.
@@macmost I did some research (which perhaps I should have done *prior* to making this comment), and you are, of course, correct. To be more precise, in order to create a clone on APFS, the clonefile(2) syscall has to be used. Examples would be ⌘D in the Finder, or "cp -c" on the command line. APFS does not check a new sector against a database of existing sectors like ZFS does, and therefore downloading the same file twice will not create a clone.
A very useful and informative video tutorial today! Thank you, Gary! 👏🏻❤️
Gary, Great content and delivery! I am trying to figure out how to find duplicate files on multiple external drives. I want to clean out unnecessary files to make room for TimeMachine to do its thing. Any help would be appreciated. Thanks
Wondering if there’s such thing as a batch file on macOS, this way you could just drag the file into the directory you want and search it easily, no need to go through terminal and find the command to copy paste
Great video Gary ! How can I move those identified duplicate files into a separate folder where I can manage/remove files manually? Thank you
If you find something you want to delete, you can just go to that folder and delete it.
duplicar]te file finder app is my best friend there!
Hello Gary and thank you for another great video! I am wondering if there is a way to add a command that will delete the duplicates?
Say I have 100 Giga of photos and backup photos and there are a lot of duplicates and I just want to keep the ones that are not duplicates.
Thank you again, Yossi
Probably, but I would be very wary of using a script that decided what to delete for you. It is too easy to make a mistake or overlook something and delete valuable photos.
@@macmost Appreciate the feedback
A fantastic way to do it for free I use a paid version "Gemini". I did not realize the command line could do it thankyou.
Hi, that's pretty cool. Do you think there would be a way to print /send the information to a file (in it is the checksum and filename at least), then modify that file to delete any of the obvious files? If there is a way, would you recommend it? Also, how would you do that? In DOSshell I can do all that but having long filenames makes it so much more specific and dangerous, at least in DOS.. Thanks
You could use some Terminal command things to send the results here to a file. But what I honestly do in cases like this is to just copy and paste into TextEdit or something and work from there. You could just go through and then copy and paste lines from one document to another to narrow down files to check out, or put * in front of some lines, etc.
can you add that string to the description?
on you mac the get info. How do you uncheck Multiple Boxes ? like The Locked in photos can you do a video on this
Hi Gary, thanks for your very helpful videos. I find my self in a situation where I made several copies of my entire hdd as a backup. Now I'm making a big storage and all my old hdd's will be copied there. This will take mote than 10TB of data and I know there is a lot of duplicates. Is there a way how to batch select all the duplicates and delete them? If I'm not mistaken I can't do anything like that in Terminal. But could this task be done in Automator?
Nothing automatic will really know what it is dealign with and how things are organized.
3rd party software seems like a much more convenient solution?
Hello Gary, thank you for the video. I have a MacBookPro with El Capitan. What App could I use to find duplicates in the PHOTOS APP? I have too many pictures to do it one by one. THanks for your help!
Don't use some "app" to do it. Your photos are too valuable. If you have too many, just do it 5 minutes here and 5 minutes there.
@@macmost You are right Gary, thank you so much for your advice!
So what do you do with the duplicates? Do these duplicate files affect performance in any way? I have what appears to be quite a few duplicates and wonder what I should do with them.
Thanks man!
If they are true duplicates (exactly the same file in two separate locations on the same drive) then I would assume you want to get rid of one of them to free up space. How much space you have available on your drive could affect performance, but as long as you have 10% or more free, as a general rule, you are fine. Make sure these really are duplicate files though. Like make sure you aren't seeing the same file in the folder and also in "Recents" or something like that.
@@macmost Thanks man! You’re the best!
Ok I wished you had included the terminal command in the description.
I do. Click on the link at the very start of the description to go to my site where I include such things.
When I back up my Mac using Time Machine is it duplicating the same files on the external hard dive every time? I have a 1 TB external hard drive that I only use for Time Machine. It is is full and my 500 GB Mac hard drive isn't full.
No. It is incremental.
Does this work also with an external drive?
Sure. Try it.
Hi Gary,
I am using Numbers for my invoicing. Could I ask how can I add up only one cell in each sheet with number which indicates service fee? By end of financial year, the number of the sheets (invoices)could be over 300. Your help would be very much appreciated.
Regards,
Frank
That will be difficult to do. 300 sheets? You could try to use the INDIRECT function if things are very consistent across sheets. Or, maybe come up with a better way than creating a new sheet for each invoice. Have you seen this? ua-cam.com/video/r_RRpZhUCzQ/v-deo.html
I should also add that you may be better off using dedicated billing software (or service) that fits whatever business you run. What do others in your field use for billing?
@@macmost Thank you for quick reply Gary. Majority of others they are using hard copy invoices. But as you mentioned, I will try to find an invoicing software.
I either get a "permission denied" message, or nothing happens. In the "nothing happens" case, I have duplicated a file intentionally before starting the terminal command. 🤔
Hi. I thought you said the code would be in the notes for the video, but I don't see it. Is the code available somewhere? Thank you.
It is. It is in the notes for the post at my site. The link is the very first thing in the description here.
@@macmost Thank you!
now how do you delete the 2nd copy and keep just one
Delete it just like you would delete any file. It won't do it for you automatically, and nothing like this ever should. Deleting files is serious and you should carefully look at each file and make a conscious decision.
Hi Gary,
I have a folder 📁 on my MacBook Air which contains family pictures and videos. The 📁 location is on the desktop and it's very large. I want to remove the duplicates before I copy the folder to the external hard drive.
Can you please help me who to remove the duplicated photos and videos from a folder on MacBook ( please note that the files are not on photos app).
Thanks for your time
How about using the methods in this video? Or, come up with a technique based on this situation. Obviously the files can't be exact duplicates (can't have a file with the same name, same location). So what do the duplicates have in common? How are the files different in some way?
@@macmost There are around 37 sub-folders within the main (mother) folder. These sub-folders are arranged using some chronological or even or any other significant order. Hence, some pictures can be found in more than one folder.
@@betemulugeta9782 Sounds like you've created quite a mess. How about searching for images in the main folder so you see everything in one list, regardless of the subfolder. Then sort by name and you'd see the duplicates that way and work through them.
@@macmost I agree it's a big mess. Didn't know this coming as it has been built gradually over time.
I will try your suggestion, thank you for your help and time.
I would use md5 instead of cksum. Even sha256 as md5 is considered weak.
In what sense? We aren't encrypting files here. We are looking for a way to compare files. Very different tasks.
@@macmost it's because the md5 or sha checksums are much less likely than cksum to generate the same result for two different files. I'm splitting hairs.
Nothing to do with encryption.
Of course, cmp tells you unequivocally if two files are identical, but you would have to work out a much more complex command to use cmp.
@@adamnealis I think cksum plus file size probably makes it astronomically improbable for a false positive though. Haven't tried md5 or sha to see how they work in this code, Wonder if it is a major speed hit too.
@@macmost I agree the odds are low. cksum produces a 32 bit checksum, md5 a 128 bit and sha256 a 256 bit checksum.
I know md5 is optimised to be fast.
Gary I was waiting to see if you show a way to actually delete the duplicates. That is really the whole purpose of finding duplicates. I have1000s of pictures and I would say 30% of them are duplicates. I know there are software that you buy that do that but does apple have a way to delete the duplicates?
Do you mean file duplicates? Or duplicates in the Photos app. I'll assume files, since that is what this video is about. The purpose of this script is to see where you have a problem. But I'm not a fan of "automatic" deletion of anything. First, I'd figure out why you have 300 duplicate images to make sure it doesn't happen again. As for how to delete them, it depends on the situation. Obviously they can't be in the same folder (won't allow files of the same name), so it is hard to list them in the Finder. A Smart Folder may work to have them all listed together, but without knowing the details it is hard for me to advise.
Can you post the command please?
Click on the link at the very start of the description to go to the post page for this video at my site. I posted it there.
why so hard a simple task that could be in finder easily?
so this would not work if you had two folders in different locations - say one on the desktop and one on a external drive.
you'd need to copy the desktop folder to the external drive then run the command on the external drive. once you had the duplicates you'd need to check what folder they are in to make sure you deleted the old desktop version (or which ever version you did not want)
unless I'm missing something with the find . command
Sounds like what you want to do is to compare the contents of two folders, not find duplicates throughout the drive. Yes, I would come up with some different method for that. Not sure of the situation where this would be needed though.
yes I guess compare (diff) would work better. I presume it will check both folder & drive and tell me if files exist on either or both locations if I use the -rq I think switches
if a file is only in the desktop folder I'd then need to copy/move it to the external drive
I do run dupeguru (free) now and then but I have found it sometimes says no duplicates when I know there are (just not always how many duplicates) so I'm looking for a way to check.
the reason for my thinking was, I often find old drives that contain files and I'll copy the drive to a folder on the desktop (more so if the drive is not 100% stable on mounting) and then I can 'go through it' to see if the files are already on my system or not by finding duplicates.
My bad habit is to take screen grabs e.g. of photos and then, a few days later, I inadvertently repeat this and end up with duplicates. I've used third party apps to find duplicates and they're pretty good. However they don't always pick up all duplicates e.g. if I deliberately save two copies of the same photo as a test to see if they're found. Looks like your method may be worth trying.
This would not work with duplicate photos as there are so many photos taken with the same file size.
Probably not exactly the same file size since compression would affect that. Plus, this uses a checksum on the bytes in the file, not just size.
Confusing
Why am I getting "grep: -: No such file or directory" on this command?????
Maybe no duplicates are found?
@@macmost I double checked the spelling AND I guaranteed there are duplicates (to test it out)...
@@tejloro Not sure what the problem could be then. Permissions maybe? Is your account an admin or standard account?
@@macmost /tmp/filelist.tmp is being created and filled...
@@tejloro Are you using Monterey and zsh? Other than a typo, I can't think of why it wouldn't be working for you then, sorry.
I FOUND DUPEGURU...
Way to advance for my Brain.
Hello Gary, thank you for your great tutorials which have often been very useful for me. Now I am stuck with a duplicate file problem and I hope you can help.
In short, I have a mid 2015 iMac running Big Sur. I use Mail for emails and I have nearly 20 years of email messages stored in different mailboxes.
When I use Mail to look inside any single mailbox (e.g. WORK_2020) I can see a number (e.g. 3000) of email messages with their respective attachments.
However, when I look at the same WORK_2020 mailbox folder within the ~Library/Mail/etc directory I find a much bigger number (something like 4800 or more) of .emlx files, corresponding to multiple copies of each email message.
I suspect that this is what slows the Mail app and causes occasional crashes.
My question is: is there a way to find and delete these duplicates ?
I will add that I already tried a number of duplicate finder apps but none was able to solve the problem. Actually, only the "Duplicate File Finder" app does spot some but not all duplicate messages while other apps do not find any duplicate.
Of course I followed your instructions in this video (ua-cam.com/video/muDhIlWc4Yc/v-deo.html) but it does not work for me.
I believe that in the case of .emlx files there is some additional complication, i.e. each multiple copy of the same email message has a different file size and different checksum. I can only confirm that these are the same email message by opening each one separately in Mail and check that the sent and received date and time are the same.
Sorry for being so long. I hope you can suggest some solution.
With my best regards
Stefano Conti