finding duplicates

[michael_s]

I've got backups of backups. Copies of copies spread out over multiple disks.
Anyone know of a good utility that will find duplicates and move or delete files?
Looking for a faster way than deleting one at a time in spotlight of in the finder.

[JEBB]

Why not make a backup of what you now use and throw the rest away? It is otherwise an impossible job.

[M A V I C]

This is one of the reasons I went with a RAID for my backups. Then I only have one version to worry about and it's redundant.

[kj4btkljv]

I was going to say that you need to consolidate your backups down into a single, good backup. Looks like I'm not alone.

Jeff

[simonm]

There are a couple of utilities for tracking duplicates. Two examples: http://www.apple.com/downloads/macosx/sy...idyup.html, http://www.apple.com/downloads/macosx/sy...eguru.html
I haven't used either.

Deduplication is one of the current buzz-words in the storage industry as big biz suffers from the same problem except way bigger. Pretty much all the big storage vendors have something. There are various implementations but in principle they identify data using a hashing algorithm (sha1, md5 or similar) and eliminate the duplicates. By doing this down at sub-file level, they can deduplicate across files aswell. Apart from saving storage, big-biz gains by really cutting back on what goes on across their network as the duplicates can be identified before the data is sent to the bback-up device.