5
What is the best duplicate file finder, that preserves my source of truth? (alien.top)
submitted 2 years ago by [B] to c/main@selfhosted.forum
26 comments fedilink hide all child comments
 

Having been so meticulous about taking back ups, I’ve perhaps not as been as careful about where I stored them, so I now have a loads of duplicate files in various places. I;ve tried various tools fdupes, czawka etc. , but none seems to do what I want.. I need a tool that I can tell which folder (and subfolders) is the source of truth, and to look for anything else, anywhere else that’s a duplicate, and give me an option to move or delete. Seems simple enough, but I have found nothing that allows me to do that.. Does anyone know of anything ?

you are viewing a single comment's thread
view the rest of the comments
[–] [B] 2 points 2 years ago (4 children)

Write a simple script which iterates over the files and generates a hash list, with the hash in the first column.

find . -type f -exec md5sum {} ; >> /tmp/foo

Repeat for the backup files.

Then make a third file by concatenating the two, sort that file, and run "uniq -d". The output will tell you the duplicated files.

You can take the output of uniq and de-duplicate.

  • permalink
  • fedilink
  • source
  • hideshow 4 child comments
    •
  • [–] [S, B] 1 point 2 years ago (2 children)

    Thanks @speculatrix - I wish I had your confidence in scripting - hence I’m hoping to find something that does all that clever stuff for me.. The key thing for me is to say something like multimedia/photos/ is the source of truth anything found elsewhere is a duplicate ..

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] [B] 1 point 2 years ago (1 child)

    I wish I had your confidence in scripting

    You know how you get it? by fucking around and finding out! I'd say give it a go!

    Do a dry run of the de-dup to make sure you don't delete anything you care about.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] [S, B] 1 point 2 years ago

    Give me a few years and maybe :P - but for now I’d rather not risk important data with my own limited skills especially if there is a product out there that it’s tried and tested and hopefully recommended by someone in this sub.. I didn’t expect my ask to be quite so unique..

  • permalink
  • fedilink
  • source
  • parent