1 .TH "FINDDUP" "1.2" "Apr 2011" "Francois Fleuret" "User Commands"
3 \" This man page was written by Francois Fleuret <francois@fleuret.org>
4 \" and is distributed under a Creative Commons Attribution-Share Alike
9 finddup \- Find files common to two directories (or not)
13 \fBfinddup\fP [OPTION]... [DIR1 [[and:|not:]DIR2]]
17 With one directory as argument, \fBfinddup\fP prints the duplicated
18 files found in it. If no directory is provided, it uses the current
21 With two directories, it prints either the files common to both DIR1
22 and DIR2 or, with the `not:' prefix, the ones present in DIR1 and not
23 in DIR2. The `and:' prefix is assumed by default and necessary only if
24 you have a directory name starting with `not:'.
26 This command compares files by first comparing their sizes, hence goes
29 When looking for identical files, \fBfinddup\fP associates a group ID
30 to every content, and prints it along the file names. Use the \fB-g\fP
35 is virtually the same as
40 \fB-v\fR, \fB--version\fR
41 print the version number and exit
43 \fB-h\fR, \fB--help\fR
44 print the help and exit
46 \fB-d\fR, \fB--ignore-dots\fR
47 ignore files and directories starting with a dot
49 \fB-0\fR, \fB--ignore-empty\fR
52 \fB-c\fR, \fB--hide-matchings\fR
53 do not show which files from DIR2 correspond to files from DIR1
54 (hence, show only the files from DIR1 which have an identical twin in
57 \fB-g\fR, \fB--no-group-ids\fR
58 do not show the file group IDs
60 \fB-t\fR, \fB--time-sort\fR
61 sort files in each group according to the modification times
63 \fB-q\fR, \fB--trim-first\fR
64 do not print the first file in each group
66 \fB-p\fR, \fB--show-progress\fR
67 show progress information in stderr
69 \fB-r\fR, \fB--real-paths\fR
70 show the real path of the files
72 \fB-i\fR, \fB--same-inodes-are-different\fR
73 files with same inode are considered as different
75 \fB-e \fI<command>\fR, \fB--exec \fI<command>\fR
76 execute the provided command for each group of identical files, with
77 their names as arguments
79 \fB-f \fI<string>\fR, \fB--result-prefix \fI<string>\fR
80 for each group of identical files, write one result file whose name is
81 the given prefix string followed by the group number, and containing
82 one file name per line
86 None known, probably many. Valgrind does not complain though.
88 Since files with same inodes are considered as different when looking
89 for duplicates in a single directory, there are weird behaviors -- not
90 bugs -- with hard links.
92 The current algorithm is dumb, as it does not use any hashing of the
95 Here are the things I tried, which did not help at all: (1) Computing
96 md5s on the whole files, which is not satisfactory because files are
97 often not read entirely, hence the md5s can not be properly computed,
98 (2) computing XORs of the first 4, 16 and 256 bytes with rejection as
99 soon as one does not match, (3) reading files in parts of increasing
100 sizes so that rejection could be done with only a small fraction read
101 when possible, (4) using mmap instead of open/read.
105 The format of the output should definitely be improved. Not clear how.
107 Their could be some fancy option to link two instances of the command
108 running on different machines to reduce network disk accesses. This
109 may not help much though.
116 List duplicated files in directory ./blah/, show a progress bar,
117 ignore empty files, and ignore files and directories starting with a
123 List all files which are duplicated in the current directory, do not
124 show the oldest in each each group of identical ones, and do not show
125 group numbers. This is what you could use to list what files to
129 .B finddup sources not:/mnt/backup
132 List all files found in \fB./sources/\fR which do not have
133 content-matching equivalent in \fB/mnt/backup/\fR.
136 .B finddup -g tralala cuicui
139 List groups of files with same content which exist both in
140 \fB./tralala/\fR and \fB./cuicui/\fR. Do not show group IDs, instead
141 write empty lines between groups of files of same content.
145 Written by Francois Fleuret <francois@fleuret.org> and distributed
146 under the terms of the GNU General Public License version 3 as
147 published by the Free Software Foundation. This is free software: you
148 are free to change and redistribute it. There is NO WARRANTY, to the
149 extent permitted by law.