.TH "FINDDUP" 1 "Mar 2010" "Francois Fleuret" "User Commands"

\" This man page was written by Francois Fleuret <francois@fleuret.org>
\" and is distributed under a Creative Commons Attribution-Share Alike
\" 3.0 License.

.SH "NAME"

finddup \- Find files common to two directories (or not)

.SH "SYNOPSIS"

\fBfinddup\fP [OPTION]... [DIR1 [[and:|not:]DIR2]]

.SH "DESCRIPTION"

With one directory as argument, \fBfinddup\fP prints the duplicated
files found in it. If no directory is provided, it uses the current
one as default.

With two directories, it prints either the files common to both DIR1
and DIR2 or, with the `not:' prefix, the ones present in DIR1 and not
in DIR2. The and: prefix is assumed by default and necessary only if
you have a directory name starting with `not:'.

This command compares files by first comparing their sizes, hence goes
reasonably fast.

When looking for identical files, \fBfinddup\fP associates a group ID
to every content, and prints it along the file names. Use the \fB-g\fP
to switch it off.

Note that
.B finddup DIR
is virtually the same as
.B finddup -i DIR DIR

.SH "OPTIONS"
.TP
\fB-h\fR, \fB--help\fR
display help and exit
.TP
\fB-d\fR, \fB--ignore-dots\fR
ignore files and directories starting with a dot
.TP
\fB-0\fR, \fB--ignore-empty\fR
ignore empty files
.TP
\fB-c\fR, \fB--hide-matchings\fR
do not show which files from DIR2 corresponds to files from DIR1
(hence, show only the files from DIR1 which have an identical twin in
DIR2)
.TP
\fB-g\fR, \fB--no-group-ids\fR
do not show the file group IDs
.TP
\fB-p\fR, \fB--show-progress\fR
show progress information in stderr
.TP
\fB-r\fR, \fB--real-paths\fR
show the real path of the files
.TP
\fB-i\fR, \fB--same-inodes-are-different\fR
files with same inode are considered as different

.SH "BUGS"

None known, probably many. Valgrind does not complain though.

The current algorithm is dumb, as it does not use any hashing of the
file content.

Here are the things I tried, which did not help at all: (1) Computing
md5s on the whole files, which is not satisfactory because files are
often never read entirely hence the md5s can not be properly computed,
(2) computing XOR of the first 4, 16 and 256 bytes with rejection as
soon as one does not match, (3) reading parts of the files of
increasing sizes so that rejection could be done with a small fraction
when possible, (4) using mmap instead of open/read.

.SH "WISH LIST"

The format of the output should definitely be improved. Not clear how.

Their could be some fancy option to link two instances of the command
running on different machines to reduce network disk accesses. This
may not help much though.

.SH "EXAMPLES"

.B finddup -p0d blah

.fi
List duplicated files in directory ./blah/, show a progress bar,
ignore empty files, and ignore files and directories starting with a
dot.

.P
.B finddup sources not:/mnt/backup

.fi
List all files found in \fB./sources/\fR which do not have
content-matching equivalent in \fB/mnt/backup/\fR.

.P
.B finddup -g tralala cuicui

.fi
List groups of files with same content which exist both in
\fB./tralala/\fR and \fB./cuicui/\fR. Do not show group IDs, instead
write an empty lines between groups of files of same content.

.SH "AUTHOR"

Written by Francois Fleuret <francois@fleuret.org> and distributed
under the terms of the GNU General Public License version 3 as
published by the Free Software Foundation. This is free software: you
are free to change and redistribute it. There is NO WARRANTY, to the
extent permitted by law.