/ Folder Synchronizer Programs Review

I need a program to keep files in sync on two machines using a USB pen drive. This is a review of the programs I looked at.

introduction
features
feature descriptions
algorithms
programs
summary
see also
other programs
other features
review updates

introduction

Synchronizer programs are typically used for:

working on the same set of files on a laptop and on a desktop, where both boxes are not connected all the time (in which case you can use shared folders)
working on the same set of files on multiple machines (e.g. work & home), moving files between them with a USB key drive, or writable CD

To work on the same set of files they should be kept 'in sync', i.e. any changes made in any file on one source ('source' in this page means a set of files stored on one machine or media) should be propagated to the other, including deleting or creating files.

These programs are different from mirroring tools, which always update one machine from another. (i.e. they assume files can only change on one box)

In cases where multiple files may be changed but only one at a time, or by different people, consider using a versioning system.
Where one copy of the files is treated as a 'master' copy, use a mirroring tool.

features

Here 'Y' means it has the feature, 'N' means it doesn't and '?' means I don't know. They are listed roughly in reverse order of importance.

	program
feature	unison	Xfiles	directorysync	jfilesync	FileSync/CmdSync(5)	foldermatch	HAS	AllwaySync	synctoy
update detection	Y	N	N	Y	N	N	N	Y(6)	N(4)
standalone	Y	N	Y	Y	Y	Y	Y	Y	Y
client server over IP	Y	Y	N	Y	N	?	N	N	N
GUI	Y(1)	Y	Y	Y	Y	Y	Y	Y	Y
open source	Y	Y	Y	Y	N	N	N	N	N
cli	Y	N(2)	Y	Y	Y	N	Y	N	N
Linux	Y	N	Y	Y	N	N	N	N	N
'token' mode	N	N	N	N	N	N	N	N	N
client-server compressed protocol	Y	N	N	N	N	?	N	N	N
client-server encrypted protocol	N	N	N	N	N	?	N	N	N
>2 machines	Y	N/A(3)	N/A(3)	?	N/A(3)	N	N	Y	N
select single files	Y	N	N	N	?	?	N	?	N
use non RW media	N	N	N	N	N	N	N	N	N
preview mode	N	?	N	Y	?	Y	?	?	Y
unicode	N	?	?	Y	N	?	?	Y	Y
price ($)	0	0	0	0	15	35	35	20(7)	0

1 only for some configuration options and sync results
2 accepts parameters but launches gui anyway
3 not relevant for programs that don't detect conflicts
4 uses update detection algorithm to detect deletions but not file updates
5 CmdSync is the command line version, FileSync is the gui version
6 The AllwaySync developer describes the update algorithm as also used by unison and jfilesync as an "innovative, proprietary algorithm" and says "This proprietary algorithm was developed by a team of prominent mathematicians, and it's currently being prepared for patent filing". He could of saved himself the 'prominent mathematicians' fees by just reading the unison source code.
7free for personal use

feature descriptions

standalone

can operate as a single program (more convenient when used with a removable disc or machines directly connected).

gui

has a graphical user interface (i.e. a window).

cli

command line interface - can be used from scripts / batch files.

linux

can run on linux as well as windows - all the programs that can run on linux can also run on Macs.

client server over IP

can operate as two separate processes to sync sources over IP ( e.g. the internet or an intranet). Standalone programs can also sync boxes using 'shared folders', but client server programs can normally be used through firewalls, and are more secure.

client-server compressed protocol

uses a compressed protocol when syncing over IP. (for example the rsync protocol that only sends diffs of files) this reduces the time it takes to sync.

client-server encrypted protocol

If files are transferred over the internet without encryption, anybody in the right place with a sniffer can get read them. For programs without this feature, the connection should be tunnelled through ssh.

update detection

This means that the program will distinguish between cases where a file has been changed on both sources since the last sync (a 'conflict'), from where it has only been changed on one source (an 'update'). Typically if it has only been changed on one source the program can automatically copy it, but if it has been changed on both it will prompt the user to merge the files. Because almost all file differences are 'updates' rather than 'conflicts' this greatly reduces the number of manual file compares you have to do. In a similar way the program can automatically delete or create files, when needed. Some of the programs reviewed will simply overwrite the least recently modified file when there is a conflict (see algorithms), losing any changes. Without this feature, you either have to manually check each file that has changed to make sure there are no changes in the older file but not in the newer, or risk losing changes. Also called 'automatic' mode

Note on SyncToy

The SyncToy description (available here) describes the update detection algorithm (as used by unison & jfilesync) but only describes it being used for detecting file deletions.
So I did an experiment with synctoy where I changed the same file in both synchronised folders, and I can confirm it does not do update detection for changed files. Even worse the program silently overwrote the older file (instead of telling me there was a conflict), losing my changes. It's a pity MS have done this, as it would take a small change in the program to detect updates as well as deletions, which would make the program almost as powerful as jfilesync

more than 2 machines

All these programs can individually sync any pair of sources, and so can potentially keep multiple sources in sync. by merging two at a time.
But there is a potential problem doing this with a removable disc. Namely, that if more than one disc is mounted under the same drive letter or directory, then the 'update detection' of the program may get confused.
A way to avoid this is for the program to store a unique identifier in the root of any directory tree to be synchronized.
This is not a problem with programs that do not detect updates and do a fresh compare of the directories each time.

tokens mode

This feature is described in my post to the unison mailing list:

I am syncing two locations via a USB flash drive (i.e. moving it between the machines and syncing each with the drive).

The flash disc is 128 MB, but I want to sync directories of about 1GB

This is not possible at the moment as unison stores a copy of all files on the USB drive.

It should not need to do this - it could tell if a file has changed just by storing the filename & date (and perhaps checksum).

So it would be nice for unison to optionally use a different algorithm when using a removable drive.

This algorithm would look like this:

Each file can be represented by a 'token' that consists of the filename, checksum & date.

On the removable disk, it stores the directory structure as before, but all files are normally replaced by their tokens.

When unison syncs, if it detects the file token is older than the file being synced with, it copies the actual file onto the disk in place of the token. If it detects an actual file on the removable disk and this is newer than the other file, then it replaces the other file with the file on the removable disk & then replaces the file on the removable disc with a token.

You could extend this to syncing > 2 sets of directories by storing 1 token for each set being synced.

This is not implemented in any of the reviewed programs, however the developer of jfilesync says he will implement this in future.

select single files

Some of these programs only allow directories to be selected directly. Although they all allow wildcards, so this limitation can be worked around.

use non RW media

The programs reviewed only allow syncing directly to a rewritable media; in other words the sources are written to directly.

Where a removable disc is used to move data between 2 machines, it would be convenient to be able to use media like CD-Rs or CD-RWs that cannot be written to on the fly.

This is not implemented in any of the reviewed programs, however the developer of jfilesync says he will implement a feature trhat will enable this in future.

open source

Open source programs use a development method that means it is easier for users to find out about any bugs and these are more likely to be fixed

preview mode

Outputs or shows what changes it will make, without making them.

unicode

Can synchronise file names with characters in non-latin alphabets, e.g. japanese.

algorithms

In a classic example of the maxim that "for any sufficiently complex problem there is a solution that is easy, simple, obvious and wrong", most of these programs just compare the file timestamps and overwrite the least recently modified file. Although some of them prompt the user to merge the files when there are timestamp differences.

This means you will either lose data if there are conflicts, or have to manually check all different files for conflicts (see update detection).

To detect updates (i.e. where only one copy has changed since the last sync) there is a need to keep a record of the timestamp of both versions of a file on the last sync. Ideally these records are stored in the source directory but they can also be in an 'application data' directory (as in unison).

Unison, at least, also stores a checksum and file attributes with these records.

To deal with multiple sources, each source should have a timestamp for each file for each other source that it may be synced with.

This algorithm can easily be be adapted for tokens.

The token algorithm can be enhanced by splitting big files into sections each of which has a token. this will enable files bigger than the media size to be synced (by only moving parts that have changed).

programs reviewed

unison
1. The developers state no new features will be added to the program.
2. The gui is quite basic, and you will need to edit configuration files as well.
3. uses the obscure OCaml language, so may be more difficult to customise if needed.
4. Uses efficient rsync protocol to sync over a network.
xfiles
jfilesync
1. needs Java 5.0, which will mean another download for most users.
2. GUI has a few quirks - e.g. you have to click 'compare' to get it to show conflicts
3. you can't launch a 'diff' program from within the GUI
4. easy to use
download here There is another program by the same name at http://home.arcor.de/mschierlm/sm-soft/jfilesyn-en.htm. The one reviewed is at the link above
directorysync
FileSync
foldermatch
HAS (Heatsoft Automatic Synchronizer)
Allway Sync
1. No mailing lists, public bug database, or other user community.
2. the only reviewed proprietary program with update detection
3. allways sync is also known (or was previously known) as GoodSync
4. note mispelling in name
MS synctoy
1. Windows XP only
2. detects rename
3. uses the update detection algorithm for deletions only, not updates. Overwrites edits in the older file when there are conflicts
4. requires .net to be installed
5. 512Mb ram & a P4 is recommended
6. requires you to download an additional program to 'validate' you have registered your installation of XP
7. only allows a single pair of folders per profile

summary

There are some features that you would expect to see in these programs that are in none - for example being able to use a non re-writable media like a cd-rw (not in packet writing mode) to synchronize two machines. Other basic features (namely client server / update detection) are only in a few programs.

I think this is because it is easy to write a simple directory differencer, so many would-be shareware developers use it as 'My first Program'.

Unfortunately the simple, obvious way of doing this is the wrong way.

In both these respects it is similar to duplicate file finders.

As usual, the open-source programs might not have the pretty GUIs of the proprietary ones, but get more of the basic features right.

If more than about a dozen files are changed with each sync, then any sync program is effectively unusable without update detection. Currently the only 3 programs that support this are unison, jfilesync & AllwaySync.

Of these 3, AllwaySync does not have a client/server mode, does not have the user community of the other 2 programs (so is more likely to have unfound bugs) and is closed source.

The unison developers state no new features will be added to the program. The unison gui is quite basic, and you will need to edit configuration files as well. Unison uses the OCaml language, so may be more difficult to customise if needed.

jfilesync needs Java 5.0, which will mean another download for most users.

JFileSync has a few UI quirks but overall it works well.

So for most purposes I recommend jfilesync.

If you need to sync a large amount of data over the internet, use unison instead because of its more efficient compressed protocol.

other features

There are some features that I have not covered in this review (often because no programs have them) but might be desirable, these are:

being able to run in the background and update files as they change
grokking zip or tar archives
detecting moved files

review updates

Since the initial draft of this page some people have pointed out that Easy2Sync also does update detection. I have had a quick look at this program and can confirm that it does do this and is very user friendly, although probably does too much hand-holding for serious use.

other programs

There are hundreds of syncing programs available. Google returns 25000 results for '"synchronize directories" software'.
And there are many more mirroring and copying tools.

I only looked at the first ten results I found on sourceforge and the the first ten commercial programs found on a search.
Below are the other programs I looked at but which offered too few features or relevance to justify a full review
CopyTo synchroniser
DCW
Active File Compare
idem
treediff
Twin Folders
Easy2Sync

These are some other results my searches turned up, but which I have not looked at:
viceversa
Visual Merge
Directory Toolkit
http://www.peersoftware.com/
SyncFolder
SyncPro Backup
vuBrief
XXCOPY
http://www.grigsoft.com/winsin.htm
powerfolder
dirsync
filemirror
weex
tassiekey
workbag
srt
IBM Lotus Notes apparently can also do syncing.

home (frames) home (no frames)