Science, Technology, and Real Estate

I regularly use 5 computers a week — one at my office at the University of Utah, one at my office at Solera Networks, one at my home office, a laptop as a research terminal, and a laptop for Solera work. This presents a serious challenge, inasmuch as there is data which I’d like to be present and modifiable in each place without worrying about copying files to-and-fro over and over again.

There are three basic solutions for a problem like this:

• Use a web storage and document platform, like Apple’s MobileMe or Amazon’s S3 service, or through a home-rolled storage platform running on my co-located server.
• Manually copy files from one computer to the others as required.
• Use synchronization software.

Because of the way I choose to use computers, there are additional constraints imposed.  First, I use Linux, which means the software I’m interested in should work in Linux.  Second, and even more important, my laptops only have periodic internet connectivity, though I want access to my documents and files all of the time, which means the documents must be cached locally.  Thirdly, I develop applications for many platforms like the GP2X, the iPhone, and the Nintendo DS — I’d like to build the development environments for each of these targets once, and let the synchronization software distribute the toolchains and source code as appropriate.  This requirement demands that synchronization software be able to handle file attributes (like the executable bit), and be able to work with many, many files.

Unfortunately for me, none of the software I’ve found fits these needs very well.  The primary contender’s that I’ve studied are: iFolder, PowerFolder, and Unision.

iFolder

Among each of the products that I tried, iFolder seemed like an immediate slam dunk.  Not only does iFolder support Linux, it supports Mac OS X and Windows 2000+.  It also stores files on the server with encryption (which is optionally not viewable by even the administrator), it operates in peer-to-peer or centralized server mode, it supports disconnected operations, and it’s a mature, Novell backed product, or so it would seem.  Unfortunately, there are some serious flaws which I encountered just during the setup process:

• iFolder is amazingly hard to compile, and the directions to do so are quite out-dated.
• iFolder setup is designed for an old version of SuSE.  None of the file-paths exist on most distributions, including newer versions of openSuSE.
• iFolder requires mod-mono and other odd-ball software extensions.
• Peer-to-peer client mode doesn’t actually work.
• Clients need big chunks of the server to compile and run.

After many hours of trying to overcome these issues and wiping my home file server to install openSuSE, I was able to get an iFolder client and server running, using the server package from the openSuSE build service and home-rolled clients based on the packaging of simias and ifolder-client in the Ubuntu Launchpad.  At least now I’d have a working solution, even if it meant I had to run openSuSE on my file server (it’s not all bad, just slow due to it’s debug kernel and access controls, and it’s non-uniform, which presents other problems).

It’s too bad, however, that things weren’t this easy.  iFolder did work properly and easily on my simple “school/” folder, which contains all of my recent school work.  After this folder, I tried to synchronize my “cabinet/” folder, which is my digital equivalent of a file cabinet.  iFolder got slower and used more memory, but was able to deal with this folder properly as well.

Finally, I tried to synchronize my “Projects/” directory, which contains a few gigabytes worth of toolchains and coding projects.  The result?  iFolder pushed my load average above 10, used all of my available memory and swap, and made the OOM killer go wild on my workstations.  Oy-vei!

And thus, we keep looking for tools.

Unison

Unison is cross platform just like iFolder, which is a plus but not a requirement.  Unlike iFolder, unison doesn’t have a persistent client which runs continuously watching for a server connection, or watching for changes.  Instead, you must run it manually whenever you want to synchronize, or use cron, scripting, or some other task to make it run periodically and automatically, and to make it support offline operations.  I wrote a simple shell script which does some of this:

#!/bin/bash
#
# Simple script to check for server connectivity and try and synchronize
# files with unison
# 

# The time to wait in-between synchronizations
SYNC_TIME=10m

# The unison profiles to synchronize
PROFILES=school cabinet Projects

# The server to check for connectivity.
SERVER=cessnaii.local

while [ /bin/true ]; do
      for project in $PROFILES; do
      	  ping -c 1 $SERVER >/dev/null 2>&1 && \
            	  unison $project -batch
      done
      sleep $SYNC_TIME
done

There are a few problems with this script, but it serves well with some caveats.  The problems:

• You must have pre-configured the unison profiles.  You can do this with unison-gtk.
• You should have ssh key exchange pre-configured.

The other problems are general problems with unison — you must store to a VFS file system, and there is a master copy.  (Alternatives might be storing to davfs, storing to a database, storing to sshfs, and similar.)

PowerFolder

PowerFolder is a commercial file synchronization software with an open source component branch.  Because PowerFolder is a java application, it works on any platform that supports java (in theory).  I did all of my testing with the open source version of PowerFolder, because I didn’t want to pay a fair amount of money for their commercial offering which has some limitations (to be discussed shortly):

PowerFolder worked somewhat well, but it suffered from similar problems to iFolder.  I won’t go into the details because I believe the free software alternatives are better than PowerFolder, but the overview is:

• PowerFolder didn’t properly handle Unix file attributes.
• PowerFolder’s memory usage grew and grew on big folders.
• PowerFolder didn’t support offline operation.
• Java requires a JVM, which typically requires a 32-bit machine in Intel land (though IcedTea and OpenJDK are correcting that, and GCJ is always a viable alternative.)

The Brave, New World

In the end, I’ve decided to write a better synchronizer, one that is smarter and easier than the current offerings, and one that doesn’t require 64GB of memory to operate on large folders! (I’ll consider 640k  as about the right amount of memory usage.)  I’m planning to implement this tool as a QT based systray applet, with the following properties:

• Automatically detect server presence, synchronize as required.
• Exteremely low CPU and memory usage.
• Simple GUI configuration tool.
• Allow encrypted storage.
• Use storage plugins, specifically a plugin for generic filesystems and a plugin for KIO slaves (which will automatically support DAV, SSH, NFS, CIFS, Cameras, etc).
• Portable between Windows, Linux/UNIX, and Mac OS X.
• Use Kde wallet and similar services for password storage as required.

Stay tuned for more updates!