The rsync Process
Articles and Tips: tip
01 Feb 2003
The rsync open-source utility is based on the highly efficient rsync algorithm, which updates files on one computer so that they are identical with the files on another computer. The two computers are linked by a low-bandwidth, high-latency, bidirectional communications link.
To perform this feat, the algorithm computes which parts of a source file identically match parts of a destination file. The algorithm sends across the link only the parts of the source file that do not identically match parts of the destination file. The server with access to the destination file then reconstructs an identical copy of the source file.
To give you a better (but still basic) idea of how this process works, suppose you have two computers, Computer A and Computer B. Computer A has access to File 1 (the source file) and Computer B has access to File 2 (the destination file), and the two files are similar.
The file-transfer process begins when Computer B splits File 2 into a series of non-overlapping fixed-sized blocks. For each of the blocks in File 2 (the destination file), Computer B calculates two checksums (one weak, one strong) and sends these checksums to Computer A.
Computer A searches File 1 to find blocks that have the same weak and strong checksum as the blocks that Computer B sent. This is done in a single pass.
Computer A then sends Computer B a sequence of instructions for constructing a copy of File 1. Each instruction is either a reference to a block in File B or actual data. In other words, the algorithm sends over the link only the blocks from File 1 that do not match blocks in File 2 plus a small amount of data for checksums and block indexes.
Computer B uses the information from Computer A to modify File 2 to be an exact copy of File 1. The process requires only one round trip, which minimizes the impact of the link latency.
* Originally published in Novell Connection Magazine
Disclaimer
The origin of this information may be internal or external to Novell. While Novell makes all reasonable efforts to verify this information, Novell does not make explicit or implied claims to its validity.