Esync files are encrypted files which can be efficiently transferred using the rsync algorithm, see esync-whitepaper.html
An Esync file conventionally has the same name as the original
file with .e suffix.
An Esync file conventionally has the same timestamp as the original file; this makes unchanged file detection easier.
An Esync file size is between a few bytes longer and 1%+30 bytes longer than the original file. This can be used with the timestamp to validate unchanged files are of a legal size.
The file consists of 4 sections:
|
Magic (4 bytes) |
|
Encrypted file |
|
Cipher Use table |
|
Cipher Use table size (4 bytes) |
An esync file begins with a 4 byte magic number: The first 4 bytes are 45 53 59 1F.
The file, encrypted with the cipherstream, is in the second section. There are no framing bytes, this section is identical in size to the original file. Details of the cipher used on this section is specified in the next section.
Each block being encrypted is described by a USE record (see below).
Each 16 bit item in such a block, X[i], is the sum of the plaintext P[i] and the cipher stream C[i]. The elements are little endian because that makes the case of odd-size blocks easier to handle. If the block size is odd, the last element is only a single byte.
The Cipher Use Table (sometimes just called the Use Table) is a table of USE records and other records which record which part of the cipher stream is used to encrypt which parts of plaintext file. The cipher stream is broken down into sections of 216 bytes, each with a separate key. Up to 232 keys may be used. Thus a location in the cipher stream can be specified using a 32 bit keynumber and a 16 bit offset.
All multi-byte values in this table are in big endian format (ie, network byte order).
The Cipher Use Table consists of records of the form <RecordType> <details>
RecordType is a 1 byte field with values:
enum { USE0 (0) ... USE120 (120), USEB, USEW, USEL, CIPHER,
NEXTL, END (126), USE0Z (128) ... USE120Z (248), USEBZ, USEWZ, USELZ
}
The first record is always of type CIPHER.
The last record is always of type END which takes no parameters.
CIPHER <ciphertype: byte> <xkey: byte[16]>
ciphertype is a 1 byte field describing the cipher type. Currently only type 1 (RC4-MD5) is defined. For ciphertype 1 records there is a 16 byte xkey specified which is appended to the 16 byte MASTER KEY (a secret key not stored in the file) and the resulting 32 byte object is digested using the MD5 message digest to produce a 16 byte xmasterkey which is used as an RC4 key to produce a stream of 16 byte keys. Each key so generated is used as an RC4 key and provides 216 bytes of the cipherstream.
It is possible (but rare) for additional CIPHER records to be present in the file. Subsequent records have an extra field keynumber:
CIPHER <keynumber: uint32> <ciphertype: byte> <xkey: byte[16]>
This record defines all keys from keynumber and up such that the first key this cipher generates is key keynumber. Previous CIPHER blocks define keys below keynumber.
USEn <offset: uint16> <bytes: uint16>
USEB <keynumber: byte> <offset: uint16> <bytes: uint16>
USEW <keynumber: uint16> <offset: uint16> <bytes: uint16>
USEL <keynumber: uint32> <offset: uint16> <bytes: uint16>
USEnZ <bytes: uint16>
USEBZ <keynumber: byte> <bytes: uint16>
USEWZ <keynumber: uint16> <bytes: uint16>
USELZ <keynumber: uint32> <bytes: uint16>
The USEn fields (USE0...USE120) define use for keynumber 0 to 120 respectively. USEB fields have an explicit byte to encode the keynumber, likewise USEW and USEL have longer fields. All these take an unsigned 16 bit integer (uint16) offset into the cipherstream and a uint16 number of bytes to be encrypted with that stream. The bytes field encoding of all zero represents 0x10000 bytes.
USExZ fields are like the variants without Z but they omit the offset, assuming it to be zero, since this is a common case.
A USE field encyphers the next bytes of plaintext with cipher key keynumber starting at offset offset. USE records are put into the table in the order of the plaintext (and ciphertext), however the cipherstream
NEXTL <keynumber: uint32> <offset: uint16>
NEXTL is used to indicate the next available part of the cipher stream. It is not usually encoded in the Use Table because it is not used for decryption and information supplied from an encrypted file on remote machine cannot be trusted for encryption purposes. This record is often used in local caches to indicate which parts of the cipherstream have already been used and therefore ensure no part is reused (which can compromise security).
END
END Indicates the end of the Use Table, it may occur only once.
A 4 byte, big endian, field giving the size of the Cipher Use table + Cipher Use Table Size field. These are the last 4 bytes in the file and they allow the start of the Cipher Use table to be located.
[ This section doesn't strictly belong here, but it might be useful. ]
Once transferred to a remote machine, unencrypted fields in the encrypted file should not be trusted by any encryption program. In particular the xkey specified in the CIPHER record and the next keynumber and offset sepecified in NEXTL should not be trusted: these could be faked in permit an attack on the cipher – no part of the cipherstream should be reused to encrypt different data.
Of course you can use the remote file's xkey to decrypt the file – that's why it's there. Just don't use it to encrypt.
Since we cannot trust the remote fields, it is useful to keep a local cache of the CIPHER and NEXTL records for each file transferred. If this is missing, a new xkey must be randomly generated, and only that be used to encrypt new data.
The local cache file has magic 45 53 1E 59, followed by exactly one CIPHER record and exactly one NEXTL record.
On Unix / Linux machines this cache is typically under
/var/cache/esync/fullpathname. These files may be
deleted, but such files should never be copied since this gives the
chance of an attack on the cipher.
Here's a simple example of a 5 byte file containing hello
being encrypted. The result is a 35 byte file.
45 53 59 1F – magic number
34 FE 84 94 94 – ciphertext (ie, encrypted
plaintext).
FC 01 11 22 33 44 55 66 77 88 99 AA BB CC DD EE FF 00
– CIPHER Type=01 xkey
80 00 05 – USE0Z 5
FE – END
00 00 00 1A – size field (26 bytes: 22 bytes of Use
Table plus these 4).
Written by Andy Henson , February 2004. Copyright © 2004 by Zexia Access Ltd
If you want to contact me regarding this, please email <esync@zexia.co.uk>
Please feel free to link to this document.