Internal File Format

The data in OnePile is synchronized using a specific file format. On Apple devices the inner folder structure is hidden by the use of package flags but in fact it is just a structure like this:

Example.onepile/
  assets/
  transactions/
  info.json

The info.json file containes meta data like the databaseID which is also a synonym for the internal workspace ID.

Instances

When you look into the assets or transactions folder you will find unique ID (UUID) like named folders. Each of therese IDs represents a local instance ID. That means when you start the OnePile app on a device this device will get a UUID and start writing into such a folder when conten changes. And it will only write into this folder and no other! This way conflicts are avoided.

Inside the folders an incremental numbering applied in order to have expectable names for easier synching on dumb services. For each 1000 files a new folder is created. To distribute folders and files equally the following algorithm is used to create folder and file structure, where n is the incrementing number:

function getDeepNames(n: number, max: number = FOLDER_MAX): Array<string> {
    if (n === 0) {
      return ['1', '0']
    }
    let p = []
    while (n > 0) {
      p.unshift(Math.floor(n % max).toString())
      n = Math.floor(n / max)
    }
    p.unshift(p.length.toString())
    return p
  }

Assets

Larger binary data like files are saved separately as assets to avoid the database to overload. Detailed information about their content type and name are stored in transactions.

Transactions

This ist the heart of the storage. Each change step in OnePile creates a new entry here. The structure of these entries is inspired by blockchains. A typical entry looks like this:

SEA
{"i":1,"p":"MVvrN5VpUrc3PwTG7zKWf+L0Tr+8GndOkAtAngc3iYA=","c":"qAJbqhdroPsRXjMIVEQSn37WufMPWPGw703+bwLXnN0=","t":1567758506}[
  {
    "_cid" : "6dda519070b2437f89a60285221e85b0",
    "_id" : "S-JrCZq0TuuPlJCBcNozBA",
    "_index" : 0,
    "_time" : 1567758506,
    "$set" : { 
      "title" : "Test"
    }
  }
]

The first line is the identifier, it is always SEA\n.

The next line containes the block properties, where i is the index and should be identical to the previously described n of the file name. p is a SHA256 checksum of the previous transactions data i.e. of i - 1. c is the checksum of the body of this transaction. And t is the timestamp when the transaction was created.

The follwing lines are the body of the transaction containing the actual change data. Describing this would prbably exceed the scope of this document but as you can see it is in JSON format and besides some references to database entries like id it contains change information like the one in $set. The pattern here is pretty simple to understand and aims to makes sync quality best it can be.

Encryption

If the user decides to encrypt the data this will affect all files in assets and transactions folder. All relevant details enable encryption are stored in info.json. Example:

{
  "cryptoInfo" : "5ejE\/aEuddTjdBGx3X7mmTDVhmdUF6DQghJYACwTl420aw2rslm7TzC3nGRUROxmnBVhEBwrVBj6UMKVn9sKBQ==",
  "cryptoIterations" : 500000,
  "cryptoMode" : "AES256-CBC",
  "cryptoSalt" : "2hHeg+WXDTSFJBVjjKdcHuvXwF5jMG88y2LymL4MxOSJ361np5rr6Y+QKntTebGNcGZ8jZG8\/wvqZ4oqRIIZig==
}

In this case AES256 is used with CBC mode. The key is secured by PBKDF2 with 500000 iterations. The used salt data is available as well as the cryptInfo. This contains the password encrypted by itself as a check for password correctness.

The encryption is than applied on the files as a whole in the content files. More transformations of the binary data like compression are to come in future releases.