The read-write file store provides transaction based persistent update.
The design aim of the read-write store is to provide a lightweight but scalable file-based persistence mechanism. It can be used effectively to store either a small number of small objects or many millions of objects of various sizes totalling many terabytes of storage.
The addressing scheme is linked with the reallocation structures. This allows a 32 bit integer to reference an address via an addressing block. Essentially this means that the addressing range is not limited to a 32 bit value, but that there can be a maximum of 2000 milllion addressable storage blocks.
Another advantage of the addressing scheme is that a storage block can be freed in constant time, where other methods might require a search to find the allocation blocks.
The first four bytes of the store point to the currently committed store header.
The store header contains the meta-allocation data and values that indicate where to read the allocation blocks.
The meta-allocation data is a bit map that indicates where active allocation blocks are to be found, all the allocation blocks are at the end of the file. The idea is that the allocation blocks grow down from the end of the file, and the data grows from the start. This approach means that when the file needs to be resized - when the meta-allocation would otherwise collide with the data allocation - the allocation blocks are simply shifted to the new end of file. The meta-allocation addressing is all relative to the current end-of-file.
The allocation blocks are structures that each manage specific areas of fixed size storage allocation.
The allocatable sizes range from 64 bytes to 8192 bytes thro' powers of two, so : 64, 128, 256, 512, 1024, 2048, 4096 and 8192.
Each block allocates a number of these areas together, never in batches of less than 16K, and never less than 32 at a time. So the 64 byte allocator will grab 16384/64 blocks in a single allocation => 256 blocks, while a 1024 byte allocator will grab 32 blocks in total requiring a 32K data allocation. The minimum of 16K data allocation, means that the block can store the native address/16k, so a 32bit value can be used to indicate an address of 16K * 2000MB, or 32 thousand terabytes. The actual address the store returns is a 32 bit value where the first 16 bits indicate the allocator used, and the second 16 bits the offset into an allocator bitmap - yep, the minimum of 32 blocks is just an ints worth of bits.
Rather than use a Binary Large OBject (BLOB) allocation mechanism, instead a stream protocol is used to allocate and retrieve data.
This allows huge data streams to be stored in the file simply using linked 8K storage blocks. This does add some complexity but resolves the problems of scalability when reading in large areas, as well as removing issues of BLOB reallocation. For example, if an application continually re-wrote the same BLOB data, each time a little larger than before, conventional BLOB structures would never be reused, and the storage requirements would grow in proportion to both the size and the frequency of updates, while using the new stream allocation scheme means that the store will simply grow in order to accommodate the growing data.