Streams vs. Arrays

For in-memory allocation, the accepted approach is to be able to specify the size of the data to be stored. This is certainly the base-level requirement. The assumption is that the allocation will return the address of an array of bytes to which data can be copied, and that at some later time the data can be retrieved from the same address.

There are however problems when the size of data allocation increases. Commonly special mechanisms are used to allocate BLOBs - Binary Large OBjects. This is due to a scalability problem in the allocation and reallocation algorithms. But is also made more difficult by the direct use of the memory addresses.

Data Streams

For file-based allocation, the data cannot be directly addressed on disk as in memory - even if a virtual memory scheme was used, this is only a convenient mockup by the operating system.

Once it is recognised that data has to be read and written to the filestore it is possible to free ourselves from the expectation that file-based allocation must directly mimic in-memory allocation.

Instead, a stream can be used rather than an array. Consider the following interface:

  public interface IStore {
    public PSOutputStream allocStream();
    public PSInputStream getStream(int streamID);
    public void freeStream(int streamID);
    public PSOutputStream reallocStream(int streamID);
  }
  public interface PSOutputStream extends OutputStream {
    public int commit(); // returns streamID
  }

The idea here is that rather than first writing data to a stream and then, after determining the size of the data, to allocate the space required to save the contents, instead a "special" persistent stream is allocated, written to and then committed when complete. The commit stream method will provide the ID that can later be used to retrieve the stream for later reading or to free it.

..and the advantage?

It can be seen that in general use this is at least as convenient as array based allocation. The key advantage is in scalability.

It is now possible to represent large storage allocation, as the allocation of smaller linked allocated blocks, since the stream reader can make this appear as a contiguous stream of bytes.

This will mean that BLOB reallocation can be handled in a similar way to large file re-allocation, where a large file is split over many linked blocks on disk.

Even more important though, is that data which is essentially streamed data - such as audio and video data - can be stored and retrieved in the same store as other system objects, and within the same transaction scopes.