Computing has developed with an acceptance that memory is split into at least two types. Tape/disk storage that can be saved and reloaded, and computer main memory, used as fast access working storage. Data is typically moved from semi-permanent storage, into main memory, and then after some processing, saved back to the semi-permanent storage system.
But need it always be this way?
Many new memory technologies under current development, promise both enormous capacity and non-volatility.
This is extremely attractive for emerging technologies such as video and audio processing, since it will enable large, rapid access storage, with minimal power requirements - no disks to spin and no memory to keep charged.
But this new technology also promises to transform the way computer systems can be built.
There are currently a range of solid state memory devices, from EEPROM chips, to compact flash devices, memory sticks and a number of variations.
Some manufacturers are now pruducing solid state disks. From the USB based pen-drive to the IDE and firewire interfaces to solid state disks.
At present, pretending to be a disk is the simplest way to "slot in" a solid state store. The performance improvements can be enormous. Mainly due to the minimal time to access data.
While this is in itself a good idea, the real advances will come when interfaces are provided to allow applications to address data directly. This is because most computer operating systems and system utilities make a lot of effort to improve what would otherwise be very slow access to data. This is achieved by various caching and buffering strategies.
Most disk drives provide some level of caching - a copy of data than can be accessed relatively quickly and conveniently.
Operating systems then provide a file cache, essential for performance of "modern" applications and their resource hungry designs.
Within language libraries different file access functions provide further buffering.
..and lastly the application will itself keep copies of data in various quantities for performance as well as for functional reasons.
The point here is that it should be possible to simplify systems at a number of levels if the access to the persistent storage media can be assumed to be high performance. Or more to the point, high enough that the compexity and resource overhead of maintaining a cache is not worth it.
With the Cut The Crap object model the object manager maintains a cache of directly usable objects, so there would be no need for any other buffering support. Object representations could be moved directly between the in-memory structures and the serialized external representation.
This is what I am waiting for.
A direct Firewire type interface to external solidstate storage.
At present the main performance problem with solid state devices is the data write and access speed. Typically, write speed is ~1-2Mb per second, and read ~2-4Mb per second.
But, since the access performace is so quick, these devices can be arranged in parallel - say 16 devices, striping data in blocks of 64bytes?
So block of data any larger than 16 * 64 bytes could be written and read 16 times faster.
A 2K object could easily be retrieved in less than 10 microseconds.
All the time the raw performance is increasing anyhow.
Given the advantage of access times measured in microseconds, Random access to large amounts of randomly accessed data can easily be serviced.
Other RAID techniques can be used to provide data resilience at various levels.
Relational database performance is linked closely with data access.
Relational databases perform best when the table data is sorted and stored contiguously on the disk store. This is a major reason why database are presized.
The search/query algorithms are then tuned so that table data is read in streams. The objective is to minimise the amount of "disk head" movement - and maximise data throughput.
In Object databases, each object is "normally" stored as a single chunk of data, but reading in lists of objects usually requires rapid random access to different areas of the store.
so....
With solid state storage, although both Relational and Object storage systems will both have improved performance, the relative benefit for object based storage will be huge.
While providing a direct access to the non-volatile store is a good step, it really gets interesting when all memory is non-volatile.
How could operating systems and applications change?
Clearly, they need not change. Different memory areas could be designated as "disk" store, and "working" store. But it seems to me that there is a huge opportunity here.
If anyone has any ideas they'd like to share here, please let me know.