Network shared memory - NetMemDistributed computing | |||||||||
|
This is a piece - in fact several pieces - of software destined to programmers who want to distribute theirs software over several machines and they need a shared space over the network that can be accessed in a transactional way. It was designed from the need of accessing some data in distributed genetic software that simulate a population of individuals that share some kind of resources, but any kind of distributed software can profits from this. Because sharing memory over IP have speed limitation, cluster environments can benefit the most of it. The solution consists of a server part and a client part. The server part can be a standalone application that allocates and manages operations on the chunks of memory: read, lock, unlock etc. The client is an API implemented in a library that will be used in one's distributed application. The server part can also be a library if needed. The implementation and the API are kept simple in order to get the bests benchmarks results. The C++ is the language of choice for this kind of tasks - at least the server part - but the client can be written in any language if necessary. The principle is simple: server start listening on a socket and the client connects and send commands to server in order to manipulate the chunks of memory. First version will be basic and then next releases can evoluate as needed. Installation
You will need CMake to build the project - the version tested is 2.6 - and
the Boost library version 1.35 - this is needed because of some changes in
theirs mutexes classes.
Download the sources and unzip them; the package contains three directories:
Version 0.1
First version is a page-based memory controller: at start-up the server will
allocate a number a fixed-length memory pages and then accept commands over
TCP/IP. The clients can connnect and send several types of commands:
On the clients side there is a duplicate of the allocated server pages - a simple cache management is done: the data for a page is dowloaded only if the server page version and the client page version do not match. The client code is not synchonized, that means that two separates threads on one client computer cannot use the same instance (socket and client-side cache) of the client API class to connect and use the server pages. This is the major down-side of this first version that will be resolved in the future version. Version 0.2A copy constructor was introduced in order to use the same cache in 2 separate threads. One instance of client NetworkMemory can be used to create several copies that are thread-safe and share the same cache system. In the precedent version the thread-safe way was to create 2 instances of NetworkMemory that consume double the memory, each one with another cache for the server memory. A new command can be used now - UpdatePage - this command does what the lock-unlock commands do in one shot. |