Network shared memory - NetMem

Distributed computing


This is a piece - in fact several pieces - of software destined to programmers who want to distribute theirs software over several machines and they need a shared space over the network that can be accessed in a transactional way.
It was designed from the need of accessing some data in distributed genetic software that simulate a population of individuals that share some kind of resources, but any kind of distributed software can profits from this. Because sharing memory over IP have speed limitation, cluster environments can benefit the most of it.
The solution consists of a server part and a client part. The server part can be a standalone application that allocates and manages operations on the chunks of memory: read, lock, unlock etc. The client is an API implemented in a library that will be used in one's distributed application. The server part can also be a library if needed.
The implementation and the API are kept simple in order to get the bests benchmarks results. The C++ is the language of choice for this kind of tasks - at least the server part - but the client can be written in any language if necessary. The principle is simple: server start listening on a socket and the client connects and send commands to server in order to manipulate the chunks of memory.
First version will be basic and then next releases can evoluate as needed.

Installation

You will need CMake to build the project - the version tested is 2.6 - and the Boost library version 1.35 - this is needed because of some changes in theirs mutexes classes.
Download the sources and unzip them; the package contains three directories:
  • src - the sources for client server and some common parts - utils
  • tests - some tests for several fonctionalities - can be used to get insight of the fonctionalities
  • samples - for now just one sample - distributed integer factorization using some kind of genetic algorithm
The configuration process in CMake is straightforward: run the CMake (in Windows is a GUI, in Linux is a text-based-GUI but the fields to complete are the same) and then you will need the introduce the directories for executables and libraries (in Windows you have to complete also an output folder where CMake will put the project files). Press "Configure" (or press "c" on Linux)( ignoring the warning about CMake version) and then OK or Generate (in Linux "g") and you are done with the configuration. You now can build/edit/debug with your prefered tools on you prefered platform. The project was  tested on Windows XP/Visual Studio 2008 Express Edition and Ubuntu 8.10/Eclipse 3.2.

Version 0.1

First version is a page-based memory controller: at start-up the server will allocate a number a fixed-length memory pages and then accept commands over TCP/IP. The clients can connnect and send several types of commands:
  • get server version - to check that client ans server are synch
  • get server parameters - return the number of pages disponible on the server and the length of each one
  • read page - send the number of the wanted page and gets the data
  • lock page
  • unlock page

On the clients side there is a duplicate of the allocated server pages - a simple cache management is done: the data for a page is dowloaded only if the server page version and the client page version do not match.

The client code is not synchonized, that means that two separates threads on one client computer cannot use the same instance (socket and client-side cache) of  the client API class to connect and use the server pages. This is the major down-side of this first version that will be resolved in the future version.

Version 0.2

A copy constructor was introduced in order to use the same cache in 2 separate threads. One instance of client NetworkMemory can be used to create several copies that are thread-safe and share the same cache system. In the precedent version the thread-safe way was to create 2 instances of NetworkMemory that consume double the memory, each one with another cache for the server memory.

A new command can be used now - UpdatePage - this command does what the lock-unlock commands do in one shot.