Index of /~matloff/R/Rdsm

 Name                    Last modified       Size  Description

 Parent Directory        22-Nov-2009 18:33      -  
 MakingPackages/         13-Nov-2009 14:59      -  
 Rdsm/                   15-Nov-2009 16:09      -  
 Timing/                 17-Nov-2009 16:56      -  
 misc/                   08-Jul-2009 13:50      -  
 old/                    04-Nov-2009 23:03      -  
 rdsm070709.tar.bz2      07-Jul-2009 23:16     8k  
 rdsm070709.zip          07-Jul-2009 23:18    12k  

Rdsm: an R Package for Distributed Shared Memory

Professor Norm Matloff
Dept. of Computer Science
University of California at Davis
Davis, CA 95616

Contents:

• overview
• quick start
• how it works
• available client functions
• future improvements

Overview:

Rdsm implements a distributed shared-memory system for R, presenting the programmer with a shared-memory view of data shared by multiple R processes running on the same or separate machines.

The performance should be similar to that of existing message-passing systems such as Rmpi, snow and ParallelR. In other words, except for clusters using special networks such as Myrinet or Infiniband, communications costs render the software suitable mainly for coarse-grained parallel applications. Possible future performance improvements are discussed below.

The advantage of Rdsm is its shared-memory view, which many in the parallel processing community prefer to message passing, as they consider the shared-memory view to make for clearer and cleaner code. Shared-memory, either with threads or with the highly popular OpenMP is the standard way of programming today's multicore machines.

It is easy to port sequential R code to Rdsm. To see this, compare the files KNN.r and KNNSeq.r in the examples/ directory.

Comments and suggestions highly welcome: Send to Norm Matloff, matloff@cs.ucdavis.edu.

Quick start:

Read and run the code in examples/MatMult1.r, which explains not only the core Rdsm features but also how to run the program. You may find it useful to run the test programs in tests/ as well.

How it works:

Rdsm operates by creating special classes for vectors and matrices (and in future versions, lists), and redefining "[" etc. on those classes. An access triggers a socket transaction to the server, where the shared data is actually stored. This is basically transparent to the application programmer. Rdsm application code looks identical to normal R code except for calls to newdsm() to create share variables, and except for the use of 0s for subscript " wild cards."

Of course, for correctness and efficiency, the programmer must be aware of the what is going on behind the scenes in Rdsm, but in composing his/code, the programmer can act as if there is a true shared-memory system in place.

Available client functions:

(See examples/MatMult1.r for usage examples.)

• init(host,port): Client joins Rdsm set with server having the specified location and port.
• newdsm(vn,type,val,size): Creates an Rdsm shared variable with the name vn, of class type, initial value val and specified size.
• barr(): Standard shared-memory barrier, a "gathering point" for all clients.
• lock(lockname), unlock(lockname) Standard shared-memory lock/unlock operations.
• fa(varname): Atomic fetch-and-add op, useful mainly if communications costs are very low.
• qput(qn,val), qget(qn) Atomic work queue ops, similar to those in Python's threading package. Again, useful mainly if communications costs are low.
• dsmexit(): To ensure a graceful shutdown of the connections and server.

Future improvements:

• "launcher," using rsh/ssh/etc. to automate the startup of clients (short term)
• remote procedure call (short term)
• higher-level operations with the "look and feel" of OpenMP (medium term)
• non-sockets server, implemented in C or maybe Unix sockets (long term)
• multiple, cache coherent servers (long term)