Re: Berkeley UPC on Cray XD1

From: Adam Leko (leko_at_hcs_dot_ufl_dot_edu)
Date: Sat Oct 07 2006 - 23:30:41 PDT

  • Next message: Dan Bonachea: "Planned downtime for & NERSC (October 30-31)"
    After some quick digging, it looks like there is a userspace library 
    provided for accessing the RapidArray stuff:
    The hardware uses GART to automagically map RDMA accesses to physical
    addresses.  After glancing over the /usr/local/include/rapl.h file, it
    seems like it might be possible to create a rapl GASNet conduit, but it
    might take a bit of work to do so.  There also doesn't seem to be any 
    docs available other than that header file :-(
    Since I was able to get a simple test case to show the really really bad
    MPI conduit performance on the XD1, I'll go ahead and file a bug for the
    On Sat, Oct 07, 2006 at 06:22 PM, Adam Leko wrote:
    > What's the best way to run Berkeley UPC on a Cray XD1?  I notice from 
    > the GASNet conduit there is support for the MPI and SMP conduits.  I 
    > think the built-in MPICH on the XD1 takes advantage of the RapidArray, 
    > and it seems to have pretty good performance (advertised 1.7us latency, 
    > and on a chassis I've seen even less in unscientific measurements).  The 
    > MPI conduit currently has OK performance but has some anomolies which 
    > might be due to the polling-based implementation interacting with the 
    > specialized MPICH version.
    > Are there any other options that better suit the XD1 hardware?  I'm not 
    > sure if RapidArray is going to make it into any future Cray products, 
    > but if so it seems worth supporting.  Our lab is interested because we 
    > might be doing something with both UPC and reconfigurable computing, and 
    > a platform with excellent RC and good UPC support is interesting.
    > Some notes: The Cray website claims support for SHMEM, but it is
    > provided via GPSHMEM and a BUPC -> SHMEM conduit -> GPSHMEM -> ARMCI
    > solution seems kind of gross.  There's also a "Global Arrays" interface
    > which looks like it is also layered above ARMCI.  However, either
    > option seems better suited than the MPI conduit.
    > (As a side note, anyone with an HCS lab account can access our XD1 at 
    > but let us know before you pound it with long jobs)
    > -Adam Leko

  • Next message: Dan Bonachea: "Planned downtime for & NERSC (October 30-31)"