From: Adam Leko (leko_at_hcs_dot_ufl_dot_edu)
Date: Sat Oct 07 2006 - 23:30:41 PDT
After some quick digging, it looks like there is a userspace library provided for accessing the RapidArray stuff: https://lists.scl.ameslab.gov/pipermail/xd1-kernel/2005-June/000000.html The hardware uses GART to automagically map RDMA accesses to physical addresses. After glancing over the /usr/local/include/rapl.h file, it seems like it might be possible to create a rapl GASNet conduit, but it might take a bit of work to do so. There also doesn't seem to be any docs available other than that header file :-( Since I was able to get a simple test case to show the really really bad MPI conduit performance on the XD1, I'll go ahead and file a bug for the problem. -Adam On Sat, Oct 07, 2006 at 06:22 PM, Adam Leko wrote: > What's the best way to run Berkeley UPC on a Cray XD1? I notice from > the GASNet conduit there is support for the MPI and SMP conduits. I > think the built-in MPICH on the XD1 takes advantage of the RapidArray, > and it seems to have pretty good performance (advertised 1.7us latency, > and on a chassis I've seen even less in unscientific measurements). The > MPI conduit currently has OK performance but has some anomolies which > might be due to the polling-based implementation interacting with the > specialized MPICH version. > > Are there any other options that better suit the XD1 hardware? I'm not > sure if RapidArray is going to make it into any future Cray products, > but if so it seems worth supporting. Our lab is interested because we > might be doing something with both UPC and reconfigurable computing, and > a platform with excellent RC and good UPC support is interesting. > > Some notes: The Cray website claims support for SHMEM, but it is > provided via GPSHMEM and a BUPC -> SHMEM conduit -> GPSHMEM -> ARMCI > solution seems kind of gross. There's also a "Global Arrays" interface > which looks like it is also layered above ARMCI. However, either > option seems better suited than the MPI conduit. > > (As a side note, anyone with an HCS lab account can access our XD1 at > xd1.hcs.ufl.edu but let us know before you pound it with long jobs) > > -Adam Leko >