From: Paul H. Hargrove (PHHargrove_at_lbl_dot_gov)
Date: Mon Jan 07 2008 - 15:04:18 PST
Steven,
I am bcc:ing this reply to a former member of the Berkeley UPC team who
is now at QLogic. He may respond with additional information.
I am the author of the ibv-conduit code in Berkeley UPC/GASNet. I am
afraid that I have not encountered the specific error you see. However,
I see that you are using the InifiPath adapters, which I am not certain
support a full implementation of the OpenIB verbs. Certainly all the
MPI implementations that support QLogic's adapters have been modified to
use their "PSM" interface rather than IB Verbs.
The only thing that I can suggest is to ensure that running
"ibv_devinfo" produces output something like the following:
$ /opt/ofed/bin/ibv_devinfo
hca_id: mthca0
fw_ver: 1.2.0
node_guid: 0005:ad00:0005:85a4
sys_image_guid: 0005:ad00:0005:85a7
vendor_id: 0x02c9
vendor_part_id: 25204
hw_ver: 0xA0
board_id: MT_0230000002
phys_port_cnt: 1
port: 1
state: PORT_ACTIVE (4)
max_mtu: 2048 (4)
active_mtu: 2048 (4)
sm_lid: 1
port_lid: 1010
port_lmc: 0x00
If ibv_devinfo fails, then that means that the OpenIB verbs support is
not present. If you do get output like the above, but don't see
"PORT_ACTIVE" then there is probably some configuration problem. If you
do get output that indicates at least one ACTIVE port, then we can start
looking at GASNet details to figure where the problem lies.
-Paul
Steven D. Vormwald wrote:
> Hello,
>
> I'm having trouble using the ibv-conduit in the 2.6.0 release:
>
> [sdvormwa@gilbert UPC]$ GASNET_BACKTRACE=1
> /usr/local/berkeley_upc-2.6.0/bin/upcrun -np 1 ./a.out
> libibverbs: Warning: no userspace device-specific driver found for
> /sys/class/infiniband_verbs/uverbs0
> GASNet gasnetc_init returning an error code: GASNET_ERR_RESOURCE
> (Problem with requested resource)
> at
> /usr/local/src/berkeley_upc-2.6.0/gasnet/vapi-conduit/gasnet_core.c:986
> reason: unable to open any HCA ports
> GASNet gasnet_init_GASNET_SEQFASTnodebugnotracenostatsnosrclines
> returning an error code: GASNET_ERR_RESOURCE (Problem with requested
> resource)
> at
> /usr/local/src/berkeley_upc-2.6.0/gasnet/vapi-conduit/gasnet_core.c:1546
> UPC Runtime: GASNet error GASNET_ERR_RESOURCE(Problem with requested
> resource): gasnet_init failed!
> [0] 0: /home/sdvormwa/UPC/./a.out [0x440913] gasneti_bt_execinfo ??:0
> [0] 1: /home/sdvormwa/UPC/./a.out [0x442db3] gasneti_print_backtrace ??:0
> [0] 2: /home/sdvormwa/UPC/./a.out [0x404d14] upcri_gaserr ??:0
> [0] 3: /home/sdvormwa/UPC/./a.out [0x4288bc] bupc_init_reentrant ??:0
> [0] 4: /home/sdvormwa/UPC/./a.out [0x403591] main ??:0
> [0] 5: /lib64/tls/libc.so.6(__libc_start_main+0xdb) [0x3ca971c3fb] ??
> ??:0
> [0] 6: /home/sdvormwa/UPC/./a.out [0x40318a] _start ??:0
> [sdvormwa@gilbert UPC]$
>
> openib was installed using prebuilt packages, configured and tested as
> documented in the user manual that came with the infiniband cards (see
> download.qlogic.com/manual/62612/IB6054601-00E.pdf). The libibverbs
> documentation for the warning says to ensure that
> /usr/local/lib/infiniband/mthca.so exists and is executable, which I
> have verified on all the nodes. I've run through the rest of the
> included troubleshooting documentation, and have yet to find a
> solution. Any help would be much appreciated.
>
> Thanks,
> Steven Vormwald
>
>
> This is upcc (the Berkeley Unified Parallel C compiler), v. 2.6.0
> (getting remote translator settings...)
> ----------------------+---------------------------------------------------------
>
> UPC Runtime | v. 2.6.0, built on Dec 3 2007 at 13:21:31
> ----------------------+---------------------------------------------------------
>
> UPC-to-C translator | v. 2.6.0, built on Oct 15 2007 at 15:50:19
> ----------------------+---------------------------------------------------------
>
> Translator location | http://upc-translator.lbl.gov/upcc-2.6.0.cgi
> ----------------------+---------------------------------------------------------
>
> networks supported | udp smp ibv
> ----------------------+---------------------------------------------------------
>
> default network | ibv
> ----------------------+---------------------------------------------------------
>
> pthreads support | available (if used, default is 2 pthreads per
> process)
> ----------------------+---------------------------------------------------------
>
> Configured with |
> '--with-translator=http://upc-translator.lbl.gov/upcc-2
> | .6.0.cgi' '--enable-ibv' '--verbose'
> | 'IBV_LIBS=-libverbs -ldl' 'LDFLAGS=-ldl'
> | '--without-mpi-cc' '--disable-mpi'
> | '--prefix=/usr/local/berkeley_upc-2.6.0//opt'
> | '--with-multiconf-magic=opt'
> ----------------------+---------------------------------------------------------
>
> Configure features | berkeleyupc,upcr,gasnet,upc_collective,upc_io,
> | upc_memcpy_async,upc_ptradd,upc_thread_distance,
> | upc_tick,upc_sem,upc_dump_shared,upc_trace_printf,
> |
> upc_trace_mask,upc_local_to_shared,upc_atomics,pupc,
> | upc_memcpy_vis,nodebug,notrace,nostats,nogasp,
> | segment_fast,os_linux,cpu_x86_64,cpu_64,cc_gnu,
> | packedsptr
> ----------------------+---------------------------------------------------------
>
> Configure id | gilbert.cse.mtu.edu Mon Dec 3 13:16:56 EST
> 2007 root
> ----------------------+---------------------------------------------------------
>
> Binary interface | 64-bit x86_64-unknown-linux-gnu
> ----------------------+---------------------------------------------------------
>
> Runtime interface # | Runtime supports 3.0 -> 3.9: Translator uses 3.6
> ----------------------+---------------------------------------------------------
>
> | --- BACKEND SETTINGS (for ibv network) ---
> ----------------------+---------------------------------------------------------
>
> C compiler | /usr/bin/gcc
> | GNU/3.4.6/3.4.6 20060404 (Red Hat 3.4.6-3)
> | gcc version 3.4.6 20060404 (Red Hat 3.4.6-3)
> | Reading specs from
> | /usr/lib/gcc/x86_64-redhat-linux/3.4.6/specs
> | Configured with: ../configure --prefix=/usr
> | --mandir=/usr/share/man
> --infodir=/usr/share/info
> | --enable-shared --enable-threads=posix
> | --disable-checking --with-system-zlib
> | --enable-__cxa_atexit
> --disable-libunwind-exceptions
> | --enable-java-awt=gtk --host=x86_64-redhat-linux
> ----------------------+---------------------------------------------------------
>
> C compiler flags | -O3 --param max-inline-insns-single=35000 --param
> | inline-unit-growth=10000 --param
> | large-function-growth=200000 -Winline
> ----------------------+---------------------------------------------------------
>
> linker | /usr/bin/gcc
> | GNU/3.4.6/3.4.6 20060404 (Red Hat 3.4.6-3)
> | gcc version 3.4.6 20060404 (Red Hat 3.4.6-3)
> | Reading specs from
> | /usr/lib/gcc/x86_64-redhat-linux/3.4.6/specs
> | Configured with: ../configure --prefix=/usr
> | --mandir=/usr/share/man
> --infodir=/usr/share/info
> | --enable-shared --enable-threads=posix
> | --disable-checking --with-system-zlib
> | --enable-__cxa_atexit
> --disable-libunwind-exceptions
> | --enable-java-awt=gtk --host=x86_64-redhat-linux
> ----------------------+---------------------------------------------------------
>
> linker flags | -O3 --param max-inline-insns-single=35000 --param
> | inline-unit-growth=10000 --param
> | large-function-growth=200000 -Winline -ldl
> | -L/usr/local/berkeley_upc-2.6.0//opt/lib
> -lupcr-ibv-seq
> | -lumalloc -L/usr/local/berkeley_upc-2.6.0//opt/lib
> | -L/usr/lib64 -lgasnet-ibv-seq -libverbs -ldl
> -lpthread
> | -L/usr/lib/gcc/x86_64-redhat-linux/3.4.6 -lgcc -lm
> ----------------------+---------------------------------------------------------
>
--
Paul H. Hargrove PHHargrove_at_lbl_dot_gov
Future Technologies Group
HPC Research Department Tel: +1-510-495-2352
Lawrence Berkeley National Laboratory Fax: +1-510-486-6900