Re: trouble using ibv-conduit

From: Paul H. Hargrove (PHHargrove_at_lbl_dot_gov)
Date: Mon Jan 07 2008 - 15:04:18 PST

  • Next message: Steven D. Vormwald: "Re: trouble using ibv-conduit"
    Steven,
    
    I am bcc:ing this reply to a former member of the Berkeley UPC team who 
    is now at QLogic.  He may respond with additional information.
    
    I am the author of the ibv-conduit code in Berkeley UPC/GASNet.  I am 
    afraid that I have not encountered the specific error you see.  However, 
    I see that you are using the InifiPath adapters, which I am not certain 
    support a full implementation of the OpenIB verbs.  Certainly all the 
    MPI implementations that support QLogic's adapters have been modified to 
    use their "PSM" interface rather than IB Verbs.
    
    The only thing that I can suggest is to ensure that running 
    "ibv_devinfo" produces output something like the following:
     
    $ /opt/ofed/bin/ibv_devinfo
    hca_id: mthca0
            fw_ver:                         1.2.0
            node_guid:                      0005:ad00:0005:85a4
            sys_image_guid:                 0005:ad00:0005:85a7
            vendor_id:                      0x02c9
            vendor_part_id:                 25204
            hw_ver:                         0xA0
            board_id:                       MT_0230000002
            phys_port_cnt:                  1
                    port:   1
                            state:                  PORT_ACTIVE (4)
                            max_mtu:                2048 (4)
                            active_mtu:             2048 (4)
                            sm_lid:                 1
                            port_lid:               1010
                            port_lmc:               0x00
    
    If ibv_devinfo fails, then that means that the OpenIB verbs support is 
    not present.  If you do get output like the above, but don't see 
    "PORT_ACTIVE" then there is probably some configuration problem.  If you 
    do get output that indicates at least one ACTIVE port, then we can start 
    looking at GASNet details to figure where the problem lies.
    
    -Paul
    
    Steven D. Vormwald wrote:
    > Hello,
    >
    > I'm having trouble using the ibv-conduit in the 2.6.0 release:
    >
    > [sdvormwa@gilbert UPC]$ GASNET_BACKTRACE=1 
    > /usr/local/berkeley_upc-2.6.0/bin/upcrun -np 1 ./a.out
    > libibverbs: Warning: no userspace device-specific driver found for 
    > /sys/class/infiniband_verbs/uverbs0
    > GASNet gasnetc_init returning an error code: GASNET_ERR_RESOURCE 
    > (Problem with requested resource)
    >  at 
    > /usr/local/src/berkeley_upc-2.6.0/gasnet/vapi-conduit/gasnet_core.c:986
    >  reason: unable to open any HCA ports
    > GASNet gasnet_init_GASNET_SEQFASTnodebugnotracenostatsnosrclines 
    > returning an error code: GASNET_ERR_RESOURCE (Problem with requested 
    > resource)
    >  at 
    > /usr/local/src/berkeley_upc-2.6.0/gasnet/vapi-conduit/gasnet_core.c:1546
    > UPC Runtime: GASNet error GASNET_ERR_RESOURCE(Problem with requested 
    > resource): gasnet_init failed!
    > [0] 0: /home/sdvormwa/UPC/./a.out [0x440913] gasneti_bt_execinfo ??:0
    > [0] 1: /home/sdvormwa/UPC/./a.out [0x442db3] gasneti_print_backtrace ??:0
    > [0] 2: /home/sdvormwa/UPC/./a.out [0x404d14] upcri_gaserr ??:0
    > [0] 3: /home/sdvormwa/UPC/./a.out [0x4288bc] bupc_init_reentrant ??:0
    > [0] 4: /home/sdvormwa/UPC/./a.out [0x403591] main ??:0
    > [0] 5: /lib64/tls/libc.so.6(__libc_start_main+0xdb) [0x3ca971c3fb] ?? 
    > ??:0
    > [0] 6: /home/sdvormwa/UPC/./a.out [0x40318a] _start ??:0
    > [sdvormwa@gilbert UPC]$
    >
    > openib was installed using prebuilt packages, configured and tested as 
    > documented in the user manual that came with the infiniband cards (see 
    > download.qlogic.com/manual/62612/IB6054601-00E.pdf).  The libibverbs 
    > documentation for the warning says to ensure that 
    > /usr/local/lib/infiniband/mthca.so exists and is executable, which I 
    > have verified on all the nodes.  I've run through the rest of the 
    > included troubleshooting documentation, and have yet to find a 
    > solution.  Any help would be much appreciated.
    >
    > Thanks,
    > Steven Vormwald
    >
    >
    > This is upcc (the Berkeley Unified Parallel C compiler), v. 2.6.0
    >  (getting remote translator settings...)
    > ----------------------+--------------------------------------------------------- 
    >
    > UPC Runtime          | v. 2.6.0, built on Dec  3 2007 at 13:21:31
    > ----------------------+--------------------------------------------------------- 
    >
    > UPC-to-C translator  | v. 2.6.0, built on Oct 15 2007 at 15:50:19
    > ----------------------+--------------------------------------------------------- 
    >
    > Translator location  | http://upc-translator.lbl.gov/upcc-2.6.0.cgi
    > ----------------------+--------------------------------------------------------- 
    >
    > networks supported   | udp smp ibv
    > ----------------------+--------------------------------------------------------- 
    >
    > default network      | ibv
    > ----------------------+--------------------------------------------------------- 
    >
    > pthreads support     | available (if used, default is 2 pthreads per 
    > process)
    > ----------------------+--------------------------------------------------------- 
    >
    > Configured with      | 
    > '--with-translator=http://upc-translator.lbl.gov/upcc-2
    >                      | .6.0.cgi' '--enable-ibv' '--verbose'
    >                      | 'IBV_LIBS=-libverbs -ldl' 'LDFLAGS=-ldl'
    >                      | '--without-mpi-cc' '--disable-mpi'
    >                      | '--prefix=/usr/local/berkeley_upc-2.6.0//opt'
    >                      | '--with-multiconf-magic=opt'
    > ----------------------+--------------------------------------------------------- 
    >
    > Configure features   | berkeleyupc,upcr,gasnet,upc_collective,upc_io,
    >                      | upc_memcpy_async,upc_ptradd,upc_thread_distance,
    >                      | upc_tick,upc_sem,upc_dump_shared,upc_trace_printf,
    >                      | 
    > upc_trace_mask,upc_local_to_shared,upc_atomics,pupc,
    >                      | upc_memcpy_vis,nodebug,notrace,nostats,nogasp,
    >                      | segment_fast,os_linux,cpu_x86_64,cpu_64,cc_gnu,
    >                      | packedsptr
    > ----------------------+--------------------------------------------------------- 
    >
    > Configure id         | gilbert.cse.mtu.edu Mon Dec  3 13:16:56 EST 
    > 2007 root
    > ----------------------+--------------------------------------------------------- 
    >
    > Binary interface     | 64-bit x86_64-unknown-linux-gnu
    > ----------------------+--------------------------------------------------------- 
    >
    > Runtime interface #  | Runtime supports 3.0 -> 3.9: Translator uses 3.6
    > ----------------------+--------------------------------------------------------- 
    >
    >                      |  --- BACKEND SETTINGS (for ibv network) ---
    > ----------------------+--------------------------------------------------------- 
    >
    > C compiler           | /usr/bin/gcc
    >                      |   GNU/3.4.6/3.4.6 20060404 (Red Hat 3.4.6-3)
    >                      |   gcc version 3.4.6 20060404 (Red Hat 3.4.6-3)
    >                      |   Reading specs from
    >                      |   /usr/lib/gcc/x86_64-redhat-linux/3.4.6/specs
    >                      |   Configured with: ../configure --prefix=/usr
    >                      |   --mandir=/usr/share/man 
    > --infodir=/usr/share/info
    >                      |   --enable-shared --enable-threads=posix
    >                      |   --disable-checking --with-system-zlib
    >                      |   --enable-__cxa_atexit 
    > --disable-libunwind-exceptions
    >                      |   --enable-java-awt=gtk --host=x86_64-redhat-linux
    > ----------------------+--------------------------------------------------------- 
    >
    > C compiler flags     | -O3 --param max-inline-insns-single=35000 --param
    >                      | inline-unit-growth=10000 --param
    >                      | large-function-growth=200000 -Winline
    > ----------------------+--------------------------------------------------------- 
    >
    > linker               | /usr/bin/gcc
    >                      |   GNU/3.4.6/3.4.6 20060404 (Red Hat 3.4.6-3)
    >                      |   gcc version 3.4.6 20060404 (Red Hat 3.4.6-3)
    >                      |   Reading specs from
    >                      |   /usr/lib/gcc/x86_64-redhat-linux/3.4.6/specs
    >                      |   Configured with: ../configure --prefix=/usr
    >                      |   --mandir=/usr/share/man 
    > --infodir=/usr/share/info
    >                      |   --enable-shared --enable-threads=posix
    >                      |   --disable-checking --with-system-zlib
    >                      |   --enable-__cxa_atexit 
    > --disable-libunwind-exceptions
    >                      |   --enable-java-awt=gtk --host=x86_64-redhat-linux
    > ----------------------+--------------------------------------------------------- 
    >
    > linker flags         | -O3 --param max-inline-insns-single=35000 --param
    >                      | inline-unit-growth=10000 --param
    >                      | large-function-growth=200000 -Winline -ldl
    >                      | -L/usr/local/berkeley_upc-2.6.0//opt/lib 
    > -lupcr-ibv-seq
    >                      | -lumalloc -L/usr/local/berkeley_upc-2.6.0//opt/lib
    >                      | -L/usr/lib64 -lgasnet-ibv-seq -libverbs -ldl 
    > -lpthread
    >                      | -L/usr/lib/gcc/x86_64-redhat-linux/3.4.6 -lgcc -lm
    > ----------------------+--------------------------------------------------------- 
    >
    
    
    -- 
    Paul H. Hargrove                          PHHargrove_at_lbl_dot_gov
    Future Technologies Group
    HPC Research Department                   Tel: +1-510-495-2352
    Lawrence Berkeley National Laboratory     Fax: +1-510-486-6900
    

  • Next message: Steven D. Vormwald: "Re: trouble using ibv-conduit"