Re: UPC Benchmarks

From: Nenad Vukicevic (nenad_at_intrepid_dot_com)
Date: Thu Feb 12 2009 - 11:08:27 PST

  • Next message: Steven D. Vormwald: "Re: UPC Benchmarks"
    Copy of our benchmark results that Gary mentioned can be found at:
    We can also provide you with the sources of all the benchmarks and scripts 
    you can run them and tabulate results.
    From: "Gary Funck" <gary_at_intrepid_dot_com>
    Sent: Thursday, February 12, 2009 10:50 AM
    To: "Steven D. Vormwald" <sdvormwa_at_mtu_dot_edu>
    Cc: <upc-users_at_lbl_dot_gov>
    Subject: Re: UPC Benchmarks
    > On 02/12/09 12:49:32, Steven D. Vormwald wrote:
    >> Hello,
    >> Are there any "standard" UPC benchmarks available, preferably with
    >> fine-grained (few upc_mem{cpy|get|put} or collective calls) remote
    >> memory accesses?  A cursory search of the various University project
    >> pages and the wiki hasn't revealed any, but I thought I'd ask before
    >> implementing some on my own.
    > Steven,
    > A student at MTU, Zhang Zhang, presented some UPC benchmark results
    > back in 2004/2005:
    > We looked at those his paper, and those benchmarks, and noted
    > some methodological errors.  Notably, a buggy version of the NPB benchmark
    > developed by GWU was utilized which skewed results and led to some
    > false indications of failures when run on various platforms.
    > This led to apparent "no shows" by various compilers.
    > A couple of years ago, we collected UPC benchmarks from various
    > sources, and re-worked them so that they (1) execute enough iterations
    > to be meaningful on modern hardware, (2) did not print extraneous
    > output during the timing run part of the benchmark, and (3) were run
    > in a dedicated OS environment (run level 1 on Linux) to avoid
    > extraneous timing noise created by normal OS activities (4) sufficient
    > runs of the benchmarks were made to obtain a representative timing
    > sample.  We found that all these steps were necessary to obtain
    > reasonable timing results.  During that process, we did not attempt
    > to verify that each benchmark measured exactly what it was trying
    > to measure in an effective fashion.  Further, we didn't try to
    > verify that complex benchmarks (like NPB) produced correct results.
    > Although I commend Zhang Zhang for advancing knowledge in the
    > area of UPC performance -- due to methodological errors it is
    > unfortunate that his paper is the seminal work in this area.
    > I'd like to see his experiments re-done with the errors corrected,
    > and run against current compilers and runtime systems.
    > A procedural recommendation: while developing and selecting
    > benchmarks and collecting initial results, I'd encourage
    > that the results be run by each vendor involved to ensure that
    > the compiler was executed with appropriate paramaters and to
    > give the vendor the opportunity to fix small errors/bugs,
    > and to verify that the benchmarks in fact measure the
    > feature as intended.
    > - Gary

  • Next message: Steven D. Vormwald: "Re: UPC Benchmarks"