Hi Jay,
Thank you for the very detailed reply to my question, and I apologize for my initial
naiveté on the matter. It is certainly an issue that everything on this list is a one off!
Nevertheless, it does allow us to make some interesting comparisons. I hope to get my data
in such a state that I'm comfortable with posting it soon.
--Ken
On Jan 24, 2019, at 12:00 PM, Lofstead, Gerald F II
<gflofst(a)sandia.gov> wrote:
Hi Ken,
BTW, most authors have a personal page with all of their papers posted. It is a good way
to avoid the digital library requirements. Those are not supposed to be camera ready
versions, but they are really close to what is published. Also, if you email the authors,
I would expect they would be happy to share their paper. Someone wanting to read your work
is a good thing. :-)
Sorry for the long delay. The committee wanted to talk about this before responding.
I've also done some work on this issue exactly in my SC10 paper:
Jay Lofstead, Fang Zheng, Qing Liu, Scott Klasky, Ron Oldfield, Todd Kordenbrock, Karsten
Schwan, Matthew Wolf. "Managing Variability in the IO Performance of Petascale
Storage Systems". In Proceedings of SC 10. New Orleans, LA. November 2010.
I agree that variability is a serious problem for storage performance. Other work
I've been involved in has tried to look at this over time:
Bing Xie, Yezhou Huang, Jeffrey S. Chase, Jong Youl Choi, Scott Klasky, Jay Lofstead, and
Sarp Oral. 2017. Predicting Output Performance of a Petascale Supercomputer. In
Proceedings of the 26th International Symposium on High-Performance Parallel and
Distributed Computing (HPDC '17). ACM, New York, NY, USA, 181-192. DOI:
https://doi.org/10.1145/3078597.3078614
The bottom line we've seen is that the variability can be pretty extreme. Over the
course of an application run, the performance is relatively stable and predictable though.
The other way to address trying to get the best overall performance is to work around hot
spots (such as my SC10 paper describes). Quality of service is making inroads, but few are
willing to give up possibly achieving the best possible performance for a guaranteed
performance. UCSC has some work I don't think has been published yet that does a
fantastic job at QoS reservations and sharing.
Ivo Jimenez's (UCSC) thesis work has investigated how to get performance
predictability from platform to platform. There are so many variables--even for a single
node--that getting with 5-10% is considered success. As a community, since we have bespoke
platforms others cannot replicate and we cannot have 100% control over our experimental
environment due to production workloads and shared storage arrays among various platforms,
we have to have a different metric for replication (as opposed to reproducibility*). Ivo
hasn't been able to get the community to come to agreement yet and he is far from the
only person trying to drive this (I know of at least 4 or 5 others that are also pushing).
What we can mostly agree on is that what the data shows in terms of trends and magnitudes,
under controlled conditions, should represent the same results indicating the same
conclusions.
All of this aside, 100x difference in performance is extreme. I've seen 10x plenty
of times and 50x+ on a few machines at Sandia (capacity machines with really odd workloads
and highly shared storage arrays). Since we are trying to get a number that shows what the
best possible result is on a platform for a single set of benchmark runs (meaning you
can't pick the ior easy from one and ior hard from another), we hope that things are
somewhat stable. Unless you can control everything to avoid any simultaneous users, and
still deal with thermal issues or other hardware related performance variations, getting
to exactly the same results is not a reasonable expectation.
Ultimately, I think the question is more should IO-500 show the best possible or average
cases? Our attempt to make this more relevant for ordinary users is to allow the easy
tests with full disclosure of how to achieve those results. If a user tunes using that
information, they should expect to be better than the hard results, but probably not as
good as easy since they are unlikely to be able to form their data into exactly the same
IO pattern that easy uses. Letting someone run multiple times, even with wide swings in
performance, should not change the ability to use the easy settings and tunings to help
users achieve better results than the hard configuration.
* All of science other than computer science uses reproducibility to mean that you can
take the raw data and generate the same analysis results. Replication means rerunning the
experiments to generate data that shows the same results.
Best,
Jay Lofstead on behalf of the IO-500 Committee.
On 1/10/19, 8:53 AM, "IO-500 on behalf of Carlile, Ken via IO-500"
<io-500-bounces(a)vi4io.org on behalf of io-500(a)vi4io.org> wrote:
The second paper you cited is fascinating and points to a number of issues I see not
only in this field of academic research but many others. I can't tell you how many
times I've heard that extant data is critical since it can't be reproduced... if
you can't reproduce it, are you really doing science? (gonna get myself into trouble
here).
I don't have immediate access to the first, as I am not a full ACM member, but
regardless, thank you for passing along this information.
--Ken
On Jan 10, 2019, at 10:36 AM, Benson Muite
<benson.muite(a)ut.ee> wrote:
There are a few cases where an absolute performance guarantee is
helpful. In other cases, cost considerations and data utility may allow
for expected but not guaranteed performance - how to measure this
effectively in a broadly applicable manner seems challenging.
Regards,
Benson
On 1/10/19 5:30 PM, Carlile, Ken wrote:
> I will have a look at those papers. I can get system load information to some degree,
and that's probably an avenue I should investigate more fully, considering I'm
sharing this test load with people doing actual work on the HPC cluster, and in certain
cases on the storage as well, although I look for down times on both.
>
> I have to simultaneously agree and disagree on your last statement. Considering that
the end goal here is to advise on the strengths and weaknesses of various storage systems,
if we throw our hands in the air and say that typical performance varies wildly, the point
of the exercise becomes somewhat moot. If you can't trust your results to stand up to
an average (or other mathematical leveling), are they really valid?
>
> --Ken
>
>> On Jan 10, 2019, at 10:20 AM, Benson Muite via IO-500 <io-500(a)vi4io.org>
wrote:
>>
>> There are a number of relevant papers. Ones that come to mind are:
>>
>> Kevin A. Brown, Nikhil Jain, Satoshi Matsuoka, Martin Schulz , Abhinav
>> Bhatele "Interference between I/O and MPI Traffic on Fat-tree
Networks"
>>
http://dx.doi.org/10.1145/3225058.3225144
>>
>> T. Hoefler, R. Belli "Scientific Benchmarking of Parallel Computing
>> Systems"
https://htor.inf.ethz.ch/publications/index.php?pub=222
>>
>> Are you able to get system load information? Figuring out conditions to
>> get best and worst performance may be helpful. Typical performance will
>> likely depend on system design and system load. This may be difficult to
>> quantify with just an average.
>>
>> Regards,
>>
>> Benson
>>
>> On 1/10/19 5:10 PM, Carlile, Ken via IO-500 wrote:
>>
>>> As I've been playing with the IO500 benchmark on various systems,
I'm seeing a fair amount of variability between runs. This may ultimately be to my
detriment, but have there been any discussions of perhaps requiring the mean of 3 or 5
runs for the official numbers? I think the most striking examples are when I've
managed to hit some kind of storage side caching just right and produced numbers,
particularly on mdtest_hard_stat that are an order of magnitude greater on one run vs
other runs. Now, I really LIKE having a higher score, but it doesn't seem exactly
fair or representative...
>>>
>>> --Ken Carlile
>>> _______________________________________________
>>> IO-500 mailing list
>>> IO-500(a)vi4io.org
>>>
https://www.vi4io.org/mailman/listinfo/io-500
>>
>> _______________________________________________
>> IO-500 mailing list
>> IO-500(a)vi4io.org
>>
https://www.vi4io.org/mailman/listinfo/io-500
_______________________________________________
IO-500 mailing list
IO-500(a)vi4io.org
https://www.vi4io.org/mailman/listinfo/io-500