The second paper you cited is fascinating and points to a number of issues I see not only
in this field of academic research but many others. I can't tell you how many times
I've heard that extant data is critical since it can't be reproduced... if you
can't reproduce it, are you really doing science? (gonna get myself into trouble
here).
I don't have immediate access to the first, as I am not a full ACM member, but
regardless, thank you for passing along this information.
--Ken
On Jan 10, 2019, at 10:36 AM, Benson Muite <benson.muite(a)ut.ee>
wrote:
There are a few cases where an absolute performance guarantee is
helpful. In other cases, cost considerations and data utility may allow
for expected but not guaranteed performance - how to measure this
effectively in a broadly applicable manner seems challenging.
Regards,
Benson
On 1/10/19 5:30 PM, Carlile, Ken wrote:
> I will have a look at those papers. I can get system load information to some degree,
and that's probably an avenue I should investigate more fully, considering I'm
sharing this test load with people doing actual work on the HPC cluster, and in certain
cases on the storage as well, although I look for down times on both.
>
> I have to simultaneously agree and disagree on your last statement. Considering that
the end goal here is to advise on the strengths and weaknesses of various storage systems,
if we throw our hands in the air and say that typical performance varies wildly, the point
of the exercise becomes somewhat moot. If you can't trust your results to stand up to
an average (or other mathematical leveling), are they really valid?
>
> --Ken
>
>> On Jan 10, 2019, at 10:20 AM, Benson Muite via IO-500 <io-500(a)vi4io.org>
wrote:
>>
>> There are a number of relevant papers. Ones that come to mind are:
>>
>> Kevin A. Brown, Nikhil Jain, Satoshi Matsuoka, Martin Schulz , Abhinav
>> Bhatele "Interference between I/O and MPI Traffic on Fat-tree
Networks"
>>
http://dx.doi.org/10.1145/3225058.3225144
>>
>> T. Hoefler, R. Belli "Scientific Benchmarking of Parallel Computing
>> Systems"
https://htor.inf.ethz.ch/publications/index.php?pub=222
>>
>> Are you able to get system load information? Figuring out conditions to
>> get best and worst performance may be helpful. Typical performance will
>> likely depend on system design and system load. This may be difficult to
>> quantify with just an average.
>>
>> Regards,
>>
>> Benson
>>
>> On 1/10/19 5:10 PM, Carlile, Ken via IO-500 wrote:
>>
>>> As I've been playing with the IO500 benchmark on various systems, I'm
seeing a fair amount of variability between runs. This may ultimately be to my detriment,
but have there been any discussions of perhaps requiring the mean of 3 or 5 runs for the
official numbers? I think the most striking examples are when I've managed to hit some
kind of storage side caching just right and produced numbers, particularly on
mdtest_hard_stat that are an order of magnitude greater on one run vs other runs. Now, I
really LIKE having a higher score, but it doesn't seem exactly fair or
representative...
>>>
>>> --Ken Carlile
>>> _______________________________________________
>>> IO-500 mailing list
>>> IO-500(a)vi4io.org
>>>
https://www.vi4io.org/mailman/listinfo/io-500
>>
>> _______________________________________________
>> IO-500 mailing list
>> IO-500(a)vi4io.org
>>
https://www.vi4io.org/mailman/listinfo/io-500