There are a few cases where an absolute performance guarantee is
helpful. In other cases, cost considerations and data utility may allow
for expected but not guaranteed performance - how to measure this
effectively in a broadly applicable manner seems challenging.
Regards,
Benson
On 1/10/19 5:30 PM, Carlile, Ken wrote:
I will have a look at those papers. I can get system load information
to some degree, and that's probably an avenue I should investigate more fully,
considering I'm sharing this test load with people doing actual work on the HPC
cluster, and in certain cases on the storage as well, although I look for down times on
both.
I have to simultaneously agree and disagree on your last statement. Considering that the
end goal here is to advise on the strengths and weaknesses of various storage systems, if
we throw our hands in the air and say that typical performance varies wildly, the point of
the exercise becomes somewhat moot. If you can't trust your results to stand up to an
average (or other mathematical leveling), are they really valid?
--Ken
> On Jan 10, 2019, at 10:20 AM, Benson Muite via IO-500 <io-500(a)vi4io.org>
wrote:
>
> There are a number of relevant papers. Ones that come to mind are:
>
> Kevin A. Brown, Nikhil Jain, Satoshi Matsuoka, Martin Schulz , Abhinav
> Bhatele "Interference between I/O and MPI Traffic on Fat-tree Networks"
>
http://dx.doi.org/10.1145/3225058.3225144
>
> T. Hoefler, R. Belli "Scientific Benchmarking of Parallel Computing
> Systems"
https://htor.inf.ethz.ch/publications/index.php?pub=222
>
> Are you able to get system load information? Figuring out conditions to
> get best and worst performance may be helpful. Typical performance will
> likely depend on system design and system load. This may be difficult to
> quantify with just an average.
>
> Regards,
>
> Benson
>
> On 1/10/19 5:10 PM, Carlile, Ken via IO-500 wrote:
>
>> As I've been playing with the IO500 benchmark on various systems, I'm
seeing a fair amount of variability between runs. This may ultimately be to my detriment,
but have there been any discussions of perhaps requiring the mean of 3 or 5 runs for the
official numbers? I think the most striking examples are when I've managed to hit some
kind of storage side caching just right and produced numbers, particularly on
mdtest_hard_stat that are an order of magnitude greater on one run vs other runs. Now, I
really LIKE having a higher score, but it doesn't seem exactly fair or
representative...
>>
>> --Ken Carlile
>> _______________________________________________
>> IO-500 mailing list
>> IO-500(a)vi4io.org
>>
https://www.vi4io.org/mailman/listinfo/io-500
>
> _______________________________________________
> IO-500 mailing list
> IO-500(a)vi4io.org
>
https://www.vi4io.org/mailman/listinfo/io-500