On 4/6/17, 12:09 PM, "IO-500 on behalf of Julian Kunkel"
<io-500-bounces(a)vi4io.org on behalf of juliankunkel(a)googlemail.com> wrote:
Hi,
>1. On the other hand, Top 500 is not called HPC Top 500.
As you say, Green500, Graph500, Top500, IO500 does make sense.
2. > Benchmarks get what benchmarks get. What about performance when
another job
> is running?
> The idea was data-easy, data-hard, metadata-easy, metadata-hard and then
> figure out how to combine these four into one number. Perhaps we add a
> fifth test which is running all four simultaneously.
We may also have useful equations to combine these values but also
supercomputer characteristics into one value.
Based on the needs, one retrieves the appropriate compound value to
sort for a "balanced system"
I don't see the benefit of running them simultaneously right now. This
is difficult as parts of the test could be served by other storage
systems, e.g., acceleration for small files. Also how to balance?
Well, maybe this thought is important for later.
> 3. Storage performance degrades with age. You run Linpack on day 1 and
>get a
> number. You run Linpack on day N and get the same number. If your
>purpose
> is to bound user expectation, then a number from day 1 may no longer be
>the
> correct bound on day N when the storage is fragmented.
I believe the way we do benchmarking is adding to the problem. We
should indeed reserve a tiny fraction of space for debugging of
performance and resilience, a test partition on which the same
benchmark can be run over and over again with comparable performance.
At least during maintenance.
Unavoidable factors are degradation in storage hardware and impact of
software / firmware upgrades.
> 4. Another challenge with IO 500 is that Linpack is so easy. You run
>it and
> you get two indisputable answers: the result which can be verified to be
> correct and the time that it took to get the result. IO benchmarks are
>much
> harder. Are people allowed to set up RAID0 RAM disks and do the
>benchmark
> into them?
> Sure, if ... provide to users
Yes, I agree too. They would show up as another file system in the
IO500 list. Less capacity but good performance :-)
> 5. IO 500 is great because it enables people to look at historical
>trends.
>
> For example, when do various systems start showing up? If I¹m
>procuring a
> new storage system and I see that Ceph, BeeGFS, or OrangeFS are high up
>in
> the IO 500 then I¹m much likely consider them instead of just Lustre and
> GPFS.
Also one could compare data centers similar to the current system or
application profile and see how they improved.
FYI: Soon the list will be extended with several BeeGFS systems that
have low TOP500 ranks, but they will show up.
> 6. NREL just doubled their flops on a system but didn¹t touch storage.
>It¹d be
> nice if the IO 500 could somehow capture this. Both before and after
>will
> have the same storage performance but the after system is worse because
>it
> is imbalanced.
Indeed, and it will. As the HPSL covers *many* other metrics and
(will) also cover costs, comparing those systems is possible.
Can you remind me what HPSL is? Also, what costs does it cover and how
would those be obtained?
Right now it is already interesting to compare peak performance vs.
storage capacity vs. memory capacity.
> 7. Aren¹t you all behind the schedule that you yourself John Bent
>proposed
> after the SC BoF?
We are. Somehow the activity here goes also up & down.
I don't know why only few people respond to me, if there is a reason,
tell me :-)
Since I didn¹t actually say that, I think John was joking! :-).
-Ilene