4:18 a.m.
All,
I met with Ilene Carpenter about a week ago and she had a bunch of interesting thoughts
about IO-500. Her’s are top-level than my thoughts are the sub-bullets.
1. Maybe it should be called HPC IO-500 to distinguish it from hyper scalars since the
focus is on HPC and the machines on the Top 500 list.
* On the other hand, Top 500 is not called HPC Top 500.
2. Benchmarks get what benchmarks get. What about performance when another job is
running?
* The idea was data-easy, data-hard, metadata-easy, metadata-hard and then figure
out how to combine these four into one number. Perhaps we add a fifth test which is
running all four simultaneously.
3. Storage performance degrades with age. You run Linpack on day 1 and get a number.
You run Linpack on day N and get the same number. If your purpose is to bound user
expectation, then a number from day 1 may no longer be the correct bound on day N when the
storage is fragmented.
4. Another challenge with IO 500 is that Linpack is so easy. You run it and you get
two indisputable answers: the result which can be verified to be correct and the time that
it took to get the result. IO benchmarks are much harder. Are people allowed to set up
RAID0 RAM disks and do the benchmark into them?
5. IO 500 is great because it enables people to look at historical trends.
* For example, when do various systems start showing up? If I’m procuring a new
storage system and I see that Ceph, BeeGFS, or OrangeFS are high up in the IO 500 then I’m
much likely consider them instead of just Lustre and GPFS.
6. NREL just doubled their flops on a system but didn’t touch storage. It’d be nice if
the IO 500 could somehow capture this. Both before and after will have the same storage
performance but the after system is worse because it is imbalanced.
7. Aren’t you all behind the schedule that you yourself John Bent proposed after the SC
BoF?
* ….
My impression was that Ilene, who is on this list (Hi Ilene!) likes the idea and supports
the idea and hopes we are successful with the idea but just wants to identify some of the
reasons why it is important and why it is challenging.
Thanks,
John