On Apr 5, 2017, at 10:18 PM, John Bent <John.Bent(a)seagategov.com&gt; wrote: > > All, > > I met with Ilene Carpenter about a week ago and she had a bunch of >interesting thoughts about IO-500. Her¹s are top-level than my thoughts >are the sub-bullets. > > € Maybe it should be called HPC IO-500 to distinguish it from hyper >scalars since the focus is on HPC and the machines on the Top 500 list. > € On the other hand, Top 500 is not called HPC Top 500. > € Benchmarks get what benchmarks get. What about performance when >another job is running? > € The idea was data-easy, data-hard, metadata-easy, metadata-hard and >then figure out how to combine these four into one number. Perhaps we >add a fifth test which is running all four simultaneously. I'm not against running all of them together, but the aggregate shouldn't be worse than data-hard + metadata-hard in the end, or by definition those are not the "hard" numbers we were looking for? > € Storage performance degrades with age. You run Linpack on day 1 and >get a number. You run Linpack on day N and get the same number. If >your purpose is to bound user expectation, then a number from day 1 may >no longer be the correct bound on day N when the storage is fragmented. Agreed. While the "-easy" numbers may degrade over time, at least the "-hard" numbers shouldn't get worse with age. Also, there is value to knowing how the "-easy" numbers decline over time. The amount that the "-easy" performance declines is a function of the underlying filesystem and storage technology (e.g. SSD vs. HDD) so unlike Top 500 there would be value in re-running the benchmarks every year and amending the storage system's entry (though I think the ranking should be based on the peak number). > € Another challenge with IO 500 is that Linpack is so easy. You run >it and you get two indisputable answers: the result which can be >verified to be correct and the time that it took to get the result. IO >benchmarks are much harder. Are people allowed to set up RAID0 RAM >disks and do the benchmark into them? Sure, if that is a storage option that they actually provide for users. I know of a few sites that use(d) RAM-based Lustre filesystems for shared storage since their application didn't have checkpoint-restart so if any node crashed the test run was lost anyway. > € IO 500 is great because it enables people to look at historical >trends. > € For example, when do various systems start showing up? If I¹m >procuring a new storage system and I see that Ceph, BeeGFS, or OrangeFS >are high up in the IO 500 then I¹m much likely consider them instead of >just Lustre and GPFS. Definitely one of the things I've wanted to see in the past was which filesystem each system on the Top-500 was using. We've had to generate these results manually in the past. > € NREL just doubled their flops on a system but didn¹t touch storage. >It¹d be nice if the IO 500 could somehow capture this. Both before and >after will have the same storage performance but the after system is >worse because it is imbalanced. While the number of clients running the tests should be part of the results, I don't think a later change in the client system should affect previous results (see my earlier comment about submitting updated numbers periodically). There could be the ability to link the IO-500 storage results to the Top-500 system results, but there isn't necessarily a 1:1 relationship between them. In particular, many sites have Lustre or GPFS site-wide filesystems that are accessed by multiple (and changing) compute clusters, so while a particular benchmark result may depend on the number of clients actually running the test, the result itself shouldn't be a function of the total number of clients accessing the storage.

On Apr 6, 2017, at 10:38 AM, Carpenter, Ilene <Ilene.Carpenter(a)nrel.gov&gt; wrote: > On 4/6/17, 10:18 AM, "IO-500 on behalf of Andreas Dilger" > <io-500-bounces(a)vi4io.org on behalf of adilger(a)dilger.ca&gt; wrote: > >> On Apr 5, 2017, at 10:18 PM, John Bent <John.Bent(a)seagategov.com&gt; >>wrote: >>> >>> All, >>> >>> I met with Ilene Carpenter about a week ago and she had a bunch of >>> interesting thoughts about IO-500. Her¹s are top-level than my >>>thoughts >>> are the sub-bullets. >>> >>> € Maybe it should be called HPC IO-500 to distinguish it from hyper >>> scalars since the focus is on HPC and the machines on the Top 500 >>>list. >>> € On the other hand, Top 500 is not called HPC Top 500. >>> € Benchmarks get what benchmarks get. What about performance when >>> another job is running? >>> € The idea was data-easy, data-hard, metadata-easy, metadata-hard >>>and >>> then figure out how to combine these four into one number. Perhaps we >>> add a fifth test which is running all four simultaneously. >> >> I'm not against running all of them together, but the aggregate >>shouldn't >> be worse than data-hard + metadata-hard in the end, or by definition >> those are not the "hard" numbers we were looking for? >> >>> € Storage performance degrades with age. You run Linpack on day 1 >>>and >>> get a number. You run Linpack on day N and get the same number. If >>> your purpose is to bound user expectation, then a number from day 1 >>>may >>> no longer be the correct bound on day N when the storage is >>>fragmented. >> >> Agreed. While the "-easy" numbers may degrade over time, at least the >> "-hard" numbers shouldn't get worse with age. Also, there is value to >> knowing how the "-easy" numbers decline over time. The amount that the >> "-easy" performance declines is a function of the underlying filesystem >> and storage technology (e.g. SSD vs. HDD) so unlike Top 500 there would >> be value in re-running the benchmarks every year and amending the >>storage >> system's entry (though I think the ranking should be based on the peak >> number). >> >>> € Another challenge with IO 500 is that Linpack is so easy. You run >>> it and you get two indisputable answers: the result which can be >>> verified to be correct and the time that it took to get the result. >>>IO >>> benchmarks are much harder. Are people allowed to set up RAID0 RAM >>> disks and do the benchmark into them? >> >> Sure, if that is a storage option that they actually provide for users. >> I know of a few sites that use(d) RAM-based Lustre filesystems for >>shared >> storage since their application didn't have checkpoint-restart so if >>any >> node crashed the test run was lost anyway. >> >>> € IO 500 is great because it enables people to look at historical >>> trends. >>> € For example, when do various systems start showing up? If I¹m >>> procuring a new storage system and I see that Ceph, BeeGFS, or >>>OrangeFS >>> are high up in the IO 500 then I¹m much likely consider them instead >>>of >>> just Lustre and GPFS. >> >> Definitely one of the things I've wanted to see in the past was which >> filesystem each system on the Top-500 was using. We've had to generate >> these results manually in the past. >> >>> € NREL just doubled their flops on a system but didn¹t touch storage. >>> It¹d be nice if the IO 500 could somehow capture this. Both before >>>and >>> after will have the same storage performance but the after system is >>> worse because it is imbalanced. >> >> While the number of clients running the tests should be part of the >> results, I don't think a later change in the client system should >>affect >> previous results (see my earlier comment about submitting updated >>numbers >> periodically). There could be the ability to link the IO-500 storage >> results to the Top-500 system results, but there isn't necessarily a >>1:1 >> relationship between them. >> >> In particular, many sites have Lustre or GPFS site-wide filesystems >>that >> are accessed by multiple (and changing) compute clusters, so while a >> particular benchmark result may depend on the number of clients >>actually >> running the test, the result itself shouldn't be a function of the >>total >> number of clients accessing the storage. > > The discussion we had was about system balance, how top 500 has caused a > tendency to value flops above having a balanced system and the >possibility > of somehow assigning a balance score - storage capacity and performance > relative to flop rate. This would only apply to systems that have > dedicated HPC-oriented filesystems, I don¹t think we¹d want to try to > assess balance for a site-wide filesystem. I think if there is a link from the IO-500 results to the Top-500 results then this is something that readers can work out themselves, but I don't think the posted IO-500 results should depend on the compute nodes directly. Otherwise, this implies that the IO-500 result would change if the Linpack number is updated (e.g. better compiler, add GPUs) but that doesn't necessarily make sense. > Thinking about how to deal with site-wide filesystems that makes me > wonder, what is the *identifier* of entries in the proposed io-500 list? > Is it system name (ie. Peregrine, Cori, Summit, etc.) ? > Is it a system name plus a file system name (as in Peregrine:/scratch, > Jim:/scratch1, Jim:/scratch2, Bob:/ptmp)? > Maybe it is a system and the aggregation of multiple parallel file >systems > (as in Jim:/scratch1+Jim:/scratch2)? > Maybe it is a site name plus a file system name (as in > NERSC:/site-wide-GPFS, NREL:Peregrine:/scratch)? > > We¹ve focused a fair amount of energy on defining benchmarks but I don¹t > recall any discussion on this. Definitely the filesystem should be identified separately from (or in addition to) the cluster that it is attached to. Some sites have names for site-wide filesystems themselves, since it is attached to multiple compute clusters.

Hi, >1. On the other hand, Top 500 is not called HPC Top 500. As you say, Green500, Graph500, Top500, IO500 does make sense. 2. > Benchmarks get what benchmarks get. What about performance when another job > is running? > The idea was data-easy, data-hard, metadata-easy, metadata-hard and then > figure out how to combine these four into one number. Perhaps we add a > fifth test which is running all four simultaneously. We may also have useful equations to combine these values but also supercomputer characteristics into one value. Based on the needs, one retrieves the appropriate compound value to sort for a "balanced system" I don't see the benefit of running them simultaneously right now. This is difficult as parts of the test could be served by other storage systems, e.g., acceleration for small files. Also how to balance? Well, maybe this thought is important for later. > 3. Storage performance degrades with age. You run Linpack on day 1 and >get a > number. You run Linpack on day N and get the same number. If your >purpose > is to bound user expectation, then a number from day 1 may no longer be >the > correct bound on day N when the storage is fragmented. I believe the way we do benchmarking is adding to the problem. We should indeed reserve a tiny fraction of space for debugging of performance and resilience, a test partition on which the same benchmark can be run over and over again with comparable performance. At least during maintenance. Unavoidable factors are degradation in storage hardware and impact of software / firmware upgrades. > 4. Another challenge with IO 500 is that Linpack is so easy. You run >it and > you get two indisputable answers: the result which can be verified to be > correct and the time that it took to get the result. IO benchmarks are >much > harder. Are people allowed to set up RAID0 RAM disks and do the >benchmark > into them? > Sure, if ... provide to users Yes, I agree too. They would show up as another file system in the IO500 list. Less capacity but good performance :-) > 5. IO 500 is great because it enables people to look at historical >trends. > > For example, when do various systems start showing up? If I¹m >procuring a > new storage system and I see that Ceph, BeeGFS, or OrangeFS are high up >in > the IO 500 then I¹m much likely consider them instead of just Lustre and > GPFS. Also one could compare data centers similar to the current system or application profile and see how they improved. FYI: Soon the list will be extended with several BeeGFS systems that have low TOP500 ranks, but they will show up. > 6. NREL just doubled their flops on a system but didn¹t touch storage. >It¹d be > nice if the IO 500 could somehow capture this. Both before and after >will > have the same storage performance but the after system is worse because >it > is imbalanced. Indeed, and it will. As the HPSL covers *many* other metrics and (will) also cover costs, comparing those systems is possible.

Right now it is already interesting to compare peak performance vs. storage capacity vs. memory capacity. > 7. Aren¹t you all behind the schedule that you yourself John Bent >proposed > after the SC BoF? We are. Somehow the activity here goes also up & down. I don't know why only few people respond to me, if there is a reason, tell me :-)

On Apr 6, 2017, at 11:16 AM, Carpenter, Ilene <Ilene.Carpenter(a)nrel.gov&gt; wrote: Since I didn¹t actually say that, I think John was joking! :-).