As I've been playing with the IO500 benchmark on various systems, I'm seeing a fair amount of variability between runs. This may ultimately be to my detriment, but have there been any discussions of perhaps requiring the mean of 3 or 5 runs for the official numbers? I think the most striking examples are when I've managed to hit some kind of storage side caching just right and produced numbers, particularly on mdtest_hard_stat that are an order of magnitude greater on one run vs other runs. Now, I really LIKE having a higher score, but it doesn't seem exactly fair or representative... --Ken Carlile _______________________________________________ IO-500 mailing list IO-500(a)vi4io.org https://www.vi4io.org/mailman/listinfo/io-500

I will have a look at those papers. I can get system load information to some degree, and that's probably an avenue I should investigate more fully, considering I'm sharing this test load with people doing actual work on the HPC cluster, and in certain cases on the storage as well, although I look for down times on both. I have to simultaneously agree and disagree on your last statement. Considering that the end goal here is to advise on the strengths and weaknesses of various storage systems, if we throw our hands in the air and say that typical performance varies wildly, the point of the exercise becomes somewhat moot. If you can't trust your results to stand up to an average (or other mathematical leveling), are they really valid? --Ken > On Jan 10, 2019, at 10:20 AM, Benson Muite via IO-500 <io-500(a)vi4io.org&gt; wrote: > > There are a number of relevant papers. Ones that come to mind are: > > Kevin A. Brown, Nikhil Jain, Satoshi Matsuoka, Martin Schulz , Abhinav > Bhatele "Interference between I/O and MPI Traffic on Fat-tree Networks" > http://dx.doi.org/10.1145/3225058.3225144 > > T. Hoefler, R. Belli "Scientific Benchmarking of Parallel Computing > Systems" https://htor.inf.ethz.ch/publications/index.php?pub=222 > > Are you able to get system load information? Figuring out conditions to > get best and worst performance may be helpful. Typical performance will > likely depend on system design and system load. This may be difficult to > quantify with just an average. > > Regards, > > Benson > > On 1/10/19 5:10 PM, Carlile, Ken via IO-500 wrote: > >> As I've been playing with the IO500 benchmark on various systems, I'm seeing a fair amount of variability between runs. This may ultimately be to my detriment, but have there been any discussions of perhaps requiring the mean of 3 or 5 runs for the official numbers? I think the most striking examples are when I've managed to hit some kind of storage side caching just right and produced numbers, particularly on mdtest_hard_stat that are an order of magnitude greater on one run vs other runs. Now, I really LIKE having a higher score, but it doesn't seem exactly fair or representative... >> >> --Ken Carlile >> _______________________________________________ >> IO-500 mailing list >> IO-500(a)vi4io.org >> https://www.vi4io.org/mailman/listinfo/io-500 > > _______________________________________________ > IO-500 mailing list > IO-500(a)vi4io.org > https://www.vi4io.org/mailman/listinfo/io-500

On Jan 10, 2019, at 10:36 AM, Benson Muite <benson.muite(a)ut.ee&gt; wrote: There are a few cases where an absolute performance guarantee is helpful. In other cases, cost considerations and data utility may allow for expected but not guaranteed performance - how to measure this effectively in a broadly applicable manner seems challenging. Regards, Benson On 1/10/19 5:30 PM, Carlile, Ken wrote: > I will have a look at those papers. I can get system load information to some degree, and that's probably an avenue I should investigate more fully, considering I'm sharing this test load with people doing actual work on the HPC cluster, and in certain cases on the storage as well, although I look for down times on both. > > I have to simultaneously agree and disagree on your last statement. Considering that the end goal here is to advise on the strengths and weaknesses of various storage systems, if we throw our hands in the air and say that typical performance varies wildly, the point of the exercise becomes somewhat moot. If you can't trust your results to stand up to an average (or other mathematical leveling), are they really valid? > > --Ken > >> On Jan 10, 2019, at 10:20 AM, Benson Muite via IO-500 <io-500(a)vi4io.org&gt; wrote: >> >> There are a number of relevant papers. Ones that come to mind are: >> >> Kevin A. Brown, Nikhil Jain, Satoshi Matsuoka, Martin Schulz , Abhinav >> Bhatele "Interference between I/O and MPI Traffic on Fat-tree Networks" >> http://dx.doi.org/10.1145/3225058.3225144 >> >> T. Hoefler, R. Belli "Scientific Benchmarking of Parallel Computing >> Systems" https://htor.inf.ethz.ch/publications/index.php?pub=222 >> >> Are you able to get system load information? Figuring out conditions to >> get best and worst performance may be helpful. Typical performance will >> likely depend on system design and system load. This may be difficult to >> quantify with just an average. >> >> Regards, >> >> Benson >> >> On 1/10/19 5:10 PM, Carlile, Ken via IO-500 wrote: >> >>> As I've been playing with the IO500 benchmark on various systems, I'm seeing a fair amount of variability between runs. This may ultimately be to my detriment, but have there been any discussions of perhaps requiring the mean of 3 or 5 runs for the official numbers? I think the most striking examples are when I've managed to hit some kind of storage side caching just right and produced numbers, particularly on mdtest_hard_stat that are an order of magnitude greater on one run vs other runs. Now, I really LIKE having a higher score, but it doesn't seem exactly fair or representative... >>> >>> --Ken Carlile >>> _______________________________________________ >>> IO-500 mailing list >>> IO-500(a)vi4io.org >>> https://www.vi4io.org/mailman/listinfo/io-500 >> >> _______________________________________________ >> IO-500 mailing list >> IO-500(a)vi4io.org >> https://www.vi4io.org/mailman/listinfo/io-500

On Jan 24, 2019, at 12:00 PM, Lofstead, Gerald F II <gflofst(a)sandia.gov&gt; wrote: Hi Ken, BTW, most authors have a personal page with all of their papers posted. It is a good way to avoid the digital library requirements. Those are not supposed to be camera ready versions, but they are really close to what is published. Also, if you email the authors, I would expect they would be happy to share their paper. Someone wanting to read your work is a good thing. :-) Sorry for the long delay. The committee wanted to talk about this before responding. I've also done some work on this issue exactly in my SC10 paper: Jay Lofstead, Fang Zheng, Qing Liu, Scott Klasky, Ron Oldfield, Todd Kordenbrock, Karsten Schwan, Matthew Wolf. "Managing Variability in the IO Performance of Petascale Storage Systems". In Proceedings of SC 10. New Orleans, LA. November 2010. I agree that variability is a serious problem for storage performance. Other work I've been involved in has tried to look at this over time: Bing Xie, Yezhou Huang, Jeffrey S. Chase, Jong Youl Choi, Scott Klasky, Jay Lofstead, and Sarp Oral. 2017. Predicting Output Performance of a Petascale Supercomputer. In Proceedings of the 26th International Symposium on High-Performance Parallel and Distributed Computing (HPDC '17). ACM, New York, NY, USA, 181-192. DOI: https://doi.org/10.1145/3078597.3078614 The bottom line we've seen is that the variability can be pretty extreme. Over the course of an application run, the performance is relatively stable and predictable though. The other way to address trying to get the best overall performance is to work around hot spots (such as my SC10 paper describes). Quality of service is making inroads, but few are willing to give up possibly achieving the best possible performance for a guaranteed performance. UCSC has some work I don't think has been published yet that does a fantastic job at QoS reservations and sharing. Ivo Jimenez's (UCSC) thesis work has investigated how to get performance predictability from platform to platform. There are so many variables--even for a single node--that getting with 5-10% is considered success. As a community, since we have bespoke platforms others cannot replicate and we cannot have 100% control over our experimental environment due to production workloads and shared storage arrays among various platforms, we have to have a different metric for replication (as opposed to reproducibility*). Ivo hasn't been able to get the community to come to agreement yet and he is far from the only person trying to drive this (I know of at least 4 or 5 others that are also pushing). What we can mostly agree on is that what the data shows in terms of trends and magnitudes, under controlled conditions, should represent the same results indicating the same conclusions. All of this aside, 100x difference in performance is extreme. I've seen 10x plenty of times and 50x+ on a few machines at Sandia (capacity machines with really odd workloads and highly shared storage arrays). Since we are trying to get a number that shows what the best possible result is on a platform for a single set of benchmark runs (meaning you can't pick the ior easy from one and ior hard from another), we hope that things are somewhat stable. Unless you can control everything to avoid any simultaneous users, and still deal with thermal issues or other hardware related performance variations, getting to exactly the same results is not a reasonable expectation. Ultimately, I think the question is more should IO-500 show the best possible or average cases? Our attempt to make this more relevant for ordinary users is to allow the easy tests with full disclosure of how to achieve those results. If a user tunes using that information, they should expect to be better than the hard results, but probably not as good as easy since they are unlikely to be able to form their data into exactly the same IO pattern that easy uses. Letting someone run multiple times, even with wide swings in performance, should not change the ability to use the easy settings and tunings to help users achieve better results than the hard configuration. * All of science other than computer science uses reproducibility to mean that you can take the raw data and generate the same analysis results. Replication means rerunning the experiments to generate data that shows the same results. Best, Jay Lofstead on behalf of the IO-500 Committee. On 1/10/19, 8:53 AM, "IO-500 on behalf of Carlile, Ken via IO-500" <io-500-bounces(a)vi4io.org on behalf of io-500(a)vi4io.org&gt; wrote: The second paper you cited is fascinating and points to a number of issues I see not only in this field of academic research but many others. I can't tell you how many times I've heard that extant data is critical since it can't be reproduced... if you can't reproduce it, are you really doing science? (gonna get myself into trouble here). I don't have immediate access to the first, as I am not a full ACM member, but regardless, thank you for passing along this information. --Ken > On Jan 10, 2019, at 10:36 AM, Benson Muite <benson.muite(a)ut.ee&gt; wrote: > > There are a few cases where an absolute performance guarantee is > helpful. In other cases, cost considerations and data utility may allow > for expected but not guaranteed performance - how to measure this > effectively in a broadly applicable manner seems challenging. > > Regards, > > Benson > > On 1/10/19 5:30 PM, Carlile, Ken wrote: >> I will have a look at those papers. I can get system load information to some degree, and that's probably an avenue I should investigate more fully, considering I'm sharing this test load with people doing actual work on the HPC cluster, and in certain cases on the storage as well, although I look for down times on both. >> >> I have to simultaneously agree and disagree on your last statement. Considering that the end goal here is to advise on the strengths and weaknesses of various storage systems, if we throw our hands in the air and say that typical performance varies wildly, the point of the exercise becomes somewhat moot. If you can't trust your results to stand up to an average (or other mathematical leveling), are they really valid? >> >> --Ken >> >>> On Jan 10, 2019, at 10:20 AM, Benson Muite via IO-500 <io-500(a)vi4io.org&gt; wrote: >>> >>> There are a number of relevant papers. Ones that come to mind are: >>> >>> Kevin A. Brown, Nikhil Jain, Satoshi Matsuoka, Martin Schulz , Abhinav >>> Bhatele "Interference between I/O and MPI Traffic on Fat-tree Networks" >>> http://dx.doi.org/10.1145/3225058.3225144 >>> >>> T. Hoefler, R. Belli "Scientific Benchmarking of Parallel Computing >>> Systems" https://htor.inf.ethz.ch/publications/index.php?pub=222 >>> >>> Are you able to get system load information? Figuring out conditions to >>> get best and worst performance may be helpful. Typical performance will >>> likely depend on system design and system load. This may be difficult to >>> quantify with just an average. >>> >>> Regards, >>> >>> Benson >>> >>> On 1/10/19 5:10 PM, Carlile, Ken via IO-500 wrote: >>> >>>> As I've been playing with the IO500 benchmark on various systems, I'm seeing a fair amount of variability between runs. This may ultimately be to my detriment, but have there been any discussions of perhaps requiring the mean of 3 or 5 runs for the official numbers? I think the most striking examples are when I've managed to hit some kind of storage side caching just right and produced numbers, particularly on mdtest_hard_stat that are an order of magnitude greater on one run vs other runs. Now, I really LIKE having a higher score, but it doesn't seem exactly fair or representative... >>>> >>>> --Ken Carlile >>>> _______________________________________________ >>>> IO-500 mailing list >>>> IO-500(a)vi4io.org >>>> https://www.vi4io.org/mailman/listinfo/io-500 >>> >>> _______________________________________________ >>> IO-500 mailing list >>> IO-500(a)vi4io.org >>> https://www.vi4io.org/mailman/listinfo/io-500 _______________________________________________ IO-500 mailing list IO-500(a)vi4io.org https://www.vi4io.org/mailman/listinfo/io-500

On Jan 24, 2019, at 1:56 PM, Lofstead, Gerald F II <gflofst(a)sandia.gov&gt; wrote: Hi Ken, Glad to help. If you need anything to help get things going, one of us will be happy to help. Best, Jay On 1/24/19, 11:02 AM, "Carlile, Ken" <carlilek(a)janelia.hhmi.org&gt; wrote: Hi Jay, Thank you for the very detailed reply to my question, and I apologize for my initial naiveté on the matter. It is certainly an issue that everything on this list is a one off! Nevertheless, it does allow us to make some interesting comparisons. I hope to get my data in such a state that I'm comfortable with posting it soon. --Ken > On Jan 24, 2019, at 12:00 PM, Lofstead, Gerald F II <gflofst(a)sandia.gov&gt; wrote: > > Hi Ken, > > BTW, most authors have a personal page with all of their papers posted. It is a good way to avoid the digital library requirements. Those are not supposed to be camera ready versions, but they are really close to what is published. Also, if you email the authors, I would expect they would be happy to share their paper. Someone wanting to read your work is a good thing. :-) > > Sorry for the long delay. The committee wanted to talk about this before responding. I've also done some work on this issue exactly in my SC10 paper: > Jay Lofstead, Fang Zheng, Qing Liu, Scott Klasky, Ron Oldfield, Todd Kordenbrock, Karsten Schwan, Matthew Wolf. "Managing Variability in the IO Performance of Petascale Storage Systems". In Proceedings of SC 10. New Orleans, LA. November 2010. > > I agree that variability is a serious problem for storage performance. Other work I've been involved in has tried to look at this over time: > Bing Xie, Yezhou Huang, Jeffrey S. Chase, Jong Youl Choi, Scott Klasky, Jay Lofstead, and Sarp Oral. 2017. Predicting Output Performance of a Petascale Supercomputer. In Proceedings of the 26th International Symposium on High-Performance Parallel and Distributed Computing (HPDC '17). ACM, New York, NY, USA, 181-192. DOI: https://doi.org/10.1145/3078597.3078614 > > The bottom line we've seen is that the variability can be pretty extreme. Over the course of an application run, the performance is relatively stable and predictable though. The other way to address trying to get the best overall performance is to work around hot spots (such as my SC10 paper describes). Quality of service is making inroads, but few are willing to give up possibly achieving the best possible performance for a guaranteed performance. UCSC has some work I don't think has been published yet that does a fantastic job at QoS reservations and sharing. > > Ivo Jimenez's (UCSC) thesis work has investigated how to get performance predictability from platform to platform. There are so many variables--even for a single node--that getting with 5-10% is considered success. As a community, since we have bespoke platforms others cannot replicate and we cannot have 100% control over our experimental environment due to production workloads and shared storage arrays among various platforms, we have to have a different metric for replication (as opposed to reproducibility*). Ivo hasn't been able to get the community to come to agreement yet and he is far from the only person trying to drive this (I know of at least 4 or 5 others that are also pushing). What we can mostly agree on is that what the data shows in terms of trends and magnitudes, under controlled conditions, should represent the same results indicating the same conclusions. > > All of this aside, 100x difference in performance is extreme. I've seen 10x plenty of times and 50x+ on a few machines at Sandia (capacity machines with really odd workloads and highly shared storage arrays). Since we are trying to get a number that shows what the best possible result is on a platform for a single set of benchmark runs (meaning you can't pick the ior easy from one and ior hard from another), we hope that things are somewhat stable. Unless you can control everything to avoid any simultaneous users, and still deal with thermal issues or other hardware related performance variations, getting to exactly the same results is not a reasonable expectation. > > Ultimately, I think the question is more should IO-500 show the best possible or average cases? Our attempt to make this more relevant for ordinary users is to allow the easy tests with full disclosure of how to achieve those results. If a user tunes using that information, they should expect to be better than the hard results, but probably not as good as easy since they are unlikely to be able to form their data into exactly the same IO pattern that easy uses. Letting someone run multiple times, even with wide swings in performance, should not change the ability to use the easy settings and tunings to help users achieve better results than the hard configuration. > > * All of science other than computer science uses reproducibility to mean that you can take the raw data and generate the same analysis results. Replication means rerunning the experiments to generate data that shows the same results. > > Best, > > Jay Lofstead on behalf of the IO-500 Committee. > > On 1/10/19, 8:53 AM, "IO-500 on behalf of Carlile, Ken via IO-500" <io-500-bounces(a)vi4io.org on behalf of io-500(a)vi4io.org&gt; wrote: > > The second paper you cited is fascinating and points to a number of issues I see not only in this field of academic research but many others. I can't tell you how many times I've heard that extant data is critical since it can't be reproduced... if you can't reproduce it, are you really doing science? (gonna get myself into trouble here). > > I don't have immediate access to the first, as I am not a full ACM member, but regardless, thank you for passing along this information. > > --Ken > >> On Jan 10, 2019, at 10:36 AM, Benson Muite <benson.muite(a)ut.ee&gt; wrote: >> >> There are a few cases where an absolute performance guarantee is >> helpful. In other cases, cost considerations and data utility may allow >> for expected but not guaranteed performance - how to measure this >> effectively in a broadly applicable manner seems challenging. >> >> Regards, >> >> Benson >> >> On 1/10/19 5:30 PM, Carlile, Ken wrote: >>> I will have a look at those papers. I can get system load information to some degree, and that's probably an avenue I should investigate more fully, considering I'm sharing this test load with people doing actual work on the HPC cluster, and in certain cases on the storage as well, although I look for down times on both. >>> >>> I have to simultaneously agree and disagree on your last statement. Considering that the end goal here is to advise on the strengths and weaknesses of various storage systems, if we throw our hands in the air and say that typical performance varies wildly, the point of the exercise becomes somewhat moot. If you can't trust your results to stand up to an average (or other mathematical leveling), are they really valid? >>> >>> --Ken >>> >>>> On Jan 10, 2019, at 10:20 AM, Benson Muite via IO-500 <io-500(a)vi4io.org&gt; wrote: >>>> >>>> There are a number of relevant papers. Ones that come to mind are: >>>> >>>> Kevin A. Brown, Nikhil Jain, Satoshi Matsuoka, Martin Schulz , Abhinav >>>> Bhatele "Interference between I/O and MPI Traffic on Fat-tree Networks" >>>> http://dx.doi.org/10.1145/3225058.3225144 >>>> >>>> T. Hoefler, R. Belli "Scientific Benchmarking of Parallel Computing >>>> Systems" https://htor.inf.ethz.ch/publications/index.php?pub=222 >>>> >>>> Are you able to get system load information? Figuring out conditions to >>>> get best and worst performance may be helpful. Typical performance will >>>> likely depend on system design and system load. This may be difficult to >>>> quantify with just an average. >>>> >>>> Regards, >>>> >>>> Benson >>>> >>>> On 1/10/19 5:10 PM, Carlile, Ken via IO-500 wrote: >>>> >>>>> As I've been playing with the IO500 benchmark on various systems, I'm seeing a fair amount of variability between runs. This may ultimately be to my detriment, but have there been any discussions of perhaps requiring the mean of 3 or 5 runs for the official numbers? I think the most striking examples are when I've managed to hit some kind of storage side caching just right and produced numbers, particularly on mdtest_hard_stat that are an order of magnitude greater on one run vs other runs. Now, I really LIKE having a higher score, but it doesn't seem exactly fair or representative... >>>>> >>>>> --Ken Carlile >>>>> _______________________________________________ >>>>> IO-500 mailing list >>>>> IO-500(a)vi4io.org >>>>> https://www.vi4io.org/mailman/listinfo/io-500 >>>> >>>> _______________________________________________ >>>> IO-500 mailing list >>>> IO-500(a)vi4io.org >>>> https://www.vi4io.org/mailman/listinfo/io-500 > > _______________________________________________ > IO-500 mailing list > IO-500(a)vi4io.org > https://www.vi4io.org/mailman/listinfo/io-500 > >