Just my 0.02$ here, but I think the usefulness of the IO-500 would be to get the details of the how the storage system is tuned and what tuning options were used when running the job. Reporting of say all the Lustre/GPFS tuning parameters for a given storage system would be nice to know. For example, an IOR test done using the exact same compute system and storage system would give different results if it was GPFS was configured with a 256 KiB block size as opposed to a 8 MiB block size. I think people submitting "optimized" results with the result setup documented and all of the storage/FS configuration documented is more useful than trying to converge on a set of run rules. For example, 5 minutes of runtime sounds good but what happens if someone deploys "burst buffer type" storage that can write 100% of memory in less than a minute? kevin ________________________________________ From: IO-500 <io-500-bounces(a)vi4io.org&gt; on behalf of John Bent <John.Bent(a)seagategov.com&gt; Sent: Friday, June 16, 2017 3:56:09 AM To: io-500(a)vi4io.org Subject: [IO-500] detailed benchmark proposal All, Sorry for the long silence on the mailing list. However, we have made some substantial progress recently as we prepare for our ISC BOF next week. For those of you at ISC, please join us from 11 to 12 on Tuesday in Substanz 1&2. The progress that we have made recently happened because a bunch of us were attending a German workshop last month at Dagstuhl and had multiple discussions about the benchmark. Here’s the highlights from what was discussed and the progress that we made at Dagstuhl: 1. General agreement that the IOR-hard, IOR-easy, mdtest-hard, mdtest-easy approach is appropriate. 2. We should add a ‘find’ command as this is a popular and important workload. 3. The multiple bandwidth measurements should be combined via geometric mean into one bandwidth. 4. The multiple IOPs measurements should also be combined via geometric mean into one IOPs. 5. The bandwidth and the IOPs should be multiplied to create one final score. 6. The ranking uses that final score but the webpage can be sorted using other metrics. 7. The webpage should allow filtering as well so, for example, people can look at only the HDD results. 8. We should separate the write/create phases from the read/stat phases to help ensure that caching is avoided 9. Nathan Hjelm volunteered to combine the mdtest and IOR benchmarks into one git repo and has now done so. This removes the #ifdef mess from mdtest and now they both share the nice modular IOR backend So the top-level summary of the benchmark in pseudo-code has become: # write/create phase bw1 = ior_easy -write [user supplies their own parameters maximizing data writes that can be done in 5 minutes] md1 = md_test_easy -create [user supplies their own parameters maximizing file creates that can be done in 5 minutes] bw2 = ior_hard -write [we supply parameters: unaligned strided into single shared file] md2 = md_test_hard -create [we supply parameters: creates of 3900 byte files into single shared directory] # read/stat phase bw3 = ior_easy -read [cross-node read of everything that was written in bw1] md3 = md_test_easy -stat [cross-node stat of everything that was created in md1] bw4 = ior_hard -read md4 = md_test_hard -stat # find phase md5 = [we supply parameters to find a subset of the files that were created in the tests] # score phase bw = geo_mean( bw1 bw2 bw3 bw4) md = geo_mean( md1 md2 md3 md4 bd5) total = bw * md Now we are moving on to precisely define what the parameters should look like for the hard tests and to create a standard so that people can start running it on their systems. By doing so, we will define the formal process so we can actually make this an official benchmark. Please see the attached file in which we’ve started precisely defining these parameters. Let’s start iterating please on this file to get these parameters correct. Thanks, John

On Jun 20, 2017, at 1:04 AM, Harms, Kevin <harms@alcf.anl.gov<mailto:harms@alcf.anl.gov>> wrote: Just my 0.02$ here, but I think the usefulness of the IO-500 would be to get the details of the how the storage system is tuned and what tuning options were used when running the job. Reporting of say all the Lustre/GPFS tuning parameters for a given storage system would be nice to know. For example, an IOR test done using the exact same compute system and storage system would give different results if it was GPFS was configured with a 256 KiB block size as opposed to a 8 MiB block size. I think people submitting "optimized" results with the result setup documented and all of the storage/FS configuration documented is more useful than trying to converge on a set of run rules. Thanks Kevin, I’ve been sending a flurry of emails while on the plane and some of this I’ve already discussed which is that we absolutely are in agreement with you that the value is to collect and document performance tuning and IO patterns that maximize storage performance. In terms of your point about it being more "useful than trying to converge on a set of run rules,” I think we are also in agreement. That is definitely the value. Converging on the set of run rules however is the means to the end. We believe we will have more much participation and much better collection of useful community information if we turn it into one of the 500’s. Thanks, John