Dear all,

This email is being sent to the IO500 community mailing list with bcc’s to all historical submitters. This email pertains to changes made to the benchmark for SC’19 as well as possible implications from those changes.  Much, if not all, of this information has already been discussed in our Slack channel but, at the risk of being redundant, we are sending it again to ensure maximum visibility.

We have introduced one major change for SC’19:  we have added a “shift” between the mdtest phases so that the set of processes doing creates, the set of processes doing stats, the set of processes doing reads, and the set of processes doing deletes are disjoint sets.  Additionally, but perhaps less materially to this email, bugs in IOR/mdtest source code were fixed to ensure correctly reported results and we have insisted on using stonewalling to more accurately and homogeneously measure the “straggler” effect.

The mdtest “shift” is designed to guarantee that there are no client-side caching effects artificially inflating any of the mdtest results.  We had hoped that client-side caching effects were already avoided due to the interleaving of the phases and due to the 300 second required minimum runtime.  However, we were starting to see a few outlier submissions that were achieving results that were obviously benefiting from client-side cache since they were physically higher than would be possible without caching.

It has been our hope that this change will not impact the historical results because the only submissions that were benefiting from client-side caching were the obvious outliers which we have already removed.  However, if our hope is unfounded and the new rules cause discernible differences, then we may decide a new list is warranted and we may accordingly archive the old results and start anew with SC’19.  

After we get all of the SC19 submissions, we will analyze them and make this decision.  We do not yet have firm quantitative criteria for making this judgment; we are considering simply looking at scatter-plots and using our best judgement.  Certainly, if anyone in the community can suggest better statistical analysis methods, we would like to hear about them.  

Given all this, we strongly encourage all historical submitters to submit new results for SC’19.  This will better enable us to make our decision and this will ensure full inclusion if a new list is warranted.  To provide further motivation, and in the interests of full disclosure, some early results that we have seen do suggest that the rule change might be more significant than we hoped.

Ultimately, we are happy about this change which frankly was overdue even if it may unfortunately result in a new list.  As is evidenced by the fact that IOR has always had the comparable “shift” and by the interleaving of the phases, it has always been our intention to avoid client-side caching.  Unfortunately, the mdtest “shift” was never properly implemented until just a few months ago. Change is never easy and we appreciate the patience of the community as we continue to strive to ensure that IO500 is fair, representative, and useful to all.

Thanks,

The IO500 Committee