3:11 p.m.
Dear all,
This email is being sent to the IO500 community mailing list with bcc’s to
all historical submitters. This email pertains to changes made to the
benchmark for SC’19 as well as possible implications from those changes.
Much, if not all, of this information has already been discussed in our
Slack channel but, at the risk of being redundant, we are sending it again
to ensure maximum visibility.
We have introduced one major change for SC’19: we have added a “shift”
between the mdtest phases so that the set of processes doing creates, the
set of processes doing stats, the set of processes doing reads, and the set
of processes doing deletes are disjoint sets. Additionally, but perhaps
less materially to this email, bugs in IOR/mdtest source code were fixed to
ensure correctly reported results and we have insisted on using
stonewalling to more accurately and homogeneously measure the “straggler”
effect.
The mdtest “shift” is designed to guarantee that there are no client-side
caching effects artificially inflating any of the mdtest results. We had
hoped that client-side caching effects were already avoided due to the
interleaving of the phases and due to the 300 second required minimum
runtime. However, we were starting to see a few outlier submissions that
were achieving results that were obviously benefiting from client-side
cache since they were physically higher than would be possible without
caching.
It has been our hope that this change will not impact the historical
results because the only submissions that were benefiting from client-side
caching were the obvious outliers which we have already removed. However,
if our hope is unfounded and the new rules cause discernible differences,
then we may decide a new list is warranted and we may accordingly archive
the old results and start anew with SC’19.
After we get all of the SC19 submissions, we will analyze them and make
this decision. We do not yet have firm quantitative criteria for making
this judgment; we are considering simply looking at scatter-plots and using
our best judgement. Certainly, if anyone in the community can suggest
better statistical analysis methods, we would like to hear about them.
Given all this, we strongly encourage all historical submitters to submit
new results for SC’19. This will better enable us to make our decision and
this will ensure full inclusion if a new list is warranted. To provide
further motivation, and in the interests of full disclosure, some early
results that we have seen do suggest that the rule change might be more
significant than we hoped.
Ultimately, we are happy about this change which frankly was overdue even
if it may unfortunately result in a new list. As is evidenced by the fact
that IOR has always had the comparable “shift” and by the interleaving of
the phases, it has always been our intention to avoid client-side caching.
Unfortunately, the mdtest “shift” was never properly implemented until just
a few months ago. Change is never easy and we appreciate the patience of
the community as we continue to strive to ensure that IO500 is fair,
representative, and useful to all.
Thanks,
The IO500 Committee