6:09 p.m.
On Oct 23, 2019, at 8:11 PM, Ruibo Wang via IO-500 <io-500(a)vi4io.org> wrote:
Dear All,
I've noticed that there is a newly added "-N 1" parameter to the mdtest
in the lastest SC19 version. I guess it would influence the stat result dramatically.
If we use the new parameter, how could we compete the old results in the IO500 list?
The old and new results will not be directly comparable.
An idea I proposed to the chat yesterday was to add a new "mdtest-harder-read"
test that uses the "-N" parameter to be run some time *after* the original
"mdtest-hard-read" test is run, so that the new test results are (mostly) a
superset of the existing results. That will allow computing scores like the
old isc-19 results using only "mdtest-hard-read" (they will be similar, but
not exactly the same), and then using only "mdtest-harder-read" for sc-19
and beyond.
We were also discussing to add an "ior-harder-read" test that is doing 4KB
*random* IOPS to better simulate AI/ML workloads. As above, it would be
better to add this as a new test in addition to the existing "ior-hard-read"
so that old and new test runs could be compared more easily (the IO500
website makes it easy to change the formula for computing the scores), yet
new runs could include the new result. Old runs would automatically be
marked "invalid" because they are missing one of the results (essentially
the geometric mean of anything with a zero result in it is zero).
It _might_ be a bit late for adding the separate mdtest-harder-read test
for io500-sc-19, but it would be (IMHO) a far superior solution to the
current change that breaks all of the old results, so I would definitely
be in favour of this change even if it is late. It wouldn't invalidate
any results that *didn't* have the separate mdtest-harder-read results,
but submissions that came in with "-N" before this change would have to
map their mdtest-hard-read results to mdtest-harder-read in the database.
Cheers, Andreas