Dear John,
thanks for sharing.
The below is an out-of-band conversation I had about “gaming” the
system that I wanted to share with all of us. I believe the summary is that ‘gaming’
might be possible and just requires us to periodically update our benchmarks if it
happens.
agree what you say: The general rules should not allow cheating that
does not result in better systems.
I hope that we won't have to adjust the benchmark -- at least not periodically.
A proper definition of the intention of the benchmarks should help:
- IO-easy: upper bound for optimized IO-heavy workloads. An optimized
application should be able to extract it.
That usually means: Sequential large-bulk I/O with good domain
decomposition within the application.
- IO-hard: expected performance for non-optimized applications. i.e.
more random workload, shared object.
- MD-easy: well-behaving small-object/metadata workload, for an
optimized workflow needing these objects
- MD-workload: typical non-coordinated small-object/metadata workload,
i.e., interactive usage by many users
Just some thoughts.
Regards,
Julian