4:27 p.m.
All,
The below is an out-of-band conversation I had about “gaming” the system that I wanted to
share with all of us. I believe the summary is that ‘gaming’ might be possible and just
requires us to periodically update our benchmarks if it happens. Also that attempts to
game will generally result in better systems.
Thanks,
John
--------------------------------
>> The idea is that this bounds the performance of a system. A complaint at the BoF
was from users who port their code to run in a new place and then don’t know if they are
getting reasonable IO performance. With these measurements, the users should at least
know their upper and lower bounds for both metadata and data.
>
> Agree. (Until someone figures out how to game the “really hard” situations)
>
Yes, this was a popular sentiment at our SC BOF. To which my response is, “Gaming the
really hard situations improves the storage system for IO workloads that have really hard
IO patterns.”
Hopefully, yes.
I suppose by definition “gaming” means cheating to get a good result
on the benchmark without actually improving the system for real workloads. I suppose I
just don’t think that’s possible. If you can get a good score on this benchmark, you’ll
have made a general good and useful storage system. That’s my current belief anyway.
Happy to be proved wrong.
I agree with you in general, and if it turns out that someone is getting great scores on
the benchmark but users aren’t seeing good performance, then the benchmark needs to be
updated.
10:02 a.m.
Dear John,
thanks for sharing.
The below is an out-of-band conversation I had about “gaming” the
system that I wanted to share with all of us. I believe the summary is that ‘gaming’
might be possible and just requires us to periodically update our benchmarks if it
happens.
agree what you say: The general rules should not allow cheating that
does not result in better systems.
I hope that we won't have to adjust the benchmark -- at least not periodically.
A proper definition of the intention of the benchmarks should help:
- IO-easy: upper bound for optimized IO-heavy workloads. An optimized
application should be able to extract it.
That usually means: Sequential large-bulk I/O with good domain
decomposition within the application.
- IO-hard: expected performance for non-optimized applications. i.e.
more random workload, shared object.
- MD-easy: well-behaving small-object/metadata workload, for an
optimized workflow needing these objects
- MD-workload: typical non-coordinated small-object/metadata workload,
i.e., interactive usage by many users
Just some thoughts.
Regards,
Julian
2583
days inactive
2586
days old
1 comments
2 participants
participants (2)
-
John Bent -
Julian Kunkel