Glenn and all,
To add to what Julian said about anonymous: some of you may know that I
work for one of the afore-mentioned vendors that uses IO500 for bragging
rights. As such, people who want anonymous submissions are absolutely
welcome to keep them shrouded from me.
In order to do so, please email submissions directly to Julian <
juliankunkel(a)googlemail.com> and to Jay <gflofst(a)sandia.gov> and ask that
your anonymity be protected. If you email to submit(a)io500.org or use the
online submission tool, I *will* see that. We will update our
documentation and submission instructions to reflect this. If you are not
worried about me but do want an anonymous submission, you can submit
regularly and just ask for anonymity in the publication of your results.
Also, please let me take this opportunity to reassure the community that I
do not discuss any submissions outside of the committee before the list is
announced.
Thanks,
John
On Wed, Jun 13, 2018 at 10:42 AM, Julian Kunkel <juliankunkel(a)googlemail.com
wrote:
> Dear Glenn,
> thanks for your request.
>
> If by anonymous, you mean we won't mention your name on the public
> page, that would be fine!
> During the submission, we need the submitters name to ensure the data
> is correct and sincere.
> We will treat the name confidentially, will add something to the
> online submission.
> However, generally, we may want to add the name of the person
> submitted to give him/her credit for doing the work.
>
> Vendors using it for promotion: The goal is not to have vendors
> promote it, but of course, the results are open and can be used for
> that purpose. If even these hero numbers tell me personally something
> that is metadata is still a problem. I actually had submitted for DKRZ
> two new values. Metadata performance is not that bad, but I see that
> DKRZ has room to improve IO throughput ^-^
>
> Which benchmark:
> > 1. the official version, where each benchmark must run for at least five
> minutes
> That is the one to go.
>
> > 2. the stonewall version, where each benchmark is allowed to stop after
> five minutes
> We experimented with this approach but have to push more changes to
> IOR and mdtest and allow for communication of the amount of data
> between the stages (write => read phase).
> This is something we should discuss in respect to the IOR roadmap during
> ISC.
>
> In the future, I'd expect we allow both, the reason that I had not
> moved further with (2, the stonewall version) is that in some settings
> it does not work well.
> Depending on the file system: some processes run much faster than
> others leading to long waiting periods of processes to catch up. I had
> this on one machine for metadata and one for data.
> An example: in 5 minutes the fastest processor did 20000 creates while
> the average creates was 1000, since all processes must then achieve
> the maximum number of creates, it will roughly take
> 100 minutes! That imbalance actually teached me something suboptimal
> about the system but leads to practical issues.
>
> I agree, we did a suboptimal job with pfind as second option, we
> should make it the default in the future as it confused people and the
> default with sfind may lead to extreme waiting times in first runs.
>
> Thanks,
> julian
>
> 2018-06-12 22:47 GMT+01:00 Glenn Lockwood <glock(a)lbl.gov>:
> > Hi John & Committee
> >
> > I'd like to run IO-500 on our systems, but I have a few concerns that
> other
> > sites may also share. In the interests of gaining clarity and resolution
> > before the deadline, I figure I would ask them here rather than in
> private.
> >
> > Concern #1: The current state of the authoritative IO-500 benchmark
> > distribution is a little unclear to me . As I understand it, there are
> two
> > versions:
> >
> > 1. the official version, where each benchmark must run for at least five
> > minutes
> > 2. the stonewall version, where each benchmark is allowed to stop after
> five
> > minutes
> >
> > In addition, I've been confused by the different options of parallel
> find.
> > It looks like the most sensible one, pfind, is in the
> "utilities/find/old/"
> > directory whose name suggests it is old and I shouldn't be using it. Is
> > this true?
> >
> > Concern #2: I can't help but notice that several HPC storage vendors have
> > been using the IO-500 results for marketing material ("IME is
> holistically
> > faster than DataWarp" and "DataWarp has the fastest peak flash
> > performance"). It is therefore conceivable that submitting anything but
> > hero numbers could be used to make me, my employer, or our vendor
> partners
> > look bad. I don't want my center's results being used to show how bad
> our
> > storage solution is, especially if the numbers are only low because I
> didn't
> > tune the benchmarks optimally.
> >
> > As such, is it possible to submit results, hero or otherwise,
> anonymously?
> > Even though there's only a few 1.6 TB/sec file systems in production in
> the
> > world, even the pretense of anonymity would make me feel more secure in
> > submitting sub-optimal (or embarrassing) numbers.
> >
> > Thanks!
> >
> > Glenn
> >
> >
> > On Tue, Jun 12, 2018 at 2:25 PM John Bent <johnbent(a)gmail.com
wrote:
> >>
> >> Hello IO500 friends,
> >>
> >> We are at a critical junction for IO500. Hopefully most of you joined
> >> this list because you agree with the motivation behind IO500 and only a
> few
> >> of you joined to laugh at its painful demise.
> >>
> >> To the former, the committee wants to remind you all that we will unveil
> >> the second list at ISC18. As of now, we do have a few submissions but
> we
> >> fear they may be too few to be sufficiently impressive to ensure our
> >> continued relevance. We are clearly still too new to have achieved
> critical
> >> mass.
> >>
> >> We remain committed to the community's need for an IO500. Reporting
> only
> >> hero numbers as was the pre-IO500 status quo hurts us all. Collecting a
> >> large historical data set of IO performance benefits us all.
> >>
> >> Please help ensure the success of our effort by submitting results
> >> yourselves and by encouraging and soliciting others to do so. The
> community
> >> stands ready to provide assistance as is necessary although please
> remember
> >> that the benchmark is very easy to run.
> >>
> >> Thanks!
> >>
> >> John (on behalf of the committee)
> >>
> >>
> >> _______________________________________________
> >> IO-500 mailing list
> >> IO-500(a)vi4io.org
> >>
https://www.vi4io.org/mailman/listinfo/io-500
> >
> >
> > _______________________________________________
> > IO-500 mailing list
> > IO-500(a)vi4io.org
> >
https://www.vi4io.org/mailman/listinfo/io-500
> >
>
>
>
> --
> Dr. Julian Kunkel
> Lecturer, Department of Computer Science
> +44 (0) 118 378 8218
>
http://www.cs.reading.ac.uk/
>
https://hps.vi4io.org/
> _______________________________________________
> IO-500 mailing list
> IO-500(a)vi4io.org
>
https://www.vi4io.org/mailman/listinfo/io-500
>