Glenn and all,

To add to what Julian said about anonymous: some of you may know that I work for one of the afore-mentioned vendors that uses IO500 for bragging rights.  As such, people who want anonymous submissions are absolutely welcome to keep them shrouded from me.  

In order to do so, please email submissions directly to Julian <juliankunkel@googlemail.com> and to Jay <gflofst@sandia.gov> and ask that your anonymity be protected.  If you email to submit@io500.org or use the online submission tool, I will see that.  We will update our documentation and submission instructions to reflect this.  If you are not worried about me but do want an anonymous submission, you can submit regularly and just ask for anonymity in the publication of your results.

Also, please let me take this opportunity to reassure the community that I do not discuss any submissions outside of the committee before the list is announced.

Thanks,

John

On Wed, Jun 13, 2018 at 10:42 AM, Julian Kunkel <juliankunkel@googlemail.com> wrote:
Dear Glenn,
thanks for your request.

If by anonymous, you mean we won't mention your name on the public
page, that would be fine!
During the submission, we need the submitters name to ensure the data
is correct and sincere.
We will treat the name confidentially, will add something to the
online submission.
However, generally, we may want to add the name of the person
submitted to give him/her credit for doing the work.

Vendors using it for promotion: The goal is not to have vendors
promote it, but of course, the results are open and can be used for
that purpose. If even these hero numbers tell me personally something
that is metadata is still a problem. I actually had submitted for DKRZ
two new values. Metadata performance is not that bad, but I see that
DKRZ has room to improve IO throughput ^-^

Which benchmark:
> 1. the official version, where each benchmark must run for at least five minutes
That is the one to go.

> 2. the stonewall version, where each benchmark is allowed to stop after five minutes
We experimented with this approach but have to push more changes to
IOR and mdtest and allow for communication of the amount of data
between the stages (write => read phase).
This is something we should discuss in respect to the IOR roadmap during ISC.

In the future, I'd expect we allow both, the reason that I had not
moved further with (2, the stonewall version) is that in some settings
it does not work well.
Depending on the file system: some processes run much faster than
others leading to long waiting periods of processes to catch up. I had
this on one machine for metadata and one for data.
An example: in 5 minutes the fastest processor did 20000 creates while
the average creates was 1000, since all processes must then achieve
the maximum number of creates, it will roughly take
100 minutes! That imbalance actually teached me something suboptimal
about the system but leads to practical issues.

I agree, we did a suboptimal job with pfind as second option, we
should make it the default in the future as it confused people and the
default with sfind may lead to extreme waiting times in first runs.

Thanks,
julian

2018-06-12 22:47 GMT+01:00 Glenn Lockwood <glock@lbl.gov>:
> Hi John & Committee
>
> I'd like to run IO-500 on our systems, but I have a few concerns that other
> sites may also share.  In the interests of gaining clarity and resolution
> before the deadline, I figure I would ask them here rather than in private.
>
> Concern #1: The current state of the authoritative IO-500 benchmark
> distribution is a little unclear to me .  As I understand it, there are two
> versions:
>
> 1. the official version, where each benchmark must run for at least five
> minutes
> 2. the stonewall version, where each benchmark is allowed to stop after five
> minutes
>
> In addition, I've been confused by the different options of parallel find.
> It looks like the most sensible one, pfind, is in the "utilities/find/old/"
> directory whose name suggests it is old and I shouldn't be using it.  Is
> this true?
>
> Concern #2: I can't help but notice that several HPC storage vendors have
> been using the IO-500 results for marketing material ("IME is holistically
> faster than DataWarp" and "DataWarp has the fastest peak flash
> performance").  It is therefore conceivable that submitting anything but
> hero numbers could be used to make me, my employer, or our vendor partners
> look bad.  I don't want my center's results being used to show how bad our
> storage solution is, especially if the numbers are only low because I didn't
> tune the benchmarks optimally.
>
> As such, is it possible to submit results, hero or otherwise, anonymously?
> Even though there's only a few 1.6 TB/sec file systems in production in the
> world, even the pretense of anonymity would make me feel more secure in
> submitting sub-optimal (or embarrassing) numbers.
>
> Thanks!
>
> Glenn
>
>
> On Tue, Jun 12, 2018 at 2:25 PM John Bent <johnbent@gmail.com> wrote:
>>
>> Hello IO500 friends,
>>
>> We are at a critical junction for IO500.  Hopefully most of you joined
>> this list because you agree with the motivation behind IO500 and only a few
>> of you joined to laugh at its painful demise.
>>
>> To the former, the committee wants to remind you all that we will unveil
>> the second list at ISC18.  As of now, we do have a few submissions but we
>> fear they may be too few to be sufficiently impressive to ensure our
>> continued relevance.  We are clearly still too new to have achieved critical
>> mass.
>>
>> We remain committed to the community's need for an IO500.  Reporting only
>> hero numbers as was the pre-IO500 status quo hurts us all.  Collecting a
>> large historical data set of IO performance benefits us all.
>>
>> Please help ensure the success of our effort by submitting results
>> yourselves and by encouraging and soliciting others to do so.  The community
>> stands ready to provide assistance as is necessary although please remember
>> that the benchmark is very easy to run.
>>
>> Thanks!
>>
>> John (on behalf of the committee)
>>
>>
>> _______________________________________________
>> IO-500 mailing list
>> IO-500@vi4io.org
>> https://www.vi4io.org/mailman/listinfo/io-500
>
>
> _______________________________________________
> IO-500 mailing list
> IO-500@vi4io.org
> https://www.vi4io.org/mailman/listinfo/io-500
>



--
Dr. Julian Kunkel
Lecturer, Department of Computer Science
+44 (0) 118 378 8218
http://www.cs.reading.ac.uk/
https://hps.vi4io.org/
_______________________________________________
IO-500 mailing list
IO-500@vi4io.org
https://www.vi4io.org/mailman/listinfo/io-500