On Jun 24, 2017, at 8:44 AM, Georgios Markomanolis
<georgios.markomanolis(a)kaust.edu.sa> wrote:
I have tried the command find and it is really slow. It takes 3-4
minutes when I have 1-1.5 million files (I don't remember exact time, I wait my flight
for my holidays). I agree to include find, there is no reason to remove it. The trap here
is, less the MPI processes, less the files you create, the duration of the command find is
shorter but also the result is lower as it searched across smaller number of files. I
don't know if we want to define a rule that minimum X number of files or something
like that.
The time that find takes to run depends on the previously-run mdtest runs, which have a
maximum score when a large number of files are created. We should make sure that the find
result is not at a significantly advantage if it completes faster because the file
creation is very slow and creates few files. That means the result needs to be in
"files per second" or similar and not just the time it takes for find to finish.
I don't think running find in a loop to get a larger runtime is an improvement, since
this would benefit from slower create rates (fewer files) by being able to re-scan the
same files from cache.
Cheers, Andreas
> On Jun 24, 2017, at 5:14 PM, John Bent
<John.Bent(a)seagategov.com> wrote:
>
> Mixed requires a bit of work but I think is worth it.
> Find requires no effort so I see no reason to drop it.
>
> Perhaps one reason not to include it is that it might take an intractable amount of
time. George, have you tried the find part yet? Just how bad is it?
>
> However, this is exactly the point. It's a hard workload and it will make most
systems look bad. If it is optional, then they won't do it because they don't want
to look bad.
>
> We had agreement on find, the community asked for it, we gave it to them. It makes
our steering committee look bad if we change now. I suggest we leave find and figure out
how to add mixed. If we still have our own discussion about this, we should move it to the
full mailing list I think.
>
> Thx
>
> John
>
>> On Jun 24, 2017, at 1:05 AM, Julian Kunkel <juliankunkel(a)googlemail.com>
wrote:
>>
>> Dear John,
>> would move find and (any) concurrent benchmark or other application
>> specific mode to the extended benchmark.
>> Makes it easier to rollout the initial version.
>>
>> Cheers,
>> Julian
>>
>> 2017-06-24 2:05 GMT+02:00 John Bent <John.Bent(a)seagategov.com>:
>>>
>>>> On Jun 23, 2017, at 6:02 PM, Andreas Dilger <adilger(a)dilger.ca>
wrote:
>>>>
>>>> On Jun 23, 2017, at 4:57 PM, John Bent <John.Bent(a)seagategov.com>
wrote:
>>>>>
>>>>> All,
>>>>>
>>>>> We had a great session at ISC (about 30 people I think) and made
great progress in the weeks leading up to it as well. Thanks to Satoshi we even got the
two attached slides added into the official slides being released from the Top 500
session! We had 6 people sign up at the BOF saying that they’ll run the benchmark when it
is finalized.
>>>>>
>>>>> I know I always say that we have ‘almost’ finalized the benchmark.
But we really are getting much closer; it helped so much that Nathan combined the
benchmarks and George worked on the script.
>>>>>
>>>>> I think we only have two open questions right now:
>>>>>
>>>>> 1. do 47K random IO in the IOR-hard or do 47K simple strided? My
original thinking was strided but someone pointed out that the idea is to create the
bounding box and random is harder than strided. Also random might be increasingly
prevalent these days with more analytics and machine learning and graph analytics, etc.
So I propose that we do random unless there are objections here.
>>>>
>>>> May as well go random at that point.
>>>>
>>>>> 2. Should we do some sort of mixed IO workload in addition to running
the 4 tests serially? I like the idea but am not sure how exactly to do it. Do we need
to merely mix IOR-hard and IOR-easy or md-hard and md-easy or both or mix all 4 at once?
Do we just launch multiple command lines in the background and hope that the mpirun launch
times are fast enough that they overlap? Do we need to modify IOR/mdtest to split the
ranks in half and do different workloads with the two halves? Thoughts?
>>>>
>>>> This would be a bit of a dog's breakfast, and very hard to specify
the test parameters. Do all of the loads need to be running for the whole duration? How
does this work if some jobs finish early? What if, for example, there was a workload
scheduler in the storage that (automatically?) segregated the IO of each workload and they
actually ran serially and didn't contend at all? Would that be considered an
improvement, since this could help real-world jobs as well?
>>>>
>>> Agree about the scheduler. But what about a modified IOR that split the
ranks in two and half did easy and the other half did hard?
>>>
>>> Thanks,
>>>
>>> John
>>>
>>>> Maybe something for V2?
>>>>
>>>>> We also made some procedural decisions. The initial steering
committee will be Jay Lofstead, Julian Kunkle, and myself. That steering committee
membership will last until IO500 is up and running and stable at which point the community
can nominate new members. All decisions will be discussed first on the mailing list and
we will try for as much consensus as possible. The VI4IO organization will host the
IO500.
>>>>>
>>>>> Thanks very much,
>>>>>
>>>>> John
>>>>>
<io500_two_slides.pdf>_______________________________________________
>>>>> IO-500 mailing list
>>>>> IO-500(a)vi4io.org
>>>>>
https://www.vi4io.org/cgi-bin/mailman/listinfo/io-500
>>>>
>>>>
>>>> Cheers, Andreas
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>> _______________________________________________
>>> IO-500 mailing list
>>> IO-500(a)vi4io.org
>>>
https://www.vi4io.org/cgi-bin/mailman/listinfo/io-500
>>
>>
>>
>> --
>>
http://wr.informatik.uni-hamburg.de/people/julian_kunkel
> _______________________________________________
> IO-500 mailing list
> IO-500(a)vi4io.org
>
https://www.vi4io.org/cgi-bin/mailman/listinfo/io-500
________________________________
This message and its contents including attachments are intended solely for the original
recipient. If you are not the intended recipient or have received this message in error,
please notify me immediately and delete this message from your computer system. Any
unauthorized use or distribution is prohibited. Please consider the environment before
printing this email.
Cheers, Andreas