2:05 a.m.
Hi,
I would like to remind an issue regarding mdtest-hard benchmark. It
creates files in a single directory for at least five minutes. When
the metadata performance is more improved, the greater number of files
will be created. This will hit the limit of the number of files in a
single directory.
Actually, the limit of Lustre file system is about 8M files although
it depends the length of the pathname. When the metadata performance
is better than 27K IOPS, it is not possible to execute the mdtest-hard
create benchmark for five minutes since the number of files is larger
than 8M.
This is a reason we cannot submit the Lustre result. I expect this
will be quite common when the metadata performance is more improved.
Regards,
Osamu
---
Osamu Tatebe, Ph.D.
Center for Computational Sciences, University of Tsukuba
1-1-1 Tennodai, Tsukuba, Ibaraki 3058577 Japan
9:57 a.m.
Dear Osama,
thanks for pointing this out.
I was not aware of this problem.
I now made some modifications to mdtest to compensate for this limitation.
Now it can limit the number of files per directories creating one
extra top-level directory if needed.
E.g. with a limit of 50 files and a total of 200 files, one would see
in the top-level testing directory:
#test-dir.0-0
#test-dir.0-1
#test-dir.0-2
#test-dir.0-3
The subdirectory tree of each directory is the complete tree as before
under #test-dir.0-0.
There is no synchronization when switching between these trees.
I expect that this emulates very well the behavior of a single large
directory but well there is no perfect solution.
The stuff is already integrated into the testing branch of io-500-dev.
To use it:
In io-500-dev
$ git checkout testing
$ ./utilities/prepare.sh
in io500.sh
Line 85:
io500_mdtest_hard_other_options=""
Change that to:
io500_mdtest_hard_other_options="-I 8000000"
Which effectively will create a directory for each batch of 8M files.
Note that the number of items (-n) must be a multiple of (-I)
Note that that feature cannot be used with stonewalling at the moment!
At some point, the mdtest code needs a redesign to allow such changes.
Hence, set in L30:
io500_stonewall_timer=0
If you want to give it a try, I would welcome it...
Best,
Julian
2018-07-26 3:05 GMT+01:00 Osamu Tatebe <tatebe(a)cs.tsukuba.ac.jp>:
Hi,
I would like to remind an issue regarding mdtest-hard benchmark. It
creates files in a single directory for at least five minutes. When
the metadata performance is more improved, the greater number of files
will be created. This will hit the limit of the number of files in a
single directory.
Actually, the limit of Lustre file system is about 8M files although
it depends the length of the pathname. When the metadata performance
is better than 27K IOPS, it is not possible to execute the mdtest-hard
create benchmark for five minutes since the number of files is larger
than 8M.
This is a reason we cannot submit the Lustre result. I expect this
will be quite common when the metadata performance is more improved.
Regards,
Osamu
---
Osamu Tatebe, Ph.D.
Center for Computational Sciences, University of Tsukuba
1-1-1 Tennodai, Tsukuba, Ibaraki 3058577 Japan
_______________________________________________
IO-500 mailing list
IO-500(a)vi4io.org
https://www.vi4io.org/mailman/listinfo/io-500
--
Dr. Julian Kunkel
Lecturer, Department of Computer Science
+44 (0) 118 378 8218
http://www.cs.reading.ac.uk/
https://hps.vi4io.org/
6:43 p.m.
Very nice Julian.
However, if we are to consider allowing this as a valid submissions then I propose we add
synchronization between directory switching to ensure fully N-1 behavior. Yes, it does
somewhat unfairly penalize Osamu here. However, not synchronizing somewhat unfairly
advantages him as it allows his fast processes non-contended access to subsequent
directories. If a submission cannot abide by official rules, then it seems better to
unfairly penalize instead of unfairly advantaging it.
Anyone else have thoughts here?
Thanks
John
On Jul 26, 2018, at 5:57 AM, Julian Kunkel
<juliankunkel(a)googlemail.com> wrote:
Dear Osama,
thanks for pointing this out.
I was not aware of this problem.
I now made some modifications to mdtest to compensate for this limitation.
Now it can limit the number of files per directories creating one
extra top-level directory if needed.
E.g. with a limit of 50 files and a total of 200 files, one would see
in the top-level testing directory:
#test-dir.0-0
#test-dir.0-1
#test-dir.0-2
#test-dir.0-3
The subdirectory tree of each directory is the complete tree as before
under #test-dir.0-0.
There is no synchronization when switching between these trees.
I expect that this emulates very well the behavior of a single large
directory but well there is no perfect solution.
The stuff is already integrated into the testing branch of io-500-dev.
To use it:
In io-500-dev
$ git checkout testing
$ ./utilities/prepare.sh
in io500.sh
Line 85:
io500_mdtest_hard_other_options=""
Change that to:
io500_mdtest_hard_other_options="-I 8000000"
Which effectively will create a directory for each batch of 8M files.
Note that the number of items (-n) must be a multiple of (-I)
Note that that feature cannot be used with stonewalling at the moment!
At some point, the mdtest code needs a redesign to allow such changes.
Hence, set in L30:
io500_stonewall_timer=0
If you want to give it a try, I would welcome it...
Best,
Julian
2018-07-26 3:05 GMT+01:00 Osamu Tatebe <tatebe(a)cs.tsukuba.ac.jp>:
> Hi,
>
> I would like to remind an issue regarding mdtest-hard benchmark. It
> creates files in a single directory for at least five minutes. When
> the metadata performance is more improved, the greater number of files
> will be created. This will hit the limit of the number of files in a
> single directory.
>
> Actually, the limit of Lustre file system is about 8M files although
> it depends the length of the pathname. When the metadata performance
> is better than 27K IOPS, it is not possible to execute the mdtest-hard
> create benchmark for five minutes since the number of files is larger
> than 8M.
>
> This is a reason we cannot submit the Lustre result. I expect this
> will be quite common when the metadata performance is more improved.
>
> Regards,
> Osamu
>
> ---
> Osamu Tatebe, Ph.D.
> Center for Computational Sciences, University of Tsukuba
> 1-1-1 Tennodai, Tsukuba, Ibaraki 3058577 Japan
> _______________________________________________
> IO-500 mailing list
> IO-500(a)vi4io.org
> https://www.vi4io.org/mailman/listinfo/io-500
--
Dr. Julian Kunkel
Lecturer, Department of Computer Science
+44 (0) 118 378 8218
http://www.cs.reading.ac.uk/
https://hps.vi4io.org/
_______________________________________________
IO-500 mailing list
IO-500(a)vi4io.org
https://www.vi4io.org/mailman/listinfo/io-500
6:45 p.m.
New subject: [EXTERNAL] Re: mdtest-hard issue
Should we pursue making this a bug report for the file system instead?
On 7/30/18, 12:44 PM, "IO-500 on behalf of John Bent"
<io-500-bounces(a)vi4io.org on behalf of johnbent(a)gmail.com> wrote:
Very nice Julian.
However, if we are to consider allowing this as a valid submissions then I propose we
add synchronization between directory switching to ensure fully N-1 behavior. Yes, it does
somewhat unfairly penalize Osamu here. However, not synchronizing somewhat unfairly
advantages him as it allows his fast processes non-contended access to subsequent
directories. If a submission cannot abide by official rules, then it seems better to
unfairly penalize instead of unfairly advantaging it.
Anyone else have thoughts here?
Thanks
John
On Jul 26, 2018, at 5:57 AM, Julian Kunkel
<juliankunkel(a)googlemail.com> wrote:
Dear Osama,
thanks for pointing this out.
I was not aware of this problem.
I now made some modifications to mdtest to compensate for this limitation.
Now it can limit the number of files per directories creating one
extra top-level directory if needed.
E.g. with a limit of 50 files and a total of 200 files, one would see
in the top-level testing directory:
#test-dir.0-0
#test-dir.0-1
#test-dir.0-2
#test-dir.0-3
The subdirectory tree of each directory is the complete tree as before
under #test-dir.0-0.
There is no synchronization when switching between these trees.
I expect that this emulates very well the behavior of a single large
directory but well there is no perfect solution.
The stuff is already integrated into the testing branch of io-500-dev.
To use it:
In io-500-dev
$ git checkout testing
$ ./utilities/prepare.sh
in io500.sh
Line 85:
io500_mdtest_hard_other_options=""
Change that to:
io500_mdtest_hard_other_options="-I 8000000"
Which effectively will create a directory for each batch of 8M files.
Note that the number of items (-n) must be a multiple of (-I)
Note that that feature cannot be used with stonewalling at the moment!
At some point, the mdtest code needs a redesign to allow such changes.
Hence, set in L30:
io500_stonewall_timer=0
If you want to give it a try, I would welcome it...
Best,
Julian
2018-07-26 3:05 GMT+01:00 Osamu Tatebe <tatebe(a)cs.tsukuba.ac.jp>:
> Hi,
>
> I would like to remind an issue regarding mdtest-hard benchmark. It
> creates files in a single directory for at least five minutes. When
> the metadata performance is more improved, the greater number of files
> will be created. This will hit the limit of the number of files in a
> single directory.
>
> Actually, the limit of Lustre file system is about 8M files although
> it depends the length of the pathname. When the metadata performance
> is better than 27K IOPS, it is not possible to execute the mdtest-hard
> create benchmark for five minutes since the number of files is larger
> than 8M.
>
> This is a reason we cannot submit the Lustre result. I expect this
> will be quite common when the metadata performance is more improved.
>
> Regards,
> Osamu
>
> ---
> Osamu Tatebe, Ph.D.
> Center for Computational Sciences, University of Tsukuba
> 1-1-1 Tennodai, Tsukuba, Ibaraki 3058577 Japan
> _______________________________________________
> IO-500 mailing list
> IO-500(a)vi4io.org
> https://www.vi4io.org/mailman/listinfo/io-500
--
Dr. Julian Kunkel
Lecturer, Department of Computer Science
+44 (0) 118 378 8218
http://www.cs.reading.ac.uk/
https://hps.vi4io.org/
_______________________________________________
IO-500 mailing list
IO-500(a)vi4io.org
https://www.vi4io.org/mailman/listinfo/io-500
_______________________________________________
IO-500 mailing list
IO-500(a)vi4io.org
https://www.vi4io.org/mailman/listinfo/io-500
8:35 p.m.
New subject: [EXTERNAL] Re: mdtest-hard issue
I think this speaks to a more fundamental problem with the 300-second
walltime requirement. I maintain that this requirement be removed and
replaced with something that speaks more directly to the intent of that
requirement (which was to eliminate caching effects, right?).
In an ideal world, one would be able to flush the entire contents of system
memory to a storage system in zero seconds, and the capacity of that
storage system would be sized to be as small as possible so as to minimize
its overall cost. As we move towards that goal, we move further away from
being able to meet the 300-second requirement; either one has to
1. slow the file system down so that it takes a full 300 seconds (which
becomes less representative of what users actually _want_ to do with a
parallel storage system), or
2. size the performance tier so large that it can absorb a full 300
seconds' worth of continuous dumping (which may be much larger than the
system memory)
I think the IO-500 benchmarks should be modeled off of real I/O tasks; for
example, read/write 80% of the aggregate node memory for whatever degree of
parallelism you choose to run to get the hero numbers, then work backwards
from there. Throw in some language about "walltime must include time to
persistence" and use the IOR -C option (and the analogous mdtest one).
Nobody intentionally performs I/O for a minimum period of time, so nobody
architects storage systems to handle that workload. This is only going to
become a more glaring issue as the ratio of bandwidth/capacity of
performance tiers goes up and storage architects tune their desired
$/bandwidth and $/capacity.
Glenn
On Mon, Jul 30, 2018 at 11:50 AM Lofstead, Gerald F II <gflofst(a)sandia.gov>
wrote:
Should we pursue making this a bug report for the file system
instead?
On 7/30/18, 12:44 PM, "IO-500 on behalf of John Bent" <
io-500-bounces(a)vi4io.org on behalf of johnbent(a)gmail.com> wrote:
Very nice Julian.
However, if we are to consider allowing this as a valid submissions
then I propose we add synchronization between directory switching to ensure
fully N-1 behavior. Yes, it does somewhat unfairly penalize Osamu here.
However, not synchronizing somewhat unfairly advantages him as it allows
his fast processes non-contended access to subsequent directories. If a
submission cannot abide by official rules, then it seems better to unfairly
penalize instead of unfairly advantaging it.
Anyone else have thoughts here?
Thanks
John
> On Jul 26, 2018, at 5:57 AM, Julian Kunkel <
juliankunkel(a)googlemail.com> wrote:
>
> Dear Osama,
> thanks for pointing this out.
> I was not aware of this problem.
>
> I now made some modifications to mdtest to compensate for this
limitation.
> Now it can limit the number of files per directories creating one
> extra top-level directory if needed.
> E.g. with a limit of 50 files and a total of 200 files, one would see
> in the top-level testing directory:
> #test-dir.0-0
> #test-dir.0-1
> #test-dir.0-2
> #test-dir.0-3
>
> The subdirectory tree of each directory is the complete tree as
before
> under #test-dir.0-0.
> There is no synchronization when switching between these trees.
> I expect that this emulates very well the behavior of a single large
> directory but well there is no perfect solution.
>
>
> The stuff is already integrated into the testing branch of
io-500-dev.
> To use it:
>
> In io-500-dev
> $ git checkout testing
> $ ./utilities/prepare.sh
>
> in io500.sh
> Line 85:
> io500_mdtest_hard_other_options=""
> Change that to:
> io500_mdtest_hard_other_options="-I 8000000"
> Which effectively will create a directory for each batch of 8M files.
> Note that the number of items (-n) must be a multiple of (-I)
>
> Note that that feature cannot be used with stonewalling at the
moment!
> At some point, the mdtest code needs a redesign to allow such
changes.
>
> Hence, set in L30:
> io500_stonewall_timer=0
>
> If you want to give it a try, I would welcome it...
>
> Best,
> Julian
>
> 2018-07-26 3:05 GMT+01:00 Osamu Tatebe <tatebe(a)cs.tsukuba.ac.jp>:
>> Hi,
>>
>> I would like to remind an issue regarding mdtest-hard benchmark. It
>> creates files in a single directory for at least five minutes. When
>> the metadata performance is more improved, the greater number of
files
>> will be created. This will hit the limit of the number of files in
a
>> single directory.
>>
>> Actually, the limit of Lustre file system is about 8M files although
>> it depends the length of the pathname. When the metadata
performance
>> is better than 27K IOPS, it is not possible to execute the
mdtest-hard
>> create benchmark for five minutes since the number of files is
larger
>> than 8M.
>>
>> This is a reason we cannot submit the Lustre result. I expect this
>> will be quite common when the metadata performance is more improved.
>>
>> Regards,
>> Osamu
>>
>> ---
>> Osamu Tatebe, Ph.D.
>> Center for Computational Sciences, University of Tsukuba
>> 1-1-1 Tennodai, Tsukuba, Ibaraki 3058577 Japan
>> _______________________________________________
>> IO-500 mailing list
>> IO-500(a)vi4io.org
>> https://www.vi4io.org/mailman/listinfo/io-500
>
>
>
> --
> Dr. Julian Kunkel
> Lecturer, Department of Computer Science
> +44 (0) 118 378 8218
> http://www.cs.reading.ac.uk/
> https://hps.vi4io.org/
> _______________________________________________
> IO-500 mailing list
> IO-500(a)vi4io.org
> https://www.vi4io.org/mailman/listinfo/io-500
_______________________________________________
IO-500 mailing list
IO-500(a)vi4io.org
https://www.vi4io.org/mailman/listinfo/io-500
_______________________________________________
IO-500 mailing list
IO-500(a)vi4io.org
https://www.vi4io.org/mailman/listinfo/io-500
8:59 p.m.
New subject: [EXTERNAL] Re: mdtest-hard issue
Hi Glenn,
Thanks for the comments. This is part of what we wrestled with for a long time. Cache
flushing was part of it. 80% of node memory was another thought, but few have that high a
requirement (LANL/SNL and maybe LLNL have an app that needs that for
checkpoint/restart).
What we settled on was 90% forward progress for an app for an hour. That gave 6 minutes
for C/R. 5 minutes seemed like a simpler number for people to work on. Seeing how much you
could do in that time seemed like the right metric. I don't think any of us expected
that we'd run into file system limitations.
Our goal was to try to limit the ability to game things too much and encouraged people to
try it since it was clear that careful tuning wouldn't buy that much. Finding
something that was general like, keep writing for 5 minutes rather than saying 80% of node
memory so people start limiting the number of nodes or carefully selecting the number of
nodes and their mapping to the storage targets. With a time limit, it strictly becomes how
much can you do because of the fixed time rather than how carefully you can configure
it.
We were stumped trying to figure out a different metric that would cause the same effect
as saying do as much as you can within the time limit. Suggestions are welcome. Ideas how
to keep the spirit of the 5 minute do-all-you-can test for file systems that cannot
sustain for 5 minutes are also welcome.
The newly announced 10 node challenge is intended to address more per node performance
rather than ultimate scale up performance. If that doesn't look like it'll achieve
the right results in your opinion, please let us know. We can adjust things still.
Best,
Jay
From: IO-500 <io-500-bounces(a)vi4io.org> on behalf of Glenn Lockwood
<glock(a)lbl.gov>
Date: Monday, July 30, 2018 at 2:45 PM
To: "io-500(a)vi4io.org" <io-500(a)vi4io.org>
Subject: Re: [IO-500] [EXTERNAL] Re: mdtest-hard issue
I think this speaks to a more fundamental problem with the 300-second walltime
requirement. I maintain that this requirement be removed and replaced with something that
speaks more directly to the intent of that requirement (which was to eliminate caching
effects, right?).
In an ideal world, one would be able to flush the entire contents of system memory to a
storage system in zero seconds, and the capacity of that storage system would be sized to
be as small as possible so as to minimize its overall cost. As we move towards that goal,
we move further away from being able to meet the 300-second requirement; either one has
to
1. slow the file system down so that it takes a full 300 seconds (which becomes less
representative of what users actually _want_ to do with a parallel storage system), or
2. size the performance tier so large that it can absorb a full 300 seconds' worth of
continuous dumping (which may be much larger than the system memory)
I think the IO-500 benchmarks should be modeled off of real I/O tasks; for example,
read/write 80% of the aggregate node memory for whatever degree of parallelism you choose
to run to get the hero numbers, then work backwards from there. Throw in some language
about "walltime must include time to persistence" and use the IOR -C option (and
the analogous mdtest one).
Nobody intentionally performs I/O for a minimum period of time, so nobody architects
storage systems to handle that workload. This is only going to become a more glaring
issue as the ratio of bandwidth/capacity of performance tiers goes up and storage
architects tune their desired $/bandwidth and $/capacity.
Glenn
On Mon, Jul 30, 2018 at 11:50 AM Lofstead, Gerald F II
<gflofst@sandia.gov<mailto:gflofst@sandia.gov>> wrote:
Should we pursue making this a bug report for the file system instead?
On 7/30/18, 12:44 PM, "IO-500 on behalf of John Bent"
<io-500-bounces@vi4io.org<mailto:io-500-bounces@vi4io.org> on behalf of
johnbent@gmail.com<mailto:johnbent@gmail.com>> wrote:
Very nice Julian.
However, if we are to consider allowing this as a valid submissions then I propose we
add synchronization between directory switching to ensure fully N-1 behavior. Yes, it does
somewhat unfairly penalize Osamu here. However, not synchronizing somewhat unfairly
advantages him as it allows his fast processes non-contended access to subsequent
directories. If a submission cannot abide by official rules, then it seems better to
unfairly penalize instead of unfairly advantaging it.
Anyone else have thoughts here?
Thanks
John
On Jul 26, 2018, at 5:57 AM, Julian Kunkel
<juliankunkel@googlemail.com<mailto:juliankunkel@googlemail.com>> wrote:
Dear Osama,
thanks for pointing this out.
I was not aware of this problem.
I now made some modifications to mdtest to compensate for this limitation.
Now it can limit the number of files per directories creating one
extra top-level directory if needed.
E.g. with a limit of 50 files and a total of 200 files, one would see
in the top-level testing directory:
#test-dir.0-0
#test-dir.0-1
#test-dir.0-2
#test-dir.0-3
The subdirectory tree of each directory is the complete tree as before
under #test-dir.0-0.
There is no synchronization when switching between these trees.
I expect that this emulates very well the behavior of a single large
directory but well there is no perfect solution.
The stuff is already integrated into the testing branch of io-500-dev.
To use it:
In io-500-dev
$ git checkout testing
$ ./utilities/prepare.sh
in io500.sh
Line 85:
io500_mdtest_hard_other_options=""
Change that to:
io500_mdtest_hard_other_options="-I 8000000"
Which effectively will create a directory for each batch of 8M files.
Note that the number of items (-n) must be a multiple of (-I)
Note that that feature cannot be used with stonewalling at the moment!
At some point, the mdtest code needs a redesign to allow such changes.
Hence, set in L30:
io500_stonewall_timer=0
If you want to give it a try, I would welcome it...
Best,
Julian
2018-07-26 3:05 GMT+01:00 Osamu Tatebe
<tatebe@cs.tsukuba.ac.jp<mailto:tatebe@cs.tsukuba.ac.jp>>:
> Hi,
>
> I would like to remind an issue regarding mdtest-hard benchmark. It
> creates files in a single directory for at least five minutes. When
> the metadata performance is more improved, the greater number of files
> will be created. This will hit the limit of the number of files in a
> single directory.
>
> Actually, the limit of Lustre file system is about 8M files although
> it depends the length of the pathname. When the metadata performance
> is better than 27K IOPS, it is not possible to execute the mdtest-hard
> create benchmark for five minutes since the number of files is larger
> than 8M.
>
> This is a reason we cannot submit the Lustre result. I expect this
> will be quite common when the metadata performance is more improved.
>
> Regards,
> Osamu
>
> ---
> Osamu Tatebe, Ph.D.
> Center for Computational Sciences, University of Tsukuba
> 1-1-1 Tennodai, Tsukuba, Ibaraki 3058577 Japan
> _______________________________________________
> IO-500 mailing list
> IO-500@vi4io.org<mailto:IO-500@vi4io.org>
> https://www.vi4io.org/mailman/listinfo/io-500
--
Dr. Julian Kunkel
Lecturer, Department of Computer Science
+44 (0) 118 378 8218
http://www.cs.reading.ac.uk/
https://hps.vi4io.org/
_______________________________________________
IO-500 mailing list
IO-500@vi4io.org<mailto:IO-500@vi4io.org>
https://www.vi4io.org/mailman/listinfo/io-500
_______________________________________________
IO-500 mailing list
IO-500@vi4io.org<mailto:IO-500@vi4io.org>
https://www.vi4io.org/mailman/listinfo/io-500
_______________________________________________
IO-500 mailing list
IO-500@vi4io.org<mailto:IO-500@vi4io.org>
https://www.vi4io.org/mailman/listinfo/io-500
6:19 a.m.
New subject: [EXTERNAL] Re: mdtest-hard issue
Glenn,
I think the 5-minute IO requirement (for IOR at least) is quite representative of real
systems. While I agree that everyone _wants_
to have a system with 1-second checkpoints, the practical outcome for
most large systems is that the budget/power/space considerations mean
that the maximum acceptable IO time is 10% of the runtime, which is
5-6 minutes per hour. That means the funding for storage will not be
more than what provides this minimum requirement (even though some
people would want it to be more), since otherwise more money could be
spent on compute. At least this is true for all of the systems I've
been involved in.
The other reason to have a time-based result is that it avoids the
"write only to RAM" issue, without making the test run excessively
long. If the requirement is 50% of RAM for the clients, then this
can easily get a great result by writing into the other 50% of RAM
on all of the clients, or by running with a small number of clients
that dump directly into the RAM on the servers, but that isn't a real
HPC workload result either.
The goal is to report the performance of the storage system itself.
Also, the benchmark should be able to complete in a reasonable time,
so 2x client RAM or other cache avoidance metrics may cause the test
to run for many hours if a large number of clients are involved.
I agree that caching layers are definitely desirable to have, so I
don't think there should be a "must write to disk" requirement, but
rather "report the results for each layer separately". That way we
can see that the flash layer gets X GB/s, Y IOPS, and the disk layer
gets 10% of those numbers, or whatever.
Initially I wasn't sure whether the 300s runtime for the metadata
tests was reasonable or not, but as HPC moves away from the typically
well-structured workloads (e.g. file per process, but only once per
hour) into genomics and AI, the "create many millions of files" is
becoming a very common workload.
The limitation of 8M files/directory that Osama reported for Lustre
(we report 10M in our manual, but it depends on filename length and
other factors) is (or at least was) a real issue for ldiskfs/ext4
MDTs. In newer versions of Lustre and with the appropriate harware
configuration, it is possible to exceed the 8/10M files per directory
by using DNE to stripe a single directory across multiple MDTs, with
ZFS MDTs, or (with an upcoming release of e2fsprogs and the right
settings) exceed 10M entries per directory with ldiskfs/ext4 itself
(up to about 5B entries/directory in theory).
Cheers, Andreas
On Jul 30, 2018, at 2:35 PM, Glenn Lockwood <glock(a)lbl.gov> wrote:
I think this speaks to a more fundamental problem with the 300-second walltime
requirement. I maintain that this requirement be removed and replaced with something that
speaks more directly to the intent of that requirement (which was to eliminate caching
effects, right?).
In an ideal world, one would be able to flush the entire contents of system memory to a
storage system in zero seconds, and the capacity of that storage system would be sized to
be as small as possible so as to minimize its overall cost. As we move towards that goal,
we move further away from being able to meet the 300-second requirement; either one has
to
1. slow the file system down so that it takes a full 300 seconds (which becomes less
representative of what users actually _want_ to do with a parallel storage system), or
2. size the performance tier so large that it can absorb a full 300 seconds' worth of
continuous dumping (which may be much larger than the system memory)
I think the IO-500 benchmarks should be modeled off of real I/O tasks; for example,
read/write 80% of the aggregate node memory for whatever degree of parallelism you choose
to run to get the hero numbers, then work backwards from there. Throw in some language
about "walltime must include time to persistence" and use the IOR -C option (and
the analogous mdtest one).
Nobody intentionally performs I/O for a minimum period of time, so nobody architects
storage systems to handle that workload. This is only going to become a more glaring
issue as the ratio of bandwidth/capacity of performance tiers goes up and storage
architects tune their desired $/bandwidth and $/capacity.
Glenn
On Mon, Jul 30, 2018 at 11:50 AM Lofstead, Gerald F II <gflofst(a)sandia.gov> wrote:
Should we pursue making this a bug report for the file system instead?
On 7/30/18, 12:44 PM, "IO-500 on behalf of John Bent"
<io-500-bounces(a)vi4io.org on behalf of johnbent(a)gmail.com> wrote:
Very nice Julian.
However, if we are to consider allowing this as a valid submissions then I propose we
add synchronization between directory switching to ensure fully N-1 behavior. Yes, it does
somewhat unfairly penalize Osamu here. However, not synchronizing somewhat unfairly
advantages him as it allows his fast processes non-contended access to subsequent
directories. If a submission cannot abide by official rules, then it seems better to
unfairly penalize instead of unfairly advantaging it.
Anyone else have thoughts here?
Thanks
John
> On Jul 26, 2018, at 5:57 AM, Julian Kunkel <juliankunkel(a)googlemail.com>
wrote:
>
> Dear Osama,
> thanks for pointing this out.
> I was not aware of this problem.
>
> I now made some modifications to mdtest to compensate for this limitation.
> Now it can limit the number of files per directories creating one
> extra top-level directory if needed.
> E.g. with a limit of 50 files and a total of 200 files, one would see
> in the top-level testing directory:
> #test-dir.0-0
> #test-dir.0-1
> #test-dir.0-2
> #test-dir.0-3
>
> The subdirectory tree of each directory is the complete tree as before
> under #test-dir.0-0.
> There is no synchronization when switching between these trees.
> I expect that this emulates very well the behavior of a single large
> directory but well there is no perfect solution.
>
>
> The stuff is already integrated into the testing branch of io-500-dev.
> To use it:
>
> In io-500-dev
> $ git checkout testing
> $ ./utilities/prepare.sh
>
> in io500.sh
> Line 85:
> io500_mdtest_hard_other_options=""
> Change that to:
> io500_mdtest_hard_other_options="-I 8000000"
> Which effectively will create a directory for each batch of 8M files.
> Note that the number of items (-n) must be a multiple of (-I)
>
> Note that that feature cannot be used with stonewalling at the moment!
> At some point, the mdtest code needs a redesign to allow such changes.
>
> Hence, set in L30:
> io500_stonewall_timer=0
>
> If you want to give it a try, I would welcome it...
>
> Best,
> Julian
>
> 2018-07-26 3:05 GMT+01:00 Osamu Tatebe <tatebe(a)cs.tsukuba.ac.jp>:
>> Hi,
>>
>> I would like to remind an issue regarding mdtest-hard benchmark. It
>> creates files in a single directory for at least five minutes. When
>> the metadata performance is more improved, the greater number of files
>> will be created. This will hit the limit of the number of files in a
>> single directory.
>>
>> Actually, the limit of Lustre file system is about 8M files although
>> it depends the length of the pathname. When the metadata performance
>> is better than 27K IOPS, it is not possible to execute the mdtest-hard
>> create benchmark for five minutes since the number of files is larger
>> than 8M.
>>
>> This is a reason we cannot submit the Lustre result. I expect this
>> will be quite common when the metadata performance is more improved.
>>
>> Regards,
>> Osamu
>>
>> ---
>> Osamu Tatebe, Ph.D.
>> Center for Computational Sciences, University of Tsukuba
>> 1-1-1 Tennodai, Tsukuba, Ibaraki 3058577 Japan
>> _______________________________________________
>> IO-500 mailing list
>> IO-500(a)vi4io.org
>> https://www.vi4io.org/mailman/listinfo/io-500
>
>
>
> --
> Dr. Julian Kunkel
> Lecturer, Department of Computer Science
> +44 (0) 118 378 8218
> http://www.cs.reading.ac.uk/
> https://hps.vi4io.org/
> _______________________________________________
> IO-500 mailing list
> IO-500(a)vi4io.org
> https://www.vi4io.org/mailman/listinfo/io-500
_______________________________________________
IO-500 mailing list
IO-500(a)vi4io.org
https://www.vi4io.org/mailman/listinfo/io-500
_______________________________________________
IO-500 mailing list
IO-500(a)vi4io.org
https://www.vi4io.org/mailman/listinfo/io-500
_______________________________________________
IO-500 mailing list
IO-500(a)vi4io.org
https://www.vi4io.org/mailman/listinfo/io-500
Cheers, Andreas
2:08 p.m.
Hello Julian,
There is another customer who is struggling with this same problem and also
wants to use stonewall. Are the two features compatible yet?
Thanks,
John
On Thu, Jul 26, 2018 at 3:57 AM Julian Kunkel <juliankunkel(a)googlemail.com>
wrote:
Dear Osama,
thanks for pointing this out.
I was not aware of this problem.
I now made some modifications to mdtest to compensate for this limitation.
Now it can limit the number of files per directories creating one
extra top-level directory if needed.
E.g. with a limit of 50 files and a total of 200 files, one would see
in the top-level testing directory:
#test-dir.0-0
#test-dir.0-1
#test-dir.0-2
#test-dir.0-3
The subdirectory tree of each directory is the complete tree as before
under #test-dir.0-0.
There is no synchronization when switching between these trees.
I expect that this emulates very well the behavior of a single large
directory but well there is no perfect solution.
The stuff is already integrated into the testing branch of io-500-dev.
To use it:
In io-500-dev
$ git checkout testing
$ ./utilities/prepare.sh
in io500.sh
Line 85:
io500_mdtest_hard_other_options=""
Change that to:
io500_mdtest_hard_other_options="-I 8000000"
Which effectively will create a directory for each batch of 8M files.
Note that the number of items (-n) must be a multiple of (-I)
Note that that feature cannot be used with stonewalling at the moment!
At some point, the mdtest code needs a redesign to allow such changes.
Hence, set in L30:
io500_stonewall_timer=0
If you want to give it a try, I would welcome it...
Best,
Julian
2018-07-26 3:05 GMT+01:00 Osamu Tatebe <tatebe(a)cs.tsukuba.ac.jp>:
> Hi,
>
> I would like to remind an issue regarding mdtest-hard benchmark. It
> creates files in a single directory for at least five minutes. When
> the metadata performance is more improved, the greater number of files
> will be created. This will hit the limit of the number of files in a
> single directory.
>
> Actually, the limit of Lustre file system is about 8M files although
> it depends the length of the pathname. When the metadata performance
> is better than 27K IOPS, it is not possible to execute the mdtest-hard
> create benchmark for five minutes since the number of files is larger
> than 8M.
>
> This is a reason we cannot submit the Lustre result. I expect this
> will be quite common when the metadata performance is more improved.
>
> Regards,
> Osamu
>
> ---
> Osamu Tatebe, Ph.D.
> Center for Computational Sciences, University of Tsukuba
> 1-1-1 Tennodai, Tsukuba, Ibaraki 3058577 Japan
> _______________________________________________
> IO-500 mailing list
> IO-500(a)vi4io.org
> https://www.vi4io.org/mailman/listinfo/io-500
--
Dr. Julian Kunkel
Lecturer, Department of Computer Science
+44 (0) 118 378 8218
http://www.cs.reading.ac.uk/
https://hps.vi4io.org/
_______________________________________________
IO-500 mailing list
IO-500(a)vi4io.org
https://www.vi4io.org/mailman/listinfo/io-500
8:47 p.m.
If the test is running on a non-production Lustre filesystem, another option is to enable
the "large_dir" feature for ldiskfs to allow larger single directories.
The code for the feature has existed for a long time in ldiskfs, but was only added to
e2fsprogs-1.44, so older e2fsprogs-1.42.12 will refuse to touch such a filesystem. While
there is an e2fsprogs-1.44.3-wc1 version that has support for it today, it is still in
testing, so I'm reluctant to advise people to enable the feature in production.
To enable the feature on an existing MDT run the following command on the MDS as root:
tune2fs -O large_dir /dev/<mdtdev>
which should also work with older e2fsprogs.
This should allow a single directory to be up to 6B entries in theory, though ldiskfs has
a cap of 4B inodes today. Performance will start to decline somewhat after 10M entries as
it adds another level to the tree, and it will need enough RAM on the MDS to cache
effectively (about 1GB per 10M entries).
Cheers, Andreas
On Sep 27, 2018, at 10:08, John Bent <johnbent(a)gmail.com>
wrote:
Hello Julian,
There is another customer who is struggling with this same problem and also wants to use
stonewall. Are the two features compatible yet?
Thanks,
John
> On Thu, Jul 26, 2018 at 3:57 AM Julian Kunkel <juliankunkel(a)googlemail.com>
wrote:
> Dear Osama,
> thanks for pointing this out.
> I was not aware of this problem.
>
> I now made some modifications to mdtest to compensate for this limitation.
> Now it can limit the number of files per directories creating one
> extra top-level directory if needed.
> E.g. with a limit of 50 files and a total of 200 files, one would see
> in the top-level testing directory:
> #test-dir.0-0
> #test-dir.0-1
> #test-dir.0-2
> #test-dir.0-3
>
> The subdirectory tree of each directory is the complete tree as before
> under #test-dir.0-0.
> There is no synchronization when switching between these trees.
> I expect that this emulates very well the behavior of a single large
> directory but well there is no perfect solution.
>
>
> The stuff is already integrated into the testing branch of io-500-dev.
> To use it:
>
> In io-500-dev
> $ git checkout testing
> $ ./utilities/prepare.sh
>
> in io500.sh
> Line 85:
> io500_mdtest_hard_other_options=""
> Change that to:
> io500_mdtest_hard_other_options="-I 8000000"
> Which effectively will create a directory for each batch of 8M files.
> Note that the number of items (-n) must be a multiple of (-I)
>
> Note that that feature cannot be used with stonewalling at the moment!
> At some point, the mdtest code needs a redesign to allow such changes.
>
> Hence, set in L30:
> io500_stonewall_timer=0
>
> If you want to give it a try, I would welcome it...
>
> Best,
> Julian
>
> 2018-07-26 3:05 GMT+01:00 Osamu Tatebe <tatebe(a)cs.tsukuba.ac.jp>:
> > Hi,
> >
> > I would like to remind an issue regarding mdtest-hard benchmark. It
> > creates files in a single directory for at least five minutes. When
> > the metadata performance is more improved, the greater number of files
> > will be created. This will hit the limit of the number of files in a
> > single directory.
> >
> > Actually, the limit of Lustre file system is about 8M files although
> > it depends the length of the pathname. When the metadata performance
> > is better than 27K IOPS, it is not possible to execute the mdtest-hard
> > create benchmark for five minutes since the number of files is larger
> > than 8M.
> >
> > This is a reason we cannot submit the Lustre result. I expect this
> > will be quite common when the metadata performance is more improved.
> >
> > Regards,
> > Osamu
> >
> > ---
> > Osamu Tatebe, Ph.D.
> > Center for Computational Sciences, University of Tsukuba
> > 1-1-1 Tennodai, Tsukuba, Ibaraki 3058577 Japan
> > _______________________________________________
> > IO-500 mailing list
> > IO-500(a)vi4io.org
> > https://www.vi4io.org/mailman/listinfo/io-500
>
>
>
> --
> Dr. Julian Kunkel
> Lecturer, Department of Computer Science
> +44 (0) 118 378 8218
> http://www.cs.reading.ac.uk/
> https://hps.vi4io.org/
> _______________________________________________
> IO-500 mailing list
> IO-500(a)vi4io.org
> https://www.vi4io.org/mailman/listinfo/io-500
_______________________________________________
IO-500 mailing list
IO-500(a)vi4io.org
https://www.vi4io.org/mailman/listinfo/io-500
12:52 a.m.
Thank you Osamu for reminding us of this. This is important.
My request would be that you size the submission such that it finishes
successfully even if it doesn't run for 300 seconds. I guess you are
running with ldiskfs and not enough DNE servers?
If you do so, there is the possibility that the community and committee can
agree to consider it a valid submission. What I would consider would be to
artificially extend the runtime to the full 300 seconds. So, if you
created 8 million in 100 seconds, your calculated rate would be 8 million
in 300 seconds.
There has been concern when this was discussed before that a person could
cheat in this way by using a small cache for a short period of time and
then the extrapolation would not be fair. However, in this case, you are
not using a cache and mdtest does call sync/flush so we know that your
reported rate is a fair one. Forcing you to artificially extend to 300
seconds is a bit unfair to your system but I think it is a safer policy
than allow invalid results.
What do you think about this proposal Osamu?
Here's some more thoughts about this general question:
Is it fair for someone to do 300 TB in 1 second into a cache and then asks
IO500 to artificially extends the runtime to the specified 300 seconds?
This is not fair. IO500 is supposed to measure the sustained rate of the
storage system. In the above example, if the backend system took more than
299 seconds to absorb the cached data, then the extrapolated score would be
unfairly high.
However, if the work done in a short period of time has been
flushed/synched to the appropriate location, then extrapolation in this way
is fair. Imagine in the above scenario that the 300 TB in 1 second has
been fully flushed. This means that the submitter could have used the
remaining 299 seconds to get more work done. Thus the extrapolated score
is strictly less than or equal to the actual sustained rate of the system.
Thanks,
John
On Wed, Jul 25, 2018 at 10:05 PM, Osamu Tatebe <tatebe(a)cs.tsukuba.ac.jp>
wrote:
Hi,
I would like to remind an issue regarding mdtest-hard benchmark. It
creates files in a single directory for at least five minutes. When
the metadata performance is more improved, the greater number of files
will be created. This will hit the limit of the number of files in a
single directory.
Actually, the limit of Lustre file system is about 8M files although
it depends the length of the pathname. When the metadata performance
is better than 27K IOPS, it is not possible to execute the mdtest-hard
create benchmark for five minutes since the number of files is larger
than 8M.
This is a reason we cannot submit the Lustre result. I expect this
will be quite common when the metadata performance is more improved.
Regards,
Osamu
---
Osamu Tatebe, Ph.D.
Center for Computational Sciences, University of Tsukuba
1-1-1 Tennodai, Tsukuba, Ibaraki 3058577 Japan
_______________________________________________
IO-500 mailing list
IO-500(a)vi4io.org
https://www.vi4io.org/mailman/listinfo/io-500
2:53 p.m.
Hi John,
OK. Anyway, I will submit the score.
Regards,
Osamu
On Thu, 26 Jul 2018 20:52:23 -0400
John Bent <johnbent(a)gmail.com> wrote:
Thank you Osamu for reminding us of this. This is important.
My request would be that you size the submission such that it finishes
successfully even if it doesn't run for 300 seconds. I guess you are
running with ldiskfs and not enough DNE servers?
If you do so, there is the possibility that the community and committee can
agree to consider it a valid submission. What I would consider would be to
artificially extend the runtime to the full 300 seconds. So, if you
created 8 million in 100 seconds, your calculated rate would be 8 million
in 300 seconds.
There has been concern when this was discussed before that a person could
cheat in this way by using a small cache for a short period of time and
then the extrapolation would not be fair. However, in this case, you are
not using a cache and mdtest does call sync/flush so we know that your
reported rate is a fair one. Forcing you to artificially extend to 300
seconds is a bit unfair to your system but I think it is a safer policy
than allow invalid results.
What do you think about this proposal Osamu?
Here's some more thoughts about this general question:
Is it fair for someone to do 300 TB in 1 second into a cache and then asks
IO500 to artificially extends the runtime to the specified 300 seconds?
This is not fair. IO500 is supposed to measure the sustained rate of the
storage system. In the above example, if the backend system took more than
299 seconds to absorb the cached data, then the extrapolated score would be
unfairly high.
However, if the work done in a short period of time has been
flushed/synched to the appropriate location, then extrapolation in this way
is fair. Imagine in the above scenario that the 300 TB in 1 second has
been fully flushed. This means that the submitter could have used the
remaining 299 seconds to get more work done. Thus the extrapolated score
is strictly less than or equal to the actual sustained rate of the system.
Thanks,
John
On Wed, Jul 25, 2018 at 10:05 PM, Osamu Tatebe <tatebe(a)cs.tsukuba.ac.jp>
wrote:
> Hi,
>
> I would like to remind an issue regarding mdtest-hard benchmark. It
> creates files in a single directory for at least five minutes. When
> the metadata performance is more improved, the greater number of files
> will be created. This will hit the limit of the number of files in a
> single directory.
>
> Actually, the limit of Lustre file system is about 8M files although
> it depends the length of the pathname. When the metadata performance
> is better than 27K IOPS, it is not possible to execute the mdtest-hard
> create benchmark for five minutes since the number of files is larger
> than 8M.
>
> This is a reason we cannot submit the Lustre result. I expect this
> will be quite common when the metadata performance is more improved.
>
> Regards,
> Osamu
>
> ---
> Osamu Tatebe, Ph.D.
> Center for Computational Sciences, University of Tsukuba
> 1-1-1 Tennodai, Tsukuba, Ibaraki 3058577 Japan
> _______________________________________________
> IO-500 mailing list
> IO-500(a)vi4io.org
> https://www.vi4io.org/mailman/listinfo/io-500
>
--
Osamu Tatebe <tatebe(a)cs.tsukuba.ac.jp>
2036
days inactive
2100
days old
10 comments
6 participants
participants (6)
-
Andreas Dilger -
Glenn Lockwood -
John Bent -
Julian Kunkel -
Lofstead, Gerald F II -
Osamu Tatebe