9:54 a.m.
Dear all,
based on the feedback of people and my own experience,
I have now drafted a simplified IO-500 run-suite:
https://github.com/VI4IO/io-500-script
I believe it is much easier to use, to debug and to optimize and quite minimal.
The design depends on phases:
1) A generation phase able to check for errors creates a hard-coded
run-script without branches. An example generated execution script is
enclosed.
=> during the generation phase you may use
https://www.vi4io.org/io500-info-creator/ to create the info tags.
It also creates a parallel minimal cleanup that purges the generated
files => again minimal.
2) The io-500.sh run-script can then be inspected / further modified
to include allowed tuning settings and the batch-submission commands
(which unfortunately were often missing in the last submissions).
3) An analysis phase which parses the output of the execution, checks
the validity of the result and computes the scores
Therewith, it minimizes the potential errors that may occur during the
*costly* execution and outsources them into the trivial preprocessing
and post-processing steps.
Outlook:
* I have not yet implemented step (3) or the build scripts, but will
do so depending on the feedback of the community.
* I reckon, the automatic detection of system information can be done
in the phases (1) as much as possible while some would need to go to
(2), but these calls can be added flexibly, too...
Best,
Julian
--
Dr. Julian Kunkel
Lecturer, Department of Computer Science
+44 (0) 118 378 8218
http://www.cs.reading.ac.uk/
https://hps.vi4io.org/
11:42 p.m.
On Dec 1, 2018, at 2:54 AM, Julian Kunkel via IO-500 <io-500(a)vi4io.org> wrote:
Dear all,
based on the feedback of people and my own experience,
I have now drafted a simplified IO-500 run-suite:
https://github.com/VI4IO/io-500-script
I believe it is much easier to use, to debug and to optimize and quite minimal.
The design depends on phases:
1) A generation phase able to check for errors creates a hard-coded
run-script without branches. An example generated execution script is
enclosed.
=> during the generation phase you may use
https://www.vi4io.org/io500-info-creator/ to create the info tags.
It also creates a parallel minimal cleanup that purges the generated
files => again minimal.
2) The io-500.sh run-script can then be inspected / further modified
to include allowed tuning settings and the batch-submission commands
(which unfortunately were often missing in the last submissions).
Minor suggestions here:
- it would be useful to have a comment like "This script was automatically
generated by io-500-gen.sh, any modifications (below this line) will be
lost if io-500-gen.sh is run again", or equivalent
- why not run "io-500-score.sh" automatically at the end, instead of just
printing a message to do so?
- for consistency, "clean-io-500.sh" should be named
"io-500-clean.sh"
Also, I put some comments inline in github after going through the patch.
Cheers, Andreas
3) An analysis phase which parses the output of the execution,
checks
the validity of the result and computes the scores
Therewith, it minimizes the potential errors that may occur during the
*costly* execution and outsources them into the trivial preprocessing
and post-processing steps.
Outlook:
* I have not yet implemented step (3) or the build scripts, but will
do so depending on the feedback of the community.
* I reckon, the automatic detection of system information can be done
in the phases (1) as much as possible while some would need to go to
(2), but these calls can be added flexibly, too...
Best,
Julian
--
Dr. Julian Kunkel
Lecturer, Department of Computer Science
+44 (0) 118 378 8218
http://www.cs.reading.ac.uk/
https://hps.vi4io.org/
<io-500.sh>_______________________________________________
IO-500 mailing list
IO-500(a)vi4io.org
https://www.vi4io.org/mailman/listinfo/io-500
Cheers, Andreas
10:08 a.m.
Hi Andreas,
Thanks for the feedback!
I have incorporated most of the suggestions.
- why not run "io-500-score.sh" automatically at the end,
instead of just
printing a message to do so?
The reason is to keep everything separated as much
as possible.
We may have different scoring tools that e.g. upload the data to the
webpage, compare data and so forth.
Community ideas :-)
Besides:
* It reduces dependencies (e.g., awk on compute nodes).
* Invoking the scoring automatically but producing an error may be too
easily ignored by users. They may also be tempted to rerun the test if
some output produces an error.
* running on 1000 nodes, I don't want to waste CPU time for post-processing.
Best,
Julian
Best,
Julian
7:18 p.m.
On Dec 2, 2018, at 3:08 AM, Julian Kunkel <juliankunkel(a)googlemail.com> wrote:
Hi Andreas,
Thanks for the feedback!
I have incorporated most of the suggestions.
> - why not run "io-500-score.sh" automatically at the end, instead of just
> printing a message to do so?
The reason is to keep everything separated as much as possible.
We may have different scoring tools that e.g. upload the data to the
webpage, compare data and so forth. Community ideas :-)
Julian,
one problem I see with the current implementation is that it hard-codes the command lines
used for the IO-500 tests into the same config file that needs to be edited by users.
This is problematic if this file lives in Git, because any changes in Git will conflict
with the parameters changed by users. I had similar problems when running IO-500
previously.
I think it would make more sense to have a separate file (marked read-only in Git to avoid
it being edited easily) that *only* contains user-editable parameters, and has a comment
at the top to make a copy (e.g. io-500-local.sh) for local usage/editing. This local
config file is sourced by the various scripts to generate the config (possibly as a
command-line argument, so that it is easy to have multiple different config files).
Besides:
* It reduces dependencies (e.g., awk on compute nodes).
Are there nodes that do *not* have awk on them?
* Invoking the scoring automatically but producing an error may be
too
easily ignored by users. They may also be tempted to rerun the test if
some output produces an error.
Conversely, if a user is running multiple iterations of io-500 in batch (e.g. to find
optimal parameters), it is better if the scores are computed as part of the test script,
rather than depending on them to run it manually on all of the output files. Also, it may
not be clear that their test runs are invalid (e.g. under 300s write duration) if this is
not show in the output. Having multiple separate steps makes it harder for users to get a
result.
* running on 1000 nodes, I don't want to waste CPU time for
post-processing.
The post-processing would obviously only be run once on as part of the job submission
script, and not on every job node. Given that io-500 takes about an hour to run, the 1s
to post-process the test output into an io-500 score file shouldn't be too much
overhead.
Cheers, Andreas
1:24 p.m.
Hi Andreas,
one problem I see with the current implementation is that it
hard-codes the command lines used for the IO-500 tests into the same config file that
needs to be edited by users.
This is problematic if this file lives in Git, because any changes in Git will conflict
with the parameters changed by users. I had similar problems when running IO-500
previously.
I cannot completely follow.
So far we had these problems that are related:
1) changes of the "fixed-io500.sh" script that forced users to change
parameters in the io-500.sh
2) that they had not included the batch submission paramters as well.
3) changes to io-500.sh was not in Git Repository either.
What I do for me is that I change the io-500-gen.sh to include the
most relevant parameters then generate io-500.sh scripts (e.g. for 10
nodes), then I version control them.
When an update to io-500-gen.sh happens (like it did before to
io-500.sh), I create the delta to my script and inspect it. Then apply
the changes needed.
A problem is always how to incorporate changes to the execution in the
way one has created it.
I absolutely expect that we will create a version number, freeze the
script for an upcoming submission.
The script will output for which version it is. "This script generates
a valid submission for the IO-500 list in 11/2018. Do not use it for
any other list!"
Users must use the right script.
I think it would make more sense to have a separate file (marked
read-only in Git to avoid it being edited easily) that *only* contains user-editable
parameters, and has a comment at the top to make a copy (e.g. io-500-
local.sh) for local usage/editing. This local config file is sourced by the various
scripts to generate the config (possibly as a command-line argument, so that it is easy to
have multiple different config files).
This can be done.
Do you think something like:
./io-500-gen.sh config-1.sh
Would do?
That one could then generate io-500-run-config-1.sh with the stuff?
> Besides:
> * It reduces dependencies (e.g., awk on compute nodes).
Are there nodes that do *not* have awk on them?
Yes.
Conversely, if a user is running multiple iterations of io-500 in
batch (e.g. to find optimal parameters), it is better if the scores are computed as part
of the test script, rather than depending on them to run it manually on all of the output
files. Also, it may not be clear that their test runs are invalid (e.g. under 300s write
duration) if this is not show in the output. Having multiple separate steps makes it
harder for users to get a result.
It is very good practice that I adapted from the big-data community to
keep raw-data separated from any data product as it allows to run not
only different types of analysis but to improve the analysis over
time, e.g. generate new data products.
You can add the execution of the post-processing to the generated
batch file or the generator if you want! You just don't have to.
I will include a comprehensive python validator that will check every
parameter and output error that we have seen to make sure that the
output is correct.
It is not useful to encode this in bash, however, at the same time, we
faced compatibility problems running Python on the compute nodes.
That said, your choice to include it to automatize the calculation.
I rather run the full checker to ensure I have not missed anything and
compute even more derived performance values than we do for the lists.
Best,
Julian
8:38 p.m.
On Dec 6, 2018, at 6:24 AM, Julian Kunkel <juliankunkel(a)googlemail.com> wrote:
Hi Andreas,
> one problem I see with the current implementation is that it hard-codes
> the command lines used for the IO-500 tests into the same config file
> that needs to be edited by users.
> This is problematic if this file lives in Git, because any changes in
> Git will conflict with the parameters changed by users. I had similar
> problems when running IO-500 previously.
I cannot completely follow.
So far we had these problems that are related:
1) changes of the "fixed-io500.sh" script that forced users to change
parameters in the io-500.sh
2) that they had not included the batch submission paramters as well.
3) changes to io-500.sh was not in Git Repository either.
What I do for me is that I change the io-500-gen.sh to include the
most relevant parameters then generate io-500.sh scripts (e.g. for 10
nodes), then I version control them.
When an update to io-500-gen.sh happens (like it did before to
io-500.sh), I create the delta to my script and inspect it. Then apply
the changes needed.
A problem is always how to incorporate changes to the execution in the
way one has created it.
The "create the delta ... and apply changes" part is what is problematic
IMHO. That is fine for you to do as author of the scripts, but for other
users it can become more complex (e.g. if there is a conflict in the patch
they may not know whether the change is mandatory or optional).
Conversely, they may modify (accidentally?) the parameters that make
IOR-hard "hard" and find they get a much better performance result...
Having the user-specified parameters separate from the execution of the
tests themselves is what I'm looking for. If the user-supplied parameters
are listed first on the execution command-line, and the mandatory (easy,
hard) parameters are listed afterward, then the mandatory ones should
override any conflicting options that were specified by the user.
I absolutely expect that we will create a version number, freeze the
script for an upcoming submission.
The script will output for which version it is. "This script generates
a valid submission for the IO-500 list in 11/2018. Do not use it for
any other list!" Users must use the right script.
I'm not strictly against this, but it means the scripts need to be
updated for each list. I think it should be done in the opposite manner,
that the scripts have a specific version, and is included in the results,
but the version is independent of the list version (e.g. "IO-500 v1.2.5"
and this is valid for 2018-11, 2019-06, ...).
Then, we can determine which version of the results is valid for a given
IO-500 list (maybe some changes are inconsequential, and the same results
are usable across many lists). Users should get a reply that their
result is not acceptable for the next top list if it is not running an
approved version, but the result can still be accepted into the DB with
the old version.
> I think it would make more sense to have a separate file (marked
> read-only in Git to avoid it being edited easily) that *only* contains
> user-editable parameters, and has a comment at the top to make a copy
> (e.g. io-500-local.sh) for local usage/editing. This local config
> file is sourced by the various scripts to generate the config (possibly
> as a command-line argument, so that it is easy to have multiple different
> config files).
This can be done.
Do you think something like:
./io-500-gen.sh config-1.sh
Would do? That one could then generate io-500-run-config-1.sh with the stuff?
Yes, this is what I was thinking, but why generate the separate static
run script at this point? The "io-500.sh config-1.sh" command would be
able to run the specific commands directly, without the (IMHO not useful)
separate step of generating a static list of commands. Otherwise, there
is also a danger that "config-1.sh" is updated, but io-500-gen.sh is not
run again, and the old static script is used for the job submission.
IMHO, the fewer steps that are needed, the better. The main difference
from the current io-500.sh script is that the user parameters are kept
in a separate config file that is not in the main git repo, so it does
not see conflicts when the script is updated, and it is easy to have
multiple different configs without having to update them all when there
are changes to the run script.
>> Besides:
>> * It reduces dependencies (e.g., awk on compute nodes).
> Are there nodes that do *not* have awk on them?
Yes.
> Conversely, if a user is running multiple iterations of io-500 in
> batch (e.g. to find optimal parameters), it is better if the scores
> are computed as part of the test script, rather than depending on
> them to run it manually on all of the output files. Also, it may
> not be clear that their test runs are invalid (e.g. under 300s
> write duration) if this is not show in the output. Having multiple
> separate steps makes it harder for users to get a result.
It is very good practice that I adapted from the big-data community to
keep raw-data separated from any data product as it allows to run not
only different types of analysis but to improve the analysis over
time, e.g. generate new data products.
You can add the execution of the post-processing to the generated
batch file or the generator if you want! You just don't have to.
I will include a comprehensive python validator that will check every
parameter and output error that we have seen to make sure that the
output is correct.
It is not useful to encode this in bash, however, at the same time, we
faced compatibility problems running Python on the compute nodes.
That said, your choice to include it to automatize the calculation.
I rather run the full checker to ensure I have not missed anything and
compute even more derived performance values than we do for the lists.
Sure, I can agree with this, as long as it is clear to users what needs
to be done to get a valid result.
Cheers, Andreas
11:32 a.m.
Hi Andreas,
Thanks for your email. I appreciate the discussion.
Disclaimer: as before, the following is my personal opinion.
Yes, this is what I was thinking, but why generate the separate
static
run script at this point? The "io-500.sh config-1.sh" command would be
able to run the specific commands directly, without the (IMHO not useful)
separate step of generating a static list of commands.
Otherwise, there
is also a danger that "config-1.sh" is updated, but io-500-gen.sh is not
run again, and the old static script is used for the job submission.
I agree with
this one, that is a danger! But it ain't be more danger
than an incompatible config-1.sh file.
What I faced a couple of times, however, was that new "variables" have
been introduced that broke the old config-1.sh file.
I have to wait often quite long (day+) to get an allocation with > 100
nodes, it is a bit annoying to then figure out something does not work
as planned.
I totally go with the generator approach and already used it to
explore various things by changing the resulting execution file.
It is easy to analyze that file and give it to someone that can say,
yes that is correct while a complex execution script cannot provide
that.
All the conditions and stuff are a problem at runtime and barely debuggable.
I can't even count how many runs I have wasted on the 4 supercomputers
I run the current IO-500 script with varying configurations.
And I faced the discussion problem with others as the debugability is
really bad.
IMHO, the fewer steps that are needed, the better.
I generally
agree, there are some requirements:
* Usability
* Debugability / traceability
* Error resistant
* Cheap to run
* Version controlled (you brought that up, and I agree)
The existing scripts do not get a *star* from me in any of these cases.
I do agree it requires people to understand that there are *phases*,
but well that is a low hurdle.
Once it is in a config file, the situation for version control will
improve (see below).
The main difference from the current io-500.sh script is that the
user parameters are kept
in a separate config file that is not in the main git repo, so it does
not see conflicts when the script is updated, and it is easy to have
multiple different configs without having to update them all when there
are changes to the run script.
I have now checked in a version that does that with config files.
Now users won't have to fiddle with the generator at all, hence that
complexity can increase again.
I also moved the user parameters for ior / mdtest as first arguments
reducing the risk that users overwrite the wrong parameter.
Best,
Julian
Am Fr., 7. Dez. 2018 um 20:38 Uhr schrieb Andreas Dilger <adilger(a)dilger.ca>:
On Dec 6, 2018, at 6:24 AM, Julian Kunkel <juliankunkel(a)googlemail.com> wrote:
>
> Hi Andreas,
>
>> one problem I see with the current implementation is that it hard-codes
>> the command lines used for the IO-500 tests into the same config file
>> that needs to be edited by users.
>> This is problematic if this file lives in Git, because any changes in
>> Git will conflict with the parameters changed by users. I had similar
>> problems when running IO-500 previously.
>
> I cannot completely follow.
> So far we had these problems that are related:
> 1) changes of the "fixed-io500.sh" script that forced users to change
> parameters in the io-500.sh
> 2) that they had not included the batch submission paramters as well.
> 3) changes to io-500.sh was not in Git Repository either.
>
> What I do for me is that I change the io-500-gen.sh to include the
> most relevant parameters then generate io-500.sh scripts (e.g. for 10
> nodes), then I version control them.
> When an update to io-500-gen.sh happens (like it did before to
> io-500.sh), I create the delta to my script and inspect it. Then apply
> the changes needed.
>
> A problem is always how to incorporate changes to the execution in the
> way one has created it.
The "create the delta ... and apply changes" part is what is problematic
IMHO. That is fine for you to do as author of the scripts, but for other
users it can become more complex (e.g. if there is a conflict in the patch
they may not know whether the change is mandatory or optional).
Conversely, they may modify (accidentally?) the parameters that make
IOR-hard "hard" and find they get a much better performance result...
Having the user-specified parameters separate from the execution of the
tests themselves is what I'm looking for. If the user-supplied parameters
are listed first on the execution command-line, and the mandatory (easy,
hard) parameters are listed afterward, then the mandatory ones should
override any conflicting options that were specified by the user.
> I absolutely expect that we will create a version number, freeze the
> script for an upcoming submission.
> The script will output for which version it is. "This script generates
> a valid submission for the IO-500 list in 11/2018. Do not use it for
> any other list!" Users must use the right script.
I'm not strictly against this, but it means the scripts need to be
updated for each list. I think it should be done in the opposite manner,
that the scripts have a specific version, and is included in the results,
but the version is independent of the list version (e.g. "IO-500 v1.2.5"
and this is valid for 2018-11, 2019-06, ...).
Then, we can determine which version of the results is valid for a given
IO-500 list (maybe some changes are inconsequential, and the same results
are usable across many lists). Users should get a reply that their
result is not acceptable for the next top list if it is not running an
approved version, but the result can still be accepted into the DB with
the old version.
>> I think it would make more sense to have a separate file (marked
>> read-only in Git to avoid it being edited easily) that *only* contains
>> user-editable parameters, and has a comment at the top to make a copy
>> (e.g. io-500-local.sh) for local usage/editing. This local config
>> file is sourced by the various scripts to generate the config (possibly
>> as a command-line argument, so that it is easy to have multiple different
>> config files).
>
> This can be done.
> Do you think something like:
> ./io-500-gen.sh config-1.sh
> Would do? That one could then generate io-500-run-config-1.sh with the stuff?
Yes, this is what I was thinking, but why generate the separate static
run script at this point? The "io-500.sh config-1.sh" command would be
able to run the specific commands directly, without the (IMHO not useful)
separate step of generating a static list of commands. Otherwise, there
is also a danger that "config-1.sh" is updated, but io-500-gen.sh is not
run again, and the old static script is used for the job submission.
IMHO, the fewer steps that are needed, the better. The main difference
from the current io-500.sh script is that the user parameters are kept
in a separate config file that is not in the main git repo, so it does
not see conflicts when the script is updated, and it is easy to have
multiple different configs without having to update them all when there
are changes to the run script.
>>> Besides:
>>> * It reduces dependencies (e.g., awk on compute nodes).
>> Are there nodes that do *not* have awk on them?
> Yes.
>
>> Conversely, if a user is running multiple iterations of io-500 in
>> batch (e.g. to find optimal parameters), it is better if the scores
>> are computed as part of the test script, rather than depending on
>> them to run it manually on all of the output files. Also, it may
>> not be clear that their test runs are invalid (e.g. under 300s
>> write duration) if this is not show in the output. Having multiple
>> separate steps makes it harder for users to get a result.
>
> It is very good practice that I adapted from the big-data community to
> keep raw-data separated from any data product as it allows to run not
> only different types of analysis but to improve the analysis over
> time, e.g. generate new data products.
>
> You can add the execution of the post-processing to the generated
> batch file or the generator if you want! You just don't have to.
> I will include a comprehensive python validator that will check every
> parameter and output error that we have seen to make sure that the
> output is correct.
> It is not useful to encode this in bash, however, at the same time, we
> faced compatibility problems running Python on the compute nodes.
>
> That said, your choice to include it to automatize the calculation.
> I rather run the full checker to ensure I have not missed anything and
> compute even more derived performance values than we do for the lists.
Sure, I can agree with this, as long as it is clear to users what needs
to be done to get a valid result.
Cheers, Andreas
--
Dr. Julian Kunkel
Lecturer, Department of Computer Science
+44 (0) 118 378 8218
http://www.cs.reading.ac.uk/
https://hps.vi4io.org/
5:39 p.m.
So I'm looking at your proposed io-500-script stuff
(https://github.com/VI4IO/io-500-script) and one thing that jumped out at me is that there
doesn't appear to be a way to add further flags to the hard tests. In my environment
(all NFS storage), I need the -B flag for ior in order for it to complete successfully
(we've talked about this on the slack channel). Can we add the setting of permissible
flags to the config.sh script?
--Ken
On Dec 11, 2018, at 6:32 AM, Julian Kunkel via IO-500
<io-500@vi4io.org<mailto:io-500@vi4io.org>> wrote:
Hi Andreas,
Thanks for your email. I appreciate the discussion.
Disclaimer: as before, the following is my personal opinion.
Yes, this is what I was thinking, but why generate the separate static
run script at this point? The "io-500.sh config-1.sh" command would be
able to run the specific commands directly, without the (IMHO not useful)
separate step of generating a static list of commands.
Otherwise, there
is also a danger that "config-1.sh" is updated, but io-500-gen.sh is not
run again, and the old static script is used for the job submission.
I agree with this one, that is a danger! But it ain't be more danger
than an incompatible config-1.sh file.
What I faced a couple of times, however, was that new "variables" have
been introduced that broke the old config-1.sh file.
I have to wait often quite long (day+) to get an allocation with > 100
nodes, it is a bit annoying to then figure out something does not work
as planned.
I totally go with the generator approach and already used it to
explore various things by changing the resulting execution file.
It is easy to analyze that file and give it to someone that can say,
yes that is correct while a complex execution script cannot provide
that.
All the conditions and stuff are a problem at runtime and barely debuggable.
I can't even count how many runs I have wasted on the 4 supercomputers
I run the current IO-500 script with varying configurations.
And I faced the discussion problem with others as the debugability is
really bad.
IMHO, the fewer steps that are needed, the better.
I generally agree, there are some requirements:
* Usability
* Debugability / traceability
* Error resistant
* Cheap to run
* Version controlled (you brought that up, and I agree)
The existing scripts do not get a *star* from me in any of these cases.
I do agree it requires people to understand that there are *phases*,
but well that is a low hurdle.
Once it is in a config file, the situation for version control will
improve (see below).
The main difference from the current io-500.sh script is that the user parameters are
kept
in a separate config file that is not in the main git repo, so it does
not see conflicts when the script is updated, and it is easy to have
multiple different configs without having to update them all when there
are changes to the run script.
I have now checked in a version that does that with config files.
Now users won't have to fiddle with the generator at all, hence that
complexity can increase again.
I also moved the user parameters for ior / mdtest as first arguments
reducing the risk that users overwrite the wrong parameter.
Best,
Julian
Am Fr., 7. Dez. 2018 um 20:38 Uhr schrieb Andreas Dilger
<adilger@dilger.ca<mailto:adilger@dilger.ca>>:
On Dec 6, 2018, at 6:24 AM, Julian Kunkel
<juliankunkel@googlemail.com<mailto:juliankunkel@googlemail.com>> wrote:
Hi Andreas,
one problem I see with the current implementation is that it hard-codes
the command lines used for the IO-500 tests into the same config file
that needs to be edited by users.
This is problematic if this file lives in Git, because any changes in
Git will conflict with the parameters changed by users. I had similar
problems when running IO-500 previously.
I cannot completely follow.
So far we had these problems that are related:
1) changes of the "fixed-io500.sh" script that forced users to change
parameters in the io-500.sh
2) that they had not included the batch submission paramters as well.
3) changes to io-500.sh was not in Git Repository either.
What I do for me is that I change the io-500-gen.sh to include the
most relevant parameters then generate io-500.sh scripts (e.g. for 10
nodes), then I version control them.
When an update to io-500-gen.sh happens (like it did before to
io-500.sh), I create the delta to my script and inspect it. Then apply
the changes needed.
A problem is always how to incorporate changes to the execution in the
way one has created it.
The "create the delta ... and apply changes" part is what is problematic
IMHO. That is fine for you to do as author of the scripts, but for other
users it can become more complex (e.g. if there is a conflict in the patch
they may not know whether the change is mandatory or optional).
Conversely, they may modify (accidentally?) the parameters that make
IOR-hard "hard" and find they get a much better performance result...
Having the user-specified parameters separate from the execution of the
tests themselves is what I'm looking for. If the user-supplied parameters
are listed first on the execution command-line, and the mandatory (easy,
hard) parameters are listed afterward, then the mandatory ones should
override any conflicting options that were specified by the user.
I absolutely expect that we will create a version number, freeze the
script for an upcoming submission.
The script will output for which version it is. "This script generates
a valid submission for the IO-500 list in 11/2018. Do not use it for
any other list!" Users must use the right script.
I'm not strictly against this, but it means the scripts need to be
updated for each list. I think it should be done in the opposite manner,
that the scripts have a specific version, and is included in the results,
but the version is independent of the list version (e.g. "IO-500 v1.2.5"
and this is valid for 2018-11, 2019-06, ...).
Then, we can determine which version of the results is valid for a given
IO-500 list (maybe some changes are inconsequential, and the same results
are usable across many lists). Users should get a reply that their
result is not acceptable for the next top list if it is not running an
approved version, but the result can still be accepted into the DB with
the old version.
I think it would make more sense to have a separate file (marked
read-only in Git to avoid it being edited easily) that *only* contains
user-editable parameters, and has a comment at the top to make a copy
(e.g. io-500-local.sh) for local usage/editing. This local config
file is sourced by the various scripts to generate the config (possibly
as a command-line argument, so that it is easy to have multiple different
config files).
This can be done.
Do you think something like:
./io-500-gen.sh config-1.sh
Would do? That one could then generate io-500-run-config-1.sh with the stuff?
Yes, this is what I was thinking, but why generate the separate static
run script at this point? The "io-500.sh config-1.sh" command would be
able to run the specific commands directly, without the (IMHO not useful)
separate step of generating a static list of commands. Otherwise, there
is also a danger that "config-1.sh" is updated, but io-500-gen.sh is not
run again, and the old static script is used for the job submission.
IMHO, the fewer steps that are needed, the better. The main difference
from the current io-500.sh script is that the user parameters are kept
in a separate config file that is not in the main git repo, so it does
not see conflicts when the script is updated, and it is easy to have
multiple different configs without having to update them all when there
are changes to the run script.
Besides:
* It reduces dependencies (e.g., awk on compute nodes).
Are there nodes that do *not* have awk on them?
Yes.
Conversely, if a user is running multiple iterations of io-500 in
batch (e.g. to find optimal parameters), it is better if the scores
are computed as part of the test script, rather than depending on
them to run it manually on all of the output files. Also, it may
not be clear that their test runs are invalid (e.g. under 300s
write duration) if this is not show in the output. Having multiple
separate steps makes it harder for users to get a result.
It is very good practice that I adapted from the big-data community to
keep raw-data separated from any data product as it allows to run not
only different types of analysis but to improve the analysis over
time, e.g. generate new data products.
You can add the execution of the post-processing to the generated
batch file or the generator if you want! You just don't have to.
I will include a comprehensive python validator that will check every
parameter and output error that we have seen to make sure that the
output is correct.
It is not useful to encode this in bash, however, at the same time, we
faced compatibility problems running Python on the compute nodes.
That said, your choice to include it to automatize the calculation.
I rather run the full checker to ensure I have not missed anything and
compute even more derived performance values than we do for the lists.
Sure, I can agree with this, as long as it is clear to users what needs
to be done to get a valid result.
Cheers, Andreas
--
Dr. Julian Kunkel
Lecturer, Department of Computer Science
+44 (0) 118 378 8218
http://www.cs.reading.ac.uk/
https://hps.vi4io.org/
_______________________________________________
IO-500 mailing list
IO-500@vi4io.org<mailto:IO-500@vi4io.org>
https://www.vi4io.org/mailman/listinfo/io-500
5:43 p.m.
Oh shoot, looks like that's present in the config-test.sh script... sorry about that,
I was just looking at config-normal.sh
On Dec 22, 2018, at 12:39 PM, Ken Carlile
<carlilek@janelia.hhmi.org<mailto:carlilek@janelia.hhmi.org>> wrote:
So I'm looking at your proposed io-500-script stuff
(https://github.com/VI4IO/io-500-script) and one thing that jumped out at me is that there
doesn't appear to be a way to add further flags to the hard tests. In my environment
(all NFS storage), I need the -B flag for ior in order for it to complete successfully
(we've talked about this on the slack channel). Can we add the setting of permissible
flags to the config.sh script?
--Ken
On Dec 11, 2018, at 6:32 AM, Julian Kunkel via IO-500
<io-500@vi4io.org<mailto:io-500@vi4io.org>> wrote:
Hi Andreas,
Thanks for your email. I appreciate the discussion.
Disclaimer: as before, the following is my personal opinion.
Yes, this is what I was thinking, but why generate the separate static
run script at this point? The "io-500.sh config-1.sh" command would be
able to run the specific commands directly, without the (IMHO not useful)
separate step of generating a static list of commands.
Otherwise, there
is also a danger that "config-1.sh" is updated, but io-500-gen.sh is not
run again, and the old static script is used for the job submission.
I agree with this one, that is a danger! But it ain't be more danger
than an incompatible config-1.sh file.
What I faced a couple of times, however, was that new "variables" have
been introduced that broke the old config-1.sh file.
I have to wait often quite long (day+) to get an allocation with > 100
nodes, it is a bit annoying to then figure out something does not work
as planned.
I totally go with the generator approach and already used it to
explore various things by changing the resulting execution file.
It is easy to analyze that file and give it to someone that can say,
yes that is correct while a complex execution script cannot provide
that.
All the conditions and stuff are a problem at runtime and barely debuggable.
I can't even count how many runs I have wasted on the 4 supercomputers
I run the current IO-500 script with varying configurations.
And I faced the discussion problem with others as the debugability is
really bad.
IMHO, the fewer steps that are needed, the better.
I generally agree, there are some requirements:
* Usability
* Debugability / traceability
* Error resistant
* Cheap to run
* Version controlled (you brought that up, and I agree)
The existing scripts do not get a *star* from me in any of these cases.
I do agree it requires people to understand that there are *phases*,
but well that is a low hurdle.
Once it is in a config file, the situation for version control will
improve (see below).
The main difference from the current io-500.sh script is that the user parameters are
kept
in a separate config file that is not in the main git repo, so it does
not see conflicts when the script is updated, and it is easy to have
multiple different configs without having to update them all when there
are changes to the run script.
I have now checked in a version that does that with config files.
Now users won't have to fiddle with the generator at all, hence that
complexity can increase again.
I also moved the user parameters for ior / mdtest as first arguments
reducing the risk that users overwrite the wrong parameter.
Best,
Julian
Am Fr., 7. Dez. 2018 um 20:38 Uhr schrieb Andreas Dilger
<adilger@dilger.ca<mailto:adilger@dilger.ca>>:
On Dec 6, 2018, at 6:24 AM, Julian Kunkel
<juliankunkel@googlemail.com<mailto:juliankunkel@googlemail.com>> wrote:
Hi Andreas,
one problem I see with the current implementation is that it hard-codes
the command lines used for the IO-500 tests into the same config file
that needs to be edited by users.
This is problematic if this file lives in Git, because any changes in
Git will conflict with the parameters changed by users. I had similar
problems when running IO-500 previously.
I cannot completely follow.
So far we had these problems that are related:
1) changes of the "fixed-io500.sh" script that forced users to change
parameters in the io-500.sh
2) that they had not included the batch submission paramters as well.
3) changes to io-500.sh was not in Git Repository either.
What I do for me is that I change the io-500-gen.sh to include the
most relevant parameters then generate io-500.sh scripts (e.g. for 10
nodes), then I version control them.
When an update to io-500-gen.sh happens (like it did before to
io-500.sh), I create the delta to my script and inspect it. Then apply
the changes needed.
A problem is always how to incorporate changes to the execution in the
way one has created it.
The "create the delta ... and apply changes" part is what is problematic
IMHO. That is fine for you to do as author of the scripts, but for other
users it can become more complex (e.g. if there is a conflict in the patch
they may not know whether the change is mandatory or optional).
Conversely, they may modify (accidentally?) the parameters that make
IOR-hard "hard" and find they get a much better performance result...
Having the user-specified parameters separate from the execution of the
tests themselves is what I'm looking for. If the user-supplied parameters
are listed first on the execution command-line, and the mandatory (easy,
hard) parameters are listed afterward, then the mandatory ones should
override any conflicting options that were specified by the user.
I absolutely expect that we will create a version number, freeze the
script for an upcoming submission.
The script will output for which version it is. "This script generates
a valid submission for the IO-500 list in 11/2018. Do not use it for
any other list!" Users must use the right script.
I'm not strictly against this, but it means the scripts need to be
updated for each list. I think it should be done in the opposite manner,
that the scripts have a specific version, and is included in the results,
but the version is independent of the list version (e.g. "IO-500 v1.2.5"
and this is valid for 2018-11, 2019-06, ...).
Then, we can determine which version of the results is valid for a given
IO-500 list (maybe some changes are inconsequential, and the same results
are usable across many lists). Users should get a reply that their
result is not acceptable for the next top list if it is not running an
approved version, but the result can still be accepted into the DB with
the old version.
I think it would make more sense to have a separate file (marked
read-only in Git to avoid it being edited easily) that *only* contains
user-editable parameters, and has a comment at the top to make a copy
(e.g. io-500-local.sh) for local usage/editing. This local config
file is sourced by the various scripts to generate the config (possibly
as a command-line argument, so that it is easy to have multiple different
config files).
This can be done.
Do you think something like:
./io-500-gen.sh config-1.sh
Would do? That one could then generate io-500-run-config-1.sh with the stuff?
Yes, this is what I was thinking, but why generate the separate static
run script at this point? The "io-500.sh config-1.sh" command would be
able to run the specific commands directly, without the (IMHO not useful)
separate step of generating a static list of commands. Otherwise, there
is also a danger that "config-1.sh" is updated, but io-500-gen.sh is not
run again, and the old static script is used for the job submission.
IMHO, the fewer steps that are needed, the better. The main difference
from the current io-500.sh script is that the user parameters are kept
in a separate config file that is not in the main git repo, so it does
not see conflicts when the script is updated, and it is easy to have
multiple different configs without having to update them all when there
are changes to the run script.
Besides:
* It reduces dependencies (e.g., awk on compute nodes).
Are there nodes that do *not* have awk on them?
Yes.
Conversely, if a user is running multiple iterations of io-500 in
batch (e.g. to find optimal parameters), it is better if the scores
are computed as part of the test script, rather than depending on
them to run it manually on all of the output files. Also, it may
not be clear that their test runs are invalid (e.g. under 300s
write duration) if this is not show in the output. Having multiple
separate steps makes it harder for users to get a result.
It is very good practice that I adapted from the big-data community to
keep raw-data separated from any data product as it allows to run not
only different types of analysis but to improve the analysis over
time, e.g. generate new data products.
You can add the execution of the post-processing to the generated
batch file or the generator if you want! You just don't have to.
I will include a comprehensive python validator that will check every
parameter and output error that we have seen to make sure that the
output is correct.
It is not useful to encode this in bash, however, at the same time, we
faced compatibility problems running Python on the compute nodes.
That said, your choice to include it to automatize the calculation.
I rather run the full checker to ensure I have not missed anything and
compute even more derived performance values than we do for the lists.
Sure, I can agree with this, as long as it is clear to users what needs
to be done to get a valid result.
Cheers, Andreas
--
Dr. Julian Kunkel
Lecturer, Department of Computer Science
+44 (0) 118 378 8218
http://www.cs.reading.ac.uk/
https://hps.vi4io.org/
_______________________________________________
IO-500 mailing list
IO-500@vi4io.org<mailto:IO-500@vi4io.org>
https://www.vi4io.org/mailman/listinfo/io-500
1945
days inactive
1966
days old
8 comments
3 participants
participants (3)
-
Andreas Dilger -
Carlile, Ken -
Julian Kunkel