NAME
Slurm::Sshare - wrapper around the Slurm sshare command
SYNOPSIS
use Slurm::Sshare;
my $Sshare = 'Slurm::Sshare';
$Sshare->verbose(1); #Print sshare commands as running them
my $shares = $Sshare->sshare_list(clusters=>'dt2');
foreach $share (@$shares)
{ print $share->cluster, $share->account, $share->user,
$share->raw_usage, $share->grpcpumins, "\n";
}
my $usage_hash = $Sshare->sshare_usage(clusters=>'dt2', users=>'nixon');
...DESCRIPTION
This is a wrapper around the Slurm sshare command. Basically, it allows the Slurm sshare command to called from within Perl, with some processing of the output into a more Perlish form, thereby enabling Perl scripts to have access to the share information for Slurm associations. The sshare command is called using forks and pipes, so no additional shell is spawned, and shell expansion is not done, making things a bit more secure.
The interface to this package is object oriented, mainly to reduce namespace pollution.
Since the sshare command is only useful when Slurm is running with the priority/multifactor plugin, this Perl class has the same restrictions on its usefulness.
Class Data Members
There are a couple of class data members that control the behavior of this package, that can be controlled by the following accessor/mutators:
-
The path to the Slurm sshare command. Normally, this defaults to just "sshare", i.e. it will look for sshare in your current path. Systems staff can set a different default by changing the value of $SSHARE_CMD at the top of this file.
- verbose:
-
If this is set, the module will work in verbose mode, which means that every sshare command will be printed to STDERR before execution. Default is false. If you wish to explicitly set this to false, you will need to provide a defined but false value (e.g. 0, but not undef) to the mutator.
These methods are both accessors and mutators, so in order to turn off verbose mode you need to supply a defined but false argument (e.g. 0) to the verbose function; if the value undef is provided, the call will be treated as a pure accessor (rather than mutator) call, and the new value will not be set.
The sshare method accepts zero, one, or two explicit arguments. With zero arguments, it acts as an accessor and returns the current path to the Slurm sshare command. With a single argument, it sets the path. With two arguments, the first argument sets the path, and the second argument is either a Slurm version number, or a hash ref describing what capabilities/features that version of the Slurm sshare command supports (see the section SLURM/Sshare versions for more information). If the second argument is omitted, the package will assume it needs to detect the capabilities of sshare if needed (and any previously cached information about that is deleted).
Because the Slurm sshare command only reads the Slurm databases and does not update them, there is no dryrun mode for this class.
CONSTRUCTOR and DATA members
Typically, one does not need to explicitly call the constructor for this class; the main methods for external consumption are class methods, and might return one or more instances of the class. But we include this section for completeness, and also to discuss the data members of this class.
The constructor new takes key => value pairs to set the initial value of data members. The instance data members are:
- account:
-
The account for this association.
- user:
-
The user for this association.
- partition:
-
The partition for this association. This is only available on newer Slurm sshares (v 15 and greater???).
- cluster:
-
The cluster for this association.
-
The raw shares assigned to this user/account.
-
The shares assigned to this user/account, normalized to the total number of assigned shares.
- raw_usage:
-
The number of cpu-seconds of all the jobs that charged this account by this user. This number will decay over time when PriorityDecayHalfLife is defined.
- normalized_usage:
-
The number of cpu-seconds of all the jobs that charged this account by this user, normalized to the total number of cpu-seconds of all jobs run on the cluster, subject to PriorityDecayHalfLife when defined.
- effective_usage:
-
Like normalized_usage, but augmented to account for usage from sibling accounts.
-
The fair share factorfor this account/user.
- grpcpumins:
-
The CPU minutes limit set on the account.
- cpurunmins:
-
The estimated (based an walltime limits) number of CPU minutes needed to complete all currently running jobs for this user/account.
- grptresmins:
-
This is a hash ref with the name of the trackable resource and the value is the limit imposed on this resource for this association and its children. Units will vary by the resource being tracked. For "cpu" it is in cpu-minutes. For "mem", it is in MB-minutes.
- trescpumins:
-
This is a hash ref with the name of the trackable resource and the value is the estimated amount of this resource that will be consumed by currently running jobs under this association. Units will vary by the resource being tracked. For "cpu" it is in cpu-minutes, for "mem", in MB-minutes.
Note that the cluster data member is not included in the sshare output, but will be filled in if possible by the sshare_list command if it is called on a specific cluster.
Also, older (before Slurm version 15.xx) versions of sshare output grpcpumins and cpurunmins, but later versions output grptresmins and trescpumins. This module will parse either format and produce both fields; obviously, if an older version of sshare is being used there will not be any data for any trackable resource except for "cpu".
The instance data members above are associated with read-only accessors with the same name. Normally they will be set when instances are created from parsing the output of the sshare command.
The sshare_list method
This class methods runs the sshare command with the appropriate arguments, and returns an array reference of instances of this class representing the rows of output returned by the sshare command. In addition to the class (an instance can also be used) invocant, it takes a (possibly empty) list of key => value pairs to provide arguments to the sshare command, with the following keys recognized:
- clusters:
-
A list of clusters to issue the sshare command to. I.e., the --clusters argument to the sshare command.
- accounts:
-
A list of accounts to report on. The --accounts argument to the sshare command.
- users:
-
A list of users to report on. The special value ALL, if given as the only user, will result in the --all flag passed to sshare, otherwise this becomes the --users argument to sshare. An undef value or an empty list ref will cause only records without an user value to be returned (in actuality, we issue --users=NO SUCH USER to sshare).
- nopartinfo:
-
A boolean value. If set, the package will not request partition information (i.e. the --partition flag will not be provided to the sshare command) even if the sshare command supports it. In particular, setting this flag might cause the package to forego an extra call to sshare to determine if the flag is supported.
For all the lists above, you can give either a list ref or a scalar CSV string.
The accounts argument is converted (if neccessary) to a scalar CSV string and added to the argument list of the sshare command with the --accounts flag. The users argument is handled similarly, except that if the user list is 'ALL', instead of setting the --users flag the -all flag will be added to the argument list of the sshare command.
The clusters argument, if given, is handled specially. For each cluster specified, the sshare command is invoked with the --clusters argument set to that cluster name (along with any flags from the accounts and users arguments), and the results for each single cluster invocation of sshare are parsed separately, and passed the name of the cluster the command was issued to in order to set the cluster data member. If the clusters argument is not given, the sshare command is invoked just once, without any --clusters argument, and the cluster data member will NOT be set when the output of the sshare command is parsed.
The sshare_list command will raise an exception if called with improper arguments. On more transient errors (like the sshare command errored), it will return a non-reference scalar error text. Otherwise, on success it returns a (possibly empty) list of Slurm::Sshare instances representing each association the sshare reported on.
The combine_tres_hashes
This takes a pair of hash references, and combines them. The hash refs are expected to represent TRES names and values, e.g. values for GrpTRESMins or TRESRunMins. Like TRES keys in both hashes are added together; if a key exists in only one of the two hash refs, it is just passed through (i.e. the missing field in the other hash is treated as a zero).
The combine_sshare_usage_records method
The Slurm sshare command produces a line of output for each Slurm association meeting the specified criteria. Depending on the version of Slurm/sshare command being run, the partition information might or might not be available to the Slurm::Sshare package (and even if it is available, package configuration might prevent its being requested). Even if partition information is available, it might not be of interest in certain cases. Usually in such cases what one is interested is the total usage statistics over all of a set of related associations.
The combine_sshare_usage_records method takes a list ref of such Slurm::Sshare records/instances and returns a hash reference with the combined usage statistics. The returned hash reference can have the following keys:
- raw_usage: The total number of raw_usage for all records in the input list. This is the total number of CPU seconds used by these associations.
- normalized_usage: The total number of normalized_usage for all records in the input list.
- effective_usage: The total number of effective_usage for all records in the input list.
- grpcpumins: The total number of grpcpumins for all records in the input list. This is in CPU minutes.
- cpurunmins: The total number of cpurunmins for all records in the input list. This is the total number of CPU minutes expected to be needed to complete all currently running jobs in these associations.
- grptresmins: A hash ref representing the account limits on various TRESes.
- tresrunmins: A hash ref representing the amount of various TRESes estimated to be needed to complete currently running jobs.
At this time, the above fields (except for the TRES hash references grptresmins and tresrunmins) are simply summed over the input records. The TRES hash references are combined with the method combine_tres_hashes. This should be the appropriate handling of the usage fields and the cpurunmins fields; I am less certain about the shares, fairshare, and grpcpumins fields, but I cannot think of a better way to handle them either. If a value is missing or undefined in any of the input records, it does not contribute to the output hash, and indeed, fields in the output hash can be missing or undefined if they are not defined in any of the input fields.
NOTE: This routine does not do any sanity checking or verification that it makes sense to combine the records given to it; it blindly combined the usage statistics on all records given trusting that you know what you are doing.
Although it is typically expected that the input list ref will consist of instances of Slurm::Sshare, this method will also accept a hash ref with the same fields as the output hash instead of an instance of Slurm::Sshare.
The collect_usage_records_by_account method
This method takes a list ref of Slurm::Sshare instances and sorts by cluster, account, and user to produce a hash ref with usage statistics. Input parameters should be given as key => value pairs, with the following keys recognized:
- records:
-
the list ref of Slurm::Sshare instances (i.e. records from sshare command) to sort.
- clusters:
-
a list ref of cluster names to include in the results. If omitted, results will include data for all clusters referenced in records. To restrict results to those for records with cluster undefined, either provide an empty list ref or use either the string 'DEFAULT' or an undef as values in the list ref.
- accounts:
-
a list ref of account names to include in the results. If omitted or an empty list ref, results will include data for all accounts referenced in records.
- users:
-
a list ref of user names to include in the results. If omitted, results will include data for all users referenced in records.
The return value is a hash ref of hash refs. The outermost hash is keyed on cluster names (with results corresponding to records for which the cluster is undefined are put under the key DEFAULT). The values are once again hash refs, this time keyed under the account name. The values again are hash refs, with keys giving the account and cluster and usage information for the account (in the specified cluster). It may also have a key users_hash, whose value is another hash ref keyed on usernames, the value being another hash ref giving usage information for that combination of user, account, and cluster. E.g., the result would be something like
$result =
{ cluster1 =>
{ account1 =>
{ account => 'account1',
cluster => 'cluster1',
raw_usage => 777777,
normalized_usage => .777777,
...
users_hash =>
{ george =>
{ user => 'george',
account => 'account1',
cluster => 'cluster1',
raw_usage => 111111,
...
},
kevin =>
{ user => 'kevin',
account => 'account1',
cluster => 'cluster1',
raw_usage => 222222,
...
},
...
},
},
account2 =>
{ account => 'account2',
cluster => 'cluster1',
raw_usage => 1234567,
...
users_hash =>
{ george => { ... },
kevin => { ... },
...
}
},
...
},
cluster2 =>
{ account1 =>
{ account => 'account1',
cluster => 'cluster2',
...
users_hash =>
{ george => { ... },
...
},
},
account2 => { ... },
...
},
...
};The usage hashes for both the account and individual users can contain the following keys:
- account:
-
The name of the account. This should always be present.
- cluster:
-
The name of the cluster. This will be undefined if the cluster was not specified.
-
These will represent the same values as given by the corresponding data members of this class. These will represent the sum over all partitions if there are multiple records for the specified cluster/account and/or user.
- grptresmins, tresrunmins:
-
These will be hash refs, keyed on TRES type/name.
The usage hashes for individual users will contain the additional key user, whose value will be the name of the user in question.
The usage hashes for accounts may contain the additional key users_hash whose value is a hash ref keyed on username with values being the usage hash for that individual user (for that account/cluster).
NOTE: restricting users,accounts, and/or clusters only effectively filters the records in records being considered. E.g., the total usage for a given account in a cluster will include usage for all users of that account, not just the ones in the users list ref. But the usage_hash will only
The sshare_usage method
This method combines the sshare_list and collect_usage_records_by_account methods, because it is expected that the two will generally be used together. For input, it takes the same key => value pairs as sshare_list does, namely
- clusters:
-
A list of clusters to issue the sshare command to. I.e., the --clusters argument to the sshare command.
- accounts:
-
A list of accounts to report on. The --accounts argument to the sshare command.
- users:
-
A list of users to report on. The special value ALL, if given as the only user, will result in the --all flag passed to sshare, otherwise this becomes the --users argument to sshare.
It then calls collect_usage_records_by_account, feeding it the results of the sshare_list command, and returns the result. The return value is the same as for collect_usage_records_by_account, namely a hash ref of hash refs for cluster, account, and users. Note that no arguments restricting the clusters, accounts, or users in the results are passed to collect_usage_records_by_account, so only the parameters passed to sshare_list restrict what is included in the results.
The sshare_usage_for_account method
This is a convenience method for calling sshare_usage to return data for a specific account (in a specific cluster). It takes key => value pairs for its arguments, recognizing the following keys:
- account:
-
The name of the account to get usage for. REQUIRED.
- cluster:
-
The name of the cluster to get usage for. The sshare default will be used if omitted.
- users:
-
A list ref of user names to return data for. This is passed to the users argument of sshare_usage. If omitted, data for all users associated with the account will be returned. Give an empty list ref (e.g. []) to have no user information returned.
- nowarnings:
-
If this boolean value is set to true, warnings about not finding any data for the account will be suppressed. By default, if not data for the account is found, a warning will be produced on STDERR.
The return value is the account level hash ref as described in collect_usage_records_by_account.
The sshare_usage_for_account_user
Like sshare_usage_for_account this is a convenience wrapper around sshare_usage, but this form returns the usage data for a specific user of an account in a cluster. It takes key => value pairs for input parameters, recognizing:
- account:
-
The name of the account to get usage for. REQUIRED.
- user:
-
The name of the user to get usage for. REQUIRED.
- cluster:
-
The name of the cluster to get usage for. The sshare default will be used if omitted.
- nowarnings:
-
If this boolean value is set to true, warnings about not finding any data for the account/user will be suppressed. By default, if not data for the account and user is found, a warning will be produced on STDERR.
The return value is the user level hash ref (e.g. the value for this username in the users_hash hash ref) as described in collect_usage_records_by_account.
The usage_for_account_in_cluster method
THIS METHOD IS DEPRECATED. Use get_usage_for_account_in_cluster instead, which basically takes the same arguments and returns the same information (and more) as a hash ref instead of an array ref.
This class method runs and parses the appropriate sshare commands to compute the usage for a specific Slurm allocation account in a specific cluster. This makes certain assumptions on how the associations, and limits for the associations, are defined in Slurm. It is assumed that there in a given cluster, there is a single association for the allocation account without an user set, and that the usage limit for the allocation account is set soley in the GrpCPUMins field of this association. It is believed that this is a fairly common arrangement, but it could fail if, for example, there are associations without an user set for this allocation account for specific partitions, with or without GrpCPUMins set on the per-partition associations. This assumption also fails if the allocation account has limits imposed on it from parent associations.
The method takes its arguments as key => value pairs, recognizing the following keys:
- cluster:
-
The name of the cluster to get information for. The underlying sshare_list command will be directed at that cluster. This parameter is REQUIRED.
- account:
-
The name of the allocation account to get information for. This parameter is REQUIRED.
- users:
-
This is a list ref of users to return data for. If omitted or undef, data will be returned for all users found. To return data for no user, set this to an empty list ref (e.g. []).
- nowarnings:
-
This takes a standard Perl boolean value. If true, warnings to STDERR for the account not being found in sshare output are suppressed. Default is false (display warnings).
The method will raise a fatal exception on certain errors, mainly for errors related to being invoked improperly (e.g. required parameters are missing, invalid data type for users, etc). For less serious errors (e.g. the underlying sshare did not succeed), an non-reference scalar error text string will be returned. On success, an array reference with the following elements will be returned:
- cpusec_used :
-
the total number of cpusecs used by all jobs charged to the specified account/cluster.
- cpumin_limit :
-
the GrpCPUMins limit for this account/cluster (see caveats above).
- cpumin_unused:
-
the difference between cpumin_limit and cpusec_used (in CPU-minutes)
- used_by_username:
-
a hash reference, keyed by username, of the total number of CPU seconds used by jobs in all associations with the specified account/cluster and username. If the users array reference was given, it will be restricted to users in the specified list.
In certain suspicious situations (e.g. no associations for the specified account in the specified cluster could be found, or some of the assumptions discussed above do not appear to be holding), the method will try to return a sensible answer, but will generate a warning to STDERR. E.g., if no assocation for the account is found, a warning will be generated but data (with all values 0) will be returned.
The usage_for_user_account_in_cluster method
This class method will invoke the Slurm sshare command to get usage information for a specific user in a specified allocation account and cluster. The sshare command will normally return a separate line for each association associated with this user/account/cluster (i.e., a separate line for each partition the user is associated with). This method will sum up the CPU seconds used for this user over all the associations, and return it.
The method takes its arguments as key => value pairs, recognizing the following keys:
- user: the name of the user to get usage for. REQUIRED.
- account: the name of the account to get usage for. REQUIRED.
- nowarnings: if true, warnings to STDERR are suppressed. Default is false.
All errors in this routine will result in an exception being raised; this includes errors running the sshare command. On success, the sum of the CPU seconds consumed for all associations with the specified user, account, and cluster will be returned. On some suspicious cases the method will return the best value it can get but also print a warning to STDERR (unless nowarnings is set to true). E.g. if no associations found for the specified user, account, and cluster, the method will print a warning and return 0 CPU-seconds.
SLURM/Sshare versions
This package has been used in various forms on version of Slurm going back to about 2.6. Version 15.x introduced some significant changes impacting the sshare command and this package, namely
These changes have necessitated some significant changes in the Slurm::Sshare package. The parsing of sshare output now can handle output with either GrpCPUMins and CPURunMins or GrpTRESMins and TRESRunMins. It can also handle output from sshare either with or without partition information. By default, the methods invoking sshare commands will attempt to request partition information if the package believes the sshare command supports such; determining whether the sshare command supports such is the tricky part.
When the path to the sshare command is set (either through the sshare mutator, or via default settings), it is possible to tell the package whether the flag for requesting partition information is supported by that sshare command. For the sshare mutator, this is done via the optional second explicit argument; one can provide a hash reference with capibilities of the sshare command as keys and the values indicating whether the capability is available or not. The values follow standard Perl boolean semantics, EXCEPT that an undef value (or a missing key in the hash ref) is interpretted as "unknown" and that the code will need to determine this on its own if necessary. If a defined value is given, the code assumes the user knows what the sshare command can handle, and the code will ignore any evidence to the contrary. The only "capability" currently recognized is "can_display_partition", so one can do something like:
Slurm::Sshare->sshare('/some/path/sshare', { can_display_partition=> 1 } );to set the path to the sshare command to '/some/path/sshare' and instruct the package that this sshare command does support the -m or --partition flag to display partition information. Similarly,
Slurm::Sshare->sshare('/some/path/sshare', { can_display_partition=> 0 } );would set the path to the sshare command and instruct the package that is does NOT support that flag. Alternatively, one can replace the capabilities hash reference with a scalar representing the Slurm version number, e.g.
Slurm::Sshare->sshare('/some/path/sshare', '15.08.2');to set the path and instruct the package that sshare is for Slurm 15.08.2, and has the corresponding capabilities. Omitting the hash ref (or providing an empty hash reference or one for which the value of the can_display_partition field is undefined) will set the path to sshare but require the package to figure out whether the partition flag is supported.
The method set_sshare_capabilities_by_version($SLURM_VERSION_NUMBER) will set the current capabilities hash based on the specified Slurm version number.
System administrators can change the default used for the sshare command by changing the $SSHARE_CMD variable at the top of this file; at the same time they can modify $SSHARE_CMD_CAPABILITIES to a hash ref providing the capabilities of the sshare command being used. (Or instead of directly setting $SSHARE_CMD_CAPABILITIES, you can invoke set_sshare_capabilities_by_version with the appropriate version number). We recommend that system administrators set both the path and version appropriately. By default, whatever sshare is found in the user's path is used, and the package needs to determine what capabilities it has.
The capabilities of the sshare command (and all cached knowledge about such) is cleared whenever the sshare method is called with a path (even if the path is the same as has been previously used).
If the package needs to determine the capabilities of the sshare command, it does so lazily. E.g., if all calls to sshare_list (direct or indirect) have the nopartinfo argument set, the code does not need to know if sshare can handle the --partition flag, and so it does not need to invoke a separate sshare command to discover this. Furthermore, since the displaying of GrpTRESMins instead of GrpCPUMins (is believed to have) occurred at the same time, the code to parse sshare output will use that to silently cache whether the command supports the partition flag. Only if actually necessary will the package silently issue an sshare command to determine this fact (currently it bases such on the output of the sshare --help command).
The class method sshare_cmd_supports allows one to query the capabilities of the current sshare command. It takes one explicit required parameter, the name of the capability being queried (currently, only 'can_display_partition' is supported), and an optional boolean variable. If this optional variable is true, only cached results will be used, and the method will not invoke sshare to discover the capabilities. If omitted or false, the method will invoke sshare to discover its capabilities as described above if needed. The return value is 1 (true) if the sshare command supports the named capability, 0 (false) if it does not, or undef (false) if it is not known. This last case can only occur if the optional boolean parameter was set to true.
It is hoped that this approach will allow for a solution to this current difference in version as well as being robust enough to adapt to future changes in Slurm.
EXPORT
None. OO interface only.
SEE ALSO
Slurmdb
Slurm::Sacctmgr
AUTHOR
Tom Payerle, payerle@umd.edu
COPYRIGHT AND LICENSE
Copyright (C) 2014-2016 by the University of Maryland
This program is free software; you can redistribute it and/or modify it under the terms of the the Artistic License (2.0). You may obtain a copy of the full license at:
http://www.perlfoundation.org/artistic_license_2_0
Any use, modification, and distribution of the Standard or Modified Versions is governed by this Artistic License. By using, modifying or distributing the Package, you accept this license. Do not use, modify, or distribute the Package, if you do not accept this license.
If your Modified Version has been derived from a Modified Version made by someone other than you, you are nevertheless required to ensure that your Modified Version complies with the requirements of this license.
This license does not grant you the right to use any trademark, service mark, tradename, or logo of the Copyright Holder.
This license includes the non-exclusive, worldwide, free-of-charge patent license to make, have made, use, offer to sell, sell, import and otherwise transfer the Package with respect to any patent claims licensable by the Copyright Holder that are necessarily infringed by the Package. If you institute patent litigation (including a cross-claim or counterclaim) against any party alleging that the Package constitutes direct or contributory patent infringement, then this Artistic License to you shall terminate on the date that such litigation is filed.
Disclaimer of Warranty: THE PACKAGE IS PROVIDED BY THE COPYRIGHT HOLDER AND CONTRIBUTORS "AS IS' AND WITHOUT ANY EXPRESS OR IMPLIED WARRANTIES. THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR NON-INFRINGEMENT ARE DISCLAIMED TO THE EXTENT PERMITTED BY YOUR LOCAL LAW. UNLESS REQUIRED BY LAW, NO COPYRIGHT HOLDER OR CONTRIBUTOR WILL BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL DAMAGES ARISING IN ANY WAY OUT OF THE USE OF THE PACKAGE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.