Please review the design proposal and provide comments.
Please review the design proposal and provide comments.
Thanks for separating this from PP-506
We’d like to see nfliter applied at the chunk level in the select statement. We certainly have jobs that require 2+ different types of nodes in a single job, so a job-wide nfilter would not suffice.
Chunk-level nfilter leads to the next complications (opportunities?), something I’ve pondered a bit as we’ve considered how to move toward an nfilter-like request mechanism. Here are a couple use cases I’m considering. I’ll use the resources_available resource “model” in my examples, at present model can be one of [san, ivy, has, bro] (representing Sandy Bridge, Ivy Bridge, Haswell, and Broadwell).
uc1) Multi-chunk job that chooses different MPI rank distributions per chunk based on memory, an example request might look like:
qsub -l select=1:mpiprocs=1:model=has:nfilter="node['resources_available'][‘mem’]>=512gb"+100:mpiprocs=12:nfilter="node['resources_available'][‘ncpus’]>=12 and node['resources_available'][‘gigs_per_core’]>=2gb"
uc2) Multi-chunk job that chooses different MPI rank distributions per chunk. The user doesn’t care which model is chosen, but they do want the same model to be chosen across all chunks. An example request might look like:
qsub -l select=1:mpiprocs=2:nfilter="node['resources_available'][‘model’]=pbs.same and bigmem=true"+100:mpiprocs=16:nfilter="node['resources_available'][‘model']=pbs.same and node['resources_available'][‘ncpus’]>=16"
In the above I’ve introduced a magic value pbs.same to accomplish the intent of the use case. I image another magic value pbs.any could satisfy a different use case where the user doesn’t care which model is provided by PBS. These values are illustrative, I’m not pushing for their use as presented here. The general notion is expressing an inter-chunk relationship. This is similar to the description of “like nodes” in the design of PP-506, but as I understand PP-506 a job-set would need to enumerate all requests that can satisfy the particular inter-chunk relationship rather than here where we constrain the matches a filter can come up with.
@gmatthew Thank you for providing your inputs.
I understand that having the filter run on each chunk would probably help in the use cases you mentioned. Although in case of uc2, would having a place=group= not help in getting “pbs.same” effect?
I actually tried a small prototype of having to filter 1000 nodes for each job and submitted 1000 jobs in the setup. It takes around 24 seconds for scheduler to run a job-wide filter on all the jobs once/scheduling cycle. Having to do this on each chunk will bring the performance down further. I think having python do the filtering may not be such a good option, although python does allow users to do a lot more while filtering (like using math module).
Do you see a use case where filtering on the basis of “resources_assigned” would help too? If it is only resources_available then maybe we can do something outside of scheduler.
If a magic value like pbs.same could be used as place=group=pbs.same, yes I could see that working for my use case. I wonder, though, how quickly we would want filtering that effectively acts as 2+ group-like resources, requiring something beyond the place=group= approach.
There are many interesting questions around the performance of nfilter based on design/use, but probably too early to get into this in great detail. For example the various bits of python overhead (spinning up an interpreter, marshaling, etc) may represent a sizable portion of that 24 seconds - or not. Bhroam has done some great work implementing our design for scheduling faster with node equivalence, that may (or may not) reduce the performance loss if python is involved. Does python have to be involved, even if it ends up not being performant in this application?
I don’t see a use case involving resources_assigned at the moment for our site, but I could easily imagine resources_assigned being handy for other sites.
@gmatthew Just to again touch base upon uc1 you previously mentioned -
Is that use case result of a one off job that you intend to submit once in a while (maybe like to benchmark) or is it something that is needed in every other job that intends to run on bigger memory nodes in their first chunk and so on?
uc1 is something our users are doing today, and we see a general trend toward setting up MPI jobs in that way in order to avoid overloading the head node with both the generally higher/different workload of rank 0 and the workload of additional ranks.
Can you name it ndfilter instead of nfilter. nfilter sounds like numeric filter
ndfilter might sound more like node-filter.
Sure I can make that change. @billnitzberg also suggested changes to “nfilter” and asked either change it to “node_filter” or just “filter”.
So do you two have any preference between “filter”, “node_filter” or “ndfilter”?
I’d prefer node_filter. Second preference would be ndfilter.
@billnitzberg asked me to provide more use cases for this feature, based on what our users are doing now. Here’s a quick bit of background on how our users request resources, then I’ll get to the use cases. Users pick from one of roughly 7 primary types of nodes using resources_available.model. If they have a need for some specialized subset of a primary type, e.g. bigmem, they request that resource as well. In some cases we have different bigmem sizes, so a user might additionally specify resources_available.mem to request the minimum mem size needed. Then we have onesy/twosey special nodes that might have a different resources_available.model, etc.
uc3) Multi-chunk job that picks a higher memory node for the head node and cheaper (lower accounting charge rate) nodes for the rest:
qsub -l select=1:model=bro:mpiprocs=4+100:model=san:mpiprocs=16
uc4) Multi-chunk job that picks all nodes from the same type, but places fewer MPI ranks on the head node:
qsub -l select=1:model=has:mpiprocs=8+199:model=has:mpiprocs=24
uc5) Multi-chunk job that picks a special onesy node for the last node of the job, and a different type for all other nodes:
qsub -l select=299:model=ivy+1:model=fasthas
a) In general there will be several model types that a user considers “cheaper” as used in uc3, so a node_filter for that particular chunk could express the acceptable set
b) For a request like uc4 we expect that users are concerned more with overall and per-core memory rather than the particular model.
c) We have thought about trying to reformulate select statements to more exactly describe the MPI/mem ratio needed and have PBS parcel out ranks to match. This would potentially recast uc3 to something like:
qsub -l select=1:model=bro:mpiprocs=4+1600:ncpus=1:mpiprocs=1:gigs_per_core=2gb
and PBS could find nodes that match, with possible contraints like pbs.same mentioned earlier in the discussion.
Currently we have moved resources away from this project. So this project is stalled for now.