How many jobs can the scheduler hold at any given time? Can it hold more than 30000-50000 jobs?
Yes, but it can slow down the scheduler in the current 14.1 version if the user(s) have not reached their limit since we will look at every job. You can put in check in the sched_config file to tell the scheduler to only look at the first x number of job based on priority level. In the master branch (what will be the next release) we added a feature that will speed up the scheduler significantly with large number of jobs.
I guess I need to ask a follow up question. Do you mean 30K - 50K jobs in a Q state or 30K - 50K jobs run per day? If it is the first then my answer above is true for 14.1 but has been addressed for the next release. If it is the later then we do have sites that run 50K - 100K jobs per day without issue.
I meant 30K-50K jobs in Q state. Currently in the torque/maui setup, we schedule about 25K jobs at once, wait till the number of jobs in Q state reduce to around 10K and schedule another 20K jobs. But we wouldn’t like to do it if that’s an option
Can you please elaborate on what you mean by the scheduler would slow down, I did not follow what you said.
When is the next release due?
What I meant by “slow” down is that the scheduling cycle would take longer since you have more jobs to consider in a scheduling cycle.
Another thing that you could do is to set up a routing queue and an execution queue. Then you could limit the number of jobs in the execution queue on a per user basis to a 100-500 hundred. This will help the scheduler since it only looks at the jobs in the execution queue. This way you can submit all of the jobs needed an PBS would route them to the execution queue as needed.
The other option it to build PBS using the master branch test that out. The next release for PBS Pro 18.1 is targeted for the end of March 2018.