I am submitting a number of jobs using for loop via python as:
for x in X: for a in A: for m in M: sample = "-".join([m,a,x]) os.chdir(sample) os.system(qsub script.sh)
The problem is in our local cluster implementation, we don’t have a
nice, and jobs are submitted in first come first serve basis. But, at any time, if my usage goes beyond 20% of nodes, sysadmin can kill it. I understand that this is not an optimal implementation of
qsub, but this is of course beyond my control.
I currently solve this problem by limiting my jobs as:
while True: numjobs = os.system("qselect |wc -l).read() if numjobs <4: break else: time.sleep(30)
Which is not very decent, but serves the purpose.
My question is, is there any way to tell
qsub how many jobs I can have at any given time so that I don’t need to check via
qselect on each loop?