Add new hook events at the server end


#41

What would be the benefit to limiting the number of times a reservation can be altered? Why impose an arbitrary limit?


#42

@agurban, there have been requests from the field for this. @smgoosen will be in a better position to tell us why.


#43

Reservation end hook is essentially synchronous and the decision on reservation should continue or end should be taken as soon as the hook execution is over.

Altering the end time of the reservation is asynchronous. We send the request, and some time later, scheduler confirms if the reservation can be extended. So theoretically it has to wait for an undefined amount of time.

Whatever be the case, we need to decide on whether to end the reservation immediately after running the hook. If we have to alter the reservation, the server has to wait for an undefined amount of time to decide. So I think it is not practical to alter the reservation end time in the reservation end hook.


#44

Hi Prakash,

Thank you for highlighting this. I feel we shall have a resv_alter event. However, As per the current UCR, we have a requirement for the reservation confirmation event. I think this event can handle the alter requests which comes from the scheduler. we shall create a resv_alter event and have a seperate set of features for that when there is a requirement.

I think @crjayadev has rightly pointed out why I was asking that question. It is mostly related to extending the reservation while running the reservation end hook.

Thank you.


#45

I have an idea of a possible use for the reservation end hook event. I’m not sure I think it’s 100% appropriate for the hook event though. The idea is to use the reservation end hook event to extend the reservation’s walltime and allow all the jobs to finish.

The real use case is that the jobs in the reservation are running long and you want to allow them to finish. Is it better to have a hook that keeps extending the reservation, or just enhance reservations. Make a new type of reservation with a soft end time. It should take N hrs for this reservation, but let the jobs finish even if they take longer.

In any case, I don’t think altering reservations is possible in hooks yet. I think the only change we’d make to the design now is to call the reservation end hook before you delete all the jobs rather than after. This will open up the ability to allow us to handle this use case once altering reservations is possible.

Bhroam


#46

Hi Bhroam,

Yes.

I liked the idea of enhancing of reservations. I have a query here. In the above statement, Did you mean a reservation with dynamic end time which will extend itself(I mean by the server) or is it the hook event which will extend the walltime of “reservations with soft end time”.

Thank you for this suggestion. I have updated the EDD.

I agree to this.

Thank you.


#47

Either will work, but I’m not sure which is the right way to design it. Hooks are a bit of a hammer. Pretty much anything can be done with them, but is it the right way to implement something? We have soft_walltime for jobs now, should we have a similar feature for reservations? I don’t know.

In any case, thanks for updating the design document. I don’t think there is much difference to the functionality of the reservation end hook if it is run before or after we delete the jobs. If we run it before we delete the jobs, we can leave the way open for implementing these soft reservations via hooks.

Bhroam


#48

It was my understanding that we already have use cases for the reservation end hook and this discussion is limited to adding new hook events.

Regardless of whether it is possible to extend a reservation through reservation end hook or not, we already have the reservation end hook in the design you have shared, so according to me it is irrelevant to discuss if we could use this event for altering reservations or not.

As far as I know, there are separate requirements that need a resv_alter hook event and @smgoosen can confirm it as well.


#49

Apart from adding new events like resv_alter, at this stage is there any further discussion needed on the three events defined in the initial scope here? We are now implementing these initial three and I want to make sure there are no further changes required.

It sounds from the discussion like the reservation end event should trigger before we delete jobs.

What about the existence of pbs.event().resv.queue as a duplicate of pbs.server().queue(pbs.event().resv.queue.name) ? I would like to hear input from the PBS Pro crowd on whether or not this is desirable – my own inclination is that duplication should be avoided as a general principle.


#50

I agree that duplication should be avoided. It is probably better if we just get the queue name from the event and then interface with it via the server.

Even better, I think, would be to extend the reservation object further to abstract away the fact that in today’s PBS Pro reservations just happen to be implemented via special queues. For example, a hook writer should be able to control who has access to a reservation directly through reservation the object, and today PBS would achieve that for them seamlessly by setting ACL etc. attributes on the queue behind the scenes.

I consider this out of scope for the current work, though, so in the meantime building in duplication for the sake of minor convenience seems like a bad idea.


#51

Apologies for the delayed response. Thank you for clarifying things.

Okay In that case, I think this shall be added in a seperate ticket.

Thank you.


#52

Hi @scc and @mbonyak ,

I agree to this and I feel it shall be doable. However, I have one thought - In other job-related hooks, we see a similar sort of duplicacy for getting the queue object from pbs.server() and pbs.job.queue(). I shall be using probably a similar sort of way for reservations.

The above suggestion seems to me like a bit of enhancement upon the currently existing features. Please provide your thoughts.

Thank you.