Right now, there seem to be multiple test suites/PTL files which test similar functionality, For example - there are 4 different test suites related to qstat, 2 different test suites related to qsub, etc., and maybe it was actually prudent to divide them up, but right now we are missing concrete guidelines for when a developer should decide to create a new test suite for their test case(es) instead of using an existing suite.
So, I wanted to seek suggestions from the community on what they think is the right approach here to prevent test suite explosion.
Here’s what I propose:
- If your code change (PR) includes tests which test a section of the code that’s not covered by any existing test, then create a new, generic test suite for the tests.
- If your PR includes multiple tests which need a common, specialized setup, then they should be put in a dedicated test suite for better performance.
- If your tests seem to test the same feature that another test suite does, but the test suite is “named” very specifically to what its existing test cases cover, then rename the test suite to broaden its scope so that your test cases can fit.
- For all other cases, use an existing test suite to add your test cases to.