Thanks @bhroam for your views,
please find my replies below:
“I find this feature interesting, but I am somewhat confused about its purpose. If we use the current configuration instead of OOB to run tests, we’re basically doing a hybrid manual/automated test. The purpose of PTL is to write automated tests, do we want this hybrid? If you want to test an old feature with a new one, isn’t the right answer to write a new test that tests both?”
As rightly said by Kumar, the intent is to test the interaction of the PBS features when working together i.e. existing feature tests with new feature enabled or disabled. Of course, the new feature tests need to be written with test focusing on the nuances related to the newly introduced feature. Writing new PTL tests for existing features might lead to multiple versions of the same test with the given new feature on.
Yes,I think you are right with “hybrid tests” that makes it partially manual. Thinking about the hybrid tests; thought comes to my mind that it would be difficult if a bug is caught in this scenario and needs to be reproduced by someone else on a different setup. This makes me feel we need a feature to record and replay PBS configuration settings; which might be taken up as a separate RFE.
“I also am worried about the --use-current-setup feature. The purpose of revert_to_defaults() is to give a test a well-known setup where they can start from. Tests need to create a certain setup to test what they need to. If the base PBS configuration is changed out from underneath them, tests could fail. This could lead to test failures where its neither the test’s nor PBS’s fault. This leads to more manual effort to determine why a test failed.”
Again as said by Kumar, this is not for production. This feature is more intended to catch bugs when new features are developed and can also be used for adhoc testing by QA to test interactions of PBS features which are not conflicting. Hence, the validation of test result plays a very important role; since it might be a false pass or fail.
“One more question, is the real use case of this feature to save time in setUp()? Right now for every test, we revert back to defaults, just to set it back up.”
The way PBS is brought back to defaults is being changed to load_configuration() method being the way to get PBS back up. revert_to_defaults() would be called only once in setUpClass() to revert to OOB if PBS has anything different (in absence of use-current-setup).
“Are we trying to save this time and set some things up once and leave them set up? If this is a use case, you should mention it in the document.”
Yes, I have added this to point B.
“I am uncertain why we would want to revert back to OOB when --use-current_setup is not passed to pbs_benchpress. If we’re going through the trouble of saving configurations now, why not return us back to the configuration we started with?”
I think you are referring to case 3 & 4 from the table. The mention of use-current-setup means that there are valid differences in current PBS when compared to OOB which the test runner is authenticating. But case 3 & 4 falls under presence of invalid differences in current PBS that need to be reverted to OOB to start test on a clean environment i.e. the differences the user is not authenticating.
I would like to mention few examples of the use cases here:
- Disable scheduler’s throughput mode and run existing tests.
- Run existing tests on a varied cluster setup like “comm not running on server node”.
Ex: In below cluster, pbs.conf needs specific variables like PBS_LEAF_ROUTERS etc.