I have posted a design document to run a test for multiple times on test framework.
Please review and provide suggestions.
@birbalsain I suggest to add a new option (like
--repeat-tests) to pbs_benchpress instead adding new key-value in -p.
Thanks @birbalsain for suggesting this feature.
I’m also agree with @hirenvadalia for the new option in pbs_benchpress.
And also can you add few details about the output of the repeated tests.
- Will there be run instance number associated in the results file(for e.g., ptl_test_results.json) ?
- If a test fails during run then PTL will stop or continue with next run ?
@birbalsain Does this work for single test case or we can pass multiple tests ?
It will work for all.
Anybody have any idea where from should i restart the test? In PTLTestRunner class, I am calling startTest method from _set_test_end_data method but test is not restarted.
@birbalsain In Details section, third point says " If repetition is 3 and test fails at 2nd run then it mark that as fail and will go to next run after the given test repetition delay" , which means, if the third run passes then overall test result shows as test has passed ?