Test Framework should allow testers to run a test for multiple times

Hi,
I have posted a design document to run a test for multiple times on test framework.
Please review and provide suggestions.
Design Doc:
https://pbspro.atlassian.net/wiki/spaces/PD/pages/1386938446/Test+Framework+should+allow+testers+to+run+a+test+for+multiple+times

@birbalsain I suggest to add a new option (like -r and/or --repeat-tests) to pbs_benchpress instead adding new key-value in -p.

Thanks @birbalsain for suggesting this feature.

I’m also agree with @hirenvadalia for the new option in pbs_benchpress.
And also can you add few details about the output of the repeated tests.

  1. Will there be run instance number associated in the results file(for e.g., ptl_test_results.json) ?
  2. If a test fails during run then PTL will stop or continue with next run ?

@birbalsain Does this work for single test case or we can pass multiple tests ?

It will work for all.

Anybody have any idea where from should i restart the test? In PTLTestRunner class, I am calling startTest method from _set_test_end_data method but test is not restarted.

@vishwaks , @hirenvadalia , @shilpakodli please review the updated design document.

@birbalsain In Details section, third point says " If repetition is 3 and test fails at 2nd run then it mark that as fail and will go to next run after the given test repetition delay" , which means, if the third run passes then overall test result shows as test has passed ?