PTL performance test automation



I have posted a document for the design of automating performance tests in PTL , So that the output can be saved and stored in db which can be useful for fetching comparison reports between builds and is useful to benchmark PBS . Please review and comment.



1> Can you please provide more details on which database(is it Postgresql database etc) we are going to use and where it resides ? How PTL is going to access such a database ?
2> The format of the JSON file should be standard so that it can be used for all other test suites not just for the performance tests alone. Thinking of the following format at the moment. Below snippet does not contain all the required fields. We can
discuss more on this and then finalise the format after which we can seek for community to help on finalising this.

% cat test_results_<data_time>.json

“tests”: {
“PerfTestSuite”: {
“testJobPerf_<instance_number>”: {
“expected”: “PASS”,
“actual”: “PASS”,
“test_start_time”: <start_time in epoch>
“test_end_time”: <end_time in epoch>
“interrupted”: false,
“path_delimiter”: “.”,
“pbs_version”: “18.2”,
“user_id” : “pbsroot”
“num_failures_by_type”: {
“FAIL”: 0,
“PASS”: 1

3> It would be better to rename perf_test_result to capture_test_results_json. Also this API should pass the result in a standard dictionary something like above. This way it is same and standard across all test cases.

4> I don’t think we no need to use separate API like perf_trial_result() for running the same test case multiple times. If we go with the above standard dictionary that can solve our problem.

5>The name of the JSON file should contain the date and time in regular format. Better not to put this in epoch.

6> We need to eliminate trial_run, trial_run_result tables. Also we need to club test_run and test_run_result into a same table. It is good to have just couple of tables to capture all these details. More redundant data and tables causes higher normalisation which results in having lot of join queries which might degrade the performance.


@vishesh, I have provided my initial comments. Please go through it.


I am working on a design proposal for JSON format support of PTL report at the framework level; which I plan to post in a couple of days. Hence, I feel this change which is specific to performance testing is completely not needed.


Thanks @saritakh ,
After your design proposal , We can discuss on a common JSON format for the report and work towards a common goal.


Hi @suresht ,
As you might know first we shall discuss with sarita on the generic JSON report for all the tests.

I will clarify on why we are using trial_run and trial_run_result .
For example:
If we consider a test for job submission time required for 100 jobs , We wont just submit 100 jobs to calculate time we will submit it 10 times and calculate an average .
It might be useful to know how much time it took each time . In that case we can use this .
For some other test case , you might want to run the test case in a loop for 5 times and check if the value we are getting is consistent then we can use trial_run data to store it .

It might not be needed for reporting but i feel it might be helpful for testing purposes .



Thanks for giving the example for trial_run. What I was initially thinking is that why can’t we store the trial run data in the test_run and test_run_result itself. To me if we run a test case say performance test case in this case consequently every day till 10 days OR if we run the same test case 10 times on a single day does not make much difference, both can be captured in the same table.

Regarding the other things as you said we need to discuss this with Sarita as they are about to post an EDD regarding storing test results in uniform way. We need to see whether all of our requirements match with them and then work towards the common goal. Let us keep this thread open so that they can all refer to it and incorporate the requirements that we need as well if appropriate to them.