Performance Testing as part of your CI/CD strategy

Introduction

Performance testing has been traditionally relegated to latest software phases. Even Agile approaches fail to define clearly how to approach this time consuming activity, as teams fails to incorporate performance conditions in their DefinitionOfDone. In Telestax we strive to get best performance from our software, and durings latest year we have progressed on incorporating this activity in our ContinousDelivery strategy.

The problem

Performance testing is hard to achieve right. Both controlling the runtime environment, and how to collect the data to provide stable and comparable results is hard enough. This activity has been conducted traditionally by senior engineers, with enough experience at different levels (HW, OS, Application, scripting…).

  1. Start the monitored process: This should involve to execute whatever process (or set of processes) involved in the testing.
  2. Start monitoring tools: This should include OS system resource monitoring (CPU,Mem,network, IO…), and particular process resource monitoring (like GC for Java processes). These processes should harvest information with proper frequency, and ensure the performance penalty of monitoring is low, so the actual system performance is not impacted.
  3. Start injecting the defined load stimulus: This may imply to use testing tools (sipp, soapui, jmeter..), and hopefully inject the load with a ramp up curve, so monitored process is not burst with unrealistic traffic profile load. The testing tool should probably add more data into the final results, since important information must be analyzed to have a complete performance model (response time, latency…).
  4. Mantain load for a period of time: After ramp-up period, the load is mantined “constantly”, and the collection of data is done in the background.
  5. Stop, and settle down: Once the defined load testing scenario is achieved, we need to stop injecting load, and probably let system resources consumption settle down. This phase is particurlay interesting to monitor to identify possible leaks of resource consumption introduced by monitored process.
  6. Stop data collection: Stop all the processes associated to monitoring.
  7. Analyse resulting data: Hopefully the monitoring tools have collected enough data to reach a conclusion. This data may be interpreted in different ways, and usually graphs generation is done to help on interpretation.
  8. Assert on final results: Finally the data is aggregated into meaningful metrics, and those are asserted against the initial performance goals (CPU under certain threshold, call failure ratio under certain threshold..).
  9. Compare results with previous runs.
  1. Reserved Environment, but manually conducted: The runtime environment is prepared and reserved for later executions. Most of the tooling and process preparation is already in place, and just a bit of update is necessary to trigger a new execution. The tester manully ensures each step is completed
  2. Reserved Environment, scripted, but manually triggered: Same as before, but the steps are run by scripts, which hopefully check conditions after each step to ensure the whole run is valid. The execution is still manually triggered by the tester when the team decides to do performance testing.
  3. Reserved Environment, scripted, automatically triggered: Same as previous, but the execution of the script is hook into the software development lifecycle somehow. The scripts needs to be designed to be remotely invoked, and hopefully configured with meaningful parameters.

Our vision

We think we need to have the highest level of automation to ensure all the potential releases meet certain minimum performance goals. That involves to have predefined hardware environment to run performance testing, and scripting to cover all steps, including the possibility to hook these scripts into the software development lifecycle.

The solution

Again, we have progressed on this vision lately this year. Lots of this progress are coming from our performance tool called PerfCorder.

Let’s see it in Action

All the previous explanation is quite reasonable, and probably we could get a consensus in the community, but as the saying goes “A picture is worth a thousand words”. So, let’s see some screenshots of our current CI/CD environment. I will be showing my work for SIPServlets project, but same strategy its been followed by the rest of projects.

SipServlet Jobs View
Performance Job Params
Performance Job Configuration
Performance job Artifacts
Analysis HTML view
Performance job test results
Performance job failing test
Performance Meas Evolution Graph

Results of this practice on latest SIPServlets release

Performance testing is not only about meeting certain goals, but to know how your system will behave by defining a performance model. This model will help you to anticipate how the system will scale against bigger loads, or identify the hotspots in your system to improve.

CMS results
G1 results

Conclusion

I hope you enjoyed the previous demo of our performance testing strategy. We think its a simple, yet effective, toolkit to track your performance, and try to get the best of it.

Software Engineer with more than 15 years experience in the Telco sector. Currently working at Telestax.

Software Engineer with more than 15 years experience in the Telco sector. Currently working at Telestax.