In the previous posts, we discussed the emerging trend of Browser Level load testing and compared it to Protocol Level load.
One of the biggest motivations for preferring Browser Level load is reusing scripts.
With the burden of script maintenance in mind, no wonder that the idea of merging two stacks of scripts for performance and for functional testing into one is highly appealing.
Now that there are Performance Testing tools that can run Browser Level Load using Selenium, is Performance Testing just letting the performance tool replay the same Selenium script used for functional testing in load? Well, not quite.
There are a few important differences between Functional and Performance testing. some steps are required in order to fit the script for both purposes:
Measuring End-User Performance
In high level, Performance testing is generating load and collecting performance metrics. Most tools can collect general metrics in protocol level and high-end tools can collect server metrics as well but collecting end user metrics requires some intervention.
Pure Selenium does not measure performance. Some tools have ways to extract these, but this sometimes requires changes in the script itself.
In addition, the best way to measure the end user experience is by defining Transactions.
A transaction represents a user flow, with a clear start and end on which the tools can collect different timing metrics on each run and perform analysis over it.
Transactions are irrelevant for functional testing and would not appear in a functional testing script.
Different Purpose of Validating Correctness
Validating correct behavior of the application is done in both functional and performance testing. While it is the main purpose in functional testing and is usually pretty deep and through, in Performance it is usually done only as a safeguard to make sure the test is testing what its meant to.
Using a functional testing script would therefore add a lot of unnecessary checks under load which may introduce unnecessary computation cost, making the load test more expensive than it needs to be.
Validate Use Cases vs Simulate Common User Flow
Functional testing is all about coverage. Testing scripts focus on optimizing the coverage for use cases and code. It does not matter how many times a specific step is run and flows are rarely considered.
A common way of defining the scripts is by defining the test cases, often using unit testing framework (such as JUnit) or use cases using a BDD framework (Cucumber, JBehave).
Performance testing is about generating realistic load – Simulating real users and the flows that are common to them.
Therefore a good script to use for generating load would be a flow emulating what a real user would do. A Reusable script would need to be designed in this way.
Quick vs Realistic execution
Functional testing optimizes for quick results. It’s desired for them to finish as soon as possible. They often use ‘bailing’ on test fails and rarely include sleep calls unless they are needed to wait for the UI to be ready.
Load Generation is about simulating real users. Real users do not run as quick as a CPU. They stop , look at the content, make a decision, choose what to do next and then operate. A Real user is very likely to have wait times of a few seconds between actions. A functional script would rarely emulate that. A load script has to emulate a real user pace or running it can produce significantly more traffic (tens of times).
Functional testing runs only once. If there is need for diverse behaviors (for example choosing a different item from a menu), separate test cases will be created – each to run exactly once.
Load generation runs the same flow hundreds, thousands and sometimes millions of times. There is often a need to test diversity as part of the load (for example using different user names in the login). This is typically done by parameterizing the script and letting the load generator manage the parameters.
To sum it up – most functional scripts do not fit to run on load generation as is.
So how do we transform them to be reusable?
There are 2 approaches for building a multi purpose script:
One ‘cross testing script:
Just like a building website running on different browsers or applications running on different OS it is certainly possible to create a script that will run effectively for both load and functional testing using a little bit of conditional code. There are a few points that need to be considered:
- It is preferable to design the script as a flow, rather than a set of use cases
- Some behavior like sleep, transactions and some validation need to conditional and depending on the type of tests
- It needs to be parameterized, letting an external tool/framework manage the parameter value (so load and functional will use it differently
Different scripts, common library:
An altogether different approach could be breaking the scripts into a library of components modelling the UI and allowing the script to activate it simply and easily. The script itself will be a thin layer on top of these components. A good example for that is the very common PageObject pattern usually used with Selenium. These components or PageObjects would work the same for both functional and load testing. You will have different scripts for functional and load testing, but the majority of the code will be in the reusable components.