While the benefits of traditional Functional Software Testing are well documented, Non Functional Testing is now taking center stage. Non Functional Testing verifies how applications fare in aspects such as performance, reliability and accessibility. This helps ensure better customer satisfaction and conversion rates, along with saved development and infrastructure costs. In this blog post, we shall focus on three increasingly important Non Functional software testing methods: Performance Testing, Load Testing, and Stress Testing – what they are, their differences and benefits and how to get started on each.
Performance Testing and Load Testing are often mixed up, but the latter is essentially a subset of the former. Performance Testing includes Load Testing and several other flavors and variants such as Stress Testing and Soak Testing. There is also UX Performance Testing which is another important aspect of Performance Testing, which will be explained in more detail later.
The main difference between UX Performance Testing and Load Testing is that UX Performance Testing (and optimization) is done when only one user is actually using the application at the same time. Load Testing requires you to simulate multiple concurrent users to make sure there is no deterioration in performance as the number of concurrent users rise. All of the aforementioned tests help ensure high levels of customer satisfaction, which is crucial in today’s competitive market.
As you will learn in the coming sections, Stress Testing, just like Load Testing, is a Performance Testing subset. This umbrella term also has other methodologies under it. Volume Testing is one of them. This kind of testing involves the populating of the database with varying amounts of data to gauge the application’s response and monitor performance fluctuations.
Other types of performance testing are Soak Testing: which aims to uncover performance problems that surface only during a long duration of time; and Capacity Testing: which helps you identify the maximum capacity of users the system can support, while not exceeding the maximum page time that has been defined in your SLA.
Performance Testing
Performance Testing mostly involves the evaluation of how an application or infrastructure responds to different workloads and how stable it can be under these fluctuating operational (usage, network, etc.) conditions. Top indicators include page response times, network responsiveness, server request processing speeds, and memory consumption patterns.
This kind of testing is crucial while developing dynamic applications that often see sudden spikes in usage. By understanding the robustness and reliability of the application, it can be worked upon before it hits the market. Developers can use the data collected to resolve inconsistencies and usability issues for better performance, resulting in enhanced user satisfaction and brand reliability.
UX performance testing is different since it measures how the application performs under the load of only one user at a time. It helps ensure the high speed and load time of a page or a transaction and makes sure that application or website is not hogging system resources. This can involve abnormal battery consumption, high data traffic, and unoptimized memory and CPU usage.
Performance Testing helps eliminate a wide range of problems by monitoring some key parameters. A few of these key parameters include:
- Processor Usage Patterns – Time spent on non-idle threads
- Disk Times – Time spent on writing requests or executing reads
- Bandwidth Usage – Bits per seconds used by network interfaces
- Throughput – Rate of requests received per second
- Number of Thread Counts – Number of active threads
- Efficiency – Garbage (unused memory) returned to the system
- Memory Usage – Virtual memory usage patterns
Common problems that can be detected with comprehensive Performance Testing include long application load times, slow response time to user input, and also poor scalability, where the app simply cannot accommodate a large number of users simultaneously. Furthermore, Performance Testing helps eliminate bottlenecks such as hardware problems and coding issues.
Detected Issues: Performance issues, low code integrity, runtime bloat, speed roadblocks, latency problems, throughput, inefficient resource usage
Load Testing
Load Testing is a key subset of Performance Testing that examines performance under predetermined loads. It’s the bread and butter of every Performance Engineer. Load Testing tests how a system behaves with a large number of users and what is the response time received for pages under different scenarios, which helps resolve performance bottlenecks. Load Testing can also help determine if the infrastructure being used is enough for the expected user load.
Here is how a Load Test is typically performed:
- First the Load Testing environment is defined and created
- Secondly, the Load Test scenarios are determined. This will typically involve transaction numbers and data for each transaction
- The predefined scenarios are executed
- Once the testing is complete, the application’s performance under various loads is analyzed and conclusions are drawn
- If the results are inconclusive, the Load Test is re-run
Just for example, a typical Load Testing procedure can test if the load time of 40 users viewing a webpage at the same time is not exceeding 2.5 seconds. It’s also recommended to perform Load Testing after every major release (part of the CI cycle) because code changes typically change the robustness of the application, which can hugely affect the end user experience.
While Response Time is undoubtedly the “King of Load Testing Metrics”, there are a few more key parameters that are uncovered with this kind of non-functional testing. Number of Transactions that have passed or failed, memory utilization, average latency, and CPU utilization are key metrics that need to be optimized after being examined by Load Testing the application.
To sum it up, good Load Testing protocols will help you determine your application’s Quality of Service (QOS) performance, a key aspect by all means.
Load Testing comes at a cost because you will need to hire expert professionals to carry it out, but it can save you money in the long run. The internet is full of examples where eCommerce websites crashed due to unexpected traffic while running promotions. Downtime can cost an organization over $650,000 per hour. Load Testing is a great long term investment that helps generate revenue.
Detected Issues: Performance bugs, bandwidth problems, load balancing issues, slow response time, throughput problems
Stress Testing
Stress Testing helps determine how a system behaves under extreme conditions such as extreme load, or with lesser memory and a weaker CPU.
You purposely try to break your system to answer a wide range of questions: Will the system start throwing out errors? How long does it take the system to recover? Will response time double? Will the entire system crash? Testers try to reach this point by doubling the number of users, using a database server with much less memory, or limiting by 5x the available network bandwidth.
Once the application or system starts faltering or suffers from performance hiccups, the information is passed on to the developers for further action. Building upon the Load Testing example from earlier, the test doesn’t stop with 40 users. The bar is raised. For example, system performance metrics can remain unharmed till the 55-user mark, after which things start deteriorating.
Also known in some circles as Endurance Testing, there are a few variations to it, all of which help determine the limit or threshold where the app will falter.
- Application Stress Testing – This testing specifically helps locate flaws and defects in data locking and blocking, along with other network issues
- Systemic Stress Testing – This is performed on systems that are running on the same server simultaneously to see if there are no blockages
- Transactional Stress Testing – As the name suggests, testers use this kind of testing to examine data transactions between two or more apps
There are also Exploratory Stress Testing procedures where the application and system are tested with unexpected parameters (often unrealistic ones). This can help find out what can happen when a high number of users log in at once or when virus scanning is commenced on multiple machines at once. It also helps check if appropriate error messages are displayed under extreme conditions.
A good time to implement Stress Testing can be when you have recently implemented a new tech stack and you wish to make sure everything works well.
How is Stress Testing performed? It all starts with planning, where goals are defined and data is gathered. Once the infrastructure is ready, automation scripts are created to simulate the required stress scenarios. Once the Stress Test scripts are executed, the results are collected and stored, after which they are analyzed to make the required application code optimization.
Detected Issues: Performance deterioration, data corruption, wrong error messages, memory leakage, security loopholes caused due to overloading
Load Testing vs Stress Testing
First and foremost, it must be clarified that Performance Testing is an important way to feel the pulse with the ongoing digitalization of organizations across all sectors. The need for speed is real, which is where Performance Testing comes into play. It also helps make sure that the application and framework it is working in are fully scalable or optimal business results.
Also, Performance Tests and Load Tests are not different processes, with the latter being more of a subset rather than a variant or competitor.
That being said, the Load Testing vs Stress Testing question often comes up while looking to start Performance Testing operations within the organization. In a nutshell, Load Testing is a methodology that is dedicated to improving performance metrics. User numbers are manipulated strategically to pinpoint the threshold where application performance metrics start deteriorating.
Stress Testing is the process used to test and understand how the application behaves when it reaches its breaking point under extreme conditions. This examination can involve error messages and data set handling to locate what can be fixed to raise the load threshold. It can also be used indirectly to detect security loopholes under these extreme conditions.
Also, Load Testing examines the entire environment and database while measuring the response time, while Stress Testing focuses on identified transactions, pushing them to a level where the functionality is terminated.
Load and Stress Testing should be performed regularly and integrated into the software development pipeline to test performance and optimize your product. Despite being different in many ways, both of these important non functional software testing methodologies are helping resolve glitches and stability issues before applications actually hit the market. Think lesser post-release patches.
| Load Testing | Stress Testing | |
| Performance Testing subset? | Yes | Yes |
| The focus of the testing | Performance | Behaviour at breakpoint, Recoverability |
| Simulated activity levels | Normal | High |
| What is manipulated? | Usage: User Numbers, transactions per second | Number of users, Memory, CPU, Network, etc. |
| Entire application / system | Yes | Focuses on identified transactions only |
| Is it done till the app breaks? | No | Yes |
| Does it help with security? | No | Yes |
Load and Stress Testing should be performed regularly and integrated into the development pipeline for testing performance and optimizing your product.
Summary
In the grand scheme of things, testing performance is no longer an option. Performance Testing remains the gold standard of non functional software testing. But depending on the requirements of your business, both Load Testing and Stress Testing should also be performed regularly and integrated into the DevOps pipeline for optimal results and enhanced customer satisfaction.
Regardless of what type of testing you are prioritizing at the moment, the solution of choice should be able to accommodate the crucial requirements for your organization. Your performance and Load testing tool should be dynamic and capable of generating varying user and network scenarios. It should also have a centralized dashboard for enhanced visibility and transparency.
Also, once you have a solution that can be automated and integrated into your CI/CD pipeline, you can also save valuable time, money, and resources.
The massive digitalization in recent years has created a need for robust and versatile applications that can be scaled up (or down) as per the demand in real-time, without suffering in the performance department. The speed, robustness, scalability, and dependability of your application is going to decide your bottom line. So pick the right testing solution and get started now.
