In the digital world, performance is measured in speed, throughput, response time, and availability.
You can test the performance of any site or software, and using this data, you can make changes to improve it. After reading this article, you’ll see how Magento performance testing can help you find weak links that are siphoning resources. By fixing the weak points, you can provide a better overall experience for site visitors.
Before getting our hands dirty with testing, let’s go over the basics.
Magento offers an in-depth testing guide on various testing strategies. In the guide, Magento divides tests into two categories: product quality tests and code quality tests. Both are important, and the results of the tests can be used in different ways.
Product quality tests are a way to verify that site visitors, support staff, administrators, and other end users get the results they expect when interacting with the Magento site.
These tests include:
Code quality tests are used by developers and Solution Architects. These tests guarantee that the site can be modified, customized, and scaled with little to no problem.
Code quality tests include:
In this article, we focus on server-side performance testing. From the screenshot above, you can see that performance testing measures how many resources (CPU, RAM, time) are used by Magento for a given store, user action, or other parameters.
Performance testing can be divided into two groups: load testing and stress testing.
A load test is a controlled, planned test that aims to evaluate the behavior of the site under a specific load (users, orders, transactions, etc.).
A stress test is similar to a load test, but the difference is that instead of increasing the load to a certain amount and then monitoring the behavior, you push the site to its breaking point. In other words, you incrementally increase the load above the expected maximum.
For most purposes, load testing is more informative. We’ll discuss the importance of load testing a bit later, but in the meantime, here’s a real-world example from one of our developers.
We interviewed Yevhenii Plakhotnyk, a Magento 2 backend developer at Elogic, about his experience with Magento performance testing, and he shared an insightful story.
“We had a client come to us who, at the time, had recently experienced a major site crash. Their site worked perfectly but had one glaring flaw, which they found out about after launching a marketing campaign for a public holiday: the site couldn’t handle the high volume of traffic and eventually went down.”
He continued, “And this is where performance testing is so important. Without running these tests, you don’t know where the bottlenecks and weak points are and, like the engineers at Chernobyl, will be surprised when it all goes up in smoke. Performance testing shows how to scale and teaches you to pay attention to details, which collectively affect the results of all your hard work.”
“Performance testing shows how to scale and teaches you to pay attention to details, which collectively affect the results of all your hard work.”
Yevhenii Plakhotnyk, Magento developer, Elogic
What’s the first step in performance testing, and where do we go from there? Here’s how you can avoid the same fate as our client.
Performance testing is complex. A single developer, QA specialist, or business analyst can’t put together a plan on their own. Testing requires a cohesive team, preferably with all three of the aforementioned roles, plus the product owner(s) and a DevOps expert.
A DevOps expert’s job is to automate the processes between software development and IT teams so that they can build, test, and deploy software in a more reliable way and at a faster rate.
Before you run the test, you’ll need a few things.
To get the most out of performance tests, we need to specify a lot of information with the client.
Specifically, we ask the client:
We also ask the client to prepare KPIs (performance acceptance criteria), or we help them do it.
Examples of KPIs for performance testing:
The great thing about performance testing is you don’t have to spend money on expensive software. The most popular tools are free and open source. For the best testing results, you’ll need three things: a performance testing tool, a profiling tool, and a tool to analyze the data and metrics.
1) Performance testing tools
2) Profiling tools
3) Analysis/reporting tools
We also need to know what the planned user journeys are. A user journey (or scenario) is a series of steps a user takes when interacting with a site. These scenarios have to be mapped out in detail because they’re used as predetermined paths for performance testing.
Example user journey:
This is the most basic user journey. The ones you’ll be using will likely be more complex and will emulate the site’s various sales funnels. Depending on the sales and marketing campaign, users land on different pages.
You can simulate simple browsing, layered browsing, logging into the account, abandoned carts, and anything else you can think of. Our advice is to stick to generic scenarios. But to get a complete picture of how deep performance tests can go, here are some ideas for advanced tests:
Please note: it’s not a good idea to include third-party extensions, gateways, or platforms in the tests like CDNs, payment gateways, shipping software, or others.
Once you have the business requirements, tools, and plans, you can start the testing process. To begin, you may need to generate data for your test environment. After you’re set with the test environment, you can move on to the testing checklist.
Depending on what you decide on the business requirements stage, you can test directly in the production environment, in a cloned production environment, or any other non-production environment.
If you want to purely measure the capabilities of your server hardware, create a benchmark to measure performance, or see how your site will perform when you increase certain parameters like products, categories, and other fixtures, you can generate sample data in Magento.
You can adjust the amount of sample data you create with profiles (small, medium, large, and extra-large). The following image shows how a product is displayed on the storefront using the small profile.
With Magento’s profile generator, you can create as many fixtures as you want. Here’s a full list of currently supported fixtures:
You can read more about generating data, as well as the commands you need to enter to generate the data in Magento DevDocs.
The details of creating a performance test are tool-specific. However, regardless of the tool you use, you will need to take the user journeys you mapped out in the preparation stage and input them into the performance testing tool to create a workload model.
Try to simulate real users when you set up the test. Jmeter has a guide on building a web test plan you can use to learn the basics. A complete test plan consists of one or more Thread Groups, logic controllers, sample generating controllers, listeners, timers, assertions, and configuration elements.
A quick tip: You can calculate the number of virtual users you need for the test with Google Analytics and this formula:
Virtual Users = (Hourly Sessions x Average Session Duration in seconds) / 3,600
After you have the testing environment in place, run through this pre-test checklist. This is an example of performance testing in a production environment (steps may vary depending on the environment).
After setting up your workload model, you can start the test. We recommend doing a trial run to ensure everything is running and data is being recorded.
Please note: Don’t change the scenarios at this point. You should only change the workload (traffic, duration).
There’s no rule to how many rounds of load tests you need to do. Start slow and gradually increase the workload. Each test round should have the same duration. Between rounds, make sure the analytics and monitoring tools are recording data.
Congratulations, you’ve just taken the first step to improving your store!
Now you’ve done the tests and you have the data, but you can’t draw conclusions from raw data alone. Yes, Jmeter offers visual HTML reports, but unfortunately, they’re fairly simple. That’s why you’ll need profiling tools, which have advanced visualization features like charts and graphs.
These are things you should keep in mind when analyzing data:
According to Microsoft’s Performance Testing Guidance for Web Applications, you should create two types of reports.
Whether you’ve done performance testing before or it’s your first time, there’s a lot you can learn from your test. You can uncover things you never thought to check, things your QA team missed, or things that went unnoticed during the development process. Experts recommend testing before deployment as well as testing any significant changes on a website.
Why is performance testing so important?
It allows you to:
Whether it’s before or after deployment, testing highlights bottlenecks caused by sub-par coding, database lag, slow components, lack of hardware resources, and a host of other reasons. By addressing them, you can significantly boost performance, and by proxy — revenue.
If you don’t know what your system is capable of, it’s impossible to develop a scaling strategy. You can still scale without testing. But as explained by Yevhen Zakharchuk, one of our senior developers, “If you decided to scale when something is eating up resources, it will be like driving around with a hole in your gas tank. You’ll have to constantly put more and more fuel in the tank to keep the car moving.”
“If you decided to scale when something is eating up resources, it will be like driving around with a hole in your gas tank. You’ll have to constantly put more and more fuel in the tank to keep the car moving.”
Yevhen Zakharchuk, Magento developer, Elogic
Like the example we mentioned earlier about our client who launched a marketing campaign without prior testing, you’re at risk of serious downtime if you don’t create scenarios that simulate increased system loads. Performance testing is a great tool to find these problems and it can help you find the right way to scale and improve.
We know that a good response time is critical for conversions and customer retention. By simulating and testing increased workloads, we can keep response time acceptable, and as a result, keep customers coming back.
No, we didn’t spend all that time communicating and consulting our clients in drawing up their KPIs for nothing. This is where you take those KPIs and see how the system stacks up against them. If it meets the requirements, great! If not, you have to consider what can be improved next time around.
Performance testing is complicated and expensive. It takes a lot of effort and resources to generate a test plan and put that plan into action. Frankly, if you’re a small merchant, it wouldn’t make much sense to do deep server-side performance tests.
However, if you have a medium, large, or enterprise ecommerce store, performance testing can be the difference between success and failure.
By testing the site’s performance, you verify that it meets your KPIs, identify the system’s capacities, and uncover bottlenecks that are holding you back. Additionally, if you plan on scaling, performance testing is highly recommended.
The most important takeaway is — test yourself before you wreck yourself.