Magento Performance Testing: A Step-by-Step Guide to Supercharging Magento

In the digital world, performance is measured in speed, throughput, response time, and availability.

You can test the performance of any site or software, and using this data, you can make changes to improve it. After reading this article, you’ll see how Magento performance testing can help you find weak links that are siphoning resources. By fixing the weak points, you can provide a better overall experience for site visitors.

Before getting our hands dirty with testing, let’s go over the basics.

Types of quality tests [according to the Magento Testing Guide]

Magento offers an in-depth testing guide on various testing strategies. In the guide, Magento divides tests into two categories: product quality tests and code quality tests. Both are important, and the results of the tests can be used in different ways.

Product quality tests are a way to verify that site visitors, support staff, administrators, and other end users get the results they expect when interacting with the Magento site.

These tests include:

Code quality tests are used by developers and Solution Architects. These tests guarantee that the site can be modified, customized, and scaled with little to no problem.

Code quality tests include:

The difference between performance testing, load testing, and stress testing

In this article, we focus on server-side performance testing. From the screenshot above, you can see that performance testing measures how many resources (CPU, RAM, time) are used by Magento for a given store, user action, or other parameters.

Performance testing can be divided into two groups: load testing and stress testing.

A load test is a controlled, planned test that aims to evaluate the behavior of the site under a specific load (users, orders, transactions, etc.).

A stress test is similar to a load test, but the difference is that instead of increasing the load to a certain amount and then monitoring the behavior, you push the site to its breaking point. In other words, you incrementally increase the load above the expected maximum.

What’s better — load testing or stress testing?

For most purposes, load testing is more informative. We’ll discuss the importance of load testing a bit later, but in the meantime, here’s a real-world example from one of our developers.

We interviewed Yevhenii Plakhotnyk, a Magento 2 backend developer at Elogic, about his experience with Magento performance testing, and he shared an insightful story.

“We had a client come to us who, at the time, had recently experienced a major site crash. Their site worked perfectly but had one glaring flaw, which they found out about after launching a marketing campaign for a public holiday: the site couldn’t handle the high volume of traffic and eventually went down.”

He continued, “And this is where performance testing is so important. Without running these tests, you don’t know where the bottlenecks and weak points are and, like the engineers at Chernobyl, will be surprised when it all goes up in smoke. Performance testing shows how to scale and teaches you to pay attention to details, which collectively affect the results of all your hard work.”

What’s the first step in performance testing, and where do we go from there? Here’s how you can avoid the same fate as our client.

What you need to prepare before testing Magento 2 performance

Performance testing is complex. A single developer, QA specialist, or business analyst can’t put together a plan on their own. Testing requires a cohesive team, preferably with all three of the aforementioned roles, plus the product owner(s) and a DevOps expert.

A DevOps expert’s job is to automate the processes between software development and IT teams so that they can build, test, and deploy software in a more reliable way and at a faster rate.

Before you run the test, you’ll need a few things.

Client questionnaires 

To get the most out of performance tests, we need to specify a lot of information with the client.

♦ Gather general information

Specifically, we ask the client:

• What are your goals?
• When is the deadline?
• Where (environment) do you want to test?

♦ Prepare KPIs

We also ask the client to prepare KPIs (performance acceptance criteria), or we help them do it.

Examples of KPIs for performance testing:

• End user response time
• Maximum concurrent users
• Throughput
• Utilization of resources
• Business metrics

♦ Choose which tools you’re going to use

The great thing about performance testing is you don’t have to spend money on expensive software. The most popular tools are free and open source. For the best testing results, you’ll need three things: a performance testing tool, a profiling tool, and a tool to analyze the data and metrics.

1) Performance testing tools

• Apache JMeter — This is the tool recommended by Magento and is included in the performance toolkit.
• Gatling
• OpenSTA
• Siege

2) Profiling tools

• Blackfire
• Tideways — We recommend using a GUI like XhGUI for better visualization.

3) Analysis/reporting tools

• Google Analytics

♦ Create user journeys

We also need to know what the planned user journeys are. A user journey (or scenario) is a series of steps a user takes when interacting with a site. These scenarios have to be mapped out in detail because they’re used as predetermined paths for performance testing.

Example user journey:

1. Open the main page
2. Click on a product category
3. Click on a specific product
4. Add the product to the shopping cart
5. Open the cart and proceed with checkout
6. Checkout as a registered user or guest user
7. Reach the checkout success page

This is the most basic user journey. The ones you’ll be using will likely be more complex and will emulate the site’s various sales funnels. Depending on the sales and marketing campaign, users land on different pages.

You can simulate simple browsing, layered browsing, logging into the account, abandoned carts, and anything else you can think of. Our advice is to stick to generic scenarios. But to get a complete picture of how deep performance tests can go, here are some ideas for advanced tests:

• Use different data sets with varying numbers of customers, orders, products, categories, and other actions.
• Emulate mistakes (wrong password, incorrect credit card information, etc.)
• Set up “think time”. This setting simulates how people interact with websites. Think time creates pauses between actions, similar to how people think, read, or consider before clicking on something.
• Consider launching background processes to emulate the Magento backend and scheduled tasks

Please note: it’s not a good idea to include third-party extensions, gateways, or platforms in the tests like CDNs, payment gateways, and shipping software, or others.

The Magento performance testing process

Once you have the business requirements, tools, and plans, you can start Magento performance optimization. To begin the testing process, you may need to generate data for your test environment. After you’re set with the test environment, you can move on to the testing checklist.

Generating data for the test site

Depending on what you decide on the business requirements stage, you can test directly in the production environment, in a cloned production environment, or in any other non-production environment.

If you want to purely measure the capabilities of your server hardware, create a benchmark to measure performance, or see how your site will perform when you increase certain parameters like products, categories, and other fixtures, you can generate sample data in Magento.

You can adjust the amount of sample data you create with profiles (small, medium, large, and extra-large). The following image shows how a product is displayed on the storefront using the small profile.

With Magento’s profile generator, you can create as many fixtures as you want. Here’s a full list of currently supported fixtures:

• Admin users
• Attribute sets
• Bundle products
• Cart price rules
• Catalog price rules
• Categories
• Configurations
• Configurable products
• Customers
• Product images
• Indexers state
• Orders
• Simple products
• Websites
• Store groups
• Store views
• Tax rates

You can read more about generating data, as well as the commands you need to enter to generate the data in Magento DevDocs.

Set up the test with your tool of choice

The details of creating a Magento performance test are tool-specific. However, regardless of the tool you use, you will need to take the user journeys you mapped out in the preparation stage and input them into the performance testing tool to create a workload model.

Try to simulate real users when you set up the test. Jmeter has a guide on building a web test plan you can use to learn the basics. A complete test plan consists of one or more Thread Groups, logic controllers, sample generating controllers, listeners, timers, assertions, and configuration elements.

A quick tip: You can calculate the number of virtual users you need for the test with Google Analytics and this formula:

Virtual Users = (Hourly Sessions x Average Session Duration in seconds) / 3,600

Magento testing checklist [for a production environment]

After you have the testing environment in place, run through this pre-test checklist. This is an example of performance testing in a production environment (steps may vary depending on the environment).

• Step 1 – Clone the existing Magento database 
• Step 2 – Ensure all the tools are working properly
• Step 3 – Pause ping, availability, or health check services
• Step 4 – Enable maintenance mode (forward load test tool IPs)
• Step 5 – Ensure no integration feeds are running
• Step 6 – Switch to the cloned site and make sure the environment is operating
• Step 7 – Ensure only system-customer communication is disabled
• Step 8 – Back up critical log files
• Step 9 – Create a “performance test start” marker in your logs

Run the test

After setting up your workload model, you can start the test. We recommend doing a trial run to ensure everything is running and data is being recorded.

Please note: Don’t change the scenarios at this point. You should only change the workload (traffic, duration).

There’s no rule as to how many rounds of load tests you need to do. Start slow and gradually increase the workload. Each test of the Magento test server round should have the same duration. Between rounds, make sure the analytics and monitoring tools are recording data.

Congratulations, you’ve just taken the first step to improving your store!

Analyze the results and create reports

Now you’ve done the tests and you have the data, but you can’t draw conclusions from raw data alone. Yes, Jmeter offers visual HTML reports, but unfortunately, they’re fairly simple. That’s why you’ll need profiling tools, which have advanced visualization features like charts and graphs.

These are things you should keep in mind when analyzing data:

• Analyze the data against your performance acceptance criteria.
• If the test fails, you need to create a plan for performance improvements.
• When you have fixed all the performance issues or bottlenecks, retest to verify if all the issues have been fixed.
• The performance test analysis can help you make informed architecture and business decisions.
• Be careful about cleaning the data, you might delete valuable information.

According to Microsoft’s Performance Testing Guidance for Web Applications, you should create two types of reports.

Technical Reports

• Description of the test, including the workload model and test environment.
• Easy-to-digest data with minimal pre-processing.
• Access to the complete dataset and test conditions.
• Short statements of observations, concerns, questions, and requests for collaboration.

Stakeholder Reports

• Performance acceptance criteria to which the results relate.
• Intuitive, visual representations of the relevant data, workload model, and test environment.
• Brief verbal summaries of the chart or graph in terms of acceptance criteria.
• Access to associated technical reports, complete datasets, and test conditions.
• Summaries of observations, concerns, and recommendations.

What you can learn from performance tests

Whether you’ve done performance testing before or it’s your first time, there’s a lot you can learn from your test. You can uncover things you never thought to check, things your QA team missed, or things that went unnoticed during the development process. Experts recommend testing before deployment as well as testing any significant changes on a website.

Why is performance testing, especially Magento performance testing so important?

It allows you to:

⇒Discover bottlenecks 

Whether it’s before or after deployment, testing highlights bottlenecks caused by sub-par coding, database lag, slow components, lack of hardware resources, and a host of other reasons. By addressing them, you can significantly boost performance, and by proxy — revenue.

Read more: Why Magento 2 is slow?

⇒Develop a scaling strategy

If you don’t know what your system is capable of, it’s impossible to develop a scaling strategy. You can still scale without testing. But as explained by Yevhen Zakharchuk, one of our senior developers, “If you decided to scale when something is eating up resources, it will be like driving around with a hole in your gas tank. You’ll have to constantly put more and more fuel in the tank to keep the car moving.”

“If you decided to scale when something is eating up resources, it will be like driving around with a hole in your gas tank. You’ll have to constantly put more and more fuel in the tank to keep the car moving.”

Yevhen Zakharchuk, Magento developer, Elogic

⇒Reduce downtime 

Like the example we mentioned earlier about our client who launched a marketing campaign without prior testing, you’re at risk of serious downtime if you don’t create scenarios that simulate increased system loads. Performance testing is a great tool to find these problems and it can help you find the right way to scale and improve.

⇒Provide a better customer experience

We know that a good response time is critical for conversions and customer retention. By simulating and testing increased workloads, we can keep response time acceptable, and as a result, keep customers coming back.

⇒Meet performance KPIs

No, we didn’t spend all that time communicating and consulting our clients in drawing up their KPIs for nothing. This is where you take those KPIs and see how the system stacks up against them. If it meets the requirements, great! If not, you have to consider what can be improved next time around.

Takeaways — Magento performance testing is vital

Performance testing is complicated and expensive. It takes a lot of effort and resources to generate a test plan and put that plan into action. Frankly, if you’re a small merchant, it wouldn’t make much sense to do deep server-side performance tests.

However, if you have a medium, large, or enterprise eCommerce store, Magento performance testing can be the difference between success and failure.

By testing the site’s performance, you verify that it meets your KPIs, identify the system’s capacities, and uncover bottlenecks that are holding you back. Additionally, if you plan on scaling, performance testing is highly recommended.

The most important takeaway is — test yourself before you wreck yourself.