BlazeMeter and LoadRunner Cloud appear similar on the surface:
They’re Both SaaS performance test tools, they both run JMeter, Gatling and other common scripts, and they both provide more than enough concurrent users for most needs.
But how do they compare to each other? And more importantly… which one should you choose?
What We Will Compare
In this article we’ll compare BlazeMeter Pro (BZM) and LoadRunner Cloud Dev Protocol Bundle (LRC) over 5 categories:
- Capability
- Additional Features
- Support
- Analysis & Reporting
- Cost
Finally, we’ll aggregate the scores, see who comes out on top, and recommend which tool you should choose.
Why We Are Comparing These Versions
While there are a few different versions of either tool, we have chosen BlazeMeter Pro and LoadRunner Cloud Dev Bundle for 4 reasons:
- They are both full cloud testing solutions
- They are both SaaS
- They have a similar price point
- They cover the same protocols
BlazeMeter v LoadRunner Cloud: Round 1 – Capability
In this section, we’ll look at the performance testing capability of each tool
Capability 1 – Protocol support
Both tools can test against the most common internet protocols including Web HTTP/HTML, JDBC, JMS, FTP, TCP/IP and WebSockets.
NB: LRC can offer additional protocol support including TruClient, ERPs such as SAP and Oracle, plus Citrix, Siebel and many others. These are more expensive bundles though, and not valid for our comparisons.
Capability 2 – Scripting
While both tools support the most common open-source scripting engines, they both have their own core scripting option. BZM uses JMeter and LRC uses DevWeb.
Scripting is much easier and quicker with LRC’s DevWeb, compared to BZM’s JMeter.
Scripting Engines Supported
Both tools offer support for multiple languages and scripting engines. There’s a slightly different coverage across each tool though, as you can see in the table below.
As you’d expect, Selenium scripting is more theoretical than realistic. It’s just not practical to run performance tests, at scale, through a GUI interface.
Capability 3 – Maximum Concurrent Users
No worries at all here – They can both test more than enough concurrent users unless you’re someone like Gazprom or McDonalds. BZM has a 5000 Vuser ceiling while LRC has a nominal 5,000,000 Vuser limit – although given we’re only comparing 80,000 Vusers, the remaining 4,920,000 Vusers are rather moot.
It’s worth noting that you can increase the Vuser ceiling in BZM by upgrading to the ‘Unleashed’ package.
Capability 4 – Maximum Concurrent Tests
With BZM you are limited to only 1 test at a time, which will create scheduling challenges if you want to execute tests across multiple workstreams or projects concurrently. Again, you can upgrade to the ‘Unleashed’ package for more tests, but this comes in at a higher price point and is outside the scope of this comparison.
LRC has no limit on concurrent tests, providing you have the necessary Vusers to cope. In theory, you could run 80,000 1-hour tests at the same time. Let us know if you manage to pull that off.
In reality, this means that each member of your team can run tests when they want.
Capability 5 – Test Duration
This is the first potential red flag against either solution. With BZM, your maximum performance test duration is only 5 hours. That means soak tests are out completely, DILO (Day In the Life Of) load tests are also out. Yet, again, you can access longer durations if you upgrade to the ‘Unleashed’ package.
No problems with LRC though, you can run your tests for 24 hours by default. This can be extended to 99 if you raise a service request.
Capability 6 – Ease of Use
It depends on what you’re used to of course, but most people will find the LRC user interface much more intuitive and easier to use than BZM.
Round 1, Capability: Verdict – LRC wins
LoadRunner Cloud is the clear winner when it comes to performance test capability
BZM 6/10: Good scripting options and concurrent users, but greatly let down by test duration and concurrent tests at this price point. Limited protocol support.
LRC 9/10: Strong showing across the board, LRC can scale and run long tests in parallel. Lost a point due to limited protocol support at this price point.
BlazeMeter v LoadRunner Cloud: Round 2 – Additional Features
In addition to out-and-out performance testing functionality, both tools offer additional features. These extras allow you to do that little bit more, whether it’s making your life easier or providing more depth and realism to your test scenarios.
Additional Features 1 – Service Virtualisation
Both products offer limited but free service virtualisation, both limited to 3 services – which is enough for most performance tests.
BZM allows you to use 3 services within their ‘Mock Services’ while LRC allows you to use 3 services with the Micro Focus ‘Service Virtualization Express’ product. Which Service Virtualization solution you prefer comes down to personal preference, although LRC does support a greater range of services out-of-the-box.
Additional Features 2 – API Monitoring
API monitoring is only available on BZM, allowing you to monitor and compare endpoints and workflow performance from multiple locations. When used in conjunction with performance testing it can potentially speed up issue resolution.
Additional Features 3 – Network Virtualization
BZM offers limited simulation of network conditions, through the JMeter httpclient.socket.http.cps and httpclient.socket.https.cps properties. This is just rudimentary throttling though, and not at all sophisticated or flexible.
LRC comes with the Micro Focus ‘Network Virtualization’ tool, which is an enterprise-level solution that allows you to simulate real-world network conditions such as bandwidth, location etc. For instance, you can simulate a slower connection in another country or someone using a mobile phone on the 10:00 train from London to Newcastle. Crucially, LRC lets you apply unique network conditions to each Vuser.
Additional Features 4 – Developer Performance Testing (Shifting Left)
BZM can be used by developers although this will consume your Vuser hours.
LRC integrates with LoadRunner Developer. This completely free tool provides each developer with 50 concurrent Vusers. Scripts can be written within and executed from their IDE of choice, and then reused by any other LoadRunner product if higher volumes are required.
Additional Features 5 – Whitelisted/Dedicated IPs
BZM doesn’t allow whitelisted IP addresses unless you go for their unleashed package.
LRC provides 10 dedicated IPs at no extra costs and allows you to purchase bundles of 30 IPs for a small fee.
Round 2, Additional Features: Verdict – LRC wins
LoadRunner Cloud comes with better extras
BZM 7/10: Lacking proper network virtualisation but provides a few neat solutions in other areas. The 3 Mock Services will be the most useful add-on for most users.
LRC 9/10: Network Virtualization and LoadRunner Developer are potential game-changers. Service Virtualization is limited, but the free whitelisted IPs are also a welcome addition.
BlazeMeter v LoadRunner Cloud: Round 3 – Support
What about when things go wrong, you run into a tricky situation, or you just need some guidance? It’s all well and good having a tool, but you need to know how to use it
Support 1 – Manufacturers Support
BZM has an online docs section that provides basic guides for most of your standard tasks, it is a bit clunky to use though. They also have a support ticket system for issues with BZM itself. Unfortunately, this is no help when you’re having problems with tool integrations – which can be the biggest source of pain. You’re dependant on the community for support in this area
LRC is a fully supported product – you get defect resolution, help queries etc. In addition, LRC has a huge knowledge base online. Within the Micro Focus help site, there are resources to help with scripting, testing, reporting, and ‘how to’ guides.
Support 2 – Community Support
There doesn’t seem to be a huge BZM community out in the wild. For the most part, you can use the highly active JMeter community, but that doesn’t always help.
LRC has more active community members, although probably not as many as JMeter. There is also a community area on the Micro Focus website, although it’s not as active as it could be. Questions are generally well answered though, eventually.
Round 3, Support: Verdict – LRC Wins
Strong win for LRC here – the professional support makes a big difference. It’s also a lot easier to find LRC answers online.
BZM 5/10: BZM’s Clunky docs section has basic guides. The BZM community is pretty small, but the JMeter community helps in a lot of cases.
LRC 8/10: Fully supported product plus lots of information on the Micro Focus Help Center. There’s a decent community out there too, just don’t rely on the Micro Focus community site.
BlazeMeter v LoadRunner Cloud: Round 4 – Analysis & Reporting
Testing is about results. Those results need to be both understood by the tester and presented to management and stakeholders. The right analysis and reporting functionality can remove days of effort.
Performance Analysis & Reporting
This is an area where BZM falters. BZM only offers extremely basic performance analysis, there are so many frustrations and limitations. For example, BZM doesn’t allow you to easily filter transactions statistic into passed or failed transactions. There are complicated workarounds for this as shown here. For a tool that costs $6k a year, you’d expect a lot more.
LRC is the complete opposite. The depth of analysis is outstanding, with the ability to drill into all sorts of statistics and information. LRC also provides great summaries and reports.
Round 4, Analysis and Reporting: Verdict – LRC Wins
LRC is the clear winner – It’s night and day. LRC looks and feels professional and capable, whereas BZM show’s its open-source roots.
BZM 5/10: Basic. BZM will require time, effort, and add-ons to get even a reasonable level of analysis.
LRC 10/10: Sophisticated and comprehensive reports and analysis are only ever a couple of clicks away.
BlazeMeter v LoadRunner Cloud: Round 5 – Cost
Both offer similar, albeit slightly different pricing models. BZM charges a fixed monthly fee, whereas LRC charges per Vuser Hour.
Cost 1 – 80,000 Vuser hours Cost Comparison
BZM allows a maximum of 80,000 Vuser hours a year – We’ll therefore use this figure for fair comparison of the different price models
NB: LRC also offers an additional pricing model where you purchase a set number of Vusers, rather than (or in addition to) Vuser hours. This does reduce the price considerably but is outside of the scope of the comparison.
Cost 2 – Vuser Hour Rounding
At first glance, BZM looks like the lower-cost option. There’s a big caveat with this cost though – Vuser Hour Roundup:
- BZM rounds up to the Vuser hour
- LRC rounds up to the Vuser minute.
This means that LRC users fewer Vusers per test – unless your test ends exactly on the hour.
See the table below for a comparison of 200 Vuser tests over various durations:
As a conservative estimate, you will easily use 10-50% fewer Vuser Hours with LRC, in normal usage. This negates the lower price of BZM. Bear in mind, if your BZM test aborts after 5 minutes you will still burn a whole hour for everyVuser running.
Vuser Rounding Examples
Let’s pull out a couple of examples for clarity:
Example 1: A 15-minute, 200 Vuser test
- With BZM this consumes 200 Vuser hours
- With LRC this only consumes 50 Vuser hours
- LRC saves you 150 Vuser Hours or 75%
Example 2: A 270-minute, 200 Vuser test
- With BZM this consumes 1000 Vuser hours
- With LRC this only consumes 900 Vuser hours
- LRC saves you 100 Vuser Hours or 10%
Cost 3 – Trial Periods
BZM has a free ‘Starter’ pack that allows up to 10 tests and 1 mock service. As far as trials go, it’s not overly generous, but it does give you a taster.
LRC offers a similar 10 test, 30-day trial period with 1000 Vuser hours and 1-hour max test duration. Again, a taster without offering true performance testing capacity.
Round 5, Cost: Verdict – Even
Both come in at a great price, with their own minor pros and cons – this round is a tie
BZM 9/10: A very affordable product. We’re giving BZM the benefit of the doubt and only marking down a point due to rounding – you can theoretically run perfect tests that end on the hour every time, with no slip-ups.
LRC 9/10: Slightly higher monthly cost, but you will likely use 10%-50% fewer Vusers per test – due to no rounding up in LRC.
BlazeMeter v LoadRunner Cloud: Overall Scores
After 5 rounds of intense head-to-head, the results are in. While the tools both score highly on cost, LoadRunner Cloud scores a lot higher overall, particularly on ‘Analysis & Reporting’ and performance testing capability.
BlazeMeter is let down by poor reporting functionality and limited performance capability – there are frustrating limitations around test durations and concurrent test runs.
As for LoadRunner Cloud, it’s fair to say that its pedigree shows.
BlazeMeter v LoadRunner Cloud: Which Should You Choose?
LoadRunner Cloud scores a resounding points victory and is easily the better choice of tool.
Why You Should Choose LoadRunner Cloud
If you’re looking to improve performance, gain detailed metrics, understand exactly how your system responds in the real world – then LoadRunner Cloud is, without doubt, the better option; it is a serious performance engineering tool at an incredibly compelling price point.
Why Not BlazeMeter?
Unfortunately, there just isn’t a persuasive reason to choose BlazeMeter. It isn’t an awful tool… it’s just not at the level of LoadRunner Cloud, and crucially it’s not any cheaper. As we’ve seen in the sections above, BlazeMeter offers less in most areas but comes in at a remarkably similar price. Even diehard open-source users can even run their JMeter scripts in LoadRunner Cloud, with more functionality.
Join our mailing list for more testing insights and product comparisons
