Open Source or Paid tools? It’s one of the biggest questions facing software projects and is a question that can deeply divide opinions. It’s a choice that often boils down to the perceived cost – free versus paid. But is it as simple as that? Once you scratch beneath the surface, there’s a lot more to consider.
I recently interviewed a host of senior test professionals across multiple industries to get their opinions. I asked the question – “What are your thoughts on open source v paid test tools?”
Naturally, these conversations took many different paths, but I was also shocked at the number of incorrect assumptions so many of them were making. After analysing the transcripts, my team and I identified 7 specific areas that needed to be urgently addressed.
Based on these, we have provided 7 real-world insights to help inform your future decisions.
Insight #1 – Open Source Tools Don’t Always Save You Money
Let’s kick things off with possibly the most important, and misunderstood factors when deciding between paid and open source tools. Our interviews confirmed our long-held belief that a tools list price has little correlation to its total cost ownership.
A test manager with a major UK Estate Agent puts it this way:
“I’ve always kept an open mind when it comes to open source vs paid tools. I go back to a role I had with a council many, many years ago, and they were implementing a new website at the time. They did everything open source and they thought they’d be saving lots and lots of money. But then when the development costs came in, to actually get it to where they needed it, the budget went way over what it should have done. I think they had a quote for a paid tool at the time which would have come under what they actually spent to get the job done using open source tools.”
One thing that I didn’t expect, was the number of interviewees, and companies who had no idea about the total cost of ownership. Most had only calculated the up-front cost.
One of our interviewees was very open about the lack of TCO consideration:
“To be honest, when I joined in 2018, we were using Selenium on a smaller scale, so I haven’t actually done the sums.”
Insight #1: Key Takeaway
Operational budgets are coming under more scrutiny again. Set against this context, it’s crucial that you understand how much a test tool will cost you over the total period of ownership, or at least the first 3 years. Making the wrong choice up front could have far-reaching impacts and limit your available testing budget for years to come.
Insight #1: Action Item: Factor in the people costs associated with implementation and maintenance effort. These will likely be much higher than the software price
Insight #2: Don’t Believe The Hype
All too often, test tools are hyped up to epic proportions. They’re claimed to offer amazing new features, be faster, be more efficient and so on. This is often supported by demo videos and presentations. Unfortunately, the reality can be quite different.
Frequently, ‘amazing’ open source solutions are pushed by consultancies who have a vested interest in selling more days. Generally, these tools don’t deliver on their promises. The demos have been curated (and potentially manipulated) to capture and showcase the best possible case scenarios. You’re then tied to a resource-intensive solution and need to buy even more consulting days.
Meanwhile, solid and proven tools aren’t spoken about as much and are positioned as expensive and somehow less able. In reality, it’s these solid and proven tools that are the most capable and cost-effective.
As a case in point, we spoke to a lead test automation engineer at a major financial services company:
Our experience with Micro-Focus UFT is that it just doesn’t lend itself quite as well to testing the web stuff. People have tried it in the past before I came along. We’ve got a partnership with a major offshore company, so they were the ones that put the frameworks in place to start with and then I came along as the lead test engineer, so this framework was already in place to start with, and they’d already decided on Selenium…”
As soon as I heard this, I knew there’d been some negative influencers somewhere down the line. For one thing, Micro Focus UFT is the most capable web test automation tool out there. Also, Micro Focus is a Gold Level Selenium partner – UFT is 100% compatible with Selenium and provides a robust framework out-of-the-box.
Insight #2: Key Takeaway
Look out for consultancies who claim it’s better to use open source tools because they are cheaper or perform the job better. In reality, they’re only interested in selling more days. If a tool sounds too good to be true, it generally is. The consultancy will likely build an elaborate framework and you’ll be locked into using their services for longer than you ever expected.
You may save on the tool but, you will spend a lot more on consultancy.
Insight #2: Action Item: Always question the validity of claims. Get second, third and even fourth opinions. Scratch beneath the surface to understand the real reason tools are being positioned.
Insight #3 – Techies Love a Challenge
Your technical test team members love problem-solving. This is a fantastic trait and one that you should bear in mind when consulting your test team about tool choice. I found many cases where tools were chosen because team members “enjoy doing techie stuff.” Rather than because they were faster or more efficient.
A product manager at a major professional services firm told me:
“I think it’s more about what the teams want is what the teams get.”
When I mentioned that some of his functional testing could be turned around faster with a paid tool, his response was one that we commonly hear:
“Whenever you start talking about investing in tools, the first opposition always comes internally from the development team. Like, why can’t you build it? You’re an engineer.”
Insight #3: Key Takeaway
While test teams understand the technology more than the business owners, they may not necessarily be making decisions that are in the interest of the business. You need to understand why the test team are recommending the tool. If the reason is anything other than saving time and money, then it is likely to be the wrong tool.
Insight #3: Action Item: Drill into why your team is recommending a certain tool. Ask them why? And ask what alternatives they have considered.
Insight #4 – People Often Put Their CV Before Your Business Needs
Your team may be recommending a certain product because it will look good on their CV, and not because it is the right solution for your business.
A senior test manager at a major insurance company puts it like this:
“From a career point of view, if you are good at using these open source tools, I personally believe you have better prospects, better opportunities. Let’s say I’m telling you that I’m good in UFT, I’m good in whatever paid tools. Then you look at the job market. If you want to switch for a higher salary or for whatever reason, and then you will find there are let’s say 10 jobs. But if you have developed skills using open source, and then you search again, I can be certain that instead of 10 jobs for the paid tools, you will find 20, 30, 40 jobs for open source tools, you know, you have more options.”
Insight #4: Key Takeaway
Test professionals will often make decisions for themselves rather than focus on what is right for their current employer. That is human nature. So, if they perceive that something will enhance their self-worth, don’t be surprised if they suggest it.
Insight #4: Action Item: Factor in your team’s recommendations, but also the needs of the project. Make sure those recommendations achieve the project/business goals over the individual goals.
Insight #5: You Can (and Should) Try Tools Before You Buy
During these interviews, I noticed a common ‘blind spot’; most interviewees didn’t realise that free trials are available for paid test tools.
A test manager at a leading betting and gaming company shared that:
“One of the big attractions, certainly for businesses is that open source tools are free. Most of the places I’ve worked at are cost sensitive, so that’s something positive. And it’s good for experimentation because you’re not committed long term.”
I went on to ask what he thought was better, paid tools or open source tools.
“I explored quite a few vendors recently to explore potential gaps in our test process, and the cost can be prohibitive. It can be hard sometimes to pull that sort of business case together without getting hands on and being able to try it, which is not necessarily an option you have with some of the paid tools.”
Insight #5: Key Takeaway
You can try most paid tools before you buy. This option might not be immediately clear. If you need help arranging a free trial, drop us a note and we can help
Insight #5: Action Item: Adopt a “try before you buy” approach. That way, you can perform a fair assessment of all potential options.
Insight #6 – Test Tool Migration Takes a Lot of Effort
A major factor that affects tool choices is the procurement process. This is especially true in the public sector. Several interviewees mentioned that they must put out a tender for a new solution every four years. If a new tool is chosen, they must scrap everything and start again.
Procurement is, of course, laser-focused on minimising spend. However, they’re often concerned with the purchase costs and pay little or no attention to the incurred effort and migration cost.
A test manager at a public sector company told me:
“We can’t ever buy something forever. We can only buy it for a period of time because it’s got to go through a fair and open evaluation. So as an example, when we took on Atlassian a couple of years ago, we were basically told, yep, you can have Atlassian for four years. And at the end of that, you need to go back to a fair and open evaluation….
The surprise during this conversation was that ‘people cost’ is not considered. The cost to migrate to a new tool is not factored in, only the direct spend.
…And if Atlassian was not at the top of the list, then you’ve got the whole migration piece. Now it’s slightly circumvented when you use open source. So JMeter is open source. It’s not really an issue as there’s no public money required. But when you have a licensed product you’re paying for it, and at a certain point you need to reevaluate, is that worth the public’s purse? And the only way to reevaluate that is to do another fair and open back to market. So, it’s quite limiting. I think this experience is going be very different compared to people you speak to in the private sector because they can just buy a license and stay with it for 12 years”.
Insight #6: Key Takeaway
For the most part, procurement are not IT professionals. They try to make the best decision they can, based on the information they have. If you can provide them with better information, they can often make more accurate and cost-effective long-term decisions.
Insight #6: Action Item: Talk with procurement and explain the total cost of ownership and cost of migration.
Insight #7: Choose a Contract Length to Suit Your Needs
A major attraction of open source tools is that you can use them for as little, or as long, as you want. After all, why would you want to be tied to a long-term contract?
What a lot of people don’t know is that, nowadays, paid tools are available on flexible, short-term contracts.
I asked a test manager at a leading betting company what his preference was:
“With open source tools, you don’t have that long-term commitment. If tech changes quickly, we can try something new. If your business case evolves, you can drop it. You can move on to something else. I’ve been in situations before where we’ve signed long-term agreements and we felt stuck with it for years and it felt like it was the right tool at the time, but actually, things have moved on and you are stuck with this long-term license in many cases, or maybe not stuck with a license, but you so invested in terms of how you’re using it. You just kind of, roll it on and roll it on. And you know, with an open source tool, it’s not quite got that same tie-in”.
This opinion is misguided on a couple of levels. Today, most companies buy annual contracts. There’s no need to commit for any longer than you’re comfortable with.
More importantly, it ignores the effort investment to set up elaborate open-source frameworks. You can’t always just ditch a tool you’ve spent months/years using. Paid tools generally require much less effort to set up – they’re pretty much ready to go out-of-the-box.
Insight #7: Key Takeaway
Once you’ve found the right tool you only need to commit for the length of time you need the tools for. Despite what many people think, it can be harder to switch between open source tools – they often require much greater migration effort. Paid tools incur an upfront or ongoing cost, but they are usually much easier to get up and running with substantially less time spent.
Insight #7: Action Item:
If you only want the tools for a short project, ask your tools vendor for a project-based test tools price. Compare that to the configuration cost of your open source framework and expert resources.
Open Source vs Paid? Our Conclusion
During our interviews, we found so many unverified assumptions, hearsay or examples that are years out of date. It was quite an eye-opening experience. Not what we were expecting.
I’ve always known that the lobby for open source is louder than the lobby for paid tools, but some of the beliefs held by our interviewees were quite staggering.
In summary, when it comes to choosing a test tool, my biggest single piece of advice is this: scratch beneath the surface. Find out the true picture (and cost) before setting off down a path that might not lead to the right destination. Ask why recommendations were made. Understand the TCO, not just the upfront cost.
Here’s a Case in Point
Recently, I was talking to a performance test manager who was unhappy with the tool he was using (JMeter).
Although JMeter is ‘free’, it involved complex scripting, long set-up times, ongoing hardware costs, the internal cost to support JMeter, and an arduous process to analyse the results.
I arranged a demo of a paid tool, LoadRunner Cloud. He loved it, however, when he heard the price, £40K, he said he would struggle to get budget approval.
Not one to be put off easily, he wanted to see if there was a business case for LoadRunner Cloud. With that in mind, we explored the relative cost of the tools. In particular, we scratched beneath the surface to find out what his current free tool was really costing.
When we looked at the effort comparison, he worked out LoadRunner Cloud would save him a staggering 61 person weeks of effort. That’s a full-time employee for well over a year (with holidays factored in).
When we looked at hardware costs, we found an additional £10k saving. LoadRunner Cloud includes all the load generation infrastructure he needed, no need to secure any additional hardware.
Armed with this information, he was easily able to secure the £40k budget required for the paid tool.
In Summary
We hope you have found these insights useful. Hopefully, they have armed you with questions, and things to consider the next time you’re choosing a test tool.
