we solve

We can help your business

Request a Free Demo / trial


Insights | From a different perspective | Functional Testing | How to | Mobile Testing | Performance Testing | Test Management Tools
18 September, 2017

7 Mistakes to Avoid When Buying Software to Improve Your Testing

It’s only natural that you have high expectations when you buy software to improve your testing.

You want to improve your ability to test and to make life easier. You may be expecting to get greater traceability, visibility, test coverage, more automation, to save time – or all of these.

However, too frequently, those benefits never pan out.

I’ve seen software bought with good intentions languishing on a shelf within a matter of weeks or months. Companies use their new software, but get even more frustrated than before because it doesn’t do what they want. Projects are delayed and more work is created.

Why does this happen? What makes it so hard to get the benefits that were so tangible when you bought your software?

Today I want to share with you some of most common and avoidable mistakes you can make when buying testing software.

Need help building the business case – see this helpful guide

It is not an easy process. There are many factors to consider, and getting them wrong will potentially cost you tens or hundreds of thousands of pounds.

In my 20+ years in the testing industry, I’ve seen companies make every possible mistake, giving me a good insight into what can go wrong and how to avoid it.

Selecting the right testing software for your business is possible. It just takes some planning – and avoiding the following pitfalls….

1. Unclear Requirements

Don’t even think of buying testing software without having a clear understanding of your requirements – what you really need the software to accomplish, what applications you will need to test today and in the foreseeable future.

Without these details, you can’t make an informed, cost-effective decision, and could end up with software that doesn’t do what you want, and that you regret.

One company that came to me was considering £250k worth of tools based on a recommendation by an IT Consultancy. They had no clear requirements or understanding of what they needed – and what the consultancy had recommended was based on information that was at least 3 years out of date, so not the best solution for them. 

This is why I take people through an in-depth fact find to determine what they’re trying to achieve. Don’t skip this step, and don’t rush through it either. Here are some questions to think about right at the beginning of the process:

  • What testing do you need to do and why?
  • What is the application under test and does it have any unusual characteristics?
  • Who is going to be using the software, and how?
  • Are there a sufficient number of people trained to use the software? If not, where will you get them from?
  • How long do you need the software for?
  • What are your success criteria? What does ‘good’ look like?

This is just the starting point and will lead to other questions. Building a picture of the requirements will lay the foundation for selecting the software and writing the business case.

2. Not getting stakeholder buy-in – all of them

Share your vision for the tools with others with a vested interest, as you may need their support to get the tools embedded and delivering real value to your company.

Have you discussed your plans with your stakeholders to ensure that they understand your vision? Were they excited by the improvements it could make to the testing process?

If not, you may have a challenge getting your plans off the ground – or worse still, people who will actively work against you. It happens!

Ensure that the Head of Development is also included and confirms buy-in. Dare I say that some developers view testing as a delay to deploying their perfect software and consider tools that bridge the void between development and test as something to be avoided.

3. Analysis paralysis

Spending so much time thinking about it, gathering details on alternative solutions or evaluating too many alternatives can lead to the decision never being made.

I have come across situations where months after the initial contact the company is no closer to a decision, as there is always one more tool to evaluate. The cost of this is wasted effort and the opportunity cost of not having tools to use must be huge. 

There are ways to avoid this. Clear requirements and limiting the evaluation to those tools that are truly relevant will go a long way to shortening the process.

4. How many licences do we need?

One of the biggest mistakes I’ve seen involved a major financial institution. They wanted to test their website to 25,000 users. But while they did have 25,000 users per hour, they discovered that website visitors never even went above 2,000 concurrent users – i.e. people using the website at the same time.

They spent over £500k on the software for this test. (Not with me, I hasten to add.)

Understanding the distinction between ‘users per hour’ and ‘concurrent users’ could have saved a mountain of cash…..

Have a clear understanding of the numbers, particularly if the software is based on a per-user model. Understanding “concurrent users” is key, however. In 50% of conversations I find that it is not well understood.

Get this wrong and you will either have:

  • Too few licenses, which can cause project delays and the embarrassment of having to buy more
  • Too Many which is a waste of money – this can be even more embarrassing.

5. Not keeping up with technology

When you decide to buy a new mobile, you check what’s new on the market. Make sure you do the same with your testing software.  

Be careful about using outdated knowledge. Don’t assume that because you once conducted tests with particular software a few years ago, that you know its capabilities today. One of my clients ruled out HPE because they thought it didn’t meet their needs. They based that decision on four-year-old information, which was completely out-of-date.

In the interim, HPE had modernised their software and introduced new rental and SaaS options. They have also introduced a number of new tools for Agile test management, a really great tool for monitoring mobile applications and new function and performance automation tools.

While this industry may not change as quickly as the mobile world, it still evolves. Make sure you have the most recent vendor information and software specifications. You don’t want to miss out on the latest innovations.

6. Free can come with a cost

Budget is always a major concern. However, be careful when considering free software.

The downfalls frequently outweigh the savings. Who will support the software if you find a defect? Will the software be enhanced in the future to support new requirements or advances in technology?

You may discover, too late, that the tool you’ve selected doesn’t offer the analysis capabilities you need, may not integrate into your other software and is generally not fit-for-purpose.

One client came to me after using a performance tool because it was free. However, it didn’t scale beyond 100 users. They needed it for 1,000 users.

If you start with a free or cheap tool, and then have to switch because of a limitation, you may not be able to save or reuse those initial test results.

Manually parsing together results to create the complete picture…. or having to start your testing all over again… or developing a capability your freeware lacks… can be more costly than picking a quality tool from the start. It can also delay your project considerably.

Consider your overall costs before you start, instead of trying to cut corners. This will save you time, money and headaches later.

7. Picking the wrong funding option

Getting your expenditure approved is not always easy.

There are different funding options nowadays, though, which can make your testing software considerably cheaper. Most testing software nowadays can be bought as SaaS, and categorised as operating expenditure rather than capital expenditure.

For many companies this is much easier to get signed off!

Added bonus: SaaS software is also much quicker to deploy because you don’t need to install any hardware, and easier to scale because you are paying-as-you-go.

Your path to success

It is so tempting in today’s fast-paced business world to cut corners, shave timelines and use free or cheap tools to save a few pounds.

Selecting quality testing software will improve your bottom line and simplify your testing. Avoid mistakes by ensuring you:

  • Go through a rigorous process to uncover your requirements
  • Get stakeholder buy-in
  • Obtain current knowledge of solutions available
  • Have a clear plan for what you will evaluate based on the requirements you have built
  • Understand how the solution will work within your organisation
  • Select the most appropriate purchase model
  • Remember that cheap can cost you more long-term

If you need help building the business case, this guide will help you.

How To Turn a No into a Yes

If selecting the right software is something you’d like help with, please do get in touch – we’d be delighted to help!

Stephen Davis
by Stephen Davis

Stephen Davis is the founder of Calleo Software, a OpenText (formerly Micro Focus) Gold Partner. His passion is to help test professionals improve the efficiency and effectiveness of software testing.

To view Stephen's LinkedIn profile and connect 

Stephen Davis LinkedIn profile

18th September 2017
cut testing costs

Cut Software Testing Costs With These Essential Tactics

Balancing rigorous testing with cost-effectiveness is a challenge. You can test too much, take too much time, and spend too much money on testing. Luckily, there are proven ways to refine your testing practices, increase efficiency, and cut your software testing costs.

performance testing is like herding cats

Herding Cats: Performance Testing Strategies for 2024

Performance testing is more challenging than ever. Technology is more complex, customer expectations are higher, and an increasing number of people and departments need to be involved to make performance testing effective. This article shines a light on strategies to help you execute successful performance test cycles in 2024.

skimping on performance test tools

Why Skimping on Performance Tools Is a False Economy

Performance tool cost is likely to be less than 10% of the performance testing budget, yet many still think that slashing this figure saves money. Common sense tells us that if you do not spend on quality tools, you will spend more on people. In this insight, I look at why investment in professional performance test tools is so important.

The Human Cost of Software Failure

Horizon: The Human Cost of Software  Failure

The Post Office Horizon scandal destroyed lives and livelihoods and is rooted in defective software. This insight looks at the scandal from a testing perspective and serves as a cautionary tale for all involved in deploying large-scale IT systems.

LoadRunner Cloud Migration

Remote Testing with LoadRunner Might Be Breaching Your Contract

Did you know that most LoadRunner Professional licenses are only valid for a specific location? This means that (unless you’ve bought the country or worldwide usage add-on) working from home, other offices or geographic locations is prohibited and doing so could incur financial penalties.

Software Testing Steps For an Easier Life

Simplify Software Testing With These 3 Easy Steps

Software test professionals like us are under constant pressure to get more done in less time. These three software testing steps will make your life easier, help you achieve your goals, and allow you to focus on your own progression.

2023 top testing articles

The Most Popular Testing Articles of 2023

in 2023 we produced 50+ testing insights to help software professionals test better than ever before and consider testing from a different perspective – today we look back at the 3 most popular of these insights.

rudolph the red nosed tester

Rudolph the Red-Nosed Tester

It was December 23rd at the North Pole, and Santa’s software team were enjoying their Christmas go-live party. They’d pushed the new CRM (Christmas Resource Management) system over the line just in time, and they were very pleased with themselves about it. Everyone apart from Rudolph, the tester.



Related Articles


To get other software testing insights, like this, direct to you inbox join the Calleo mailing list.

You can, of course, unsubscribe 

at any time!

By signing up you consent to receiving regular emails from Calleo with updates, tips and ideas on software testing along with the occasional promotion for software testing products. You can, of course, unsubscribe at any time. Click here for the privacy policy.

Sign up to receive the latest, Software Testing Insights, news and to join the Calleo mailing list.

You can, of course, unsubscribe at any time!

By signing up you consent to receiving regular emails from Calleo with updates, tips and ideas on software testing along with the occasional promotion for software testing products. You can, of course, unsubscribe at any time. Click here for the privacy policy.