we solve

We can help your business

Request a Free Demo / trial


Insights | From a different perspective | Functional Testing | How to | Mobile Testing | Performance Testing | Test Management Tools
18 September, 2017

7 Mistakes to Avoid When Buying Software to Improve Your Testing

It’s only natural that you have high expectations when you buy software to improve your testing.

You want to improve your ability to test and to make life easier. You may be expecting to get greater traceability, visibility, test coverage, more automation, to save time – or all of these.

However, too frequently, those benefits never pan out.

I’ve seen software bought with good intentions languishing on a shelf within a matter of weeks or months. Companies use their new software, but get even more frustrated than before because it doesn’t do what they want. Projects are delayed and more work is created.

Why does this happen? What makes it so hard to get the benefits that were so tangible when you bought your software?

Today I want to share with you some of most common and avoidable mistakes you can make when buying testing software.

Need help building the business case – see this helpful guide

It is not an easy process. There are many factors to consider, and getting them wrong will potentially cost you tens or hundreds of thousands of pounds.

In my 20+ years in the testing industry, I’ve seen companies make every possible mistake, giving me a good insight into what can go wrong and how to avoid it.

Selecting the right testing software for your business is possible. It just takes some planning – and avoiding the following pitfalls….

1. Unclear Requirements

Don’t even think of buying testing software without having a clear understanding of your requirements – what you really need the software to accomplish, what applications you will need to test today and in the foreseeable future.

Without these details, you can’t make an informed, cost-effective decision, and could end up with software that doesn’t do what you want, and that you regret.

One company that came to me was considering £250k worth of tools based on a recommendation by an IT Consultancy. They had no clear requirements or understanding of what they needed – and what the consultancy had recommended was based on information that was at least 3 years out of date, so not the best solution for them. 

This is why I take people through an in-depth fact find to determine what they’re trying to achieve. Don’t skip this step, and don’t rush through it either. Here are some questions to think about right at the beginning of the process:

  • What testing do you need to do and why?
  • What is the application under test and does it have any unusual characteristics?
  • Who is going to be using the software, and how?
  • Are there a sufficient number of people trained to use the software? If not, where will you get them from?
  • How long do you need the software for?
  • What are your success criteria? What does ‘good’ look like?

This is just the starting point and will lead to other questions. Building a picture of the requirements will lay the foundation for selecting the software and writing the business case.

2. Not getting stakeholder buy-in – all of them

Share your vision for the tools with others with a vested interest, as you may need their support to get the tools embedded and delivering real value to your company.

Have you discussed your plans with your stakeholders to ensure that they understand your vision? Were they excited by the improvements it could make to the testing process?

If not, you may have a challenge getting your plans off the ground – or worse still, people who will actively work against you. It happens!

Ensure that the Head of Development is also included and confirms buy-in. Dare I say that some developers view testing as a delay to deploying their perfect software and consider tools that bridge the void between development and test as something to be avoided.

3. Analysis paralysis

Spending so much time thinking about it, gathering details on alternative solutions or evaluating too many alternatives can lead to the decision never being made.

I have come across situations where months after the initial contact the company is no closer to a decision, as there is always one more tool to evaluate. The cost of this is wasted effort and the opportunity cost of not having tools to use must be huge. 

There are ways to avoid this. Clear requirements and limiting the evaluation to those tools that are truly relevant will go a long way to shortening the process.

4. How many licences do we need?

One of the biggest mistakes I’ve seen involved a major financial institution. They wanted to test their website to 25,000 users. But while they did have 25,000 users per hour, they discovered that website visitors never even went above 2,000 concurrent users – i.e. people using the website at the same time.

They spent over £500k on the software for this test. (Not with me, I hasten to add.)

Understanding the distinction between ‘users per hour’ and ‘concurrent users’ could have saved a mountain of cash…..

Have a clear understanding of the numbers, particularly if the software is based on a per-user model. Understanding “concurrent users” is key, however. In 50% of conversations I find that it is not well understood.

Get this wrong and you will either have:

  • Too few licenses, which can cause project delays and the embarrassment of having to buy more
  • Too Many which is a waste of money – this can be even more embarrassing.

5. Not keeping up with technology

When you decide to buy a new mobile, you check what’s new on the market. Make sure you do the same with your testing software.  

Be careful about using outdated knowledge. Don’t assume that because you once conducted tests with particular software a few years ago, that you know its capabilities today. One of my clients ruled out HPE because they thought it didn’t meet their needs. They based that decision on four-year-old information, which was completely out-of-date.

In the interim, HPE had modernised their software and introduced new rental and SaaS options. They have also introduced a number of new tools for Agile test management, a really great tool for monitoring mobile applications and new function and performance automation tools.

While this industry may not change as quickly as the mobile world, it still evolves. Make sure you have the most recent vendor information and software specifications. You don’t want to miss out on the latest innovations.

6. Free can come with a cost

Budget is always a major concern. However, be careful when considering free software.

The downfalls frequently outweigh the savings. Who will support the software if you find a defect? Will the software be enhanced in the future to support new requirements or advances in technology?

You may discover, too late, that the tool you’ve selected doesn’t offer the analysis capabilities you need, may not integrate into your other software and is generally not fit-for-purpose.

One client came to me after using a performance tool because it was free. However, it didn’t scale beyond 100 users. They needed it for 1,000 users.

If you start with a free or cheap tool, and then have to switch because of a limitation, you may not be able to save or reuse those initial test results.

Manually parsing together results to create the complete picture…. or having to start your testing all over again… or developing a capability your freeware lacks… can be more costly than picking a quality tool from the start. It can also delay your project considerably.

Consider your overall costs before you start, instead of trying to cut corners. This will save you time, money and headaches later.

7. Picking the wrong funding option

Getting your expenditure approved is not always easy.

There are different funding options nowadays, though, which can make your testing software considerably cheaper. Most testing software nowadays can be bought as SaaS, and categorised as operating expenditure rather than capital expenditure.

For many companies this is much easier to get signed off!

Added bonus: SaaS software is also much quicker to deploy because you don’t need to install any hardware, and easier to scale because you are paying-as-you-go.

Your path to success

It is so tempting in today’s fast-paced business world to cut corners, shave timelines and use free or cheap tools to save a few pounds.

Selecting quality testing software will improve your bottom line and simplify your testing. Avoid mistakes by ensuring you:

  • Go through a rigorous process to uncover your requirements
  • Get stakeholder buy-in
  • Obtain current knowledge of solutions available
  • Have a clear plan for what you will evaluate based on the requirements you have built
  • Understand how the solution will work within your organisation
  • Select the most appropriate purchase model
  • Remember that cheap can cost you more long-term

If you need help building the business case, this guide will help you.

How To Turn a No into a Yes

If selecting the right software is something you’d like help with, please do get in touch – we’d be delighted to help!

Stephen Davis
by Stephen Davis

Stephen Davis is the founder of Calleo Software, a Micro Focus (formerly HPE Software) Gold Partner. His passion is to help test professionals improve the efficiency and effectiveness of software testing.

To view Stephen's LinkedIn profile and connect 

Stephen Davis LinkedIn profile

18th September 2017
Using Chatbots for Test Creation

5 ChatGPT Guidelines For Software Testers

Guidelines to help you use chatbots for software testing. The structure you need to leverage AI to enhance, rather than compromise, the quality of your testing and application.

Whats new in digital lab 234

UFT Digital Lab 23.4: 3 Release Highlights

OpenText (previously Micro Focus) has released UFT Digital Lab v23.4. The latest version of their class-leading mobile device lab tool. We’ve selected our top three release highlights.

Public Sector Procurement Challenges

UK Public Sector Procurement: Wise Spending, or Flawed Processes?

Strict public sector procurement processes present a major challenge to uk public bodies. I explore why the public sector continues to make these bad purchasing decisions and how how Calleo can help procurement departments make better informed decisions.

Excel is not a test management tool

Excel: Sabotaging Software Testing Since 1987

In this insight, I dissect Excel’s test management shortcomings – from poor collaboration to the chaos of multiple spreadsheets. I’ll highlight Excel’s inherent risks, the associated effort, and how the absence of centralisation makes streamlined operations impossible.

chaos testing with steadybit

Chaos Testing: How to Integrate Steadybit with LoadRunner Professional

There is often an underlying and unspoken fear with software systems: How resilient is your solution to unexpected outages and system failures? Sure, you’ve tested your solution’s functionality and performance, but have you tested against system outages? This is where chaos testing comes in.

Avoid software license penalties

Avoid License Penalties: Software Compliance in 3 Easy Steps

Software compliance is increasingly important. Major software companies are increasingly auditing customers to check that usage falls within their license contracts, and quite often, it doesn’t. Falling foul of your contract can lead to unexpected and potentially sizeable penalties and disrupt your operations. I want to help you avoid both of these.

grumpy testers

Software Testers: Grumpy Old Dogs?

We’ve all heard it said: “You can’t teach an old dog new tricks.” But that is a complete load of old cobblers, both literally and as a metaphor. It’s spouted by people stuck in their ways, who can’t even be bothered to try and are probably more afraid of failure than anything else.

3 Unexpected Lessons Testers Can Learn from Nature

3 Unexpected Lessons Testers Can Learn from Nature

Nature’s beauty and complexity have always inspired poets and philosophers, and I thought software testers deserved to join that list. Here are three small but important lessons testers can learn from the vast wisdom of the natural world.

the pareto principle

The Pareto Principle: 6 Ways to Test Smarter With The 80-20 Rule

In many ways, software testing is the art of doing more with less. Efficiency is the name of the game. Testing teams are often encouraged to abandon best practices when timings get tight. I will show you 6 ways you can get ahead of the game and test smarter and faster with the 80-20 rule


Related Articles


Sign up to receive the latest, software testing insights, news and to join the Calleo mailing list.

You can, of course, unsubscribe at any time!

By signing up you consent to receiving regular emails from Calleo with updates, tips and ideas on software testing along with the occasional promotion for software testing products. You can, of course, unsubscribe at any time. Click here for the privacy policy.

Sign up to receive the latest, Software Testing Insights, news and to join the Calleo mailing list.

You can, of course, unsubscribe at any time!

By signing up you consent to receiving regular emails from Calleo with updates, tips and ideas on software testing along with the occasional promotion for software testing products. You can, of course, unsubscribe at any time. Click here for the privacy policy.