You can manage software testing in Excel, but more often that not, it quickly becomes a pain in the behind. Recently, I’ve been wondering: does the same apply to AI Chatbots?
After all, there are solid reasons why businesses invest in professional tools, rather than using general-purpose desktop solutions. Are there similar justifications for purpose-built testing AI tools?
I think there are. Or at least there are a few considerations worth discussing…
Note: For this discussion, I’ll assume your general-purpose chatbot meets your security and data protection requirements. That lets us focus on how well it supports your testing and what it costs to use.
How Much Does Your AI Know About Your Application?
In a similar way to Excel, general-purpose AI tools such as ChatGPT and Claude can help with all manner of tasks, including testing-related activities. They can potentially suggest test scenarios, explain an error message or produce an initial automation script.
But the quality of those outputs depends, at least partly, on the input you provide. To produce something genuinely useful, the AI may need requirements, relevant code, existing tests and details of your environment.
Someone must supply that information, keep it current and check whether the output reflects it correctly. For an occasional task, this can be manageable. Across a large testing operation, the effort involved deserves closer attention.
Generating Tests Is Part of a Wider Process
Software testing isn’t just a case of running scripts. There’s requirements management, risk prioritisation, script and coverage maintenance, failure investigation and all manner of reporting and triage activities. Your AI chatbot needs to fit into that process and help drive overall quality.
How well does a general-purpose chatbot do this, compared to a purpose-built testing AI tool?
For instance, consider what happens after a chatbot suggests a useful test case. How much manual work is involved in adding it into your testing tool? Then linking it to a requirement, checking it against existing coverage, assigning it for review, etc?
A purpose-built testing AI tool, on the other hand, could sit within your test products and support these activities within your existing workflows, using relevant project information. This integration with testing and DevOps workflows could make its output easier to use and maintain.
AI Usage Costs Can Add Up
Working with a general-purpose chatbot can involve considerable back-and-forth. That can quickly eat up testing effort and increase costs. Preparing prompts, transferring information, reviewing suggestions and fixing generated scripts aren’t trivial activities, and all contribute to the cost of completing the task.
Plus, where AI is billed by token usage, supplying requirements, code, logs and previous results contributes to token consumption. Asking for revisions, correcting assumptions and repeating unsuccessful tasks incurs additional usage.
Then, as usage expands into automated workflows, those costs can accumulate. A process that repeatedly sends large amounts of context or requires several attempts to produce an acceptable result may become expensive at scale.
Should You Choose a Purpose-Built AI Testing Tool?
Let’s recap the above:
- General-purpose chatbots need relevant application and project context.
- Moving their outputs into testing workflows can require considerable manual effort.
- Preparing prompts, reviewing outputs and correcting mistakes takes time.
- Token costs can accumulate as usage grows.
Given all this, should you choose a purpose-built testing AI tool instead of your preferred, general-purpose chatbot?
Well, potentially.
The case is very strong when it can integrate with your testing and DevOps tools, use relevant project information and produce outputs your team can review, trace and maintain.
Increasingly, our customers are turning to OpenText’s DevOps Aviator for this sort of solution.
Obviously, there’s a cost benefit calculation to factor in, but that’s the same with a general-purpose chatbot anyway.
What Do You Think?
For me, the question boils down to whether the improvement in testing outcomes justifies the time, effort and money involved. Basically, the same considerations businesses weighed up before moving away from Excel-based testing.
And I don’t see many looking back and regretting that decision.
But what do you think? Does a purpose-built testing AI tool do enough to justify the trade-offs?
