The world is abuzz with conversations about artificial intelligence (AI), and it’s fair to say that opinion is split. On one side, alarmists warn of a dystopian, Terminator-like future — where AI takes over the world, and humans are made obsolete. Conversely, optimistic technophiles are excited about working less while AI labours on our behalf.
As is often the case, the truth likely falls somewhere in the middle. However, whatever you think may happen, change is the one thing we can all be sure of.
As you’re undoubtedly aware, the primary AI model at the centre of current AI discussions is the ChatGPT chatbot — it’s talked about everywhere and anywhere, so much so that you’ll be hard-pressed to find someone in our industry who doesn’t have an opinion on it.
With this in mind, we will explore AI’s potential implications and applications within quality assurance and examine how AI and Chatbots like ChatGPT will impact software testers.
What is ChatGPT?
Before we go any further, I want to ensure we’re all on the same page. You probably have a rough idea of what AI is, but I want to take a few lines to explain what ChatGPT is and what it isn’t.
ChatGPT is a language model developed by OpenAI. It’s an iteration of the Generative Pretrained Transformer (GPT), a technology that uses machine learning techniques to generate human-like text. ChatGPT learns from vast amounts of data, finding patterns and predicting the next word in a sentence, allowing it to create coherent and contextually relevant responses.
ChatGPT does not possess consciousness or understanding. ChatGPT cannot form beliefs or desires. It cannot access or understand real-time information beyond its last training update (around two years ago at the time of writing).
ChatGPT simply mimics human-like text based on the patterns it has learned.
It’s important to note that ChatGPT isn’t the only AI chatbot out there, but it is by far the most well-known. There’s also Bard, HuggingChat, Bing AI (Based on the same GPT model as ChatGPT), and many others.
For a bit of fun and to demonstrate some of what ChatGPT is capable of, I asked ChatGPT to compare LoadRunner Cloud, JMeter and NeoLoad – in the form of a rap battle!
How Are AI Chatbots Shaking Things Up?
The disruptive nature of AI chatbots like ChatGPT stems from their ability to process natural language inputs and generate human-like text responses. This advancement in natural language processing (NLP) is transforming various industries.
In an interview with the BBC in May 2023, AI Emad Mostaque, the founder and CEO of Stability AI, stated, “the impact on our lives of AI will be greater than the impact of the pandemic, Google & Facebook.”
ChatGPT already powers sophisticated customer support chatbots and virtual assistants, answering queries quickly and efficiently. It is also aiding the creation of diverse forms of written content.
In the same interview mentioned above, Greg Jackon, CEO of Octopus Energy, said that Octopus is now using AI to help answer emails.
He stated that AI answers 34% of their customer emails and that “AI is doing the work equivalent of 250 people“. Not only that, but Greg also stated that AI-generated emails have a higher customer satisfaction rating than human-generated emails. Of course, this higher satisfaction may be due to several factors, including the context of the emails.
AI Chatbots Are Only Getting Started
To be honest, these current use cases are barely scratching the surface of ChatGPT’s potential. Specifically, when it comes to testing, software development teams are only starting to explore the potential commercial applications of ChatGPT. ChatGPT and other AI solutions will only increase in importance.
It is not an overstatement to say that ChatGPT is the most exciting technological innovation of recent years. I have absolutely no doubt that ChatGPT is the start of a genuinely transformative revolution in how we all work.
How will Chatbots and AI impact testers?
For software testers, ChatGPT currently represents both a tool and a challenge. We’ve listed just a few of the pros and cons below:
Script Generation (a Pro)
A few testers I speak with have already used ChatGPT to generate first drafts of LoadRunner scripts, albeit with mixed results. While it offers a potentially useful starting point, ChatGPT openly admits its limitations.
Just this morning, I asked ChatGPT whether it can generate a LoadRunner script. It said it could, but it told me, “Please remember that GPT is not perfect, and the code it generates should always be checked for accuracy and appropriateness in your specific use case.”
I then asked ChatGPT, “Can you help me write a LoadRunner script that sends an HTTP request to “http://example.com” and then checks if the response code is 200?”
Here’s what it came back with:
Effort Reduction (a Pro)
Testers also use AI to accelerate testing and reduce effort. Whilst automation can increase efficiency, you still need to check the accuracy. However, it is beginning to allow testers to focus on more complex and creative tasks — a win-win.
As mentioned above, current applications are only the beginning of what we can expect as ChatGPT and AI continue to evolve at a breakneck pace.
AI Interface Into Test Management Tools (a Pro)
An even more fascinating application could be an AI interface to test management tools like ALM Octane & ValueEdge Quality. Imagine asking an AI interface questions about the state of testing and receiving comprehensive, data-backed responses — Well, good news! This functionality is being developed for these tools and will be available before long. I was privileged enough to see a pre-release demo, and it’s impressive.
However, it’s not all sunshine and roses — AI and ChatGPT present real-world challenges with potentially significant impacts.
Can You Trust It? (a Con)
My biggest issue is trust. That is, how can I trust anything generated by ChatGPT? As far as I can see, you can only verify anything it creates by manually checking the outputs.
Other Collected Cons
Aside from this fundamental issue, other challenges presented by ChatGPT include the following:
- Data privacy and security concerns due to the vast amounts of data chatbots utilise.
- Chatbots lack domain-specific knowledge and expertise in testing and the application under test — As far as I know, no chatbots have ISTQB certification.
- Chatbots do not understand business risks and the potential impacts of defective software.
- Chatbots are based on training data that invariably contain biases. Chatbots can repeat these biases in their answers – potentially impacting the objectivity and reliability of the suggestions provided during software testing.
- Plus, integrating ChatGPT into existing workflows could present technical difficulties.
That being said, ChatGPT is still a valuable resource for software testers, offering general guidance, brainstorming ideas, or helping with common testing challenges.
ChatGPT is Not a Substitute for Expertise.
AI chatbots can help testers take shortcuts and accelerate some of our more tedious activities. But you must supplement their use with expertise, industry best practices, and robust testing methodologies.
You must be clear on the questions you ask, the inputs you give, and the outputs you need. Without this clarity, you could quickly end up going down a blind alley or ending up with incorrect information – leading to wasted effort or, worse, ineffective testing.
Why you should be excited about ChatGPT
ChatGPT presents a chance to rethink and redefine what software testing looks like daily. It has the potential to automate mundane and repetitive tasks – and let’s be honest, testers experience quite a bit of monotonous work.
This, in turn, will enable testers to focus on more strategic, complex, and creative work, such as designing complex testing strategies or uncovering unusual bugs that require out-of-the-box thinking.
Moreover, introducing AI-powered testing could encourage skill development and career advancement. Testers are quick learners, and they can quickly develop knowledge of AI implementation and developing expertise in AI testing tools, adding more depth to their roles.
Why You Might Be Worried About ChatGPT – And What You Can Do About It?
AI’s introduction into any industry invites concerns, and software testing is no exception. There are questions about job displacement due to automation, ethical considerations of using AI, and others.
Here are a few reasons why you might be worried about AI chatbots and potential mitigation strategies:
| Concerns | Potential Impact | Mitigation Strategies |
| Job Security | Automation can efficiently perform repetitive, low-level testing tasks. As AI advances, this could mean fewer roles for manual testers, creating insecurity about job stability. | Testers should learn about AI and automation, even specialising in these areas. By understanding and utilising these technologies, testers can remain a critical part of the testing process. |
| Poor Quality | AI’s potential inability to catch all bugs could lead to the approval of flawed software products and tarnish the tester’s reputation for thoroughness. These bugs include those that require human intuition, understanding of complex business rules, or contextual understanding, | Testers who leverage AI for low-level testing can shift their focus to exploratory, usability, and scenario-based testing. AI currently has limitations in these areas, and human expertise is vital. |
| How Can You Test AI-based systems? | Due to their non-deterministic and highly complex nature, AI systems could pose challenges for testers. A lack of understanding could lead to missed defects. | Testers must invest time and effort in learning about AI and machine learning. This will improve their ability to test AI systems and understand their outcomes effectively. |
| Over-reliance on AI | Excessive reliance on AI for testing might mean testers overlook important human-centric aspects of software usability, potentially leading to a decline in software quality and user satisfaction. | Testers must maintain a balance between AI-powered and human-led testing — Prioritise areas of testing where human insight is critical, like usability and accessibility. |
Generally, it’s important to remember that while AI can automate specific tasks, it is not a substitute for human intellect, creativity, and critical thinking. If you’re worried about ChatGPT and AI, even if you aren’t, you should focus on expanding your skills to work alongside AI, not against it. Learn to evolve with the technology, to manage and control it effectively.
Summary
The arrival of AI and further integration into our lives signifies a significant milestone in human evolution and the evolution of software testing. It brings tremendous potential for improving software testing efficiency, consistency and depth, allowing testers to engage with more complex tasks.
However, as we navigate this new landscape, we must balance the thrill of innovation with the acknowledgement of valid concerns. As the AI wave swells, we must all adapt and evolve to survive and thrive in the AI-driven testing era.
Remember – Chatbot Tools are Handy Tools, not Robust Solutions
ChatGPT and other chatbots are fantastic tools but no substitute for knowledge and experience. You need to be very specific about the problem and precisely what you need to solve that problem. Chatbots can shortcut tasks, including scripting, but you have to know precisely what you want the script to do – and you must review everything you get back.
Only time will tell whether we are heading towards a dystopian or utopian future. For now, we can focus on harnessing the power of AI, learning to integrate it into our workflows effectively, and utilising it to augment our capabilities.
After all, the one certainty we have is that AI is here, and it’s reshaping our world.
