Hello,
I need to understand if Botium could be a solution for hour needs.
We need to monitor a bot by the following Touch Points:
1. Textual interface:
Web Portal chat
Mobile Portal chat
App chat
WhatsApp
Deep link in sms messages
2. Voice interface
The purpose of the monitoring is to measure and report the bot performance over time, ideally 7 days a week and 24 hours a day (24*7).
Measures have to be performed invoking the Digital Assistant via Smartphone or Desktop
For each Touch Point, a given set of dialogues has to be run sequentially and cyclically.
Results have to be collected and aggregated in electronic format - an excel file - daily. For each test, the file shall include the following fields: Touch Point type, dialogue identifier, the question text, bot answer whatever it is, time stamp of text execution and of bot response time, test result (i.e. success/failure, where failure means that the test did not end normally).
Dialogues are constituted of a single question and a reply for the Textual channel, while for Voice cannel they include multiple questions and answers executed in predefined patterns.
Actual Dialogues are preceded by preliminary steps used to activate the channel, i.e. to launch the App or call the number, and advance through the channel till the predefined point where the Dialogue can start.
Is this possible with botium? Is it possible a demo (in italian if possible)?
Hello Paolo,
In short, Botium can do everything you are asking for, but maybe not 100% the way you requested.
First of all, with Botium you can do multi-channel-testing on several levels and with several test types. Botium can do tests on API level - with chatbots developed with IBM Watson, Dialogflow, Bot Framework and all other relevant technologies, see list of supported technologies in Botium Wiki. You can do conversational testing, NLP testing with Botium Coach, performance testing, security testing. We recently built a connector based on Twilio (alpha state) to simulate phone calls to an IVR system and are currently working on SMS integration as well (expected to be part of Botium Box by Q1/2021).
Finding the matching test mix for your use case is typically the first part of a Botium project.
A monitoring system can be set up with Botium Box either by using the integrated scheduler and email notification mechanism, or by integration with a CI/CD pipeline. Test results can be exported as Excel, JSON, CSV, Jira, PDF (customizable).
Regarding testing the conversational flow of a chatbot, everything you mentioned can be done with BotiumScript, the core functionality of Botium.
Botium itself is language agnostic, so you can write tests in all languages (certain components like TTS/STT have restrictions though). No one in our team is speaking italian, I am sorry - we can do demos in english, german, hungarian, and I think also in spanish … Please use the “Book a Demo” button on our website!
The first question is how Botium interfaces with the chatbot. Does the chatbot engine have to be
present in your network or does Botium behave like an end user and access the url where the chatbot
is present?
It depends on the technology and the test level. If you are testing a Dialogflow bot on API level, then Botium connects to the Dialogflow API running on the Google servers. When testing an Alexa Skill, Botium is sending simulated audio streams to the Alexa backend services running on the Amazon servers. If you are testing a chatbot hosted in a website widget end-2-end, then Botium uses Selenium to load the website and simulate the user interaction.
Naturally, network connectivity from the server running Botium Box to the chatbot server has to be given - again this depends on the network topology, the chatbot technology and the Botium Box deployment (Saas vs On-Premise). We can help and make suggestions if required.
To perform tests on local devices, what should be expected
Building a local device farm from scratch is not a piece of cake. Honestly, I have never done this myself, but I assume you will have to purchase the devices and attach them to a USB hub. Then you have to install and configure a Selenium Grid to use those local devices. Finally, the Selenium Grid endpoint can be used in Botium to simulate user interaction on those devices.
If the Selenium Grid is online, you have to enable network connectivity from Botium Box to the Selenium Endpoint. Again, this depends on the network topology, and we can help with suggestions.
I would like to understand better how IVR is tested
Phone number where the IVR is running is called Botium
Botium is using Text-To-Speech to synthesize speech and simulate a user talking
Botium receives the audio stream back from the IVR and uses Speech-To-Text to compare with the test case transcripts
There can be multiple conversation steps, as well as DTMF input
When the test case transcript is over, the call is ended.
Thanks for your answers Florian
I have another question on whatsapp.
I understand that a test with whatsapp is possible only by API but I don’t think it’s acceptable for our customer. Is it possible to perform a scripted test on mobile devices to reproduce the end user experience?
Sure, if you have devices with SIM card attached where Whatsapp is available - although I never tried this myself, there is no reason why this shouldn’t work with our Botium Webdriver Connector.
Testing on the mobile device itself should be part of on overall test strategy, but as these kind of tests are running near real-time only a small subset of the whole test suite should be selected for it.
But this test case is currently out-of-scope for Botium, as it is not really a conversation flow as seen by Botium - IVR is a different messaging channel than SMS, and currently there is no way in Botium to do this switch within one test case.
As with Botium the calls are done with Twilio, you will also receive the SMS response on the same number as used in Twilio, I assume. So you can have your own component listing the SMS responses from Twilio.
From what I read in the Twilio docs, it should actually be possible to receive SMS and use Voice at the same time with the Twilio Programmable Voice/SMS API. So the overall test case could be handled by extending the Botium Twilio Voice connector.