[HELP] How to use Botium

I need to understand if Botium could be a solution for hour needs.
We need to monitor a bot by the following Touch Points:

1. Textual interface:

  • Web Portal chat
  • Mobile Portal chat
  • App chat
  • WhatsApp
  • Deep link in sms messages

2. Voice interface

  • The purpose of the monitoring is to measure and report the bot performance over time, ideally 7 days a week and 24 hours a day (24*7).
  • Measures have to be performed invoking the Digital Assistant via Smartphone or Desktop
  • For each Touch Point, a given set of dialogues has to be run sequentially and cyclically.
  • Results have to be collected and aggregated in electronic format - an excel file - daily. For each test, the file shall include the following fields: Touch Point type, dialogue identifier, the question text, bot answer whatever it is, time stamp of text execution and of bot response time, test result (i.e. success/failure, where failure means that the test did not end normally).
  • Dialogues are constituted of a single question and a reply for the Textual channel, while for Voice cannel they include multiple questions and answers executed in predefined patterns.
  • Actual Dialogues are preceded by preliminary steps used to activate the channel, i.e. to launch the App or call the number, and advance through the channel till the predefined point where the Dialogue can start.

Is this possible with botium? Is it possible a demo (in italian if possible)?

Thanks in advance
Paolo Zanetti

1 Like

Hello Paolo,
In short, Botium can do everything you are asking for, but maybe not 100% the way you requested.

First of all, with Botium you can do multi-channel-testing on several levels and with several test types. Botium can do tests on API level - with chatbots developed with IBM Watson, Dialogflow, Bot Framework and all other relevant technologies, see list of supported technologies in Botium Wiki. You can do conversational testing, NLP testing with Botium Coach, performance testing, security testing. We recently built a connector based on Twilio (alpha state) to simulate phone calls to an IVR system and are currently working on SMS integration as well (expected to be part of Botium Box by Q1/2021).

Finding the matching test mix for your use case is typically the first part of a Botium project.

A monitoring system can be set up with Botium Box either by using the integrated scheduler and email notification mechanism, or by integration with a CI/CD pipeline. Test results can be exported as Excel, JSON, CSV, Jira, PDF (customizable).

Regarding testing the conversational flow of a chatbot, everything you mentioned can be done with BotiumScript, the core functionality of Botium.

Botium itself is language agnostic, so you can write tests in all languages (certain components like TTS/STT have restrictions though). No one in our team is speaking italian, I am sorry - we can do demos in english, german, hungarian, and I think also in spanish … Please use the “Book a Demo” button on our website!

I hope this answered your question.


1 Like

Thanks a lot Florian.
Just a question: we could have the professional or the enterprise solution?



Sure, both available as SaaS, enterprise available on-premise as well

  1. The first question is how Botium interfaces with the chatbot. Does the chatbot engine have to be
    present in your network or does Botium behave like an end user and access the url where the chatbot
    is present?

It depends on the technology and the test level. If you are testing a Dialogflow bot on API level, then Botium connects to the Dialogflow API running on the Google servers. When testing an Alexa Skill, Botium is sending simulated audio streams to the Alexa backend services running on the Amazon servers. If you are testing a chatbot hosted in a website widget end-2-end, then Botium uses Selenium to load the website and simulate the user interaction.

Naturally, network connectivity from the server running Botium Box to the chatbot server has to be given - again this depends on the network topology, the chatbot technology and the Botium Box deployment (Saas vs On-Premise). We can help and make suggestions if required.

  1. To perform tests on local devices, what should be expected

Building a local device farm from scratch is not a piece of cake. Honestly, I have never done this myself, but I assume you will have to purchase the devices and attach them to a USB hub. Then you have to install and configure a Selenium Grid to use those local devices. Finally, the Selenium Grid endpoint can be used in Botium to simulate user interaction on those devices.

If the Selenium Grid is online, you have to enable network connectivity from Botium Box to the Selenium Endpoint. Again, this depends on the network topology, and we can help with suggestions.

  1. I would like to understand better how IVR is tested
  • Phone number where the IVR is running is called Botium
  • Botium is using Text-To-Speech to synthesize speech and simulate a user talking
  • Botium receives the audio stream back from the IVR and uses Speech-To-Text to compare with the test case transcripts
  • There can be multiple conversation steps, as well as DTMF input
  • When the test case transcript is over, the call is ended.

Thanks for your answers Florian
I have another question on whatsapp.
I understand that a test with whatsapp is possible only by API but I don’t think it’s acceptable for our customer. Is it possible to perform a scripted test on mobile devices to reproduce the end user experience?

Sure, if you have devices with SIM card attached where Whatsapp is available - although I never tried this myself, there is no reason why this shouldn’t work with our Botium Webdriver Connector.

Testing on the mobile device itself should be part of on overall test strategy, but as these kind of tests are running near real-time only a small subset of the whole test suite should be selected for it.

Sorry, Florian, about the IVR test, how does Botium call the IVR number? By the mobile device o by a gsm telephone dialer?

Botium is using Twilio - https://github.com/codeforequity-at/botium-connector-twilio-ivr

If I’m not wrong, Twilio excute call from a landline number. Our need is to execute a call from a mobile number: is it possible?

You are wrong - on Twilio you can purchase landline as well as mobile numbers

If we don’t use local device, what do we use? Can we use emulator too?

Not sure if it is possible to launch Whatsapp on an Emulator - I think you need a valid SIM card for it - this is definitly out of scope for Botium.

Hi Florian,
is it possible the step 3 of the following scenario?

  1. Botium call the IVR by Twilio and execute the test
  2. IVR, when start the call, send an sms that will be received by Twilio
  3. Botium execute a click on the link inside the sms


clicking hyperlinks is possible with the Hyperlink Asserter.

But this test case is currently out-of-scope for Botium, as it is not really a conversation flow as seen by Botium - IVR is a different messaging channel than SMS, and currently there is no way in Botium to do this switch within one test case.

Ok, is it possible to export the test of the sms outside of Botium?
We could externally parse the sms to check the link

As with Botium the calls are done with Twilio, you will also receive the SMS response on the same number as used in Twilio, I assume. So you can have your own component listing the SMS responses from Twilio.

From what I read in the Twilio docs, it should actually be possible to receive SMS and use Voice at the same time with the Twilio Programmable Voice/SMS API. So the overall test case could be handled by extending the Botium Twilio Voice connector.