Hi, I wrote a simple convo and configured the Twilio connector to call my phone. I then answered the call and played the bot part. Here is what I found out:
Botium is slow to respond. It takes quite a few seconds after the “bot” finishes its utterance for Botium to utter its part.
There is some kind of timer that expires at 60 seconds no matter what, so the test always ends at that point.
I attempted to enclose the supporting documentation, but it contains a lot of links and it seems that I’m only allowed 2 links per post. Sent them to @Florian via e-mail, please let me know if you didn’t get them.
Yes, this connector is slow. This is due to the way the Twilio API works - with all the Text-To-Speech and Speech-to-Text involved, it always takes several seconds for a conversation turn to complete. There are IVR systems out there that will then just hang up. We know about this problem, but right now there is no other way than configuring the IVR system under test to accept longer waiting times.
By default, a Botium test has to complete within one minute, I suppose that this is what you are experiencing. Using the –timeout switch to extend it to 2-3 minutes would help in this case.
Thanks, that’s good to know. We’ll think of a different approach for the future, like perhaps streaming the audio to the connector (or rather proxy), generating our own ASR events, and using our own TTS.
Meanwhile, I’m trying to get an extremely simple test working: just call in and go through the dialog. That doesn’t seem possible though, because bot’s first prompt is delayed a few seconds (there are a few seconds of silence after the call connects - a problem with our setup), and then the bot starts uttering a very long initial prompt that is meant to be interrupted. If I wait for the “bot says” event, I run past bot’s time limit on waiting for human input. So I decided to just insert a long pause, and send a “user says” event unconditionally a few seconds after the call connects. I tried the following:
#me
PAUSE 10000
BUTTON 1
and
#begin
PAUSE 10000
#me
BUTTON 1
In both cases, there is no delay and the DTMF sounds immediately after connection.
Is there a way to reliably delay a user utterance or make sending it relative to the leading edge of bot’s utterance rather than the trailing edge?
Are there any examples of anyone successfully using the Twilio connector (or any telephony connector)?
We are also not happy with the current situation. We are thinking about some improvements (streaming to ASR engine directly for example).
The Twilio API and Botium don’t really allow to do the thing you are trying to do. The only thing supported is a simple back-and-forth conversation without any forced interruptions.
We know about some users doing smoke tests with it, and we implemented it ourselves with a couple of our clients. But the whole topic hasnt got so much traction yet for us to invest more time into it till now.