The Retry Behaviour is for individual steps of a conversation, and it will repeat exactly the one step that is failing.
For instance, occasionally an IBM Watson or Google Dialogflow endpoint is not responsive, for whatever reason - might be a cloud issue or a networking issue or something else - and in this case there is a failure message like “ECONNTIMEOUT” shown. You can configure the regular expression here to exactly this ECONNTIMEOUT to make Botium just repeat the call to IBM Watson or Google Dialogflow.
This does not work in all cases, it depends on several things that are not under the influence of Botium - like implementation details of the chatbot - but for catching endpoint timeout issues it usually works well.
I have two different errors that I want the complete test to retry.
- We get a response that tells us there was a timeout
- We get an empty response.
If we get any failures, I would like the repeat the whole test case. It looks like you can set that up in the connection, but I do not understand the correct way to right the regex response.
Currently there is no way of repeating a failed test case automatically.
In the test results view, there is the option to retry the failed test cases, but it has to be triggered manually right now.
@Florian So do I understand correctly that there is no way to automatically repeat a step where the bot failed to reply within the given amount of time (error waiting for bot- bot did not respond with in 10 secs)? Based on our observations, these cases frequently self-heal on the next run, so we really don’t want them to trigger sending alerts.
Well, retrying a step is possible, but it can break the test. (If you order a pizza, and got timeout as response, then in most cases it means that you ordered the pizza, but the answer is coming slow)
Best way to deal with this problem is to increase the timeout.
To retry just a step, you can use RETRY_USERSAYS_ONERROR_REGEXP capability.