Our customer has requested for JSON results to validate the responses and consume them further.
Below are the challenges faced while downloading the Consolidated JSON test results from Botium
For Passed test cases - Only “User Says” utterance is present in the JSON, Expected and Actual responses from the BOT appear in Botium report UI and not in the JSON file.
[Mandatory requirement]
Test Cases with multiple Assertions - Only the first assertion is displayed in the JSON file
[Medium Priority]
Cards & Buttons Response - If the test case fails and the actual response comes with Card/Buttons it is not available in JSON report
[Medium Priority]
All these details are available in the JSON pop-up inside Botium, however not available in the exportable results
How do you downloaded the test results @Sameerah? I suggest downloading using TestResults -> TestResult -> Downloads -> Download full transcript (JSON)
{
"results": [
{
"errDetails": [
{
"cause": {
"expected": [
"qwerty Here are some buttons"
],
"actual": "Here are some buttons"
},
"message": "2168 Multiple/Line 6: Bot response (on Line 3: #me - buttons) \"Here are some buttons\" expected to match \"qwerty Here are some buttons\""
},
{
"cause": {
"expected": [
"qwerty",
"Second Button"
],
"actual": [
"First Button",
"Second Button"
]
},
"message": "Line 6: Expected button(s) with text \"qwerty\""
}
],
"steps": [
{
"step": 0,
"expected": {
"sender": "me",
"messageText": "buttons"
},
"not": false,
"actual": {
"sender": "me",
"messageText": "buttons"
}
},
{
"step": 1,
"expected": {
"sender": "bot",
"messageText": "qwerty Here are some buttons",
"asserters": [
{
"name": "BUTTONS",
"args": [
"qwerty",
"Second Button"
]
}
]
},
"errDetails": [
{
"cause": {
"expected": [
"qwerty Here are some buttons"
],
"actual": "Here are some buttons"
},
"message": "2168 Multiple/Line 6: Bot response (on Line 3: #me - buttons) \"Here are some buttons\" expected to match \"qwerty Here are some buttons\""
},
{
"cause": {
"not": false,
"expected": [
"qwerty",
"Second Button"
],
"actual": [
"First Button",
"Second Button"
],
"diff": [
"qwerty"
]
},
"message": "Line 6: Expected button(s) with text \"qwerty\""
}
]
}
]
}
]
}
Example download (I removed some not related fields)
See results.steps[1] for info about the response
See results.steps[1] for all data extracted from from response and results.steps[1].errDetails for all assertation errors occured in current response
See results.steps[1] for all cards/buttons extracted from from response and results.steps[1].errDetails for cards/buttons assertation errors occured in current response
Thank you @Florian and @ujja. We are now able to download the transcript JSON file.
Below are the differences that we see, can you please suggest an optimal method to extract the expected result
Transcript file
- We are able to now see the expected and actual responses for Passed & Failed cases
- However the expected response is only provided as the Utterance file name and not the exact phrase of expected response eg.
“step”: 1,
“expected”: {
“sender”: “bot”,
“channel”: null,
“not”: false,
“optional”: false,
“messageText”: “ODA_SKILL_PERFORMANCE_INTENT_SUMMARYTOUCHPOINTS_BOT_1”,
[On contrary, for a failed case we are able to see the “Expected Response” as such]
Can you please enable the exact expected response for a Pass as well as requested earlier
Hello,
For failed test cases it is important to see in the transcript the expectations that where not met that made a test case fail. We do not persist much information about the conversation steps that actually worked as expected. Providing this information would be a client-specific customization we would have to deploy (pls contact by email if you consider this).
Florian
