Expected & Actual Responses are not displayed in Consolidated JSON results for Many testcases

Our customer has requested for JSON results to validate the responses and consume them further.
Below are the challenges faced while downloading the Consolidated JSON test results from Botium

  1. For Passed test cases - Only “User Says” utterance is present in the JSON, Expected and Actual responses from the BOT appear in Botium report UI and not in the JSON file.
    [Mandatory requirement]

  2. Test Cases with multiple Assertions - Only the first assertion is displayed in the JSON file
    [Medium Priority]

  3. Cards & Buttons Response - If the test case fails and the actual response comes with Card/Buttons it is not available in JSON report
    [Medium Priority]

All these details are available in the JSON pop-up inside Botium, however not available in the exportable results

1 Like

@ujja is working on it

Thank you very much, pls let us know once done… Thanks

How do you downloaded the test results @Sameerah? I suggest downloading using TestResults -> TestResult -> Downloads -> Download full transcript (JSON)

{
    "results": [
        {
            "errDetails": [
                {
                    "cause": {
                        "expected": [
                            "qwerty Here are some buttons"
                        ],
                        "actual": "Here are some buttons"
                    },
                    "message": "2168 Multiple/Line 6: Bot response (on Line 3: #me - buttons) \"Here are some buttons\" expected to match \"qwerty Here are some buttons\""
                },
                {
                    "cause": {
                        "expected": [
                            "qwerty",
                            "Second Button"
                        ],
                        "actual": [
                            "First Button",
                            "Second Button"
                        ]
                    },
                    "message": "Line 6: Expected button(s) with text \"qwerty\""
                }
            ],
            "steps": [
                {
                    "step": 0,
                    "expected": {
                        "sender": "me",
                        "messageText": "buttons"
                    },
                    "not": false,
                    "actual": {
                        "sender": "me",
                        "messageText": "buttons"
                    }
                },
                {
                    "step": 1,
                    "expected": {
                        "sender": "bot",
                        "messageText": "qwerty Here are some buttons",
                        "asserters": [
                            {
                                "name": "BUTTONS",
                                "args": [
                                    "qwerty",
                                    "Second Button"
                                ]
                            }
                        ]
                    },
                    "errDetails": [
                        {
                            "cause": {
                                "expected": [
                                    "qwerty Here are some buttons"
                                ],
                                "actual": "Here are some buttons"
                            },
                            "message": "2168 Multiple/Line 6: Bot response (on Line 3: #me - buttons) \"Here are some buttons\" expected to match \"qwerty Here are some buttons\""
                        },
                        {
                            "cause": {
                                "not": false,
                                "expected": [
                                    "qwerty",
                                    "Second Button"
                                ],
                                "actual": [
                                    "First Button",
                                    "Second Button"
                                ],
                                "diff": [
                                    "qwerty"
                                ]
                            },
                            "message": "Line 6: Expected button(s) with text \"qwerty\""
                        }
                    ]
                }
            ]
        }
    ]
}

Example download (I removed some not related fields)

  1. See results.steps[1] for info about the response
  2. See results.steps[1] for all data extracted from from response and results.steps[1].errDetails for all assertation errors occured in current response
  3. See results.steps[1] for all cards/buttons extracted from from response and results.steps[1].errDetails for cards/buttons assertation errors occured in current response

We are unable to download both full transcript JSON & CSV/EXCEL report. It gives us 404 response code!.
uploaded the screenshot for reference.

You will receive the latest development version with the new JSON fields in the next few hours.

We have enabled the transcript feature with the new JSON fields in your instance.

Thank you @Florian and @ujja. We are now able to download the transcript JSON file.

Below are the differences that we see, can you please suggest an optimal method to extract the expected result

Transcript file
- We are able to now see the expected and actual responses for Passed & Failed cases
- However the expected response is only provided as the Utterance file name and not the exact phrase of expected response
eg.
“step”: 1,
“expected”: {​​
“sender”: “bot”,
“channel”: null,
“not”: false,
“optional”: false,
“messageText”: “ODA_SKILL_PERFORMANCE_INTENT_SUMMARYTOUCHPOINTS_BOT_1”,

[On contrary, for a failed case we are able to see the “Expected Response” as such]
Can you please enable the exact expected response for a Pass as well as requested earlier

Hello,
For failed test cases it is important to see in the transcript the expectations that where not met that made a test case fail. We do not persist much information about the conversation steps that actually worked as expected. Providing this information would be a client-specific customization we would have to deploy (pls contact by email if you consider this).
Florian