Is there a recommended strategy for testing multi-language bots in Botium?

Botium Box
1

Question:

Is there a recommended strategy for testing multi-language bots in Botium? I mean how to structure
tests, where to store them, which Botium features to use, etc. So far I found in wiki a possibility to
use Google translate service, but we will probably not be able to use it.

If the conversation structure is the same in all of the tested languages, and the differences are in the multi-language utterances only, the suggestion from Botium Team is to split up the test sets into multiple test sets:

  • one dedicated test set for each language holding the utterance lists only (and maybe language-specific test cases)
  • one shared test set holding the convos and empty utterance lists
  • for each language a dedicated test project

Then, in the language-specific test sets, select the shared test set as dependency to combine those test sets (convos and utterance lists) on test execution. The empty utterance lists in the shared test set will be overwritten by the language-specific utterance lists.

image
image1567×660 111 KB

When using Excel files or Google Sheets, you can choose an even easier structure:

  • one dedicated test set for each language linking to the Google Sheet or uploaded the Excel file
  • The utterance lists are on language-specific worksheets which are configured in the Test Set settings for each language

image
image1565×756 128 KB

1 Like
2

Hi Florian,
Thanks for providing recommendations.
But could you please elaborate more on options for Excel/Google sheets?

  • Do I understand correctly you mean to have one Excel/Google sheet with 1 shared worksheet for common convos + language specific sheets with utterances? Language specific Test sets in Botium then connect common convos with one of language utterances to create language specific Botium project?
  • Language specific convos must be probably separated in additional Excel/Google sheet and separate Botium Test Sets?
  • Can we somehow force JSON_PATH asserters to test parameter values which differ per language? Or difference in asserter means that such test cannot be shared in common convos?
    Regards,
    Jiri
3

Do I understand correctly you mean to have one Excel/Google sheet with 1 shared worksheet for common convos + language specific sheets with utterances? Language specific Test sets in Botium then connect common convos with one of language utterances to create language specific Botium project?

Exactly. You can tell Botium what sheets to consider. Take an Excel with sheets like this:

  • Convos Generic for language-agnostic conversation structure
  • Utterances_EN for english utterance lists
  • Utterances_DE for german utterance lists

Link this sheet to two test sets:

  • My Tests EN is configured to use worksheet Convos Generic for convos and Utterances_EN* for utterances
  • My Tests DE is configured to use worksheet Convos Generic for convos and Utterances_DE* for utterances

I think the concept is clear now ?

Language specific convos must be probably separated in additional Excel/Google sheet and separate Botium Test Sets?

Add another sheet named Convos EN and add it it the sheet name list for My Tests EN to consider them for english test set, in addition to the language-agnostic convos.

Can we somehow force JSON_PATH asserters to test parameter values which differ per language? Or difference in asserter means that such test cannot be shared in common convos?

Can be done with the Scripting Memory (see Botium Docs). In short, again you have one worksheet per language holding the scripting memory table for each language - you can think of the scripting memory as a list of pre-filled variables that you can use in the convos and asserters:

      |$name |$somethingelse
german|Hans  |irgendwas

       |$name |$somethingelse
english|John |anything

Place the german scripting memory table on a sheet named Scripting Memory DE, and the english one on Scripting Memory EN, and again configure the worksheet names for the Scripting Memory. You have to enable the Expand Scripting Memory switch on the Scripting Settings as well.

You can now have language-agnostic JSON-Path assertions like this:

JSON_PATH $.session.name | $name

The $name is substituted by the scripting memory variable in action by Botium.

I hope it helps.

4

The concept is clear.
Regarding Scripting Memory files: Does it allow having/checking different value of parameter $name per each convo in Test set?

5

Don’t know what you mean, please give an example.

6

I have Test-1, Test-2 and 2 environments, QA and PROD. Test-1 and Test-2 contain parameter $ConsumerName with values as follows:
Test-1, QA -> John
Test-1, PROD -> Peter
Test-2, QA -> Jane
Test-2, PROD -> Penelope
I need to run Test-1 only once with value John, when running in QA and Peter when running in PROD while Test-2 must be run only with Jane in QA and Penelope in PROD.

7

Wouldn’t it be possible to use two variables for this instead of just one ?

$Consumer1 => John (QA) and Peter (PROD)
$Consumer2 => Jane (QA) and Penelope (PROD)

8

Do you mean to have ‘test case/convo’-specific parameters, not only values? We have test sets with 100+ convos, then we’d theoretically have 100+ parameters. Is it ok?

How to resolve different values for the same test in different environments (QA,PROD) - using scripting memory files/sheets? And in combination with multiple languages (Lang-1,Lang-2) we would have test sets/projects combining sheets:

  • Convos+Utterances(for Lang-1)+ScriptingMemory(for Lang-1,Env-QA)
  • Convos+Utterances(for Lang-1)+ScriptingMemory(for Lang-1,Env-PROD)
  • Convos+Utterances(for Lang-2)+ScriptingMemory(for Lang-2,Env-QA)
  • Convos+Utterances(for Lang-2)+ScriptingMemory(for Lang-2,Env-PROD)?
9

hm, that’s tricky question. I would say: yes, the more descriptive way is to use separate parameters for each of the values you want to assign to it in the different convos.

Separating QA and PROD test sets can be done as you suggested.

If you are not fully satisfied with this structure, can you make a suggestion for what you think would be the optimal solution for you, regarding effort, maintenance, simplicity ? We are always listing to input from our users.