Bienvenue, Invité
Nom d'utilisateur : Mot de passe : Se souvenir de moi

SUJET : SquashTA does not execute tests in the same order that SquashTM requested

SquashTA does not execute tests in the same order that SquashTM requested il y a 1 an 6 mois #6222

  • pierre
  • Portrait de pierre
  • Hors Ligne
  • Fresh Boarder
  • Messages : 4
hello,

With SquashTM 1.14.2 and SquashTA 1.9, I have a strange comportment: SquashTA does not execute the tests in the order requested by SquashTM.

The testsuite.json file SquahTM generates and provides to Jenkins job uses the same order than the one shown in SquashTM user interface.
But SquashTA does not uses this order.

Here infos picked from Jenkins logfile job:
  • testsuite.json file:
    {"test": [
       {"param":{"TC_REFERENCE":"1-PREQ-ENV-SYST-003"},   "script":"10-Systeme/1-PREQ-ENV-SYST-003.ta",           "id":"1374"},
       {"param":{"TC_REFERENCE":"1-PREQ-ENV-PGSQL-007"},  "script":"11-SGDB/postgreSQL/1-PREQ-ENV-PGSQL-007.ta",  "id":"1375"},
       {"param":{"TC_REFERENCE":"3-CFMT-PGSQL-005"},      "script":"11-SGDB/postgreSQL/3-CFMT-PGSQL-005.ta",      "id":"1376"},
       {"param":{"TC_REFERENCE":"1-PREQ-ENV-SYST-001"},   "script":"10-Systeme/1-PREQ-ENV-SYST-001.ta",           "id":"1377"},
       {"param":{"TC_REFERENCE":"3-CFMT-PGSQL-006"},      "script":"11-SGDB/postgreSQL/3-CFMT-PGSQL-006.ta",      "id":"1378"},
       {"param":{"TC_REFERENCE":"1-PREQ-ENV-SYST-002"},   "script":"10-Systeme/1-PREQ-ENV-SYST-002.ta",           "id":"1379"},
       {"param":{"TC_REFERENCE":"1-PREQ-ENV-ORA-001"},    "script":"11-SGDB/Oracle/1-PREQ-ENV-PGSQL-001.ta",      "id":"1380"},
       {"param":{"TC_REFERENCE":"3-CFMT-SYST-001"},       "script":"10-Systeme/3-CFMT-SYST-001.ta",               "id":"1381"},
       {"param":{"TC_REFERENCE":"1-PREQ-ENV-SYST-005"},   "script":"10-Systeme/1-PREQ-ENV-SYST-005.ta",           "id":"1382"},
       {"param":{"TC_REFERENCE":"1-PREQ-ENV-SYST-008"},   "script":"10-Systeme/1-PREQ-ENV-SYST-008.ta",           "id":"1383"},
       {"param":{"TC_REFERENCE":"1-PREQ-ENV-SYST-006"},   "script":"10-Systeme/1-PREQ-ENV-SYST-006.ta",           "id":"1384"},
       {"param":{"TC_REFERENCE":"3-CFMT-SYST-009"},       "script":"10-Systeme/3-CFMT-SYST-009.ta",               "id":"1385"},
       {"param":{"TC_REFERENCE":"3-CFMT-APA-005"},        "script":"12-Serveur_HTTP/Apache/3-CFMT-APA-005.ta",    "id":"1386"}
    ]}

  • SquashTA uses this order:
    [INFO] Beginning execution of test 1-PREQ-ENV-SYST-003.ta
    [INFO] Beginning execution of test 1-PREQ-ENV-SYST-001.ta
    [INFO] Beginning execution of test 1-PREQ-ENV-SYST-002.ta
    [INFO] Beginning execution of test 3-CFMT-SYST-001.ta
    [INFO] Beginning execution of test 1-PREQ-ENV-SYST-005.ta
    [INFO] Beginning execution of test 1-PREQ-ENV-SYST-008.ta
    [INFO] Beginning execution of test 1-PREQ-ENV-SYST-006.ta
    [INFO] Beginning execution of test 3-CFMT-SYST-009.ta
    [INFO] Beginning execution of test 1-PREQ-ENV-PGSQL-007.ta
    [INFO] Beginning execution of test 3-CFMT-PGSQL-005.ta
    [INFO] Beginning execution of test 3-CFMT-PGSQL-006.ta
    [INFO] Beginning execution of test 1-PREQ-ENV-PGSQL-001.ta
    [INFO] Beginning execution of test 3-CFMT-APA-005.ta
L'administrateur a désactivé l'accès en écriture pour le public.

SquashTA does not execute tests in the same order that SquashTM requested il y a 1 an 6 mois #6231

  • Eric Degenetais
  • Portrait de Eric Degenetais
  • Hors Ligne
  • Junior Boarder
  • Messages : 38
  • Remerciements reçus 5
Hi,
tests are modeled as standalone test cases, without dependencies between them. As such, there is no guarantee of order between individual test scripts.
The ordering rules are as follows:
  1. if you don't use ecosystems (ie: only one level of tests, all under src/squashTA/tests - see documentation):
    1. default ecosystem setup script (named setup.ta)
    2. all remaining tests scripts except for the default ecosystem teardown script
    3. default ecosystem teardown script (teardown.ta)
  2. if you use ecosystems:
  • all tests of a given ecosystem are executed when this ecosystem is executed
  • there is no guarantee on ecosystem execution order
  • each ecosystem is executed in the following order:
    1. ecosystem setup script (setup.ta)
    2. all other scripts except teardown script, in any order
    3. ecosystem teardown script (teardown.ta)
L'administrateur a désactivé l'accès en écriture pour le public.

SquashTA does not execute tests in the same order that SquashTM requested il y a 1 an 6 mois #6256

  • pierre
  • Portrait de pierre
  • Hors Ligne
  • Fresh Boarder
  • Messages : 4
Hello,

Thanks for your detailed reply.

I understand the "standalone test cases" model that explain this out-of-order execution by SquashTA.
However, since test cases can be fully ordered in SquashTM, it sounds somewhat incoherent with SquashTA behaviour, no ?

Do you think that SquashTA comportment might change in future releases, eg use the order of TC provided by SquashTM ?

Best regards
L'administrateur a désactivé l'accès en écriture pour le public.

SquashTA does not execute tests in the same order that SquashTM requested il y a 1 an 5 mois #6270

  • Eric Degenetais
  • Portrait de Eric Degenetais
  • Hors Ligne
  • Junior Boarder
  • Messages : 38
  • Remerciements reçus 5
Hello !
Ordering tests is not planned for now as far as I know.
Please note that, while tests execute without particular order TA side, results are posted back in the relevant TM execution results, which means that seen from Squash TM the final result will keep the on-screen order.
If you are refering to the manual test system in Squash TM, which allows ordering, it is a completely independant system, and therefore does not follow the same rules.

Sorry that I can't tell you more.

Best regards
L'administrateur a désactivé l'accès en écriture pour le public.
Modérateurs: Laurent MASSER
Temps de génération de la page : 0.155 secondes