tests: twister: using statemachine for status #80623

hakehuang · 2024-10-30T11:48:38Z

using statuemachine in status change.
in stastus.py we can generated the statuemachine
by get_graph

The purposes of this PR:

use statuemachine transition to verify the status change in twister runtime. if there is a violate, TransitionNotAllowed exception will be triggered, and call context will be print.
the transition is defined in the statues.py with class TwisterStatusMachine_inner.cycles, you can generate the status.png by run python3 ./statues.py
with this, we can see and control the status flow to avoid mis-use.
enable call stack print if a exception is triggered, in error.py
fix some status error found by this.

hakehuang · 2024-10-30T11:50:29Z

with this, we can observe the support test state transition as above

golowanow · 2024-10-31T10:15:02Z

with this, we can observe the support test state transition as above

is there already differentiation on status transitions for Harness, TestCase, TestSuite, and TestInstance ?

hakehuang · 2024-10-31T13:08:14Z

is there already differentiation on status transitions for Harness, TestCase, TestSuite, and TestInstance ?

No, but I need clean the ci failures. BTW do you happen to know why this change does not get execute?
https://github.com/zephyrproject-rtos/zephyr/actions/runs/11608284181/workflow?pr=80623#L103

nashif · 2024-10-31T13:49:01Z

is there already differentiation on status transitions for Harness, TestCase, TestSuite, and TestInstance ?

No, but I need clean the ci failures. BTW do you happen to know why this change does not get execute? zephyrproject-rtos/zephyr/actions/runs/11608284181/workflow?pr=80623#L103

because we are running in pull_request_target mode.

hakehuang · 2024-10-31T13:55:49Z

because we are running in pull_request_target mode.

then how can I change this? @nashif

scripts/pylib/twister/twisterlib/statuses.py

PerMac · 2024-11-14T09:50:29Z

@hakehuang to elaborate: pull_request_target means that the source is taken from the target pranch, not the PR. So your change has to be already in the tree to take place, which afaik doesn't allow to test the change itself within the PR. But you can split the workflow change from the rest and marge it to your github fork and then see if it works. Providing a link to a PR on a fork with the workflow change included and working could help with the review.
I think we also have repo made for testing https://github.com/zephyrproject-rtos/zephyr-testing but I never used it. We can discuss how to proceed such changes during testing wg.

PerMac · 2024-11-14T15:24:46Z

To reviewers: this PR requires manual testing before being approved

PerMac · 2024-11-14T15:26:14Z

@hakehuang Please provide more details.

hakehuang · 2024-11-14T16:31:06Z

is there already differentiation on status transitions for Harness, TestCase, TestSuite, and TestInstance ?

@golowanow , from my observation, the Harness, TestCase, TestSuite and TestInstance transtions can follow one rule set. do you see any exceptions?

hakehuang · 2024-11-14T16:37:49Z

Please provide more details.

The purposes of this PR:

use statuemachine transition to verify the status change in twister runtime. if there is a violate, TransitionNotAllowed exception will be triggered, and call context will be print.
the transition is defined in the statues.py with class TwisterStatusMachine_inner.cycles, you can generate the status.png by run python3 ./statues.py
with this, we can see and control the status flow to avoid mis-use.
enable call stack print if a exception is triggered, in error.py
fix some status error found by this.

golowanow · 2024-11-14T18:31:37Z

is there already differentiation on status transitions for Harness, TestCase, TestSuite, and TestInstance ?

@golowanow , from my observation, the Harness, TestCase, TestSuite and TestInstance transtions can follow one rule set. do you see any exceptions?

well, potentially they can be in the same space, but according to these comments, there is significant differences in transitions, e.g. only some states/transitions valid for Harness, blocked is TestCase only, etc.

zephyr/scripts/pylib/twister/twisterlib/statuses.py

Lines 54 to 68 in cc42b35

    
           # All statuses below this comment can be used for TestCase 
        
           BLOCK = 'blocked' 
        
           STARTED = 'started' 
        
           # All statuses below this comment can be used for TestSuite 
        
           # All statuses below this comment can be used for TestInstance 
        
           FILTER = 'filtered' 
        
           NOTRUN = 'not run' 
        
           # All statuses below this comment can be used for Harness 
        
           NONE = None 
        
           ERROR = 'error' 
        
           FAIL = 'failed' 
        
           PASS = 'passed' 
        
           SKIP = 'skipped'

hakehuang · 2024-11-15T11:09:10Z

only some states/transitions valid for Harness, blocked is TestCase only, etc.

I see, let me add harness transition

gchwier · 2024-11-15T10:49:37Z

scripts/pylib/twister/twisterlib/error.py

IMO: Exception handling is a good candidate for a separate PR.

IMO: Exception handling is a good candidate for a separate PR.

yeah, #80866

gchwier · 2024-11-15T10:51:41Z

scripts/pylib/twister/twisterlib/error.py


 class TwisterException(Exception):
-    pass
+    def __init__(self, message="TwisterException"):


You pass only a class name in every subclasses, maybe better to add class name to exception message in one place, e.g.:

def __init__(self, message=None): if message is None: message = self.__class__.__name__ else: message = f"{self.__class__.__name__}: {message}" super().__init__(message)

gchwier · 2024-11-15T11:00:40Z

scripts/pylib/twister/twisterlib/error.py

-    pass
+    def __init__(self, message="TwisterException"):
+        super().__init__(message)
+        for line in traceback.format_stack():


you can just call logger.exception(msg) to print the callback, however the traceback is usually more relevant when the exception is caught rather than when it is created,

gchwier · 2024-11-15T11:10:01Z

scripts/pylib/twister/twisterlib/statuses.py

+        except exceptions.TransitionNotAllowed:
+            # here we violate the pre-defined transition rules
+            self.force_state(event)
+            g()


just use

logger.exception(f"Transition from {self.current_state} to {event} is not allowed.")

instead of adding extra g() method

BTW, I received
Transition from passed to passed is not allowed
after running:
./scripts/twister -vv --no-clean -T samples/hello_world -p native_sim

using StateMachine in status change. in stastus.py we can generated the StateMachine by get_graph Signed-off-by: Hake Huang <[email protected]>

add pip install step for twister Signed-off-by: Hake Huang <[email protected]>

add trace print when exception happens Signed-off-by: Hake Huang <[email protected]>

update test cases for twister 1. test_errors.py add protection. 2. test_handlers.py change call to status 3. test_testsuite.py change call to status Signed-off-by: Hake Huang <[email protected]>

zephyrbot added the area: Twister Twister label Oct 30, 2024

zephyrbot requested review from gchwier, golowanow, KamilxPaszkiet, LukaszMrugala, nashif and PerMac October 30, 2024 11:49

zephyrbot assigned nashif Oct 30, 2024

hakehuang marked this pull request as draft October 30, 2024 11:52

hakehuang force-pushed the order_test_status branch 7 times, most recently from 673ece9 to 7c6608d Compare October 31, 2024 08:10

hakehuang force-pushed the order_test_status branch 5 times, most recently from c02927f to 43ea148 Compare November 1, 2024 04:33

hakehuang force-pushed the order_test_status branch 4 times, most recently from eddffa7 to 01598ad Compare November 14, 2024 04:31

pdgendt reviewed Nov 14, 2024

View reviewed changes

scripts/pylib/twister/twisterlib/statuses.py Outdated Show resolved Hide resolved

hakehuang force-pushed the order_test_status branch from 01598ad to 6bed985 Compare November 14, 2024 16:29

hakehuang force-pushed the order_test_status branch 3 times, most recently from d79db2e to cc42b35 Compare November 14, 2024 17:46

gchwier reviewed Nov 15, 2024

View reviewed changes

hakehuang force-pushed the order_test_status branch from cc42b35 to 2166e9f Compare November 15, 2024 13:36

hakehuang added 4 commits November 15, 2024 23:23

tests: twister: using StateMachine for status

9ce01e2

using StateMachine in status change. in stastus.py we can generated the StateMachine by get_graph Signed-off-by: Hake Huang <[email protected]>

ci: twister: add pip install befer twister test

ace1877

add pip install step for twister Signed-off-by: Hake Huang <[email protected]>

twister: add trace print in exception

cf448c0

add trace print when exception happens Signed-off-by: Hake Huang <[email protected]>

twister: test: update test case

14b588e

update test cases for twister 1. test_errors.py add protection. 2. test_handlers.py change call to status 3. test_testsuite.py change call to status Signed-off-by: Hake Huang <[email protected]>

hakehuang force-pushed the order_test_status branch from 2166e9f to 14b588e Compare November 15, 2024 15:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tests: twister: using statemachine for status #80623

tests: twister: using statemachine for status #80623

hakehuang commented Oct 30, 2024 •

edited

Loading

hakehuang commented Oct 30, 2024 •

edited

Loading

golowanow commented Oct 31, 2024

hakehuang commented Oct 31, 2024 •

edited

Loading

nashif commented Oct 31, 2024

hakehuang commented Oct 31, 2024

PerMac commented Nov 14, 2024

PerMac commented Nov 14, 2024

PerMac commented Nov 14, 2024

hakehuang commented Nov 14, 2024

hakehuang commented Nov 14, 2024

golowanow commented Nov 14, 2024 •

edited

Loading

hakehuang commented Nov 15, 2024

gchwier Nov 15, 2024

golowanow Nov 15, 2024

gchwier Nov 15, 2024

gchwier Nov 15, 2024

gchwier Nov 15, 2024

gchwier Nov 15, 2024

tests: twister: using statemachine for status #80623

Are you sure you want to change the base?

tests: twister: using statemachine for status #80623

Conversation

hakehuang commented Oct 30, 2024 • edited Loading

hakehuang commented Oct 30, 2024 • edited Loading

golowanow commented Oct 31, 2024

hakehuang commented Oct 31, 2024 • edited Loading

nashif commented Oct 31, 2024

hakehuang commented Oct 31, 2024

PerMac commented Nov 14, 2024

PerMac commented Nov 14, 2024

PerMac commented Nov 14, 2024

hakehuang commented Nov 14, 2024

hakehuang commented Nov 14, 2024

golowanow commented Nov 14, 2024 • edited Loading

hakehuang commented Nov 15, 2024

gchwier Nov 15, 2024

Choose a reason for hiding this comment

golowanow Nov 15, 2024

Choose a reason for hiding this comment

gchwier Nov 15, 2024

Choose a reason for hiding this comment

gchwier Nov 15, 2024

Choose a reason for hiding this comment

gchwier Nov 15, 2024

Choose a reason for hiding this comment

gchwier Nov 15, 2024

Choose a reason for hiding this comment

hakehuang commented Oct 30, 2024 •

edited

Loading

hakehuang commented Oct 30, 2024 •

edited

Loading

hakehuang commented Oct 31, 2024 •

edited

Loading

golowanow commented Nov 14, 2024 •

edited

Loading