Skip to content

User Testing Workflows

Jump to bottom
Eberhard Beilharz edited this page Jan 29, 2024 · 20 revisions

Contents

Introduction

The keyman repository has a bot, @keymanapp-test-bot, which watches all pull requests and issues, and tracks user testing requirements for the pull request.

Watch the videos:

Any time a pull request is opened, edited, or a comment added, the bot reviews the body of the pull request (the initial comment), and all subsequent comments in order to determine the current state of user testing for the pull request.

A developer will add a # User Testing section, in a specific parseable format, anywhere in any comment in the pull request, with a set of tests and instructions for those tests.

A tester will follow the instructions for the tests and report on the result of each test in subsequent comments. Tests can be run multiple times; the final test result reported determines the state of the test.

The bot will collect the set of tests from the # User Testing section, and review subsequent comments in the pull request to determine the state of each test along with test notes. It will create and maintain a single # User Test Results comment which reflects the current state of the user testing in the pull request.

The bot will add or remove the following two labels as required:

  • user-test-required: The pull request has tests that need to be completed. Once each test has a result, this label is removed. Any time a test is re-opened, the label is automatically re-added.

  • user-test-missing: The pull request does not yet have any tests defined.

Alongside this, the bot will also issue a check on the most recent commit in the pull request reflecting the state of user testing.

It is also possible to issue bot commands to control the behavior of the bot.

Note: the workflow for issues is almost identical to that for pull requests, except for two things: (a) the bot will not label or add comments to issues that have no # User Testing comment, and (b) there is no concept of checks for an issue. The rest of this document will talk about pull requests, because that is where we expect the majority of user testing to take place.

Playground

To experiment with user testing workflows, use https://github.com/keymanapp/keymanapp-test-bot-playground

Hint: press . on that page to open the GitHub Visual Studio Code web editor for the repository, where you can make new branches and pull requests; see the status bar in the editor to create branches.

Example

This is a short example of a single test and test result.

Test specification:

# User Testing

* **TEST_BUTTON_IS_ROUND**: make sure that the button is round, not square.

  In the main screen, look at the button and make sure it is now a pleasing
  round shape, instead of the previously grevious square shape.

Test run:

* **TEST_BUTTON_IS_ROUND**: **PASSED** I found the button to be pleasingly round

  It was so much nicer on my eyes without the sharp 90 degree corners.

Test Properties

A test can have the following properties:

  • An identifier, e.g. TEST_BUTTON_IS_ROUND. The identifier must start with TEST_.
  • An optional short description, e.g. make sure that the button is round, not square.
  • An optional longer set of instructions, e.g. In the main screen, look at the button and make sure it is now a pleasing round shape, instead of the previously grevious square shape.
  • A state, managed by the bot, one of:
    • OPEN: the test has not yet been run (or is awaiting re-test)
    • PASSED: the test passed
    • FAILED: the test failed
    • BLOCKED: the tester was not able to run the test due to external factors
    • UNKNOWN: a test state that we probably shouldn't use
    • SKIPPED: the test failed but the result was ignored because of an unrelated bug

A test can also be part of a group and/or a suite -- see the suites and groups section.

Each test run has the following properties:

  • The identifier of the test.
  • The state, one of the above states, e.g. PASSED
  • An optional one line summary, e.g. I found the button to be pleasingly round
  • An optional longer set of notes, e.g. It was so much nicer on my eyes without the sharp 90 degree corners.

Instructions for Developers

Requesting user tests

Each pull request should have a section titled # User Testing. This can be placed in the initial comment or in any subsequent comment, but must come before any actual test results. It is fine to edit this section and add, remove or modify tests, and the bot will cope.

  • Hint: If you modify a test specification which has already been tested, you will need to ask the tester to rerun that test; it may be better to rename the test in this situation to avoid confusion.

  • Hint: if you want to note that the specification is not yet ready for consumption, you could throw an emoji into the # User Testing title: # 🚧 User Testing, which will cause the parser to ignore the whole user testing section.

A test should be formatted as follows:

TEST_ID: description

instructions
  • The TEST_ID should should be a unique identifier for this test, using characters A-Z, 0-9, _, - and ., and starting with TEST_.
  • The description should be a single line description of the test.
  • If necessary, additional instructions can be included on subsequent lines. If the instructions are long, consider collapsing them with a detail/summary section: <details><summary>Steps</summary> detailed steps</details>.
  • Each test must start on a new line.
  • All tests must be defined in a single comment.
  • Each test will have an initial state of OPEN.
  • You may choose to format test declarations with bullets (* or -) and bold ** markers to make them easier to read, for example: 
    * **TEST_ONE:** Open the widget

  The widget can be found on the app home page
  • Don't use - [ ] checkbox styling - only bullets and bold markers will be recognised.
  • See also Test Suites and Groups for specifying more complex sets of tests.

Retesting

To request that any or all tests be re-run (e.g. you pushed a new commit that impacts the test), you can add a control command which the bot will process:

@keymanapp-test-bot retest [SUITE_ID] [GROUP_ID] [all|TEST_ID...] [, [SUITE_ID] [GROUP_ID] [all|TEST_ID...]]

A comment can have other content as well as the control command, but the command should be on a line on its own.

If no parameter is passed to the retest command, or all is passed, then all tests will be re-opened and marked as OPEN.

(You could also re-open a test by using TEST_ID: OPEN but it is clearer to use the bot command, and the bot command may do other cool things in the future).

If you use a TEST_ID which is repeated in other suites or groups, it will currently only mark the first test that matches as OPEN. This may be improved in the future!

If you include SUITE_ID or GROUP_ID but do not specify a TEST_ID it will mark all tests within that suite or group for retesting.

Skipping user tests

If a pull request does not require user testing, then make a control command to the bot:

@keymanapp-test-bot skip

A comment can have other content as well as the control command, but the command should be on a line on its own.

An individual test can be skipped by setting the state SKIPPED.

Instructions for testers

  • Links to pull requests and issues that require testing:
  • Each test result must start on a new line.
  • You can record multiple test results in a single comment, and you can record test results across multiple comments.
  • Test results should be formatted as follows: 
    TEST_ID: STATE summary

notes
  • The state can be OPEN, PASSED, FAILED, SKIPPED or BLOCKED.
  • The optional summary should be a one line note about how the test went
  • The optional notes content can contain additional detail about the test, including images, etc. You can choose to collapse the content if you wish using the <details><summary>Notes</summary>details...</details> pattern.
  • Remember that later comments override earlier ones, so if you test twice, just add the second result in a later comment -- don't edit your earlier one.
  • You may choose to format test results with bullets (* or -) and bold ** markers to make them easier to read, for example: 
    * **TEST_ONE: PASSED** This went really well

  The widget was excellent
  • See also Test Suites and Groups for reporting more complex sets of tests.

Test Suites and Groups

For complex sets of tests, it may be useful to group or repeat them. We have two kinds of grouping: suites and groups.

  • A test suite is a set of tests. Test results will show the tests within the suite along with the overall result of the test suite. A suite is declared with the prefix SUITE_. All tests following a suite declaration belong within that suite.

  • A test group is a way of repeating tests: define the test only once but specify that it may be run in multiple contexts. The bot will expand the test results out across each of the groups. A group is declared with the prefix GROUP_. Groups are commonly used within suites. All tests following group declarations will be repeated across each of the groups within the suite (or the entire specification, if no suite is declared).

Specifying suites and groups

Each suite and group declaration must be on its own line. The declaration may optionally be prefixed with a dot point bullet (- or *) or a header hash (#, ##, ###, etc). You can optionally add a description after the name of the suite or group, and additional instructions about the suite or group on subsequent lines.

## SUITE_ID: description

instructions
* GROUP_ID: description

  instructions

Reporting test results for suites and groups

If suites/groups are defined in the test specification, then when reporting test results, you must first declare the SUITE_ and GROUP_ to which the test run belongs, each declaration on its own line before the TEST_ report. Again, you can prefix the line with dot point bullets or header hashes. Any text following the declaration is ignored.

Test Suite and Group Example

This example defines two suites: SUITE_UX and SUITE_PERF. Within the first suite, there are three groups (GROUP_WIN, GROUP_MAC, GROUP_LINUX), meaning each test should be run three times, once for each group. The second suite has just two groups and a single test -- which should be run once for each group.

Test specification:

# User Testing

## SUITE_UX: user experience

Test how things look and feel to the end user.

* GROUP_WIN: Windows / Chrome browser
* GROUP_MAC: macOS / Firefox browser
* GROUP_LINUX: Linux / Chrome browser

* **TEST_BUTTON_IS_ROUND**: make sure that the button is round, not square.
* **TEST_LINK_IS_RED**: make sure that links are now red instead of blue.

## SUITE_PERF: performance

Here we are going to be checking out how zippy things are.

* GROUP_DESKTOP: Windows / Chrome browser
* GROUP_MOBILE: Android / Chrome browser

* **TEST_LOAD_TIME**: make sure that the page loads within 400msec

Test run:

## SUITE_UX
### GROUP_WIN
* TEST_BUTTON_IS_ROUND: pass Nice round buttons now!
* TEST_LINK_IS_RED: fail The link was green, yeeuch.

### GROUP_MAC
* TEST_BUTTON_IS_ROUND: pass Yep, pleasingly round
* TEST_LINK_IS_RED: fail The link was brown. How does that even happen?

### GROUP_LINUX
* TEST_BUTTON_IS_ROUND: pass So, so round
* TEST_LINK_IS_RED: pass Yep, it's as red as red can be

## SUITE_PERF
### GROUP_DESKTOP
* TEST_LOAD_TIME: pass I got 124msec

### GROUP_MOBILE
* TEST_LOAD_TIME: fail It was 401msec. Sorry dude.

status.keyman.com

status.keyman.com shows the current status of user testing for pull requests with a colored border:

  • red - failed or unspecified tests
  • yellow - pending tests (no failures)
  • green - all passed or no testing required

If you hover over a PR, then details of the user testing status are shown within the popup that appears.

Clone this wiki locally