Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add top guides section to a11y intro + a11y content updates #6059

Open
wants to merge 11 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/accessibility/configuration/_category_.json
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
{
"label": "Configuration",
"position": 30
"position": 40
}
65 changes: 43 additions & 22 deletions docs/accessibility/core-concepts/accessibility-score.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -9,15 +9,17 @@ sidebar_position: 30

<AccessibilityAddon />

Cypress produces a top-line percentage score that indicates a general sense of accessibility for a project. This overall score for a run is based on the **average score of** **all application snapshots** that were captured during the run, and is rooted in the ratio of passing and failing accessibility checks performed on each snapshot, weighted according to the severity of the issues.&#x20;
Cypress provides a top-line percentage score to give a general sense of accessibility for your project. This score, calculated for a run, reflects the **average score** of **all application snapshots** captured during the run. It is based on the ratio of passing versus failing accessibility checks for each snapshot, with results weighted by the severity of the issues.

The severity levels presented in Cypress Cloud match the Axe Core® `impact` level for each rule. This impact classification was developed by the creators of Axe, Deque Systems, to aid in prioritization of accessibility results, by highlighting rules that are likely to reveal the most substantial barriers (such as missing label text) so they can be addressed first.
## Severity levels

It's important to note that the Web Content Accessibility Guidelines (WCAG) standards do not have a concept of relative severity of the Success Criteria. For **compliance** purposes, all failures of the Success Criteria count and must be addressed. For **usability** purposes, however, the impact values provided by Axe Core® provide excellent guidance about which issues to take up first to bring the most benefit to disabled users of your platform as quickly as possible.
Severity levels in Cypress Accessibility align with the Axe Core® impact classification. Developed by Deque Systems, this classification helps prioritize accessibility issues by highlighting those likely to create the most substantial barriers for users (such as missing label text) so they can be addressed first. You can learn more about the Axe Core® impact classification in the [Axe Core® documentation](https://github.com/dequelabs/axe-core/blob/develop/doc/issue_impact.md).

## Algorithm
_Note: WCAG standards do not categorize Success Criteria by severity; all failures require resolution for compliance. However, Axe Core® impact levels are helpful for prioritizing fixes to maximize usability improvements quickly._

The following weights are applied to the accessibility report data:
## Scoring algorithm

The accessibility score calculation applies the following weights:

| Severity of Issue | Weight |
| ----------------- | ------ |
Expand All @@ -26,34 +28,53 @@ The following weights are applied to the accessibility report data:
| Moderate | 3 |
| Minor | 1 |

In order to create scores for each snapshot, View, and the entire run, we use the following process:
### Score Calculation Process

1. **Snapshot Score:**

- Calculate the total weight of passed checks.
- Calculate the total weight of failed checks.
- Compute the score as: `Score = (Passed Weights) / (Passed Weights + Failed Weights)`

2. **View Score:**

- Average the scores of all snapshots within the View.

3. **Run Score:**

- For each Snapshot
- Sum the weights of each passed check
- Sum the weights of each failed check
- Score equals ratio of passed weight over total weight
- [Sum of weights from passed checks] / [Sum of weights from passed checks + Sum of weights from failed checks]
- For each View
- Score equals the average of all Snapshot scores found in the View
- For the Run
- Score equals the average of all Snapshots in the Run
- Average the scores of all snapshots in the run.

## Interpreting your scores

Whether a score is good or bad for you depends on context. There is no single number that can tell you the accessibility of your application. A 99% score may be good for your goals if the score was 90% before and you are making progress, or it might be bad if you previously had no problems.
Accessibility scores should be viewed in context. There is no single number that can tell you the accessibility of your application. For example:

Even if 1000 checks pass, if there is 1 check that fails, that may happen to have a huge impact on accessibility. For example, if your login page is not accessible, disabled users may not be able to enter your application at all, even if after the login step all the scores are 100%. So even a single issue requires human judgment as to how much of a barrier it represents.
- A score of 99% may indicate progress if the previous score was lower, but a single failure could represent a significant barrier (e.g., an inaccessible login page).
- Even a 100% score does not guarantee full compliance or user-friendliness but indicates that Axe Core® detected no violations in the tested scope.

The most important thing about the score is that it is deterministic and consistent across runs, so that when changes in the application do bubble up to show an increase or decrease in the score, that is going be meaningful, and not noise.
The key value of the score is its consistency across runs. Changes in the score reliably reflect accessibility improvements or regressions.

In general, the ideal state is a clean 100% score in your Cypress Accessibility report, which you can achieve with a combination of fixing existing issues and ignoring things you know you will not fix.
In general, the ideal state is a clean 100% score in your Cypress Accessibility report, which you can achieve with a combination of fixing existing issues and ignoring things you know you won't fix.

## What a 100% score means

If there are no axe-core violations detected in what you test with Cypress, you can be confident that up to 57% of the issues ([source](https://www.deque.com/blog/automated-testing-study-identifies-57-percent-of-digital-accessibility-issues/)) that a manual audit would detect have already been addressed. This does not mean you have a fully compliant, accessible, or user-friendly application, but it does mean your team has considered the experience and needs of disabled users in the application's development process. This often means the team is well placed to understand and fix any issues that are reported by users or revealed in manual testing.&#x20;
A 100% score with Cypress Accessibility means:

Reaching this "Axe-clean" status is a good sign that basic usability needs are likely met for disabled users well enough that they can experience your application and provide you more general usability feedback as needed.&#x20;
- No Axe Core® violations were detected in the tested scope.
- Up to 57% of issues typically identified in manual audits have been addressed ([source](https://www.deque.com/blog/automated-testing-study-identifies-57-percent-of-digital-accessibility-issues/)).
- While not a guarantee of full compliance, it reflects that the team has actively considered accessibility and is well-positioned to address user-reported issues or findings from manual audits.

Reaching a 100% score, also known as "Axe-clean," signals that the application meets basic usability needs for disabled users, providing a foundation for further usability enhancements.

## Improving Accessibility scores

Many violations are easily addressed by HTML and CSS changes that can be made in your application's code. Click the "Learn More" link for any violation to understand the impact of this violation on users and how to fix it. More details about this are available in the [Inspecting Violation Details section](/accessibility/core-concepts/inspecting-violation-details).
### Address Violations

Most accessibility violations can be resolved through HTML and CSS adjustments. Each violation in Cypress Accessibility includes a "Learn More" link to understand the impact of the issue on users and get step-by-step guidance to fix it.

Refer to the [Inspecting Violation Details section](/accessibility/core-concepts/inspecting-violation-details) for more information on understanding and addressing violations.

### Ignore Non-relevant Issues

For issues you decide not to fix, update your [configuration](/accessibility/configuration/overview) to ignore them. This keeps the score focused on actionable issues while maintaining visibility into ignored elements for transparency.

By actively addressing violations and maintaining a clean score, your team can ensure consistent progress towards a more accessible application.
30 changes: 12 additions & 18 deletions docs/accessibility/core-concepts/how-it-works.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -7,19 +7,17 @@ sidebar_position: 40

# How it works

The stages of the Cypress Accessibility process are as follows:
Cypress Accessibility seamlessly integrates accessibility testing into your development pipeline. Below are the stages of the Cypress Accessibility process:

1. As your run is recorded and specs are uploaded to Cypress Cloud, accessibility checks begin, based on the incoming [Test Replay](/cloud/features/test-replay) data, on the Cypress Cloud servers.
1. When your run completes, the final specs are processed for accessibility checks. All earlier specs have already entered processing and are likely completed by the end of the run.
1. Cypress Cloud organizes all the Views (pages and components) that were rendered during the run, merging and deduplicating across tests so that page-level results and live DOM snapshots can be grouped together.
1. A combined "rule-level" report is created.
1. The completed results are displayed in Cypress Cloud and are flagged in integrations such Slack as GitHub/Gitlab comments, and available CI via the Results API so that you can optionally fail your pipeline if your standards are not met.
1. **Recording and Uploading**: As your test run is recorded, specs are uploaded to Cypress Cloud. Accessibility checks begin based on incoming [Test Replay](/cloud/features/test-replay) data on Cypress Cloud servers.
1. **Processing**: When your run completes, the final specs are processed for accessibility checks. Earlier specs have already entered processing and are likely completed by the end of the run.
1. **Organizing Views**: Cypress Cloud groups Views (pages and components) rendered during the run, merging and deduplicating across tests. This ensures page-level results and live DOM snapshots are organized effectively.
1. **Generating Reports**: A combined "rule-level" report is created, summarizing all detected accessibility violations.
1. **Displaying Results**: Results are displayed in Cypress Cloud, flagged in integrations (e.g., Slack, GitHub/GitLab comments), and accessible via the Results API for optional CI pipeline actions (e.g., failing builds based on unmet standards).

## About Axe Core®

Axe Core® is an open source library for detecting accessibility violations. It is used widely in the industry and have over 1 billion total downloads on npm.
The library may be familiar from other tools, such as Lighthouse, or tools used for code linting, mobile testing and more, and has also been used extensively in Cypress tests through open-source packages (discussed more our [accessibility testing guide](/app/guides/accessibility-testing)). This means that Cypress Accessibility's
configuration can often be easily aligned with your existing accessibility standards across different teams and platforms.
Axe Core® is an open-source library by Deque Systems, widely recognized for detecting accessibility violations. With over a billion downloads on npm and is used in numerous tools.

Axe Core® checks for accessibility violations in your website's user interface, based on common accessibility standards. A violation detected by Axe usually indicates that some users with disabilities would have a problem using or perceiving a feature. An example of a violation would be an image without alternative text provided in the markup, which means the content is not perceivable to visually-impaired users. [Learn more about Axe.](https://github.com/dequelabs/axe-core)

Expand All @@ -31,16 +29,12 @@ The value of this form of testing in Cypress Accessibility is to give you fast,

## Axe Core® updates

Axe Core® publishes new versions several times a year. Cypress will wait at least 30 days after an Axe Core® release before updating the library used in the Cloud. This gives you a known minimum amount of time to see the [changelog](https://github.com/dequelabs/axe-core/blob/develop/CHANGELOG.md) and make any necessary adjustments to your process in advance of Cypress Cloud bumping the version. Bug fixes or new rules in Axe Core® may cause previously-passing projects to fail.
Axe Core® is regularly updated, with Cypress following a controlled adoption process:

We will strive to update Axe Core® in a timely manner after that 30-day buffer, but set no specific expectation around when that will be. Each release is different, and some do not affect the way Axe Core® is executed in Cypress Cloud at all, so those releases may be skipped.

The [Cypress Accessibility changelog](/accessibility/changelog) will reflect any Axe Core® updates that are made in Cypress Cloud, starting on December 1, 2025 (30 days after the publication of this policy).

### Exceptions

In rare cases - such as if an Axe Core® update contains a critical security patch - we may need to update sooner than the 30-day window. If this happens we will communicate with affected customers.
- Update Policy: Cypress Cloud integrates new Axe Core® versions at least 30 days after their release. This provides teams time to review the Cypress Accessibility [changelog](https://github.com/dequelabs/axe-core/blob/develop/CHANGELOG.md) so you can make any necessary adjustments to your process in advance of Cypress Cloud bumping the version.
- Changelog Tracking: Updates are documented in the Cypress Accessibility [changelog](https://github.com/dequelabs/axe-core/blob/develop/CHANGELOG.md), starting December 1, 2024.
- Critical Exceptions: In cases like critical security patches, updates may occur sooner, with notifications provided to affected customers.

## Powered by Test Replay

Because Cypress Accessibility uses data captured through Cypress Test Replay, it is subject [Test Replay limitations](https://docs.cypress.io/faq/questions/cloud-faq#Is-everything-captured-and-replayed-in-Test-Replay).
Cypress Accessibility leverages data captured through Test Replay, ensuring comprehensive analysis while being subject to its [limitations](https://docs.cypress.io/faq/questions/cloud-faq#Is-everything-captured-and-replayed-in-Test-Replay).
32 changes: 22 additions & 10 deletions docs/accessibility/core-concepts/inspecting-violation-details.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -9,35 +9,47 @@ sidebar_position: 20

<AccessibilityAddon />

After clicking through to specific Rule of View from a [Run-level report](/accessibility/core-concepts/run-level-reports), you will see the Violation Detail view.
After navigating to a specific Rule of View from a [Run-level report](/accessibility/core-concepts/run-level-reports), you'll land on the Violation Detail view.

<DocsImage
src="/img/accessibility/core-concepts/cypress-accessibility-issue-detail-view.png"
alt="The detail view of an accessibility issue, containing a Rules area and s Snapshot area for a Buttons must have discernible text violation in the Cypress Realworld App project. Each section is described in detail later in the page."
/>

This page breaks the report down into two sections.
This page is divided into two main sections: **Rules** and **Snapshots**.

### Rules

On the left hand side is the main body of the report that lists all the Rules that match the current filters for this View - or for All Views, if no specific View was selected.&#x20;
The left-hand side of the detail view lists all Rules that match the current filters for the selected View. If no specific View is selected, the list applies to all Views in the run.

#### Rule details

Each Rule can be expanded to see its related Elements. The expanded rule also has extra description text and a "Learn more" link that goes to Deque University to explain more about the rule itself, where it comes from, whose experience is affected by it, how to fix issues related to the rule.
Each Rule can be expanded to reveal related elements and additional information:

- **Description**: Text explaining the rule, its origin, the affected user experiences, and how to fix the issue.
- **Learn More**: A link to Deque University for in-depth guidance.

#### Element details

Each element can be clicked to pin an example of that element in the UI and reveal more details about how to address the accessibility issues. There are two buttons associated with each element:
Clicking on an element in the expanded Rule section provides:

- Copy selector
- This is a quick way to grab the CSS selector that uniquely identifies this element in the page or component
- Print to console
- This will log a reference to the element in the JavaScript console and is the fastest way to debug something in devtools. In Chrome-base browsers for example, you can open your console, right click the logged element, and choose "Reveal in elements panel" to get to the exact right part of the live DOM in two clicks, every time, and debug from there.
- **Pinned Element**: Highlights the element in the UI for easier identification.
- **Copy Selector**: A button to copy the CSS selector uniquely identifying the element.
- **Print to Console**: Logs the element reference to the browser's console for debugging. In Chrome-based browsers, you can right-click the logged element and select "Reveal in elements panel" to navigate directly to the live DOM.
- **Share issue**: A button to copy markdown or plain text snippet to share the issue with your team or collaborators.
- **Solutions**: A list of potential solutions to address the issue.

<DocsImage
src="/img/accessibility/core-concepts/cypress-accessibility-element-details-on-click.png"
alt="Details displayed after clicking on an element in the Rules section of the Cypress Accessibility issue detail view. Options to copy the selector, print to console, share the issue, and solutions are available."
/>

### Snapshots

Note: these are not images or screenshots from the tests. They are fully hydrated HTML and CSS snapshots of your application. This makes it easy to see what each violation represents in context, as well as inspect the entire DOM, make adjustments and see how things appear in the browser's accessibility tree. This makes a big difference in understanding and debugging issues, because you don't have to run the application locally to reproduce the state, and with access to the full DOM you can understand even complex issues and identify solutions.
Snapshots are fully hydrated HTML and CSS representations of your application's state during the test. Unlike screenshots or video, these snapshots allow you to:

- Inspect the DOM and accessibility tree without running the application locally.
- Understand the context of each violation and debug directly from the snapshot.

This area also has the Test Replay button to provide access to any tests where this snapshot appeared, as well as the specific URL of the snapshot displayed at the bottom of the screen.

Expand Down
Loading