testilo

Utilities for ensemble web accessibility testing

Introduction

Testilo is an application that facilitates automated web accessibility testing.

Testilo is designed to be used in conjunction with Testaro.

Testilo and Testaro

The two applications collaborate as follows:

Testaro runs on a computer workstation (typically MacOS or Windows) and performs testing jobs.
Testilo runs on a server and manages testing jobs.
A user-facing application learns from users what testing they want done. That application uses Testilo to get that testing done. Testilo prepares testing jobs and assigns them to Testaro agents. Testaro agents perform the jobs and send reports back to Testilo. Testilo analyzes the Testaro reports and derives from them enhanced reports to satisfy user requirements.

Because Testilo supports Testaro, this README file presumes that you have access to the Testaro README file and therefore does not repeat information provided there.

Dependencies

Testilo depends on Plot for the creation of line graphs by trackers used by the track module. Plot is inserted as a script from the Plot content delivery network.

The dotenv dependency lets you set environment variables in an untracked .env file. This prevents secret data, such as passwords, from being shared as part of this package.

When Testilo is a dependency of another application, the .env file is not imported, because it is not tracked, so all needed environment variables must be provided by the importing application.

Architecture

Testilo is written in Node.js. Commands are given to Testilo in a command-line (terminal) interface or programmatically.

Shared routines, called procs, are located in the procs directory.

Testilo can be installed wherever Node.js (version 18 or later) is installed. This can be a server or the same workstation on which Testaro is installed.

The reason for Testilo being an independent package, rather than part of Testaro, is that Testilo can be installed on any host, while Testaro can run successfully only on a Windows, Macintosh, Ubuntu, or Debian workstation. Testaro runs tests similar to those that a human accessibility tester would run, using whatever browsers, input devices, system settings, simulated and attached devices, and assistive technologies tests may require. Thus, Testaro is mostly limited to functionalities that require workstation attributes. For flexibility in the management of Testaro jobs, other functionalities are located outside of Testaro. You could have software such as Testilo running on a server, communicating with multiple agents running Testaro. The agents could receive jobs from the server and return job reports to the server for further processing. In an advanced configuration, Testilo could even split jobs into segments and assign segments to different Testaro agents for faster processing.

Configuration

Environment variables for Testilo can be specified in a .env file. An example:

FUNCTIONDIR=./procs
SPECDIR=../testdir/specs
JOBDIR=../testdir/jobs
REPORTDIR=../testdir/reports
SCORED_REPORT_URL=../scored/__id__.json
DIGEST_URL=../digested/__id__.html

The use of these environment variables is explained below.

Job preparation

Introduction

Testaro executes jobs. In a job, Testaro performs acts (tests and other operations) on targets (typically, web pages). The Testaro README.md file specifies the requirements for a job.

You can create a job for Testaro directly, without using Testilo.

Testilo can, however, make job preparation more efficient in these scenarios:

You want to perform a battery of tests on multiple targets.
You want to test targets only for particular issues, using whichever tools happen to have tests for those issues.

Target lists

The simplest version of a list of targets is a target list. It is an array of arrays defining 1 or more targets.

A target is defined by 2 items:

A description
A URL

For example, a target list might be:

[
  ['World Wide Web Consortium', 'https://www.w3.org/'],
  ['Mozilla Foundation', 'https://foundation.mozilla.org/en/']
]

A target list can be represented by a text file, in which each target is specified on a line with a Vertical Line character (|) delimiting its description and its URL, which are not quoted. Such a file representing the above target list would have this content:

World Wide Web Consortium|https://www.w3.org/
Mozilla Foundation|https://foundation.mozilla.org/en/

Batches

Targets can be specified in a more complex way, too. That allows you to create jobs in which particular targets are handled distinctively in particular contexts. The more complex representation of a set of targets is a batch. Here is the start of a batch, showing its first target:

{
  id: 'clothing-stores',
  what: 'clothing stores',
  targets: {
    acme: {
      what: 'Acme Clothes',
      url: 'https://acmeclothes.com/',
      actGroups: {
        public: [],
        private: [
          {
            type: 'launch',
            what: 'Acme Clothes login page',
            url: 'https://acmeclothes.com/login.html'
          },
          {
            type: 'text',
            which: 'User Name',
            what: 'tester34'
          },
          {
            type: 'text',
            which: 'Password',
            what: '__TESTER34_PASSWORD__'
          },
          {
            type: 'button',
            which: 'Submit',
            what: 'submit the login form'
          },
          {
            type: 'wait',
            which: 'title',
            what: 'account'
          }
        ]
      }
    },
    …
  ]
}

As shown, a batch, unlike a target list, defines named groups of acts. They can be substituted for script placeholders, so various complex operations can be performed on each target.

In this example, the public act group contains no acts. The private act group contains 5 acts. A launch act in an act group is permitted to have only two properties, what and url. If either of these is omitted, its value is inherited from the corresponding property of the target.

A batch is a JavaScript object. It can be converted to JSON and stored in a file.

Target list to batch

If you have a target list, the batch module of Testilo can convert it to a simple batch. The batch will contain, for each target, only one act group, named main, containing no acts.

Invocation

There are two ways to use the batch module.

By a module

A module can invoke batch() in this way:

const {batch} = require('testilo/batch');
const id = 'divns';
const what = 'divisions';
const targets = [
  ['Energy', 'https://abc.com/energy'],
  ['Water', 'https://abc.com/water']
];
const batchObj = batch(id, what, targets);

The id argument to batch() is an identifier for the target list. The what variable describes the target list. The targets variable is an array of arrays, with each array containing the 2 items (description and URL) defining one target.

The batch() function of the batch module generates a batch and returns it as an object. Within the batch, each target is given a sequential (base-36 alphanumeric) string as an ID.

The invoking module can further dispose of the batch as needed.

By a user

A user can invoke batch() in this way:

Create a target list and save it as a text file (with Vertical-Line-delimited items in Newline-delimited lines) in the targetLists subdirectory of the SPECDIR directory. Name the file x.txt, where x is the list ID.
In the Testilo project directory, execute the statement node call batch id what.

In this statement, replace id with the list ID and what with a string describing the batch. Example: node call batch divns 'ABC company divisions'.

The call module will retrieve the named target list. The batch module will convert the target list to a batch. The call module will save the batch as a JSON file named x.json (replacing x with the list ID) in the batches subdirectory of the SPECDIR directory.

Scripts

The generic, target-independent description of a job is script. A script can contain placeholders that Testilo replaces with acts from a batch, creating one job per target. Thus, one script plus a batch containing n targets will generate n jobs.

Here is a script:

{
  id: 'ts99',
  what: 'aside mislocation',
  strict: true,
  isolate: true,
  standard: 'also',
  observe: false,
  device: {
    id: 'iPhone 8',
    windowOptions: {
      reducedMotion: 'no-preference',
      userAgent: 'Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/604.1.38 (KHTML, like Gecko) Version/17.4 Mobile/15A372 Safari/604.1',
      viewport: {
        width: 375,
        height: 667
      },
      deviceScaleFactor: 2,
      isMobile: true,
      hasTouch: true,
      defaultBrowserType: 'webkit'
    }
  },
  browserID: 'webkit',
  timeLimit: 80,
  creationTimeStamp: ''
  executionTimeStamp: '',
  target: {
    what: '',
    url: ''
  },
  sources: {
    script: 'ts99',
    batch: '',
    mergeID: '',
    requester: '[email protected]'
  },
  acts: [
    {
      type: 'placeholder',
      which: 'private',
      deviceID: 'default',
      browserID: 'chromium'
    },
    {
      type: 'test',
      which: 'axe',
      detailLevel: 2,
      rules: ['landmark-complementary-is-top-level'],
      what: 'Axe'
    },
    {
      type: 'test',
      which: 'qualWeb',
      withNewContent: false,
      rules: ['QW-BP25', 'QW-BP26']
      what: 'QualWeb'
    }
  ]
}

A script has several properties that specify facts about the jobs to be created. They include:

id: an ID. A script can be converted from a JavaScript object to JSON and saved in a file in the SPECDIR directory, where it will be named by its ID (e.g., if the ID is ts99, the file name will be ts99.json). Each script needs an id with a unique value composed of alphanumeric ASCII characters.
what: a description of the script.
strict: true if Testaro is to abort jobs when a target redirects a request to a URL differing substantially from the one specified. If false Testaro is to allow redirection. All differences are considered substantial unless the URLs differ only in the presence and absence of a trailing slash.
isolate: If true, Testilo, before creating a job, will isolate test acts, as needed, from effects of previous test acts, by inserting a copy of the latest placeholder after each target-modifying test act other than the final act. If false, placeholders will not be duplicated.
standard: When Testaro performs a job, every tool produces its own report. Testaro can convert the test results of each tool report to standard results. The standard property specifies how to handle standardization. If also, Testaro will include in its reports both the original results of the tests of tools and the Testaro-standardized results. If only, reports will include only the standardized test results. If no, reports will include only the original results, without standardization.
observe: Testaro jobs can allow granular observation. If true, the job will do so. If false, Testaro will not report job progress, but will send a report to the server only when the report is completed. It is generally user-friendly to allow granular observation, and for user applications to implement it, if they make users wait while jobs are assigned and performed, since that process typically takes a few minutes.
timeLimit: This specifies the maximum duration, in seconds, of a job. Testaro will abort jobs that are not completed within that time.
device: This specifies the ID of a device and properties that each new browser context (window) will have that correspond to that device. The permitted devices are those (about 125 in number) recognized by Playwright, as well as 'default'.
browserID: This specifies the default browser type ('chromium', 'firefox', or 'webkit') of the job.
creationTimeStamp and executionTimeStamp: These properties will have values assigned to them when jobs are created from the script.
target: This object contains blank url and what properties, which will be populated each time the script is converted to a job.
sources.script: This preserves the ID of the script. The id property may be revised to a job ID when the script is converted to a job.
sources.requester: This is the email address to which a message announcing the completion of the job is to be sent, if any.
acts: an array of acts.

In this example, the script contains 3 acts, of which the first is a placeholder. If the above batch were merged with this script, in each job the placeholder would be replaced with the acts in the private act group of a target. For example, the first act of the first job would launch a Chromium browser on a default device, navigate to the Acme login page, complete and submit the login form, wait for the account page to load, run the Axe tests, and then run the QualWeb tests. If the batch contained additional targets, additional jobs would be created, with the acts for each target specified by the private property of the actGroups object of that target.

As shown in this example, it is possible for any particular placeholder to override the default device type and/or browser type of the script by having its own optional deviceID and/or browserID property. Some rules are particularly relevant to some device types and/or can be successfully tested only with particular browser types. Overriding the default device and browser types lets you handle such constraints.

Script creation

You can use the script() function of the script module to simplify the creation of scripts.

Without options

In its simplest form, script() requires 3 string arguments:

An ID for the script
A description of the script
A device ID

Called in this way, script() produces a script that tells Testaro to perform the tests for all of the evaluation rules defined by all of the tools integrated by Testaro. In this case, the script launches a new Webkit browser before performing the tests of each tool.

With options

If you want a more focused script, you can add an additional option argument to script(). The option argument lets you restrict the rules to be tested for. You may choose between restrictions of two types:

Tools
Issues

The option argument is an object. Its properties depend on the restriction type.

For a tool restriction, it has this structure:

{
  type: 'tools',
  specs: ['toolID0', 'toolID1', 'toolIDn']
}

For an issue restriction, it has this structure:

{
  type: 'issues',
  specs: {
    issues: issueClassification,
    issueIDs: ['issue0', 'issue1', 'issueN']
  }
}

If you specify tool options, the script will prescribe the tests for all evaluation rules of the tools that you specify.

If you specify issue options, the script will prescribe the tests for all evaluation rules that are classified into the issues whose IDs you specify. Any tools that do not have any of those rules will be omitted. The value of specs.issues is an issue classification object, with a structure like the one in procs/score/tic43.js. That one classifies about 1000 rules into about 300 issues.

For example, one issue in the tic43.js file is mainNot1. Four rules are classified as belonging to that issue: rule main_element_only_one of the aslint tool and 3 more rules defined by 3 other tools. You can also create custom classifications and save them in a score subdirectory of the FUNCTIONDIR directory.

Invocation

There are two ways to use the script module.

By a module

A module can invoke script() in one of these ways:

const {script} = require('testilo/script');
const scriptObj = script('monthly', 'landmarks', 'default');

const {script} = require('testilo/script');
const options = {…};
const scriptObj = script('monthly', 'landmarks', 'default', options);

In this example, the script will have 'monthly' as its ID, 'landmarks' as its description, and 'default' as its device. It will tell Testaro to test for all evaluation rules if the first form is used, or for all rules specified by the options argument if the second form is used.

The invoking module can further modify and use the script (scriptObj) as needed.

By a user

A user can invoke script() by executing one of these statements in the Testilo project directory:

node call script id what deviceID
node call script id what deviceID tools toolID0 toolID1 toolID2 …
node call script id what deviceID issues tic99 issueID0 issueID1 …

The first form will create a script with no restrictions.

The second form will create a script that prescribes tests for all the rules of the specified tools.

The third form will create a script that prescribes tests for all the rules classified by the named issue-classification file into any of the specified issues.

In this statement, replace id with an ID for the script, such as headings1, consisting of ASCII alphanumeric characters, or with '' if you want Testilo to create an ID. Replace what with a string describing the script, or with '' if you want Testilo to create a generic description. Replace deviceID with a valid device ID.

The call module will retrieve the named classification, if any. The script module will create a script. The call module will save the script as a JSON file in the scripts subdirectory of the SPECDIR directory, using the id value as the base of the file name.

Properties

When the script module creates a script for you, it does not ask you for all of the property values that the script may require. Instead, it chooses these default values:

strict: false
isolate: true
standard: 'only'
observe: false
device.windowOptions.reduce-motion: 'no-preference'
browserID: 'webkit'
lowMotion: false
timeLimit: 50 plus 30 per tool
sources.id: script ID
sources.requester: ''
test acts: launch = {}
axe test act: detailLevel = 2
ibm test act: withItems = true, withNewContent = true
qualWeb test act: withNewContent = false
testaro test act: withItems = true, stopOnFail = false
wave test act: reportType = 4

The device.windowOptions object has, in addition to reducedMotion, other properties shown in the above script example. The script module will set them according to the specified device. If you specify 'default' as the device ID, the only property will be reducedMotion.

The webkit browser type is selected because the other browser types corrupt some tests. The ibm test is performed on the existing page content because some targets cause HTTP2 protocol errors when the ibm tool tries to visit them.

After you invoke script, you can edit the script that it creates to revise any of these options.

Merge

Testilo merges batches with scripts, producing Testaro jobs, by means of the merge module.

Invocation

There are two ways to use the merge module.

By a module

A module can invoke merge() in this way:

const {merge} = require('testilo/merge');
const script = …;
const batch = …;
const executionTimeStamp = '241215T1200';
const jobs = merge(script, batch, executionTimeStamp);

The first two arguments are a script and a batch obtained from files or from prior calls to script() and batch().

The merge() function returns an array of jobs, one job per target in the batch. The invoking module can further dispose of the jobs as needed.

By a user

A user can invoke merge() in this way:

Create a script and save it as a JSON file in the scripts subdirectory of the SPECDIR directory.
Create a batch and save it as a JSON file in the batches subdirectory of the SPECDIR directory.
In the Testilo project directory, execute the statement:

node call merge scriptID batchID executionTimeStamp todoDir

In this statement, replace:

scriptID with the ID (which is also the base of the file name) of the script.
batchID with the ID (which is also the base of the file name) of the batch.
executionTimeStamp with a time stamp in format yymmddThhMM representing the UTC date and time before which the jobs are not to be executed, or '' if it is now.
todoDir: true if the jobs are to be saved in the todo subdirectory, or false if they are to be saved in the pending subdirectory, of the JOBDIR directory.

The call module will retrieve the named script and batch from their respective directories. The merge module will create an array of jobs. The call module will save the jobs as JSON files in the todo or pending subdirectory of the JOBDIR directory.

Output

A Testaro job produced by merge will be identical to the script from which it was derived (see the example above), except that:

The id property of the job will be revised to uniquely identify the job.

The originally empty properties will be populated, as in this example:

…
creationTimeStamp: '241229T0537',
executionTimeStamp: '250110T1200',
target: {
  what: 'Real Estate Management',
  url: 'https://abccorp.com/mgmt/realproperty.html'
},
sources: {
  …,
  batch: 'departments',
  mergeID: '7f',
  …
},
…

Validation

To test the merge module, in the project directory you can execute the statement node validation/merge/validate. If merge is valid, all logging statements will begin with “Success” and none will begin with “ERROR”.

Report enhancement

Introduction

Testaro executes jobs and produces reports of test results. A report is identical to a job (see the example above), except that:

The acts contain additional data recorded by Testaro to describe the results of the performance of the acts. Acts of type test have additional data describing test results (successes, failures, and details).
Testaro also adds a jobData property, describing information not specific to any particular act.

Thus, a report produced by Testaro contains these properties:

id
what
strict
timeLimit
standard
observe
timeStamp
acts
sources
timeStamp
creationTimeStamp
jobData

Testilo can enhance such a report by:

adding scores
creating digests
creating difgests
summarizing reports
comparing scores
tracking score changes
crediting tools

Scoring

The score module of Testilo performs computations on test results and adds a score property to a report.

The score() function of the score module takes two arguments:

a scoring function
a report object

A scoring function defines scoring rules. The Testilo package contains a procs/score directory, in which there are modules that export scoring functions. You can use one of those scoring functions, or you can create your own.

Scorers

The built-in scoring functions are named scorer() and are exported by files whose names begin with tsp (for Testilo scoring proc).

Issues

Those functions make use of issues objects defined in files whose names begin with tic. An issues object defines an issue classification: a body of data about rules of tools and the tool-agnostic issues that those rules are deemed to belong to.

The properties of an issues object are issue objects: objects containing data about issues. Here is an example from tic40.js:

multipleLabelees: {
  summary: 'labeled element ambiguous',
  why: 'User cannot get help on the topic of a form item',
  wcag: '1.3.1',
  weight: 4,
  tools: {
    aslint: {
      label_implicitly_associatedM: {
        variable: false,
        quality: 1,
        what: 'Element contains more than 1 labelable element.'
      }
    },
    nuVal: {
      'The label element may contain at most one button, input, meter, output, progress, select, or textarea descendant.': {
        variable: false,
        quality: 1,
        what: 'Element has more than 1 labelable descendant.'
      },
      'label element with multiple labelable descendants.': {
        variable: false,
        quality: 1,
        what: 'Element has multiple labelable descendants.'
      }
    }
  }
},

In this example, multipleLabelees is the issue ID. The weight property represents the severity of the issue and ranges from 1 to 4. The tools property is an object containing data about the tools that have rules deemed to belong to the issue. Here there are 2 such tools: aslint and nuVal. The tool properties are objects containing data about the relevant rules of those tools: 1 from the aslint tool and 2 from the nuVal tool.

The property for each rule has the rule ID as its name.

The variable property is true if the rule ID is a regular expression or false if the rule ID is a string. Most rule IDs are strings, but some rules have patterns rather than constant strings as their identifiers, and in those cases regular expressions matching the patterns are the property names.

The quality property is usually 1, but if the test of the rule is known to be inaccurate the value is a fraction of 1, so the result of that test will be downweighted.

Some issue objects (such as flash in tic40.js) have a max property, equal to the maximum possible count of instances. That property allows a scorer to ascribe a greater weight to an instance of that issue.

Output

A scorer adds a score property to the report that it scores.

Invocation

There are two ways to invoke the score module.

By a module

A module can invoke score() in this way:

const {score} = require('testilo/score');
const {scorer} = require('testilo/procs/score/tsp99');
const report = …;
score(scorer, report);

The first argument to score() is a scoring function. In this example, it has been obtained from a module in the Testilo package, but it could be a custom function.

The second argument to score() is a report object. It may have been read from a JSON file and parsed, or parsed from the body of a POST request received from a Testaro agent.

The invoking module can further dispose of the scored report as needed.

By a user

A user can invoke score() in this way:

node call score tsp99
node call score tsp99 240922

When a user invokes score() in this example, the call module:

gets the scoring module tsp99 from its JSON file tsp99.json in the score subdirectory of the FUNCTIONDIR directory.
gets all reports, or if the third argument to call() exists the reports whose file names begin with '240922', from the raw subdirectory of the REPORTDIR directory.
adds score data to each report.
writes each scored report in JSON format to the scored subdirectory of the REPORTDIR directory.

Validation

To test the score module, in the project directory you can execute the statement node validation/score/validate. If score is valid, all logging statements will begin with “Success” and none will begin with “ERROR”.

Rescoring

The rescore module of Testilo creates a new report for a subset of the results in an existing report.

Any scored report is based on a set of tests of a set of tools. Suppose you want to disregard some of those tests and get a revised report for only the remaining tests. The rescore module does this for you.

A typical use case is your desire to examine results for only one or only some of the tools that were used for a report. All the needed information is in the report, so it is not necessary to create, perform, and await a new job and report. You want a new report whose standard results and score data are what a new job would have produced.

The rescore() function of the rescore module takes four arguments:

a scoring function
a report object
a restriction type ('tools' or 'issues')
an array of IDs of the tools or issues to be included

Then the rescore() function copies the report, removes the no-longer-relevant acts, removes the no-longer-relevant instances from and revises the totals of the standardResult properties, replaces the score property with a new one, and returns the revised report.

The new report is not identical to the report that a new job would have produced, because:

Any original (non-standardized) results and data that survive in the new report are not revised.
Any scores arising from causes other than test results, such as latency or browser warnings, are not revised.
The score property object now includes a rescore property that identifies the original report ID (in case it is later changed), the date and time of the rescoring, the restriction type, and an array of the tool or issue IDs included by the restriction.

Invocation

There are two ways to invoke the rescore module.

By a module

A module can invoke rescore() in this way:

const {rescore} = require('testilo/rescore');
const {scorer} = require('testilo/procs/score/tsp99');
const report = …;
const restrictionType = 'tools';
const restrictions = ['axe', 'nuVal'];
rescore(scorer, report, restrictionType, restrictions);

The invoking module can further dispose of the rescored report as needed. Disposal may require revising the ID of the report so that the original and the rescored reports have distinct IDs.

By a user

A user can invoke rescore() in this way:

node call rescore tsp99 '' tools axe nuVal
node call rescore tsp99 240922 tools axe nuVal

When a user invokes rescore() in this example, the call module:

gets the scoring module tsp99 from its JSON file tsp99.json in the score subdirectory of the FUNCTIONDIR directory.
gets all reports, or if the third argument to call() is nonempty the reports whose file names begin with '240922', from the scored subdirectory of the REPORTDIR directory.
defines an ID suffix.
revises the stardardResult properties in each report.
replaces the score property in each report.
appends the ID suffix to the ID of each report.
writes each rescored report in JSON format to the scored subdirectory of the REPORTDIR directory.

Validation

To test the rescore module, in the project directory you can execute the statement node validation/rescore/validate. If rescore is valid, all logging statements will begin with “Success” and none will begin with “ERROR”.

Digesting

Introduction

Reports from Testaro are JavaScript objects. When represented as JSON, they are human-readable, but not human-friendly. They are basically designed for machine tractability. This is equally true for reports that have been scored by Testilo. But Testilo can digest a scored report, converting it to a human-oriented HTML document, or digest.

The digest module digests a scored report. Its digest() function takes two arguments:

a digester (a digesting function)
a scored report object

The digester populates an HTML digest template. A copy of the template, with its placeholders replaced by computed values, becomes the digest. The digester defines the rules for replacing the placeholders with values. The Testilo package contains a procs/digest directory, in which there are subdirectories, each containing a template and a module that exports a digester. You can use one of those modules, or you can create your own.

The included templates format placeholders with leading and trailing underscore pairs (such as __issueCount__).

Invocation

There are two ways to use the digest module.

By a module

A module can invoke digest() in this way:

const {digest} = require('testilo/digest');
const digesterDir = `${process.env.FUNCTIONDIR}/digest/tdp99a`;
const {digester} = require(`${digesterDir}/index`);
const scoredReport = …;
digest(digester, scoredReport)
.then(digestedReport => {…});

The first argument to digest() is a digester. In this example, it has been obtained from a file in the Testilo package, but it could be custom-made.

The second argument to digest() is a scored report object. It may have been read from a JSON file and parsed, or may be a report scored by score().

The digest() function returns a promise resolved with a digest. The invoking module can further dispose of the digest as needed.

By a user

A user can invoke digest() in this way:

node call digest tdp99
node call digest tdp99 241105

When a user invokes digest() in this example, the call module:

gets the template and the digesting module from subdirectory tdp99 in the digest subdirectory of the FUNCTIONDIR directory.
gets all reports, or if the third argument to call() exists all reports whose file names begin with '241105', from the scored subdirectory of the REPORTDIR directory.
digests each report.
writes the digested reports to the digested subdirectory of the REPORTDIR directory.
includes in each digest a link to the scored report, with the link destination being based on SCORED_REPORT_URL.

The included digesters create digests that have links. The server that serves such a digest must also respond correctly when a user activates one of the links. One link is to the scored report. Other links are to World Wide Web Consortium documents on WCAG principles, guidelines, and success criteria.

The digests created by digest() are HTML files, and they expect a style.css file to exist in their directory. If you use an included digester, the reports/digested/style.css file in Testilo is an appropriate stylesheet to be copied into the directory where digested reports are written.

Difgesting

Introduction

A difgest is a digest that compares two reports. They can be reports of different targets, or reports of the same target from two different times or under two different conditions.

The difgest module difgests two scored reports. Its difgest() function takes five arguments:

a difgest template
a difgesting function
a scored report object
another scored report object
the URL of the digest of the first scored report
the URL of the digest of the second scored report

The difgest template and module operate like the digest ones.

Invocation

There are two ways to use the difgest module.

By a module

A module can invoke difgest() in this way:

const {difgest} = require('testilo/difgest');
const difgesterDir = `${process.env.FUNCTIONDIR}/difgest/tdp99a`;
const {difgester} = require(`${difgesterDir}/index`);
const scoredReportA = …;
const scoredReportB = …;
const digestAURL = 'https://abc.com/testing/reports/digested/241022T1458-0.html';
const digestBURL = 'https://abc.com/testing/reports/digested/241029T1458-0.html';
difgest(difgester, scoredReportA, scoredReportB, digestAURL, digestBURL)
.then(difgestedReport => {…});

The difgest will include links to the two digests, which, in turn, contain links to the full reports.

difgest() returns a difgest. The invoking module can further dispose of the difgest as needed.

By a user

A user can invoke difgest() in this way:

node call difgest tfp99 20141215T1200-x7-3 20141215T1200-x7-4

When a user invokes difgest in this example, the call module:

gets the template and the difgesting module from subdirectory tfp99 in the difgest subdirectory of the FUNCTIONDIR directory.
gets reports 20141215T1200-x7-3 and 20141215T1200-x7-4 from the scored subdirectory of the REPORTDIR directory.
writes the difgested report to the difgested subdirectory of the REPORTDIR directory.

Difgests include links to the digests of the two reports. The destinations of those links are obtained from the DIFGEST_URL environment variable.

Difgests expect a style.css file to exist in their directory, as digests do.

Validation

To test the digest module, in the project directory you can execute the statement node validation/digest/validate. If digest is valid, all logging statements will begin with “Success” and none will begin with “ERROR”.

Summarization

The summarize module of Testilo can summarize a scored report. The summary is an object that contains these properties from the report: id, endTime, targetWhat (description of the target), url (of the target), sources, and score (only the value of the score.total property of the report).

Report summaries make some operations more efficient by allowing other modules to get needed data from summaries instead of from reports. The size of a summary tends to be about 0.01% of the size of a report.

Invocation

The summarize module summarizes one report when invoked by a module, but the call module invoked by a user can call summarize multiple times to summarize multiple reports and combine those summaries into a file.

By a module

A module can invoke summarize() in this way:

const {summarize} = require('testilo/summarize');
const report = …;
const summary = summarize(report);
…

The report argument is a scored report. The summary constant is an object. The module can further dispose of summary as needed.

By a user

A user can invoke summarize() in either of these two ways:

node call summarize 'company divisions'
node call summarize 'company divisions' 2411

When a user invokes summarize in this example, the call module:

gets all the reports in the scored subdirectory of the REPORTDIR directory, or (if the third argument is present) all those whose file names begin with 2411.
creates a summary of each report.
combines the summaries into an array.
creates a summary report, an object containing three properties: id (an ID), what (a description, such as 'company divisions'), and summaries (the array of summaries).
writes the summary report in JSON format to the summarized subdirectory of the REPORTDIR directory.

Comparison

If you use Testilo to perform a battery of tests on multiple targets, you may want a single report that compares the total scores received by the targets. Testilo can produce such a comparison.

The compare module compares the scores in a summary report. The compare() function of the compare module takes two arguments:

a comparison function
a summary report

The comparison function defines the rules for generating an HTML file comparing the scored reports. The Testilo package contains a procs/compare directory, in which there are subdirectories containing modules that export comparison functions. You can use one of those functions, or you can create your own.

The built-in comparison functions compare all of the scores in the summary report. Thus, if the summary report contains multiple scores for the same target, based on tests performed at various times, those scores will all appear in the comparison, labeled identically with the what description of the target. If you want only one score per target to appear, you can create a new summary report that includes only one summary per target in its summaries array.

Invocation

There are two ways to use the compare module.

By a module

A module can invoke compare() in this way:

const {compare} = require('testilo/compare');
const comparerDir = `${process.env.FUNCTIONDIR}/compare/tcp99`;
const {comparer} = require(`${comparerDir}/index`);
const id = …;
compare(id, comparer, summaryReport)
.then(comparison => {…});

The first argument to compare() is an ID that will be named in the comparison. The second argument is a comparison function. In this example, it been obtained from a file in the Testilo package, but it could be custom-made. The third argument is a summary report. The compare() function returns a comparison. The invoking module can further dispose of the comparison as needed.

By a user

A user can invoke compare() in this way:

node call compare 'state legislators' tcp99 240813

When a user invokes compare in this example, the call module:

gets the comparison module from subdirectory tcp99 of the subdirectory compare in the FUNCTIONDIR directory.
gets the last summary report whose file name begins with '240813' from the summarized subdirectory of the REPORTDIR directory.
creates an ID for the comparison.
creates the comparison as an HTML document.
writes the comparison in the comparative subdirectory of the REPORTDIR directory, with state legislators as a description and the ID as the base of the file name.

The comparative report created by compare is an HTML file, and it expects a style.css file to exist in its directory. The reports/comparative/style.css file in Testilo is an appropriate stylesheet to be copied into the directory where comparative reports are written.

Validation

To test the compare module, in the project directory you can execute the statement node validation/compare/validate. If compare is valid, all logging statements will begin with “Success” and none will begin with “ERROR”.

Track

The track module of Testilo selects, organizes, and presents data from summaries to show changes over time in total scores. The module produces a web page, showing changes in a table and a line graph. The line graph contains a line for each target (namely, each value of the target.what property).

A typical use case for tracking is monitoring, i.e. periodic auditing of one or more web pages.

Invocation

By a module

A module can invoke track() in this way:

const {track} = require('testilo/track');
const trackerDir = `${process.env.FUNCTIONDIR}/track/ttp99a`;
const {tracker} = require(`${trackerDir}/index`);
const summaryReport = …;
const [reportID, 'main competitors', trackReport] = track(tracker, summaryReport);

The track() function returns, as an array, an ID and an HTML tracking report that shows data for all of the results in the summary report and identifies “main competitors” as its subject. The invoking module can further dispose of the tracking report as needed.

By a user

A user can invoke track() in one of these ways:

node call track ttp99a 'main competitors'
node call track ttp99a 'main competitors' 241016
node call track ttp99a 'main competitors' '' 'ABC Foundation'
node call track ttp99a 'main competitors' 241016 'ABC Foundation'

When a user invokes track() in this example, the call module:

gets the summary report from the last file in the summarized subdirectory of the REPORTDIR directory, or if the third argument to call() exists and is not empty the last one whose name begins with '241016'.
selects the summarized data for all results in the summary report, or if the fourth argument to call() exists from all results whose target.what property has the value 'ABC Foundation'.
uses tracker ttp99a to create a tracking report that identifies “main competitors” as its subject.
assigns an ID to the tracking report.
writes the tracking report to the tracking subdirectory of the REPORTDIR directory, with the ID as the base of its file name.

The tracking reports created by track() are HTML files, and they expect a style.css file to exist in their directory. The reports/tracking/style.css file in Testilo is an appropriate stylesheet to be copied into the directory where tracking reports are written.

Statistics

If you use Testaro to perform all the tests of all the tools on multiple targets and score the reports with a score proc that maps tool rules onto tool-agnostic issues, you may want to get statistics on tools and issues, aggregated from the scored reports. Testilo has two procs for this purpose.

Tool crediting

The credit module tabulates the contribution of each tool to the discovery of issue instances in a collection of scored reports. Its credit() function takes two arguments: a report description and an array of score properties of scored reports.

The function produces a credit report containing four sections:

counts: for each issue, how many instances each tool reported
onlies: for each issue of which only 1 tool reported instances, which tool it was
mosts: for each issue of which at least 2 tools reported instances, which tool(s) reported the maximum instance count
tools: for each tool, two sections:
- onlies: a list of the issues that only the tool reported instances of
- mosts: a list of the issues for which the instance count of the tool was not surpassed by that of any other tool

Invocation

There are two ways to use the credit module.

By a module

A module can invoke credit() in this way:

const {credit} = require('testilo/credit');
const reportScores = […];
const creditReport = credit('June 2025', reportScores);

The first argument to credit() is a description to be included in the credit report. The second argument is an array of score properties of scored report objects. The credit() function returns a credit report. The invoking module can further dispose of the credit report as needed.

By a user

A user can invoke credit() in one of these ways:

node call credit legislators
node call credit legislators 241106

When a user invokes credit in this example, the call module:

gets all reports, or if the third argument to call() exists all reports whose file names begin with '241106', in the scored subdirectory of the REPORTDIR directory.
gets the score properties of those reports.
creates an ID for the credit report.
writes the credit report as a JSON file, with the ID as the base of its file name and legislators as its description, to the credit subdirectory of the REPORTDIR directory.

Issue scores

The issues module tabulates total issue scores. Its issues() function takes two arguments: a report description and an array of score properties of scored reports.

The function produces an issue report, an object with issue properties, whose values are the totals of the scores of the respective issues.

Invocation

There are two ways to use the credit module.

By a module

A module can invoke issues() in this way:

const {issues} = require('testilo/issues');
const reportScores = […];
const issuesReport = issues('legislators', reportScores);

The arguments to issues() are a report description and an array of score properties of scored report objects. The issues() function returns an issues report. The invoking module can further dispose of the issues report as needed.

By a user

A user can invoke issues() in one of these ways:

node call issues legislators
node call issues legislators 241106

When a user invokes issues in this example, the call module:

gets all reports, or if the third argument to call() exists all reports whose file names begin with '241106', in the scored subdirectory of the REPORTDIR directory.
gets the score properties of those reports.
creates an ID for the issues report.
writes the issues report as a JSON file, with the ID as the base of its file name and legislators as its description, to the issues subdirectory of the REPORTDIR directory.

Origin

Work on the functionalities of Testaro and Testilo began in 2017. It was named Autotest in early 2021 and then partitioned into the more single-purpose packages Testaro and Testilo in January 2022.

On 5 May 2024 ownership of the Testilo repository was transfered from the personal account of contributor Jonathan Pool to the organization account cvs-health of CVS Health. The MIT license of the repository did not change.

Contributing

As of 5 May 2024, upon the transfer of the repository ownership to CVS Health, contributors of code to Testilo are required to execute the CVS Health OSS Project Contributor License Agreement for Testilo before any pull request will be approved and merged.

Etymology

“Testilo” means “testing utility” in Esperanto.

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. */

Name		Name	Last commit message	Last commit date
Latest commit History 857 Commits
procs		procs
reports		reports
scripts		scripts
toolRules		toolRules
validation		validation
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
batch.js		batch.js
call.js		call.js
compare.js		compare.js
credit.js		credit.js
difgest.js		difgest.js
digest.js		digest.js
issues.js		issues.js
merge.js		merge.js
package-lock.json		package-lock.json
package.json		package.json
reconcile.js		reconcile.js
rescore.js		rescore.js
score.js		score.js
script.js		script.js
summarize.js		summarize.js
track.js		track.js
wcag.json		wcag.json

License

cvs-health/testilo

Folders and files

Latest commit

History

Repository files navigation

testilo

Introduction

Testilo and Testaro

Dependencies

Architecture

Configuration

Job preparation

Introduction

Target lists

Batches

Target list to batch

Invocation

By a module

By a user

Scripts

Script creation

Without options

With options

Invocation

By a module

By a user

Properties

Merge

Invocation

By a module

By a user

Output

Validation

Report enhancement

Introduction

Scoring

Scorers

Issues

Output

Invocation

By a module

By a user

Validation

Rescoring

Invocation

By a module

By a user

Validation

Digesting

Introduction

Invocation

By a module

By a user

Difgesting

Introduction

Invocation

By a module

By a user

Validation

Summarization

Invocation

By a module

By a user

Comparison

Invocation

By a module

By a user

Validation

Track

Invocation

By a module

By a user

Statistics

Tool crediting

Invocation

By a module

By a user

Issue scores

Invocation

Packages