README.adoc

RISC-V Architectural Testing Framework

Table of Contents

• 1. Introduction
  • 1.1. About
  • 1.2. Intent of the architectural test suite
  • 1.3. Intended audience
  • 1.4. Contribute
• 2. Licensing
  • 2.1. AsciiDoc
  • 2.2. Installing tools
  • 2.3. Building the documentation
  • 2.4. Contributors
  • 2.5. Document history
• 3. Repository structure
• 4. Vocabulary
  • 4.1. The architectural test
  • 4.2. The architectural test pool
  • 4.3. The test signature
  • 4.4. The reference signature
  • 4.5. The test target
  • 4.6. The target shell
• 5. Porting a new target

1. Introduction

1.1. About

This document describes the RISC-V Architectural Testing framework which is used to test if a RISC-V device’s has understood and implemented the specifications correctly

• It explains the framework around the tests, the running of individual tests, and the suites of tests.

• It explains how to set up targets to run the tests.

This document is made freely available under a [app_cc_by_4.0].

1.2. Intent of the architectural test suite

The RISC-V Architectural Tests are an evolving set of tests that are created to help ensure that software written for a given RISC-V Profile/Specification will run on all implementations that comply with that profile.

These tests also help ensure that the implementer has both understood and implemented the specification correctly.

The RISC-V Architectural Test suite is a minimal filter. Passing the tests and having the results approved by RISC-V International is a prerequisite to licensing the RISC-V trademarks in connection with the design. Passing the RISC-V Architectural Tests does not mean that the design complies with the RISC-V Architecture. These are only a basic set of tests checking important aspects of the specification without focusing on details.

The RISC-V Architectural Tests are not a substitute for rigorous design verification.

The result that the architecture tests provide to the user is an assurance that the specification has been interpreted correctly and the implementation under test (DUT) can be declared as RISC-V Architecture Test compliant.

1.3. Intended audience

This document is intended for design and verification engineers who wish to check if their RISC-V implementation (simulation models, HDL models, etc.) is compliant to the RISC-V specification.

For those who wish to develop new architectural tests and/or to write or adapt their own test framework are suggested to read the Test Format Spec.

1.4. Contribute

You are encouraged to contribute to this repository (including changes to this document) by submitting pull requests and by commenting on pull requests submitted by other people as described in the README.md file in the top level directory.

While submitting a pull request note that some directories use ChangeLog files to track changes in the code and documentation. Please honor these, keeping them up to date and including the ChangeLog entry in the git commit message.

Make sure to also include a comment with the SPDX license identifier in all source files, for example:

// SPDX-License-Identifier: BSD-3-Clause

Note	Don’t forget to add your own name to the list of contributors in the document.

2. Licensing

In general:

• code is licensed under the BSD 3-clause license (SPDX license identifier BSD-3-Clause);

• documentation is licensed under the Creative Commons Attribution 4.0 International license (SPDX license identifier CC-BY-4.0).

The files COPYING.BSD and COPYING.CC in the top level directory contain the complete text of these licenses.

Note	The riscv-ovpsim simulator is licensed under an Imperas license. There is no dependency on this and it is included as a convenience to users.

2.1. AsciiDoc

This is a structured text format used by this document. Simple usage should be fairly self evident.

• Comprehensive information on the format is on the AsciiDoc website.

• Comprehensive information on the tooling on the AsciiDoctor website.

• You may find this cheat sheet helpful.

2.2. Installing tools

To generate the documentation as HTML you need asciidoctor and to generate as PDF you need asciidoctor-pdf.

• These are the installation instructions for asciidoctor.

• These are the installation instructions for asciidoctor-pdf.

To spell check you need aspell installed.

2.3. Building the documentation

To build HTML:

make html

To build PDF:

make pdf

To build both:

make

To check the spelling (excludes any listing or code phrases):

make spell

Any custom words for spell checking should be added to custom.wordlist.

2.4. Contributors

This document has been created by the following people (in alphabetical order of surname).

• Jeremy Bennett

• Mary Bennett

• Simon Davidmann

• Neel Gala

• Radek Hajek

• Lee Moore

• Milan Nostersky

• Marcela Zachariasova

2.5. Document history

Revision	Date	Author	Modification
1.16 Draft	23 September 2020	Neel Gala	Changed Compliance to Architecture/Architectural. Refined the steps to port a new target. Removed simulator and target specific docs. Added vocabulary from the Test Format Specification
1.15 Draft	14 March 2019	Prashanth Mundkur	Added support and instructions for using the C and OCaml simulators from the Sail RISC-V formal model as targets.
1.14 Draft	21 February 2019	Deborah Soung	Documented how to use SiFive’s RISC-V ISA Formal Specification model as a target.
1.13 Draft	29 January 2019	Deborah Soung	Added documentation on how to use Rocket Chip generated cores as targets.
1.12 Draft	22 November 2018	Simon Davidmann	Updated notes on Test Suites.
1.11 Draft	21 November 2018	Neel Gala	Added new signature format specs .
1.10 Draft	20 June 2018	Simon Davidmann, Lee Moore	Cleaned up description of updated framework and inclusion of riscvOVPsim.
1.9 Draft	12 June 2018	Jeremy Bennett	Update Future work section to take account of Codasip changes. Remove diagrammatic directory structure.
1.8 Draft	12 June 2018	Jeremy Bennett	Add Future work section.
1.7 Draft	12 June 2018	Jeremy Bennett	Add CC license as an appendix.
1.6 Draft	10 June 2018	Jeremy Bennett	Tidy up areas that are flawed in HTML version.
1.5 Draft	8 June 2018	Jeremy Bennett	General tidy up.
1.4 Draft	8 June 2018	Jeremy Bennett	Added license preamble.
1.3 Draft	5 June 2018	Simon Davidmann	Updated to reflect directory structure and trace macros.
1.2 Draft	3 June 2018	Jeremy Bennett	Converted to AsciiDoc, cleaned up and restructured.
1.1 Draft	1 June 2018	Simon Davidmann Lee Moore	Revised format and expand to describe framework, usage of many tests groups, and different Targets
1.0	24 December 2017	Radek Hajek Milan Nostersky Marcela Zachariasova	First version of the document.

3. Repository structure

The top level directory contains :

• a README.md file providing high-level details about the RISC-V Architecture Testing Framework.

• CHANGELOG.md for logging changes with respect to code.

• complete license files for the Creative Commons and BSD licenses used by the task group.

There are then five top level directories.

doc

All the documentation for the project, written using AsciiDoc.

coverage

This directory contains multiple coverpoint YAMLs which capture all the coverpoints of interest which are covered by all the tests in the riscv-test-suite directory.

riscv-test-suite

This contains a further env subdirectory which contains the arch_test.h and the encoding.h file which includes common assembly macros and routines which is used across the tests in the suite. Further directories are present which contain the actual architectural tests. The names and structure of these directories is based on the link:../test-pool structure guideline highlighted above.

4. Vocabulary

4.1. The architectural test

At the heart of the testing infrastructure is the detailed architectural test available as .S assembly files. The specification and format of the tests is defined in the Test Format Spec.

4.2. The architectural test pool

The architectural tests are grouped into different functional test suites targeting the different subsets of the RISC-V specification. For more details on the test-pool structure, hierarchy and conventions please refer to the Test Format Spec.

For information on the currently supported different test suites, look here: ../riscv-test-suite/README.md

4.3. The test signature

Each test in the architectural test pool generates a test signature, which represents the data written into specific memory locations during the execution of the test. The signature typically will record values (or sanitised values) of the operations carried out in the test. More details on the format and nature of the signatures is available in the Test Format Spec

4.4. The reference signature

In order to claim that a device/implementation has passed the RISC-V Architecture Tests, the test signatures obtained from the execution of the tests on the implementation need to be compared against a set of golden reference signature. These reference signatures are currently generated by the RISC-V SAIL formal model and statically hosted in the repository for each test.

4.5. The test target

The test target can be either a RISC-V Instruction Set Simulator (ISS), a RISC-V emulator, a RISC-V RTL model running on an HDL simulator, a RISC-V FPGA implementation or a physical chip. Each of the target types offers specific features and represents specific interface challenges. It is a role of the target shell to handle different targets while using the same architectural test pool as a test source.

4.6. The target shell

The target shell is the software and hardware environment around the test target that enables it to communicate with the framework, including assembling and linking tests, loading tests into memory, executing tests, and extracting the signature. The input to the target shell is a .S architectural test file, and the output is a test signature.

5. Porting a new target

Please refer to the RISCOF’s guide on [Running RISCV-ARCH-TESTS](https://riscof.readthedocs.io/en/stable/arch-tests.html) to run the architectural test-suite on your DUT.