Test Suite
We have Travis CI and AppVeyor CI set up to run our short test suite (the same as the pre-commit suite) for every push to the master branch and for every pull request, for 32-bit and 64-bit x86 on Linux, Mac, and Windows. We do not yet have ARM, AArch64, or Android builds.
The drmemory-devs mailing list is sent email whenever there is a (non-flaky-test) failure, or when a prior failure is fixed. If your commit causes a failure, please reply to this email explaining the failure and whether you will be reverting your commit (which you should do unless you have a trivial fix ready immediately). If any tests become flaky on Travis or AppVeyor, please contribute to addressing them to keep our continuous integration testing clean and green.
The Travis configuration is specified in .travis.yml.
The AppVeyor configuration is specified in .appveyor.yml.
When examining AppVeyor failures, it is best to download the log using the link in the upper right, as the web view of the CONSOLE output behaves in surprising ways, moving the ? characters in the regular expression into unusual places and hiding certain characters. Examine the downloaded log instead.
If Travis or AppVeyor turns red, the tree is "closed": meaning that there should be no commits to the repository unrelated to fixing the problem until both Travis and AppVeyor are green again.
Our CI setups provide "trybot" functionality for Linux gcc, Linux clang, Mac OSX clang, and Visual Studio, via pull request testing. To test a change on platforms you don't have locally, or test whether your change will pass the CI, you can create a pull request. Travis and AppVeyor will then automatically run the test suite on your change.
We also have several of our own machines running the test suite as a scheduled job either once per day or several times per week. The results are sent to our CDash dashboard. The drmemory-devs mailing list is sent email whenever there is a build or test failure.
To set up a new machine to run the test suite and report results to CDash, follow the instructions in the tests/nightly.cmake file.
The test suite for Dr. Memory resides in the tests/ directory. The tests are applications that we execute under control of Dr. Memory.
The tests are built by default along with the rest of Dr. Memory.
CTest is used to run the tests. After building, you can use the test
make target to run them:
make -j6
make test
You can also invoke ctest directly, which gives greater control over
which subsets of the tests are run. Use ctest -N to see the list of
tests and which number is assigned to each. Then you can use ctest -I x,y to run all tests from number x to number y, and the -V parameter to
display the command line and the output of the test. For example:
ctest -V -I 49,49
You can also specify tests using inclusion and exclusion regular expressions. For example:
ctest -R 'suppress|realloc'
will run all tests with the string "suppress" or the string "realloc" in their names, while
ctest -E alloc
will run all tests except those with alloc in their names.
If you are using CTest version 2.8 or later, you can run tests in
parallel by passing -jN to ctest on the command line:
ctest -j5
As an alternative to using -V to display the test command line and output
during executing, CTest stores that information (even when not run with -V)
in the Testing/Temporary/LastTest.log file in the build directory.
Each test produces two types of output: stdout and the Dr. Memory results file. Both are checked against expected output, stdout versus a .out file and the results file versus a .res file. The checking is not a regular diff and simply looks for each line. We may want to change this to a stronger diff with regular expression support in the future.
The tests/runsuite.cmake script is used to execute a series of builds and
test runs. Running make test in a single build directory is not
sufficient to test all of the configurations that we support.
The runsuite.cmake script is meant to be executed from an empty directory.
It creates a subdirectory for each build in the suite. Use ctest -S to
execute the script. Here is an example:
mkdir ../build_suite
cd ../build_suite
ctest -S ../drmemory/tests/runsuite.cmake
We recommend using Visual Studio 2010 and MSBuild, in which case pass the use_msbuild parameter:
mkdir ../build_suite
cd ../build_suite
ctest -S ../drmemory/tests/runsuite.cmake,use_msbuild
You can pass -V to ctest to see the results as the test suite runs. Note that it is normal to have ctest output strings such as "No tests were found!!!" as we have builds for which we run no tests (particularly in the short suite). It is also expected to have an error at the end of the suite (if run with -V):
CMake Error: Some required settings in the configuration file were missing:
CTEST_SOURCE_DIRECTORY = E:/derek/opensource/withwiki/trunk/suite/..
CTEST_BINARY_DIRECTORY = (Null)
CTEST_COMMAND = c:/PROGRA~2/CMAKE2~1.6/bin/ctest.exe
This warning is an artifact of how CTest assumes we've set things up and can be ignored.
At the end of the suite a file called results.txt will be created and displayed. It shows configure failures, build failures, and test failures.
Here are sample results from running the test suite:
### ============================================
RESULTS
drheapstat-dbg-32: all 19 tests passed
drheapstat-rel-32: build successful; no tests for this build
drmemory-dbg-32: all 23 tests passed
drmemory-rel-32: build successful; no tests for this build
final package: build successful; no tests for this build
Error in read script: /home/bruening/work/build/drmem_suite/src/tests/runsuite.cmake
The "Error in read script" is expected: ignore it.
Here is an example of running the test suite on Windows from the cmd shell on a new checkout, with the -V flag for more verbose output:
cd "c:\Program Files (x86)\Microsoft Visual Studio *"
VC\vcvarsall.bat
set DDKROOT=C:/derek/ddk
set FLEXROOT=C:/derek/flex_sdk_4.1
cd \derek\drmemory
git clone https://github.com/DynamoRIO/drmemory.git
cd drmemory
git submodule update --init
cd \derek\drmemory\build_suite
"c:\Program Files (x86)\CMake 2.8\bin\ctest.exe" -V -S c:/derek/drmemory/src/tests/runsuite.cmake
DynamoRIO is included inside the Dr. Memory source tree as an svn:externals property. We support both building that embedded DR along with pointing to an already-built separate DR export tree. Whenever a new feature from DynamoRIO is used within Dr. Memory, be sure to update the svn:externals following the instructions at UpdatingDR. In addition, be sure to run the pre-commit test suite using the embedded DR, or be very careful if using an external DR that it matches the svn:externals version.
The test suite includes cross-compilation tests to ensure that the ARM and Android builds are not broken. If a cross-compiler is not found on the PATH, these builds will fail, but they are considered optional, so the whole suite will not be considered a failure. However, we recommend installing the cross-compilers and placing them on your PATH for more thorough testing.
If you have an Android device set up for access through the adb shell utility, the Android build is capable of automatically copying binaries to the device and running tests. If both the Android cross-compiler and adb are on your PATH, and adb status indicates an attached device, the tests will be run.
