Open source offering of the cdisc rules engine
This project uses the black code formatter and flake8 linter for python.
It also uses pre-commit to run black and flake8 when you commit.
Both dependencies are added to requirements.txt.
Required
Setting up pre-commit requires one extra step. After installing it you have to run
pre-commit install
This installs pre-commit in your .git/hooks directory.
From the root of the project run the following command:
python -m pytest tests/unit/
Clone the repository and run core.py --help to see the full list of commands.
Run core.py validate --help to see the list of validation options.
-ca, --cache TEXT Relative path to cache files containing pre
loaded metadata and rules
-ps, --pool-size INTEGER Number of parallel processes for validation
-d, --data TEXT Path to directory containing data files
-dp, --dataset-path TEXT Absolute path to dataset file
-l, --log-level [info|debug|error|critical|disabled|warn]
Sets log level for engine logs, logs are
disabled by default
-rt, --report-template TEXT File path of report template to use for
excel output
-s, --standard TEXT CDISC standard to validate against
[required]
-v, --version TEXT Standard version to validate against
[required]
-ct, --controlled-terminology-package TEXT
Controlled terminology package to validate
against, can provide more than one
-o, --output TEXT Report output file destination
-of, --output-format [JSON|XLSX]
Output file format
-rr, --raw-report Report in a raw format as it is generated by
the engine. This flag must be used only with
--output-format JSON.
-dv, --define-version TEXT Define-XML version used for validation
--whodrug TEXT Path to directory with WHODrug dictionary
files
--meddra TEXT Path to directory with MedDRA dictionary
files
-r, --rules TEXT
-vo, --verbose-output Specify this option to print rules as they
are completed
-p, --progress [verbose_output|disabled|percents|bar]
Defines how to display the validation
progress. By default a progress bar like
"[████████████████████████████--------]
78%"is printed.
--help Show this message and exit.
To validate a folder using rules for SDTM-IG version 3.4 use the following command:
python core.py validate -s sdtmig -v 3-4 -d path/to/datasets
-
update-cache - update locally stored cache data (Requires an environment variable -
CDISC_LIBRARY_API_KEY)python core.py update-cache -
list-rules - list rules available in the cache
- list all rules:
python core.py list-rules- list rules for standard:
python core.py list-rules -s sdtmig -v 3-4 -
list-rule-sets - lists all standards and versions for which rules are available:
python core.py list-rule-sets -
test - Test authored rule given dataset in json format
-ca, --cache TEXT Relative path to cache files containing pre loaded metadata and rules -dp, --dataset-path TEXT Absolute path to dataset file -s, --standard TEXT CDISC standard to validate against [required] -v, --version TEXT Standard version to validate against [required] -ct, --controlled-terminology-package TEXT Controlled terminology package to validate against, can provide more than one -dv, --define-version TEXT Define-XML version used for validation --whodrug TEXT Path to directory with WHODrug dictionary files --meddra TEXT Path to directory with MedDRA dictionary files -r, --rule TEXT Path to rule json file. --help Show this message and exit.
EX: `python core.py test -s sdtmig -v 3-4 -dp <path to dataset json file> -r <path to rule json file> --meddra ./meddra/ --whodrug ./whodrug/`
Note: JSON dataset should match the format provided by the rule editor:
{ "datasets": [{ "filename": "cm.xpt", "label": "Concomitant/Concurrent medications", "domain": "CM", "variables": [ { "name": "STUDYID", "label": "Study Identifier", "type": "Char", "length": 10 } ], "records": { "STUDYID": [ "CDISC-TEST", "CDISC-TEST", "CDISC-TEST", "CDISC-TEST" ], } } ] }
### Creating an executable version
**Linux**
`pyinstaller core.py --add-data=venv/lib/python3.9/site-packages/xmlschema/schemas:xmlschema/schemas --add-data=resources/cache:resources/cache --add-data=resources/templates:resources/templates`
**Windows**
`pyinstaller core.py --add-data=".venv/Lib/site-packages/xmlschema/schemas;xmlschema/schemas" --add-data="resources/cache;resources/cache" --add-data="resources/templates;resources/templates"`
_Note .venv should be replaced with path to python installation or virtual environment_
This will create an executable version in the `dist` folder. The version does not require having Python installed and
can be launched by running `core` script with all necessary CLI arguments.
### Creating .whl file
All non-python files should be listed in `MANIFEST.in` to be included in the distribution.
Files must be in python package.
**Unix/MacOS**
`python3 -m pip install --upgrade build`
`python3 -m build`
To install from dist folder
`pip3 install {path_to_file}/cdisc_rules_engine-{version}-py3-none-any.whl`
To upload built distributive to pypi
`python3 -m pip install --upgrade twine`
`python3 -m twine upload --repository {repository_name} dist/*`
**Windows(Untested)**
`py -m pip install --upgrade build`
`py -m build`
To install from dist folder
`pip install {path_to_file}/cdisc_rules_engine-{version}-py3-none-any.whl`
To upload built distributive to pypi
`py -m pip install --upgrade twine`
`py -m twine upload --repository {repository_name} dist/*`