BindingScanner version 1.0.0 (#45)

* add the template file

* Add GitHub issue templates

* add meta files and update README (#5)

* add meta files and update README

* fix broken links 1

* fix broken links 2

* docs: add Zavolan Lab logo to README

* docs: remove link from logo, fix broken link in README

* ci: Add initial Travis CI YAML file

* build: add pre-run scripts, envs, test, docs

* add main and dev envs YAML files

* add conda env create scripts

* install miniconda in CI

* build conda envs in CI

* add download ATtRACT db

* fix: ATtRACT download in CI, 1

* fix: ATtRACT download in CI, 2

* fix: ATtRACT download in CI, 3

* add ATtRACT test files for re-format

* fix: ATtRACT download in CI, 4

* fix: ATtRACT download in CI, 5

* delete redundancy in ATtRACT download

* add shell linter and styler to the dev env

* add link to shell linter

* moved workflow docs

* add ATtRACT reformat script

* clean CI

* ci: removed Py2 support

* ci: back to the unzip state

* ci: clean unzip command

* docs: update Readme

* add ATtRACT backup

* add unzip the backup db

* corr typo

* add commands to reformat and test ATtRACT

* docs: add ATtRACT section

* ci: update

* ci: fix multiline python command 1

* ci: add md5 check , 1

* ci: add md5 check , 2

* ci: add md5 check , 3

* ci: remove download ATtRACT test

* docs: fix typos

* build: add pipeline, envs, tests, docs

* DIRTY: move files from macOS

* graph tests OK

* add snakemake graph tests

* add snakemake internal dir

* add main execution test script

* style the YAML file

* add integration test pwms

* first working Snakemake ver

* add pipeline test to the CI

* clean config

* add rulegraph

* update rule names

* pipeline stable

* add bash env

* clean template

* test integration md5 check added

* add description

* add workflow execution notes

* build: add one step to merge all MotEvo results at the end

* Combine results in one tsv file

* fix tests

* Use shell instead of script

* Add csv version

* Add unit testing

* Add Combine_motevo_results rule description

* Add comments in parse.py

* Remove print statements

* Replace fast with fasta

* Update expected_output.txt

* Add pip command

* Remove failing command

* Update checksum values

* Correct path

* Add licence

* update values

* Make suggested changes

* Update travis

* Remove log files

* Make changes in tsv table

* Add comments

* Change indir to input_directory

* Update tests

* Clean the code and add more comments

* Add env file

* Change path of parse.py

* Update travis

* Update travis and add new image

* Correct typo

* rename python script

* adjust newlines

* update conda env name

* update script name

* adjust newlines

* minor formatting

Co-authored-by: Maciej Bak <[email protected]>

* refactor: fix encoding paths to pipeline scripts

* --force flag does not work properly - remove it

* add config field with path to pipeline directory

* fix syntax

* fix cwd for integration test

* fix cwd for integration test

* fix cwd for integration test

* update gitignore, change pipeline dir for integration test

* refactor: sort binding sites in the final TSV table by the probability

* build: Add heatmap with binding probability

* Add heatmap and heatmap script

* Make suggested changes

* make grid square, grey background white and sep cells

* Replace value with bindin-probability

* Make env file for the script

* Add unit and integration tests

* Add R to travis.yml

* Add R libraries in dev.yml

* add r in dev

* Make a seperate directory for heatmap

* Add comments to heatmap script

* Add new snakemake image and make changes to travis

* Change variable names and add new lines

* Use svg instead of pdf and create virtual environments in travis

* Remove motevo conda

* Remove unncessary newline

* Update path of envs in travis

* test md5 checksum Maciek

* Correct unit test to produce svg file

* change plot format to PNG

* set SVG format for the heatmap

* corr typo

* turn off plots md5 test

* minor styling

Co-authored-by: Maciej Bak <[email protected]>

* build: add plotting sequence logos for all input motifs

* Fix heatmap.r

* Add sequence_logos.py

* Add to snakemake pipeline

* Add comments to script

* Add unit tests

* Add integration tests

* Modify expected_results.tx

* Change image of snakemake pipeline

* Formatted script using black

* fix tests

* Link sequence logo to create rule

* Make suggested changes

* Remove expand from output and params

* ChEdit unit tests md5 sum

* Fix indentation

* Replace T with U

* Edit unit tests md5sum

* Edit unit and integration md5sum tests

* Delete output files from unit tests

* Add code to replace T with U

* Update md5sums

* update md5 sums

* correct all indentations

* removed 2 unnecessary expand functions

* add missing benchmark section to 3 rules

* add missing log section to 3 rules

Co-authored-by: Maciej Bak <[email protected]>

* build: add sequence logos to heatmap

* Add sequence logos to heatmap

* Pass directory in command line instead of hard coding

* Add final heatmap in unit test for review

* Correct final heatmap

* Replace T with U in input sequence

* update md5sums

* minor aesthetics

Co-authored-by: Maciej Bak <[email protected]>

* refactor: change heatmap's format from SVG to PDF

* build: add SLURM cluster execution

Co-authored-by: Maciej Bak <[email protected]>

* refactor: distribute seq.logos plotting as separate jobs per motif

* Fix sequence logo bug

* Add png logos as input to heatmap

* Edit travis file

* Fix tests

* fix: logomaker plotting issue and filename parsing issue

* fix: increase heatmap visibility

* docs: update pipeline documentation

* Add documentation for pipeline

* Make suggested changes

* Correct filename in rule plot_heatmap_of_MotEvo_results

* ci: add separate travis build on osx

* specify linux OS

* specify anaconda download link

* specify anaconda download link

* add building on osx

* diversify apt/brew update

* ci: add pipeline testing inside singularity containers

* Install singularity 3.6.2 in travis

* Add singularity in Snakemake file

* Add singularity script to travis

* Fix typo

* Update travis file

* run singularity tests for linux

* Install packages

* DO not upgrade and install git

* Add separate cheksum for singularity in travis

* rebuild CI

* rebuild CI v2

* rebuild CI v3

Co-authored-by: Maciej Bak <[email protected]>

* build: upgrade snakemake ver.

* update Snakemake ver to 5.23

* add pulp package

* docs: update pipeline documentation

* docs: refactor README for ver 1.0.0

* add git installation info

* add ATtRACT WMs

* correct ATtRACT in CI

* update installation info

* specify info about paths

* docs: corrections

* docs: typo

* build: bump snakemake version

* build: snakemake ver 5.24.2

* docs: Typo fix

* Small fixes the the phrasing.

* Expanded the 'General information' section

* docs: add info on output files

Co-authored-by: Mihaela Zavolan <[email protected]>

* ci: turn off the singularity test

Co-authored-by: Krish Agarwal <[email protected]>
Co-authored-by: Maciej Bak <[email protected]>
Co-authored-by: mzavolan <[email protected]>
Co-authored-by: Mihaela Zavolan <[email protected]>

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,38 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: ''
+labels: ''
+assignees: ''
+
+---
+
+**Describe the bug**
+A clear and concise description of what the bug is.
+
+**To Reproduce**
+Steps to reproduce the behavior:
+1. Go to '...'
+2. Click on '....'
+3. Scroll down to '....'
+4. See error
+
+**Expected behavior**
+A clear and concise description of what you expected to happen.
+
+**Screenshots**
+If applicable, add screenshots to help explain your problem.
+
+**Desktop (please complete the following information):**
+ - OS: [e.g. iOS]
+ - Browser [e.g. chrome, safari]
+ - Version [e.g. 22]
+
+**Smartphone (please complete the following information):**
+ - Device: [e.g. iPhone6]
+ - OS: [e.g. iOS8.1]
+ - Browser [e.g. stock browser, safari]
+ - Version [e.g. 22]
+
+**Additional context**
+Add any other context about the problem here.

.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,20 @@
+---
+name: Feature request
+about: Suggest an idea for this project
+title: ''
+labels: ''
+assignees: ''
+
+---
+
+**Is your feature request related to a problem? Please describe.**
+A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+
+**Describe the solution you'd like**
+A clear and concise description of what you want to happen.
+
+**Describe alternatives you've considered**
+A clear and concise description of any alternative solutions or features you've considered.
+
+**Additional context**
+Add any other context or screenshots about the feature request here.

.gitignore

@@ -0,0 +1,10 @@
+# all backup emacs and vim files
+*~
+
+# mac files
+.DS_Store
+._*
+
+# snakemake internal directories for tests
+tests/integration/execution/.snakemake/
+.snakemake/

.travis.yml

@@ -0,0 +1,124 @@
+language: bash
+
+os:
+  - linux
+  - osx
+
+install:
+  - if [ "$TRAVIS_OS_NAME" = "linux" ]; then sudo apt update; fi
+  - if [ "$TRAVIS_OS_NAME" = "linux" ]; then wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh; fi
+  - if [ "$TRAVIS_OS_NAME" = "osx" ]; then brew update; fi
+  - if [ "$TRAVIS_OS_NAME" = "osx" ]; then wget https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-x86_64.sh -O miniconda.sh; fi
+  - bash miniconda.sh -b -p $HOME/miniconda
+  - source "$HOME/miniconda/etc/profile.d/conda.sh"
+  - hash -r
+  - conda config --set always_yes yes --set changeps1 no
+  - conda update -q conda
+  # Useful for debugging any issues with conda
+  - conda info -a
+  # Install singularity
+  - if [ "$TRAVIS_OS_NAME" = "linux" ]; then sudo apt-get install -y wget; fi
+  - if [ "$TRAVIS_OS_NAME" = "linux" ]; then sudo apt-get install -y build-essential; fi
+  - if [ "$TRAVIS_OS_NAME" = "linux" ]; then sudo apt-get install -y uuid-dev ; fi
+  - if [ "$TRAVIS_OS_NAME" = "linux" ]; then sudo apt-get install -y squashfs-tools; fi
+  - if [ "$TRAVIS_OS_NAME" = "linux" ]; then sudo apt-get install -y libseccomp-dev; fi
+  - if [ "$TRAVIS_OS_NAME" = "linux" ]; then sudo apt-get install -y pkg-config; fi
+  - if [ "$TRAVIS_OS_NAME" = "linux" ]; then sudo apt-get install -y cryptsetup-bin; fi
+  - if [ "$TRAVIS_OS_NAME" = "linux" ]; then eval "$(gimme 1.13.1)"; fi
+  - if [ "$TRAVIS_OS_NAME" = "linux" ]; then go version; fi
+  - if [ "$TRAVIS_OS_NAME" = "linux" ]; then export VERSION=3.5.2; fi
+  - if [ "$TRAVIS_OS_NAME" = "linux" ]; then wget https://github.com/sylabs/singularity/releases/download/v${VERSION}/singularity-${VERSION}.tar.gz; fi
+  - if [ "$TRAVIS_OS_NAME" = "linux" ]; then tar -xzf singularity-${VERSION}.tar.gz; fi
+  - if [ "$TRAVIS_OS_NAME" = "linux" ]; then cd singularity; fi
+  - if [ "$TRAVIS_OS_NAME" = "linux" ]; then ./mconfig; fi
+  - if [ "$TRAVIS_OS_NAME" = "linux" ]; then make -C ./builddir; fi
+  - if [ "$TRAVIS_OS_NAME" = "linux" ]; then sudo make -C ./builddir install; fi
+  - if [ "$TRAVIS_OS_NAME" = "linux" ]; then singularity --version; fi
+  - if [ "$TRAVIS_OS_NAME" = "linux" ]; then cd ..; fi
+
+before_script:
+  # Create conda virtual environments
+  - conda env create -f envs/main.yml
+  - conda env create -f envs/dev.yml
+  - conda env create -f workflow/envs/plot_heatmap_of_MotEvo_results.yml
+  - conda env create -f workflow/envs/combine-motevo-results.yml
+  - conda env create -f workflow/envs/plot_sequence_logos.yml
+  # Activate the main env
+  - conda activate binding-scanner && echo $CONDA_DEFAULT_ENV
+
+script:
+  # Download and extract the ATtRACT db
+  #- bash scripts/download-ATtRACT-motifs.sh -o ATtRACT # this curl/wget do not work on Travis machine...
+  # Extract the backup db
+  - mkdir ATtRACT_backup_26082020
+  - unzip resources/ATtRACT_backup_26082020.zip -d ATtRACT_backup_26082020
+  # Extract motifs for Homo sapiens
+  - mkdir tests/unit/format-ATtRACT-motifs/ATtRACT_hsa
+  - >
+    python scripts/format-ATtRACT-motifs.py
+    --pwms tests/unit/format-ATtRACT-motifs/ATtRACT/pwm.txt
+    --names tests/unit/format-ATtRACT-motifs/ATtRACT/ATtRACT_db.txt
+    --organism Homo_sapiens
+    --outdir tests/unit/format-ATtRACT-motifs/ATtRACT_hsa
+  - md5sum --check tests/unit/format-ATtRACT-motifs/expected_output_hsa.md5
+  # Extract motifs for Mus musculus
+  - mkdir tests/unit/format-ATtRACT-motifs/ATtRACT_mmu
+  - >
+    python scripts/format-ATtRACT-motifs.py
+    --pwms tests/unit/format-ATtRACT-motifs/ATtRACT/pwm.txt
+    --names tests/unit/format-ATtRACT-motifs/ATtRACT/ATtRACT_db.txt
+    --organism Mus_musculus
+    --outdir tests/unit/format-ATtRACT-motifs/ATtRACT_mmu
+  - md5sum --check tests/unit/format-ATtRACT-motifs/expected_output_mmu.md5
+  # Test sequence_logos.py script
+  - conda activate plot_sequence_logos
+  - >
+    python workflow/scripts/sequence_logos.py 
+    --input_file tests/unit/plot_sequence_logos/motif_HNRNPF_820
+    --output_location tests/unit/plot_sequence_logos
+  - >
+    python workflow/scripts/sequence_logos.py 
+    --input_file tests/unit/plot_sequence_logos/motif_HNRNPF_821
+    --output_location tests/unit/plot_sequence_logos
+  - >
+    python workflow/scripts/sequence_logos.py 
+    --input_file tests/unit/plot_sequence_logos/motif_HNRNPF_822
+    --output_location tests/unit/plot_sequence_logos
+  - >
+    python workflow/scripts/sequence_logos.py 
+    --input_file tests/unit/plot_sequence_logos/motif_HNRNPF_823
+    --output_location tests/unit/plot_sequence_logos
+  - >
+    python workflow/scripts/sequence_logos.py 
+    --input_file tests/unit/plot_sequence_logos/motif_HNRNPF_824
+    --output_location tests/unit/plot_sequence_logos
+  - md5sum --check tests/unit/plot_sequence_logos/expected_output.md5
+  # Test combine-motevo-results.py script
+  - conda activate combine-motevo-results
+  - >
+    python workflow/scripts/combine-motevo-results.py
+    --input_directories tests/unit/combine_results/motif_HNRNPF_820 tests/unit/combine_results/motif_HNRNPF_821 tests/unit/combine_results/motif_HNRNPF_822 tests/unit/combine_results/motif_HNRNPF_823 tests/unit/combine_results/motif_HNRNPF_824
+    --filename posterior_sites
+    --outfile tests/unit/combine_results/combined_MotEvo_results.tsv
+  - md5sum --check tests/unit/combine_results/expected_output.md5
+  # Test heatmap.r script
+  - conda activate plot_heatmap_of_MotEvo_results
+  - >
+    Rscript workflow/scripts/heatmap.r 
+    --input_tsv tests/unit/Plot-heatmap-for-motifs/combined_MotEvo_results.tsv 
+    --input_sequence ATGTGAGTGAAGTGTGGGAAAGATGACTCGATATATCTGGATGCTAGGGATCGGATGGCGATACG
+    --outfile tests/unit/Plot-heatmap-for-motifs/ProbabilityvsSequences.pdf
+    --sequence_logos_directory tests/unit/Plot-heatmap-for-motifs/sequence_logos
+  #- md5sum --check tests/unit/Plot-heatmap-for-motifs/expected_output.md5
+  - conda activate binding-scanner
+  # Test snakemake Rulegraph and DAG
+  - bash tests/integration/execution/snakemake_rulegraph_run.sh
+  - bash tests/integration/execution/snakemake_dag_run.sh
+  # Test pipeline execution: local, conda envs:
+  - bash tests/integration/execution/snakemake_local_run_conda_environments.sh
+  - md5sum --check tests/integration/expected_output.md5
+  - rm -rf tests/integration/output/
+  # On Linux: Test pipeline execution: local, singularity containers:
+  # - if [ "$TRAVIS_OS_NAME" = "linux" ]; then bash tests/integration/execution/snakemake_local_run_singularity_environments.sh; fi
+  # - if [ "$TRAVIS_OS_NAME" = "linux" ]; then md5sum --check tests/integration/expected_output_singularity.md5; fi
+  # - if [ "$TRAVIS_OS_NAME" = "linux" ]; then rm -rf tests/integration/output/; fi

CODE_OF_CONDUCT.md

@@ -0,0 +1,73 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+In the interest of fostering an open and welcoming environment, we as
+contributors and maintainers pledge to making participation in our project and
+our community a harassment-free experience for everyone, regardless of age, body
+size, disability, ethnicity, sex characteristics, gender identity and expression,
+level of experience, education, socio-economic status, nationality, personal
+appearance, race, religion, or sexual identity and orientation.
+
+## Our Standards
+
+Examples of behavior that contributes to creating a positive environment
+include:
+
+* Using welcoming and inclusive language
+* Being respectful of differing viewpoints and experiences
+* Gracefully accepting constructive criticism
+* Focusing on what is best for the community
+* Showing empathy towards other community members
+
+Examples of unacceptable behavior by participants include:
+
+* The use of sexualized language or imagery and unwelcome sexual attention or
+  advances
+* Trolling, insulting/derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or electronic
+  address, without explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+  professional setting
+
+## Our Responsibilities
+
+Project maintainers are responsible for clarifying the standards of acceptable
+behavior and are expected to take appropriate and fair corrective action in
+response to any instances of unacceptable behavior.
+
+Project maintainers have the right and responsibility to remove, edit, or
+reject comments, commits, code, wiki edits, issues, and other contributions
+that are not aligned to this Code of Conduct, or to ban temporarily or
+permanently any contributor for other behaviors that they deem inappropriate,
+threatening, offensive, or harmful.
+
+## Scope
+
+This Code of Conduct applies both within project spaces and in public spaces
+when an individual is representing the project or its community. Examples of
+representing a project or community include using an official project e-mail
+address, posting via an official social media account, or acting as an appointed
+representative at an online or offline event. Representation of a project may be
+further defined and clarified by project maintainers.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported by contacting the project team at <[email protected]>. All complaints will be
+reviewed and investigated and will result in a response that is deemed necessary
+and appropriate to the circumstances. The project team is obligated to maintain
+confidentiality with regard to the reporter of an incident. Further details of
+specific enforcement policies may be posted separately.
+
+Project maintainers who do not follow or enforce the Code of Conduct in good
+faith may face temporary or permanent repercussions as determined by other
+members of the project's leadership.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
+available at <https://www.contributor-covenant.org/version/1/4/code-of-conduct.html>
+
+[homepage]: https://www.contributor-covenant.org