Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci: Add coverage reports #11912

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from
+25 −23
Draft

ci: Add coverage reports #11912

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ff06066
ci: Add coverage reports
ko3n1g Jan 21, 2025
b50295c
ci: Update CODEOWNERS
ko3n1g Jan 21, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/CODEOWNERS
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
.github/ @pablo-garay @ko3n1g
Dockerfile.ci @pablo-garay @ko3n1g
.github/ @pablo-garay @ko3n1g @thomasdhc @chtruong814
Dockerfile.ci @pablo-garay @ko3n1g @thomasdhc @chtruong814
43 changes: 22 additions & 21 deletions .github/workflows/cicd-main.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -107,7 +107,7 @@ jobs:
TIMEOUT: 20
# TODO: remove this hack
SCRIPT: |
python -c "from nemo.collections.asr.models import ASRModel" && NEMO_NUMBA_MINVER=0.53 pytest tests/collections/asr -m "not pleasefixme" --with_downloads
python -c "from nemo.collections.asr.models import ASRModel" && NEMO_NUMBA_MINVER=0.53 pytest tests/collections/asr -m "not pleasefixme" --with_downloads --cov-report=term --cov-report=html --cov=nemo/collections/asr

L0_Unit_Tests_GPU_Audio:
needs: [cicd-test-container-setup]
Expand All @@ -117,7 +117,7 @@ jobs:
RUNNER: self-hosted-azure
TIMEOUT: 20
SCRIPT: |
NEMO_NUMBA_MINVER=0.53 pytest tests/collections/audio -m "not pleasefixme" --with_downloads
NEMO_NUMBA_MINVER=0.53 pytest tests/collections/audio -m "not pleasefixme" --with_downloads --cov-report=term --cov-report=html --cov=nemo/collections/audio

L0_Unit_Tests_GPU_Common:
needs: [cicd-test-container-setup]
Expand All @@ -126,7 +126,7 @@ jobs:
with:
RUNNER: self-hosted-azure
SCRIPT: |
NEMO_NUMBA_MINVER=0.53 pytest tests/collections/common -m "not pleasefixme" --with_downloads
NEMO_NUMBA_MINVER=0.53 pytest tests/collections/common -m "not pleasefixme" --with_downloads --cov-report=term --cov-report=html --cov=nemo/collections/common

L0_Unit_Tests_GPU_LLM:
needs: [cicd-test-container-setup]
Expand All @@ -135,7 +135,7 @@ jobs:
with:
RUNNER: self-hosted-azure
SCRIPT: |
NEMO_NUMBA_MINVER=0.53 pytest tests/collections/llm -m "not pleasefixme" --with_downloads
NEMO_NUMBA_MINVER=0.53 pytest tests/collections/llm -m "not pleasefixme" --with_downloads --cov-report=term --cov-report=html --cov=nemo/collections/llm

L0_Unit_Tests_GPU_Multimodal:
needs: [cicd-test-container-setup]
Expand All @@ -144,7 +144,7 @@ jobs:
with:
RUNNER: self-hosted-azure
SCRIPT: |
NEMO_NUMBA_MINVER=0.53 pytest tests/collections/multimodal -m "not pleasefixme" --with_downloads
NEMO_NUMBA_MINVER=0.53 pytest tests/collections/multimodal -m "not pleasefixme" --with_downloads--cov-report=term --cov-report=html --cov=nemo/collections/multimodal

L0_Unit_Tests_GPU_NLP:
needs: [cicd-test-container-setup]
Expand All @@ -153,7 +153,7 @@ jobs:
with:
RUNNER: self-hosted-azure
SCRIPT: |
NEMO_NUMBA_MINVER=0.53 pytest tests/collections/nlp -m "not pleasefixme" --with_downloads
NEMO_NUMBA_MINVER=0.53 pytest tests/collections/nlp -m "not pleasefixme" --with_downloads--cov-report=term --cov-report=html --cov=nemo/collections/nlp

L0_Unit_Tests_GPU_TTS:
needs: [cicd-test-container-setup]
Expand All @@ -162,7 +162,7 @@ jobs:
with:
RUNNER: self-hosted-azure
SCRIPT: |
NEMO_NUMBA_MINVER=0.53 pytest tests/collections/tts -m "not pleasefixme" --with_downloads
NEMO_NUMBA_MINVER=0.53 pytest tests/collections/tts -m "not pleasefixme" --with_downloads--cov-report=term --cov-report=html --cov=nemo/collections/tts

OPTIONAL_L0_Unit_Tests_GPU_Core:
needs: [cicd-test-container-setup]
Expand All @@ -172,7 +172,7 @@ jobs:
RUNNER: self-hosted-azure
TIMEOUT: 20
SCRIPT: |
NEMO_NUMBA_MINVER=0.53 pytest tests/core -m "not pleasefixme" --with_downloads
NEMO_NUMBA_MINVER=0.53 pytest tests/core -m "not pleasefixme" --with_downloads --cov-report=term --cov-report=html --cov=nemo/core
IS_OPTIONAL: true

L0_Unit_Tests_GPU_Hydra:
Expand All @@ -182,7 +182,7 @@ jobs:
with:
RUNNER: self-hosted-azure
SCRIPT: |
NEMO_NUMBA_MINVER=0.53 pytest tests/hydra -m "not pleasefixme" --with_downloads
NEMO_NUMBA_MINVER=0.53 pytest tests/hydra -m "not pleasefixme" --with_downloads --cov-report=term --cov-report=html --cov=nemo/core/config

L0_Unit_Tests_GPU_Lightning:
needs: [cicd-test-container-setup]
Expand All @@ -191,7 +191,7 @@ jobs:
with:
RUNNER: self-hosted-azure
SCRIPT: |
NEMO_NUMBA_MINVER=0.53 pytest tests/lightning -m "not pleasefixme" --with_downloads
NEMO_NUMBA_MINVER=0.53 pytest tests/lightning -m "not pleasefixme" --with_downloads --cov-report=term --cov-report=html --cov=nemo/lightning

L0_Unit_Tests_GPU_Others:
needs: [cicd-test-container-setup]
Expand All @@ -212,7 +212,7 @@ jobs:
--ignore=tests/core_ptl \
--ignore=tests/hydra \
--ignore=tests/lightning \
--ignore=tests/utils
--ignore=tests/utils --cov-report=term --cov-report=html --cov=nemo

# L0: CPU unit tests
L0_Unit_Tests_CPU_ASR:
Expand All @@ -232,7 +232,7 @@ jobs:
with:
RUNNER: self-hosted-azure-cpu
SCRIPT: |
CUDA_VISIBLE_DEVICES="" NEMO_NUMBA_MINVER=0.53 pytest tests/collections/audio -m "not pleasefixme" --cpu --with_downloads --relax_numba_compat
CUDA_VISIBLE_DEVICES="" NEMO_NUMBA_MINVER=0.53 pytest tests/collections/audio -m "not pleasefixme" --cpu --with_downloads --relax_numba_compat --cov-report=term --cov-report=html --cov=nemo/collections/audio

L0_Unit_Tests_CPU_Common:
needs: [cicd-test-container-setup]
Expand All @@ -242,7 +242,7 @@ jobs:
RUNNER: self-hosted-azure-cpu
TIMEOUT: 20
SCRIPT: |
CUDA_VISIBLE_DEVICES="" NEMO_NUMBA_MINVER=0.53 pytest tests/collections/common -m "not pleasefixme" --cpu --with_downloads --relax_numba_compat
CUDA_VISIBLE_DEVICES="" NEMO_NUMBA_MINVER=0.53 pytest tests/collections/common -m "not pleasefixme" --cpu --with_downloads --relax_numba_compat --cov-report=term --cov-report=html --cov=nemo/collections/common

L0_Unit_Tests_CPU_LLM:
needs: [cicd-test-container-setup]
Expand All @@ -251,7 +251,7 @@ jobs:
with:
RUNNER: self-hosted-azure-cpu
SCRIPT: |
CUDA_VISIBLE_DEVICES="" NEMO_NUMBA_MINVER=0.53 pytest tests/collections/llm -m "not pleasefixme" --cpu --with_downloads --relax_numba_compat
CUDA_VISIBLE_DEVICES="" NEMO_NUMBA_MINVER=0.53 pytest tests/collections/llm -m "not pleasefixme" --cpu --with_downloads --relax_numba_compat --cov-report=term --cov-report=html --cov=nemo/collections/llm

L0_Unit_Tests_CPU_Multimodal:
needs: [cicd-test-container-setup]
Expand All @@ -260,7 +260,7 @@ jobs:
with:
RUNNER: self-hosted-azure-cpu
SCRIPT: |
CUDA_VISIBLE_DEVICES="" NEMO_NUMBA_MINVER=0.53 pytest tests/collections/multimodal -m "not pleasefixme" --cpu --with_downloads --relax_numba_compat
CUDA_VISIBLE_DEVICES="" NEMO_NUMBA_MINVER=0.53 pytest tests/collections/multimodal -m "not pleasefixme" --cpu --with_downloads --relax_numba_compat --cov-report=term --cov-report=html --cov=nemo/collections/multimodal

L0_Unit_Tests_CPU_NLP:
needs: [cicd-test-container-setup]
Expand All @@ -270,7 +270,7 @@ jobs:
RUNNER: self-hosted-azure
TIMEOUT: 20
SCRIPT: |
CUDA_VISIBLE_DEVICES="" NEMO_NUMBA_MINVER=0.53 pytest tests/collections/nlp -m "not pleasefixme" --cpu --with_downloads --relax_numba_compat
CUDA_VISIBLE_DEVICES="" NEMO_NUMBA_MINVER=0.53 pytest tests/collections/nlp -m "not pleasefixme" --cpu --with_downloads --relax_numba_compat --cov-report=term --cov-report=html --cov=nemo/collections/nlp

L0_Unit_Tests_CPU_TTS:
needs: [cicd-test-container-setup]
Expand All @@ -279,7 +279,7 @@ jobs:
with:
RUNNER: self-hosted-azure-cpu
SCRIPT: |
CUDA_VISIBLE_DEVICES="" NEMO_NUMBA_MINVER=0.53 pytest tests/collections/tts -m "not pleasefixme" --cpu --with_downloads --relax_numba_compat
CUDA_VISIBLE_DEVICES="" NEMO_NUMBA_MINVER=0.53 pytest tests/collections/tts -m "not pleasefixme" --cpu --with_downloads --relax_numba_compat --cov-report=term --cov-report=html --cov=nemo/collections/tts

L0_Unit_Tests_CPU_Core:
needs: [cicd-test-container-setup]
Expand All @@ -289,7 +289,7 @@ jobs:
RUNNER: self-hosted-azure-cpu
TIMEOUT: 20
SCRIPT: |
CUDA_VISIBLE_DEVICES="" NEMO_NUMBA_MINVER=0.53 pytest tests/core tests/core_ptl -m "not pleasefixme" --cpu --with_downloads --relax_numba_compat
CUDA_VISIBLE_DEVICES="" NEMO_NUMBA_MINVER=0.53 pytest tests/core tests/core_ptl -m "not pleasefixme" --cpu --with_downloads --relax_numba_compat --cov-report=term --cov-report=html --cov=nemo/core

L0_Unit_Tests_CPU_Hydra:
needs: [cicd-test-container-setup]
Expand All @@ -298,7 +298,7 @@ jobs:
with:
RUNNER: self-hosted-azure-cpu
SCRIPT: |
CUDA_VISIBLE_DEVICES="" NEMO_NUMBA_MINVER=0.53 pytest tests/hydra -m "not pleasefixme" --cpu --with_downloads --relax_numba_compat
CUDA_VISIBLE_DEVICES="" NEMO_NUMBA_MINVER=0.53 pytest tests/hydra -m "not pleasefixme" --cpu --with_downloads --relax_numba_compat --cov-report=term --cov-report=html --cov=nemo/core/config

L0_Unit_Tests_CPU_Lightning:
needs: [cicd-test-container-setup]
Expand All @@ -307,7 +307,7 @@ jobs:
with:
RUNNER: self-hosted-azure-cpu
SCRIPT: |
CUDA_VISIBLE_DEVICES="" NEMO_NUMBA_MINVER=0.53 pytest tests/lightning -m "not pleasefixme" --cpu --with_downloads --relax_numba_compat
CUDA_VISIBLE_DEVICES="" NEMO_NUMBA_MINVER=0.53 pytest tests/lightning -m "not pleasefixme" --cpu --with_downloads --relax_numba_compat --cov-report=term --cov-report=html --cov=nemo/lightning

L0_Unit_Tests_CPU_Others:
needs: [cicd-test-container-setup]
Expand All @@ -328,7 +328,8 @@ jobs:
--ignore=tests/core_ptl \
--ignore=tests/hydra \
--ignore=tests/lightning \
--ignore=tests/utils
--ignore=tests/utils \
--cov-report=term --cov-report=html --cov=nemo


L0_Setup_Test_Data_And_Models:
Expand Down
1 change: 1 addition & 0 deletions requirements/requirements_test.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ click>=8.1
isort>5.1.0,<6.0.0
parameterized
pytest
pytest-coverage
pytest-mock
pytest-runner
ruamel.yaml
Expand Down
Loading