Dapt data curation tutorial fuzzy and semantic dedupe

[ruchaa-apte]

Description

The PR adds

• Versions for installations edits in requirements.txt for pip packages
• fuzzy_dedupe module implementation to DAPT data curator tutorial for chipNeMo. It performs Fuzzy Deduplication on code as well as text documents from Wikipedia and Arxiv sources.
• semantic_dedupe module implementation to DAPT data curator tutorial for chipNeMo. It performs Semantic Deduplication on text documents from Wikipedia and Arxiv sources.

Usage

cd NeMo-Curator/tutorials/dapt-curation/code/
python main.py

Checklist

• [ X] I am familiar with the Contributing Guide.
• [ X] New or Existing tests cover these changes.
• [ X] The documentation is up to date with these changes.

[ruchaa-apte]

pre-commit.ci autofix

[ayushdg]

Looks like the commits are not gpg signed. You might need to recommit your changes with commit signing enabled -Ss flags during git commit. -S for signing and -s for sigoff.
More info on signing commits here: https://docs.github.com/en/authentication/managing-commit-signature-verification/signing-commits

[ruchaa-apte]

pre-commit.ci autofix

[ruchaa-apte]

Looks like the commits are not gpg signed. You might need to recommit your changes with commit signing enabled -Ss flags during git commit. -S for signing and -s for sigoff. More info on signing commits here: https://docs.github.com/en/authentication/managing-commit-signature-verification/signing-commits

Hi Ayush, thank you for pointing that out. Looks like me email address linked to signing off was not correct. I modified it and signed off the commit.

[ruchaa-apte]

pre-commit.ci autofix

[ruchaa-apte]

pre-commit.ci autofix