Special Tokens (#2)

* add special tokens * special tokens * pylint * flake8
asyml · Apr 6, 2022 · 94eed1f · 94eed1f
1 parent f0718f0
commit 94eed1f
Show file tree

Hide file tree

Showing 2 changed files with 15 additions and 0 deletions.
diff --git a/README.md b/README.md
@@ -5,6 +5,7 @@ Some shared utilities across multiple ASYML projects.
 
 ## List of Modules
 * `asyml_utilities.hyperparams.HParams`
+* `asyml_utilities.special_tokens.SpecialTokens`
 
 ## License
 [Apache License 2.0](./LICENSE)
diff --git a/asyml_utilities/special_tokens.py b/asyml_utilities/special_tokens.py
@@ -0,0 +1,14 @@
+__all__ = [
+    'SpecialTokens',
+]
+
+
+class SpecialTokens:
+    r"""Special tokens, including :attr:`PAD`, :attr:`BOS`, :attr:`EOS`,
+    :attr:`UNK`. These tokens will by default have token ids 0, 1, 2, 3,
+    respectively.
+    """
+    PAD = "<PAD>"
+    BOS = "<BOS>"
+    EOS = "<EOS>"
+    UNK = "<UNK>"