This repository defines a Lua wrapper for sentencepiece.

Installation

1. Install Lua development libraries (optional if Torch is installed):

On Ubuntu, they can be installed with apt-get:

sudo apt-get install lua5.1 liblua5.1-dev

2. Install sentencepiece

3. Install the wrapper:

luarocks make lua-sentencepiece-scm-1.rockspec

Test and use

$(which luajit lua | head -n 1) test.lua

The module creates a sentencepiece namespace with 2 functions:

• sentencepiece.encode(model, string): encodes the string (convert into pieces table)
• sentencepiece.decode(model, table): builds a string from the table of piece