Toknizers for my research
To get the source described below, simply run make get-src.
(However, you still have to build on your own for mecab and kytea.)
Clone https://github.com/moses-smt/mosesdecoder into this folder.
For Japanese tokenization, download and build KyTea(http://www.phontron.com/kytea/index-ja.html) in this folder.
This is alos a Japanese tokenizer. (http://taku910.github.io/mecab/)