/h[2SrSSKrSSKJr "SS\5rg)a The tok-tok tokenizer is a simple, general tokenizer, where the input has one sentence per line; thus only final period is tokenized. Tok-tok has been tested on, and gives reasonably good results for English, Persian, Russian, Czech, French, German, Vietnamese, Tajik, and a few others. The input should be in UTF-8 encoding. Reference: Jon Dehdari. 2014. A Neurophysiologically-Inspired Statistical Language Model (Doctoral dissertation). Columbus, OH, USA: The Ohio State University. N) TokenizerIc\rSrSrSr\R "S5S4r\R "S5S4r\R "S5S4r \R "S5S4r \R "S 5S 4r \R "S 5S 4r \R "S 5S4r \R "S5S4r\R "S5S4r\R "S5S4r\R "S5S4r\R "S5S4r\R "S5S4r\R "S5S4r\R "S5S4r\R "S5S4r\"S5r\"S5r\"S5r\R "S\S 35S!4r\R "S\S 35S!4r\R "S\S 35S!4r\R "S"5S#4r\R "S$5S%4r\R "S&5S'4r \R "S(5S'4r!\R "S)5S*4r"\R "S+5S,4r#\R "S-5S4r$\\\ \\\ \!\ \ \ \\\\\\\\\ \\\\\$/r%S1S.jr&S/r'g0)2ToktokTokenizeru This is a Python port of the tok-tok.pl from https://github.com/jonsafari/tok-tok/blob/master/tok-tok.pl >>> toktok = ToktokTokenizer() >>> text = u'Is 9.5 or 525,600 my favorite number?' >>> print(toktok.tokenize(text, return_str=True)) Is 9.5 or 525,600 my favorite number ? >>> text = u'The https://github.com/jonsafari/tok-tok/blob/master/tok-tok.pl is a website with/and/or slashes and sort of weird : things' >>> print(toktok.tokenize(text, return_str=True)) The https://github.com/jonsafari/tok-tok/blob/master/tok-tok.pl is a website with/and/or slashes and sort of weird : things >>> text = u'¡This, is a sentence with weird» symbols… appearing everywhere¿' >>> expected = u'¡ This , is a sentence with weird » symbols … appearing everywhere ¿' >>> assert toktok.tokenize(text, return_str=True) == expected >>> toktok.tokenize(text) == [u'¡', u'This', u',', u'is', u'a', u'sentence', u'with', u'weird', u'»', u'symbols', u'…', u'appearing', u'everywhere', u'¿'] True   u1([،;؛¿!"\])}»›”؟¡%٪°±©®।॥…])z \1 u([({\[“‘„‚«‹「『])u ([–—])z& z&  z z\|z | u(?rBs   (W4jW4r