Predefined tokenizers.
getTokenizers()
A character vector with tokenizers provided by package tm.
MC_tokenizer and scan_tokenizer.
MC_tokenizer
scan_tokenizer