Package: tokenizers.bpe
Type: Package
Title: Byte Pair Encoding Text Tokenization
Version: 0.1.0
Authors@R: c(person('Jan', 'Wijffels', role = c('aut', 'cre', 'cph'), email = 'jwijffels@bnosac.be', comment = "R wrapper"), 
    person('BNOSAC', role = 'cph', comment = "R wrapper"), 
    person('VK.com', role = 'cph'), 
    person('Gregory Popovitch', role = c('ctb', 'cph'), comment = "Files at src/parallel_hashmap (Apache License, Version 2.0"), 
    person('The Abseil Authors', role = c('ctb', 'cph'), comment = "Files at src/parallel_hashmap (Apache License, Version 2.0"),
    person('Ivan Belonogov', role = c('ctb', 'cph'), email = 'xbelonogov@gmail.com', comment = "Files at src/youtokentome (MIT License)"))
Maintainer: Jan Wijffels <jwijffels@bnosac.be>
Description: Unsupervised text tokenizer focused on computational efficiency. Wraps the 'YouTokenToMe' library <https://github.com/VKCOM/YouTokenToMe> which is an implementation of fast Byte Pair Encoding (BPE) <https://www.aclweb.org/anthology/P16-1162>.
URL: https://github.com/bnosac/tokenizers.bpe
License: MPL-2.0
Encoding: UTF-8
LazyData: true
RoxygenNote: 6.1.1
Depends: R (>= 2.10)
Imports: Rcpp (>= 0.11.5)
LinkingTo: Rcpp
NeedsCompilation: yes
Packaged: 2019-07-31 21:02:18 UTC; Jan
Author: Jan Wijffels [aut, cre, cph] (R wrapper),
  BNOSAC [cph] (R wrapper),
  VK.com [cph],
  Gregory Popovitch [ctb, cph] (Files at src/parallel_hashmap (Apache
    License, Version 2.0),
  The Abseil Authors [ctb, cph] (Files at src/parallel_hashmap (Apache
    License, Version 2.0),
  Ivan Belonogov [ctb, cph] (Files at src/youtokentome (MIT License))
Repository: CRAN
Date/Publication: 2019-08-02 09:40:02 UTC
Built: R 4.2.0; x86_64-conda-linux-gnu; 2022-06-04 18:38:34 UTC; unix
