stri_enc_detect2 {stringi} | R Documentation

## [DRAFT API] Detect Locale-Sensitive Character Encoding

### Description

This function tries to detect character encoding in case the language of text is known.

[THIS IS AN EXPERIMENTAL FUNCTION]

### Usage

    stri_enc_detect2(str, locale = NULL)

### Arguments

`str` | character vector, a raw vector, or a list of `raw` vectors
`locale` | `NULL` or `""` for default locale, `NA` for just checking the UTF-* family, or a single string with locale identifier.

### Details

Vectorized over `str`.

First, the text is checked whether it is valid UTF-32BE, UTF-32LE, UTF-16BE, UTF-16LE, UTF-8 (as in `stri_enc_detect`, this slightly bases on ICU's `i18n/csrucode.cpp`, but we do it in our own way, however) or ASCII.

If `locale` is not `NA` and the above fails, the text is checked for the number of occurrences of language-specific code points (data provided by the ICU library) converted to all possible 8-bit encodings that fully cover the indicated language. The encoding is selected basing on the greatest number of total byte hits.

The guess is of course imprecise [This is DRAFT API - still does not work as expected], as it is obtained using statistics. Because of this, detection works best if you supply at least a few hundred bytes of character data that's in a single language.

If you have no initial guess on language and encoding, try with `stri_enc_detect` (uses ICU facilities). However, it turns out that (empirically) `stri_enc_detect2` works better than the ICU-based one if UTF-* text is provided. Test yourself.

### Value

Just like `stri_enc_detect`, this function returns a list of length equal to the length of `str`. Each list element is a list with the following three named components:

- `Encoding` – string; guessed encodings; `NA` on failure (iff `encodings` is empty),

- `Language` – always `NA`,

- `Confidence` – numeric in [0,1]; the higher the value, the more confidence there is in the match; `NA` on failure.

The guesses are ordered w.r.t. nonincreasing confidence.

### See Also

Other encoding_detection: `stri_enc_detect`, `stri_enc_isascii`, `stri_enc_isutf16be`, `stri_enc_isutf8`, `stringi-encoding`

Other locale_sensitive: `%s<%`, `stri_compare`, `stri_count_boundaries`, `stri_duplicated`, `stri_extract_all_boundaries`, `stri_locate_all_boundaries`, `stri_opts_collator`, `stri_order`, `stri_split_boundaries`, `stri_trans_tolower`, `stri_unique`, `stri_wrap`, `stringi-locale`, `stringi-search-boundaries`, `stringi-search-coll`

[Package stringi version 1.1.1 Index]