krippalpha {icr} | R Documentation |
krippalpha
computes Krippendorff's reliability coefficient alpha.
krippalpha(data, metric = "nominal", bootstrap = FALSE, bootnp = FALSE, nboot = 20000, nnp = 1000, cores = 1, seed = rep(12345, 6))
data |
a matrix or data frame (coercible to a matrix) of reliability data. Data of type |
metric |
metric difference function to be applied to disagreements. Supports |
bootstrap |
logical indicating whether uncertainty estimates should be obtained using the bootstrap algorithm defined by Krippendorff. Defaults to |
bootnp |
logical indicating whether non-parametric bootstrap uncertainty estimates should be computed. Defaults to |
nboot |
number of bootstraps used in Krippendorff's algorithm. Defaults to |
nnp |
number of non-parametric bootstraps. Defaults to |
cores |
number of cores across which bootstrap-computations are distributed. Defaults to 1. If more cores are specified than available, the number will be set to the maximum number of available cores. |
seed |
numeric vector of length 6 for the internal L'Ecuyer-CMRG random number generator (see details). Defaults to |
krippalpha
takes the seed vector to seed the internal random number generator of both bootstrap-routines. It does not advance R's RNG state.
When using the ratio
metric with reliability data containing scales involving negative as well as positive values, krippalpha
may return a value of NaN
. The ratio
metric difference function is defined as ((c - k)/(c + k))^2. Hence, if for any two scale values c = -k, the fraction is not defined, resulting in alpha = NaN
. In order to avoid this issue, shift your reliability data to have strictly positive values.
Returns a list of type icr
with following elements:
alpha |
value of inter-coder reliability coefficient |
metric |
integer representation of metric used to compute alpha: 1 nominal, 2 ordinal, 3 interval, 4 ratio, 6 bipolar |
n_coders |
number of coders |
n_units |
number of units to be coded |
n_values |
number of unique values in reliability data |
coincidence_matrix |
matrix containing coincidences within coder-value pairs |
delta_matrix |
matrix of metric differences depending on |
D_e |
expected disagreement |
D_o |
observed disagreement |
bootstrap |
|
nboot |
number of bootstraps |
bootnp |
|
nnp |
number of non-parametric bootstraps |
bootstraps |
vector of bootstrapped values of alpha (Krippendorff's algorithm) |
bootstrapsNP |
vector of non-parametrically bootstrapped values of alpha |
krippalpha
's bootstrap-routines use L'Ecuyer's CMRG random number generator (see L'Ecyuer et al. 2002) to create random numbers suitable for parallel computations. The routines interface to L'Ecuyer's C++ code, which can be found at https://pubsonline.informs.org/doi/abs/10.1287/opre.50.6.1073.358
Krippendorff, K. (2004) Content Analysis: An Introduction to Its Methodology. Beverly Hills: Sage.
Krippendorff, K. (2011) Computing Krippendorff's Alpha Reliability. Departmental Papers (ASC) 43. http://repository.upenn.edu/asc_papers/43.
Krippendorff, K. (2016) Bootstrapping Distributions for Krippendorff's Alpha. http://web.asc.upenn.edu/usr/krippendorff/boot.c-Alpha.pdf.
L'Ecuyer, P. (1999) Good Parameter Sets for Combined Multiple Recursive Random Number Generators. Operations Research, 47 (1), 159–164. https://pubsonline.informs.org/doi/10.1287/opre.47.1.159.
L'Ecuyer, P., Simard, R, Chen, E. J., and Kelton, W. D. (2002) An Objected-Oriented Random-Number Package with Many Long Streams and Substreams. Operations Research, 50 (6), 1073–1075. http://www.iro.umontreal.ca/~lecuyer/myftp/streams00/c++/streams4.pdf.
data(codings) # compute alpha, without uncertainty estimates krippalpha(codings) # additionally compute bootstrapped uncertainty estimates for alpha alpha <- krippalpha(codings, metric = "nominal", bootstrap = TRUE, bootnp = TRUE) alpha # plot bootstrapped alphas plot(alpha) # alternatively, use ggplot2 df <- plot(alpha, return_data = TRUE) library(ggplot2) ggplot() + geom_line(data = df[df$ci_limit == FALSE, ], aes(x, y, color = type)) + geom_area(data = df[df$ci == TRUE, ], aes(x, y, fill = type), alpha = 0.4) + theme_minimal() + theme(plot.title = element_text(hjust = 0.5)) + theme(legend.position = "bottom", legend.title = element_blank()) + ggtitle(expression(paste("Bootstrapped ", alpha))) + xlab("value") + ylab("density") + guides(fill = FALSE)