\documentclass[a4paper]{article}

\usepackage[margin=3cm]{geometry}
%%\usepackage[round]{natbib}
\usepackage[colorlinks=true,urlcolor=blue]{hyperref}

%%\newcommand{\acronym}[1]{\textsc{#1}}
%%\newcommand{\class}[1]{\mbox{\textsf{#1}}}
\newcommand{\code}[1]{\mbox{\texttt{#1}}}
\newcommand{\pkg}[1]{{\normalfont\fontseries{b}\selectfont #1}}
\newcommand{\proglang}[1]{\textsf{#1}}
\SweaveOpts{keep.source=TRUE, strip.white=all}
%% \VignetteIndexEntry{Quick introduction}

<<echo=FALSE,results=hide>>=
if (!exists("data.table",.GlobalEnv)) library(data.table)  
# In devel won't call library, but R CMD build/check will.
rm(list=as.character(tables()$NAME),envir=.GlobalEnv)
# for development when we Sweave this file repeatedly. Otherwise first tables() shows tables from last run
@

\begin{document}
\title{Introduction to the \pkg{data.table} package in \proglang{R}}
\date{Revised: \today\\(A later revision may be available on the \href{https://github.com/Rdatatable/data.table/wiki}{homepage})}
\maketitle

\section*{Introduction}

This vignette is aimed at those who are already familiar with creating and subsetting \code{data.frame} in \proglang{R}. We aim for this quick introduction
to be readable in {\bf 10 minutes}, briefly covering a few features:
1.\,Keys; 2.\,Fast Grouping; and 3.\,Fast \emph{ordered} join.

\section*{Creation}

Recall that we create a \code{data.frame} using the function \code{data.frame()}:
<<>>=
DF = data.frame(x=c("b","b","b","a","a"),v=rnorm(5))
DF
@
A \code{data.table} is created in exactly the same way:
<<>>=
DT = data.table(x=c("b","b","b","a","a"),v=rnorm(5))
DT
@
Observe that a \code{data.table} prints the row numbers with a colon so as to visually separate the row number from the first column.
We can easily convert existing \code{data.frame} objects to \code{data.table}.
<<>>=
CARS = data.table(cars)
head(CARS)
@
We have just created two \code{data.table}s: \code{DT} and \code{CARS}. It is often useful to see a list of all
\code{data.table}s in memory:
<<>>=
tables()
@

The MB column is useful to quickly assess memory use and to spot if any redundant tables can be
removed to free up memory. Just like \code{data.frame}s, \code{data.table}s must fit inside RAM. 

Some users regularly work with 20 or more tables in memory, rather like a database. 
The result of \code{tables()} is itself a \code{data.table}, returned silently, so that \code{tables()} 
can be used in programs. \code{tables()} is unrelated to the base function \code{table()}.

To see the column types :

<<>>=
sapply(DT,class)
@

You may have noticed the empty column KEY in the result of \code{tables()} above. This is the subject of the next section.


\section*{1. Keys}

Let's start by considering \code{data.frame}, specifically \code{rownames}. We know that each row has exactly one row name. However, a person (for example) has at least two names, a first name and a second name. It's useful to organise a telephone directory sorted by surname then first name.

In \code{data.table}, a \emph{key} consists of one \emph{or more} columns. These columns may be integer, factor or numeric as well as character. Furthermore, the rows are sorted by the key. Therefore, a \code{data.table} can have at most one key because it cannot be sorted in more than one way. We can think of a key as like super-charged row names; i.e., mult-column and multi-type.

Uniqueness is not enforced; i.e., duplicate key values are allowed. Since
the rows are sorted by the key, any duplicates in the key will appear consecutively.

Let's remind ourselves of our tables:
<<>>=
tables()
DT
@

No keys have been set yet. 

<<>>=
DT[2,]         # select row 2
DT[x=="b",]    # select rows where column x == "b"
@

Aside: notice that we did not need to prefix \code{x} with \code{DT\$x}. In \code{data.table} queries, we can use column names as if they are variables directly.

But since there are no rownames, the following does not work:
<<>>=
cat(try(DT["b",],silent=TRUE))
@

The error message tells us we need to use \code{setkey()}:
<<>>=
setkey(DT,x)
DT
@

Notice that the rows in \code{DT} have now been re-ordered according to the values of \code{x}. 
The two \code{"a"} rows have moved to the top.
We can confirm that \code{DT} does indeed have a key using \code{haskey()}, \code{key()},
\code{attributes()}, or just running \code{tables()}.

<<>>=
tables()
@

Now that we are sure \code{DT} has a key, let's try again:

<<>>=
DT["b",]
@

By default all the rows in the group are returned\footnote{In contrast to a \code{data.frame} where only the first rowname is returned when the rownames contain duplicates.}. The \code{mult} argument (short for \emph{multiple}) allows the first or last row of the group to be returned instead.

<<>>=
DT["b",mult="first"]
DT["b",mult="last"]
@

Also, the comma is optional.

<<>>=
DT["b"]
@

Let's now create a new \code{data.frame}. We will make it large enough to demonstrate the
difference between a \emph{vector scan} and a \emph{binary search}.
<<print=TRUE>>=
grpsize = ceiling(1e7/26^2)   # 10 million rows, 676 groups
tt=system.time( DF <- data.frame(
  x=rep(LETTERS,each=26*grpsize),
  y=rep(letters,each=grpsize),
  v=runif(grpsize*26^2),
  stringsAsFactors=FALSE)
)
head(DF,3)
tail(DF,3)
dim(DF)
@

We might say that \proglang{R} has created a 3 column table and \emph{inserted}
\Sexpr{format(nrow(DF),big.mark=",",scientific=FALSE)} rows. It took \Sexpr{format(tt[3],nsmall=3)} secs, so it inserted
\Sexpr{format(as.integer(nrow(DF)/tt[3]),big.mark=",",scientific=FALSE)} rows per second. This is normal in base \proglang{R}. Notice that we set \code{stringsAsFactors=FALSE}. This makes it a little faster for a fairer comparison, but feel free to experiment. 

Let's extract an arbitrary group from \code{DF}:

<<print=TRUE>>=
tt=system.time(ans1 <- DF[DF$x=="R" & DF$y=="h",])   # 'vector scan'
head(ans1,3)
dim(ans1)
@

Now convert to a \code{data.table} and extract the same group:

<<>>=
DT = as.data.table(DF)       # but normally use fread() or data.table() directly, originally 
system.time(setkey(DT,x,y))  # one-off cost, usually
@
<<print=TRUE>>=
ss=system.time(ans2 <- DT[list("R","h")])   # binary search
head(ans2,3)
dim(ans2)
identical(ans1$v, ans2$v)
@
<<echo=FALSE>>=
if(!identical(ans1$v, ans2$v)) stop("vector scan vs binary search not equal")
@

At \Sexpr{format(ss[3],nsmall=3)} seconds, this was {\bf\Sexpr{as.integer(tt[3]/ss[3])}} times faster than \Sexpr{format(tt[3],nsmall=3)} seconds,
and produced precisely the same result. If you are thinking that a few seconds is not much to save, it's the relative speedup that's important. The
vector scan is linear, but the binary search is O(log n). It scales. If a task taking 10 hours is sped up by 100 times to 6 minutes, that is
significant\footnote{We wonder how many people are deploying parallel techniques to code that is vector scanning}. 

We can do vector scans in \code{data.table}, too. In other words we can use data.table \emph{badly}.

<<>>=
system.time(ans1 <- DT[x=="R" & y=="h",])   # works but is using data.table badly
system.time(ans2 <- DF[DF$x=="R" & DF$y=="h",])   # the data.frame way
mapply(identical,ans1,ans2)
@


If the phone book analogy helped, the {\bf\Sexpr{as.integer(tt[3]/ss[3])}} times speedup should not be surprising. We use the key to take advantage of the fact 
that the table is sorted and use binary search to find the matching rows. We didn't vector scan; we didn't use \code{==}.

When we used \code{x=="R"} we \emph{scanned} the entire column x, testing each and every value to see if it equalled "R". We did
it again in the y column, testing for "h". Then \code{\&} combined the two logical results to create a single logical vector which was
passed to the \code{[} method, which in turn searched it for \code{TRUE} and returned those rows. These were \emph{vectorized} operations. They
occurred internally in R and were very fast, but they were scans. \emph{We} did those scans because \emph{we} wrote that R code.


When \code{i} is a \code{list} (and \code{data.table} is a \code{list} too), we say that we are \emph{joining}. In this case, we are joining DT to the 1 row, 2 column table returned by \code{list("R","h")}. Since we do this a lot, there is an alias for \code{list}: \code{.()}.

<<>>=
identical( DT[list("R","h"),],
           DT[.("R","h"),])
@
<<echo=FALSE>>=
if(!identical(DT[list("R","h"),],DT[.("R","h"),])) stop("list != . check")
@

Both vector scanning and binary search are available in \code{data.table}, but one way of using \code{data.table} is much better than the other.

The join syntax is a short, fast to write and easy to maintain. Passing a \code{data.table} into a \code{data.table} subset is analogous to \code{A[B]} syntax in base \proglang{R} where \code{A} is a matrix and \code{B} is a 2-column matrix\footnote{Subsetting a keyed \code{data.table} by a n-column 
\code{data.table} is consistent with subsetting a n-dimension array by a n-column matrix in base R}. In fact, the \code{A[B]} syntax in base R inspired the \code{data.table} package. There are
other types of ordered joins and further arguments which are beyond the scope of this quick introduction.

The merge method of \code{data.table} is very similar to \code{X[Y]}, but there are some differences. See FAQ 1.12.

This first section has been about the first argument inside \code{DT[...]}, namely \code{i}. The next section is about the 2nd and 3rd arguments: \code{j} and \code{by}.


\section*{2. Fast grouping}


The second argument to \code{DT[...]} is \code{j} and may consist of
one or more expressions whose arguments are (unquoted) column names, as if the column names were variables. Just as we saw earlier in \code{i} as well.

<<>>=
DT[,sum(v)]
@

When we supply a \code{j} expression and a 'by' expression, the \code{j} expression is repeated for each 'by' group.

<<>>=
DT[,sum(v),by=x]
@

The \code{by} in \code{data.table} is fast.  Let's compare it to \code{tapply}.

<<>>=
ttt=system.time(tt <- tapply(DT$v,DT$x,sum)); ttt
sss=system.time(ss <- DT[,sum(v),by=x]); sss
head(tt)
head(ss)
identical(as.vector(tt), ss$V1)
@
<<echo=FALSE>>=
if(!identical(as.vector(tt), ss$V1)) stop("by check failed")
@

At \Sexpr{sprintf("%0.3f",sss[3])} sec, this was {\bf\Sexpr{as.integer(ttt[3]/sss[3])}} times faster than 
\Sexpr{sprintf("%0.3f",ttt[3])} sec, and produced precisely the same result.

Next, let's group by two columns:

<<>>=
ttt=system.time(tt <- tapply(DT$v,list(DT$x,DT$y),sum)); ttt
sss=system.time(ss <- DT[,sum(v),by="x,y"]); sss
tt[1:5,1:5]
head(ss)
identical(as.vector(t(tt)), ss$V1)
@
<<echo=FALSE>>=
if(!identical(as.vector(t(tt)), ss$V1)) stop("group check failed")
@

This was {\bf\Sexpr{as.integer(ttt[3]/sss[3])}} times faster, and the syntax is a little simpler and easier to read.
\newline


\section*{3. Fast ordered joins}

This is also known as last observation carried forward (LOCF) or a \emph{rolling join}.

Recall that \code{X[Y]} is a join between \code{data.table} \code{X} and \code{data.table} \code{Y}.  If \code{Y} has 2 columns, the first column is matched
to the first column of the key of \code{X} and the 2nd column to the 2nd.  An equi-join is performed by default, meaning that the values must be equal.

Instead of an equi-join, a rolling join is :

\code{X[Y,roll=TRUE]}

As before the first column of \code{Y} is matched to \code{X} where the values are equal. The last join column in \code{Y} though, the 2nd one in
this example, is treated specially. If no match is found, then the row before is returned, provided the first column still matches.

Further controls are rolling forwards, backwards, nearest and limited staleness. 

For examples type \code{example(data.table)} and follow the output at the prompt.


\end{document}


