---
title: "Data Retrieval and Reuse"
date: "2017-06-15"
output:
  html_document:
    fig_caption: false
    toc: true
    toc_float:
      collapsed: false
      smooth_scroll: false
    toc_depth: 2
vignette: >
  %\VignetteIndexEntry{3. Data Retrieval and Reuse}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---



This vignette shows how to download data from Dataverse using the dataverse package. We'll focus on a Dataverse repository that contains supplemental files for [Jamie Monogan](http://spia.uga.edu/faculty-member/jamie-monogan/)'s book [*Political Analysis Using R*](http://www.springer.com/gb/book/9783319234458), which is stored at Harvard University's [IQSS Dataverse Network](https://dataverse.harvard.edu/):

> Monogan, Jamie, 2015, "Political Analysis Using R: Example Code and Data, Plus Data for Practice Problems", [doi:10.7910/DVN/ARKOTI](https://doi.org/10.7910/DVN/ARKOTI), Harvard Dataverse, V1, UNF:6:+itU9hcUJ8I9E0Kqv8HWHg==

This study is persistently retrievable by a "[Digital Object Identifier (DOI)](https://www.doi.org/)": https://doi.org/10.7910/DVN/ARKOTI and the citation above (taken from the Dataverse page) includes a "[Universal Numeric Fingerprint (UNF)](http://guides.dataverse.org/en/latest/developers/unf/index.html)": `UNF:6:+itU9hcUJ8I9E0Kqv8HWHg==`, which provides a versioned, multi-file hash for the entire study, which contains 32 files.

If you don't already know what datasets and files you want to use from Dataverse, see the ["Data Search" vignette](B-search.html) for guidance on data search and discovery.

## Retrieving Dataset and File Metadata

We will download these files and examine them directly in R using the **dataverse** package. To begin, we need to loading the package and using the `get_dataset()` function to retrieve some basic metadata about the dataset:


```r
library("dataverse")
Sys.setenv("DATAVERSE_SERVER" = "dataverse.harvard.edu")
(dataset <- get_dataset("doi:10.7910/DVN/ARKOTI"))
```

```
## Dataset (75170): 
## Version: 1.0, RELEASED
## Release Date: 2015-07-07T02:57:02Z
## License: CC0
## 17 Files:
##                           label version      id                  contentType
## 1                  alpl2013.tab       2 2692294    text/tab-separated-values
## 2                   BPchap7.tab       2 2692295    text/tab-separated-values
## 3                   chapter01.R       2 2692202 text/plain; charset=US-ASCII
## 4                   chapter02.R       2 2692206 text/plain; charset=US-ASCII
## 5                   chapter03.R       2 2692210 text/plain; charset=US-ASCII
## 6                   chapter04.R       2 2692204 text/plain; charset=US-ASCII
## 7                   chapter05.R       2 2692205 text/plain; charset=US-ASCII
## 8                   chapter06.R       2 2692212 text/plain; charset=US-ASCII
## 9                   chapter07.R       2 2692209 text/plain; charset=US-ASCII
## 10                  chapter08.R       2 2692208 text/plain; charset=US-ASCII
## 11                  chapter09.R       2 2692211 text/plain; charset=US-ASCII
## 12                  chapter10.R       1 2692203 text/plain; charset=US-ASCII
## 13                  chapter11.R       1 2692207 text/plain; charset=US-ASCII
## 14 comprehensiveJapanEnergy.tab       2 2692296    text/tab-separated-values
## 15         constructionData.tab       2 2692293    text/tab-separated-values
## 16             drugCoverage.csv       1 2692233 text/plain; charset=US-ASCII
## 17         hanmerKalkanANES.tab       2 2692290    text/tab-separated-values
## 18                 hmnrghts.tab       2 2692298    text/tab-separated-values
## 19                 hmnrghts.txt       1 2692238                   text/plain
## 20                   levant.tab       2 2692289    text/tab-separated-values
## 21                       LL.csv       1 2692228 text/plain; charset=US-ASCII
## 22                 moneyDem.tab       2 2692292    text/tab-separated-values
## 23            owsiakJOP2013.tab       2 2692297    text/tab-separated-values
## 24                PESenergy.csv       1 2692230 text/plain; charset=US-ASCII
## 25                  pts1994.csv       1 2692229 text/plain; charset=US-ASCII
## 26                  pts1995.csv       1 2692231 text/plain; charset=US-ASCII
## 27                 sen113kh.ord       1 2692239 text/plain; charset=US-ASCII
## 28                SinghEJPR.tab       2 2692299    text/tab-separated-values
## 29                 SinghJTP.tab       2 2692288    text/tab-separated-values
## 30                 stdSingh.tab       2 2692291    text/tab-separated-values
## 31                       UN.csv       1 2692232 text/plain; charset=US-ASCII
## 32                  war1800.tab       2 2692300    text/tab-separated-values
```

The output prints some basic metadata and then the `str()` of the `files` data frame returned by the call. This lists all of the files in the dataset along with a considerable amount of metadata about each. We can see a quick glance at these files using:

```
dataset$files[c("filename", "contentType")]
```

This shows that there are indeed 32 files, a mix of .R code files and tab- and comma-separated data files. 


You can also retrieve more extensive metadata using `dataset_metadata()`:


```r
str(dataset_metadata("doi:10.7910/DVN/ARKOTI"), 1)
```

```
## List of 2
##  $ displayName: chr "Citation Metadata"
##  $ fields     :'data.frame':	7 obs. of  4 variables:
```

We'll focus here on the code and data files for Chapter 2 from the book.

## Retrieving Files

Let's start by grabbing the code using `get_file()` (note that this always returns a raw vector):


```r
code3 <- get_file("chapter03.R", "doi:10.7910/DVN/ARKOTI")
writeBin(code3, "chapter03.R")
```

Now we'll get the corresponding data and save it locally. For this code we need two data files:


```r
writeBin(get_file("constructionData.tab", "doi:10.7910/DVN/ARKOTI"), 
         "constructionData.dta")
writeBin(get_file("PESenergy.csv", "doi:10.7910/DVN/ARKOTI"), 
         "PESenergy.csv")
```

To confirm that the data look the way we want, we can also (perhaps alternatively) load it directly into R:


```r
constructionData <- foreign::read.dta("constructionData.dta")
str(constructionData)
PESenergy <- utils::read.table("PESenergy.csv")
str(PESenergy)
```

```
## 'data.frame':	50 obs. of  55 variables:
##  $ year                      : int  1997 1997 1997 1997 1997 1997 1997 1997 1997 1997 ...
##  $ stno                      : int  1 2 3 4 5 6 7 8 9 10 ...
##  $ totalreg                  : int  329 500 314 963 2106 643 634 239 1996 880 ...
##  $ totalhealth               : int  300 424 263 834 1859 554 501 204 1640 732 ...
##  $ raneyfolded97             : num  0.58 0.69 0.85 0.63 0.5 ...
##  $ healthagenda97            : int  49 180 137 220 1409 153 324 40 408 157 ...
##  $ predictedtotalig          : num  51.8 99 81.8 111.2 224.1 ...
##  $ supplytotalhealth         : int  1168 6991 4666 9194 70014 8847 7845 1438 35363 13471 ...
##  $ totalhealthsupplysq       : int  136 4887 2177 8453 490196 7827 6154 207 125054 18147 ...
##  $ partratetotalhealth       : num  2.48 1.09 1.09 1.4 0.35 ...
##  $ ighealthcare              : int  29 76 51 129 247 89 133 35 356 148 ...
##  $ supplydirectpatientcare   : int  1137 6687 4458 8785 66960 8320 7439 1365 33793 12760 ...
##  $ dpcsupplysq               : int  129 4472 1987 7718 448364 6922 5534 186 114197 16282 ...
##  $ partratedpc               : num  1.14 0.51 0.43 0.68 0.17 ...
##  $ igdpcare                  : int  13 34 19 60 112 40 67 12 212 74 ...
##  $ supplypharmprod           : int  0 174 78 229 2288 340 202 36 962 360 ...
##  $ pharmsupplysq             : int  0 30276 6084 52441 5234944 115600 40804 1296 925444 129600 ...
##  $ partratepharmprod         : num  0 10.34 19.23 5.24 2.05 ...
##  $ igpharmprod               : int  4 18 15 12 47 23 22 12 46 32 ...
##  $ supplybusiness            : int  0 51 28 93 315 55 36 14 317 78 ...
##  $ businesssupplysq          : int  0 2601 784 8649 99225 3025 1296 196 100489 6084 ...
##  $ partratebusness           : num  0 1.96 14.29 15.05 6.03 ...
##  $ igbusiness                : int  2 1 4 14 19 5 4 2 25 6 ...
##  $ supplygovt                : int  14 26 80 23 70 71 105 2 67 176 ...
##  $ govsupplysq               : num  0.02 0.07 0.64 0.05 0.49 ...
##  $ partrategov               : num  0 38.5 2.5 30.4 10 ...
##  $ iggovt                    : int  0 10 2 7 7 1 8 0 12 2 ...
##  $ supplyadvocacy            : int  16 37 14 57 344 54 51 18 206 76 ...
##  $ advossq                   : int  256 1369 196 3249 118336 2916 2601 324 42436 5776 ...
##  $ partrateadvo              : num  31.25 16.22 28.57 31.58 8.72 ...
##  $ ig97advoc                 : int  5 6 4 18 30 7 9 4 26 17 ...
##  $ rnmedschools              : int  1 16 8 7 37 7 12 3 18 21 ...
##  $ rnmedschoolssq            : int  1 256 64 49 1369 49 144 9 324 441 ...
##  $ rnmedschoolpartrate       : num  100 0 12.5 28.57 5.41 ...
##  $ rnmedschooligs            : int  1 0 1 2 2 0 1 0 6 1 ...
##  $ healthprofessionals       : int  12890 128980 82140 122760 749620 111550 121110 22740 471270 215670 ...
##  $ healthprofessionalssquared: int  16615 1663584 674698 1507002 56193014 1244340 1466763 51711 22209541 4651355 ...
##  $ partrateprofessionals     : num  0.03 0.01 0.01 0.01 0 ...
##  $ ighealthprofessionals     : int  4 7 6 16 30 13 22 5 29 16 ...
##  $ predictdpcpartrate        : num  1.175 0.915 1.016 0.826 0.348 ...
##  $ predictdpcig              : num  23.1 49.7 39.4 58.8 103.5 ...
##  $ predictprofpartrate       : num  0.02475 0.01383 0.01788 0.01434 0.00579 ...
##  $ predictprofig             : num  7.59 12.58 10.69 12.34 22.47 ...
##  $ predictmedschoolparttrate : num  17.39 8.08 12.3 12.95 5.02 ...
##  $ predictmedschoolig        : num  0.355 1.269 0.774 0.713 2.65 ...
##  $ predictadvopartrate       : num  31.9 26.4 32.5 21.6 13 ...
##  $ predictadvoig             : num  5.96 7.98 5.76 9.83 28.53 ...
##  $ predictbuspartrate        : num  25.78 18.08 21.33 13.1 7.27 ...
##  $ predictbusig              : num  2.58 7.96 5.66 11.66 20.04 ...
##  $ predictpharmpartrate      : num  21.38 15.22 18.52 13.44 4.14 ...
##  $ predictpharmig            : num  11.3 18.1 14.4 20.1 45.1 ...
##  $ predictgovpartrate        : num  14.41 12.61 5.84 13.03 6.93 ...
##  $ predictgovig              : num  2.06 2.43 3.78 2.35 3.57 ...
##  $ predicttotalpartrate      : num  2.41 1.823 2.047 1.623 0.752 ...
##  $ predicttotalig            : num  54.2 99.2 81.9 114.8 228.3 ...
##  - attr(*, "datalabel")= chr ""
##  - attr(*, "time.stamp")= chr " 1 Jun 2013 16:59"
##  - attr(*, "formats")= chr  "%8.0g" "%8.0g" "%8.0g" "%8.0g" ...
##  - attr(*, "types")= int  252 251 252 252 254 252 254 253 253 254 ...
##  - attr(*, "val.labels")= chr  "" "" "" "" ...
##  - attr(*, "var.labels")= chr  "Year" "StNo." "97 TotalReg" "97Total-Health" ...
##  - attr(*, "version")= int 12
## 'data.frame':	181 obs. of  1 variable:
##  $ V1: Factor w/ 181 levels "Apr-69,5,3.4,60,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,39.2",..: 31 62 47 107 1 122 92 77 16 167 ...
```

In addition to visual inspection, we can compare the UNF signatures for each dataset against what is reported by Dataverse to confirm that we received the correct files:


```r
library("UNF")
unf(constructionData)
unf(PESenergy)
dataset$files[c("label", "UNF")]
```

```
## UNF6:+4pc5114xS0ryr1sSvdX6g== 
## UNF6:TD7TEMZyrX4iGTlTsUKQDg== 
##                           label                            UNF
## 1                  alpl2013.tab UNF:6:d9ZNXvmiPfiunSAiXRpVfg==
## 2                   BPchap7.tab UNF:6:B3/HJbnzktaX5eEJA2ItiA==
## 3                   chapter01.R                           <NA>
## 4                   chapter02.R                           <NA>
## 5                   chapter03.R                           <NA>
## 6                   chapter04.R                           <NA>
## 7                   chapter05.R                           <NA>
## 8                   chapter06.R                           <NA>
## 9                   chapter07.R                           <NA>
## 10                  chapter08.R                           <NA>
## 11                  chapter09.R                           <NA>
## 12                  chapter10.R                           <NA>
## 13                  chapter11.R                           <NA>
## 14 comprehensiveJapanEnergy.tab UNF:6:Vhb3oZb9m4Nk9N7s6UAHGg==
## 15         constructionData.tab UNF:6:+4pc5114xS0ryr1sSvdX6g==
## 16             drugCoverage.csv                           <NA>
## 17         hanmerKalkanANES.tab UNF:6:lrQrhDAXFc8lSRP9muJslw==
## 18                 hmnrghts.tab UNF:6:uEg24jBA2ht0P4WeNLjI+w==
## 19                 hmnrghts.txt                           <NA>
## 20                   levant.tab UNF:6:zlgG7+JXsIZYvS383eQOvA==
## 21                       LL.csv                           <NA>
## 22                 moneyDem.tab UNF:6:7M/QM5i6IM/VUM94UJjJUQ==
## 23            owsiakJOP2013.tab UNF:6:0ZEvCFuUQms2zYD57hmwNQ==
## 24                PESenergy.csv                           <NA>
## 25                  pts1994.csv                           <NA>
## 26                  pts1995.csv                           <NA>
## 27                 sen113kh.ord                           <NA>
## 28                SinghEJPR.tab UNF:6:iDGp9dXOl4SiR+rCBWo8Tw==
## 29                 SinghJTP.tab UNF:6:lDCyZ7YQF5O++SRsxh2kGA==
## 30                 stdSingh.tab UNF:6:A5gwtn5q/ewkTMpcQEQ73w==
## 31                       UN.csv                           <NA>
## 32                  war1800.tab UNF:6:jJ++mepKcv9JbJOOPLMf2Q==
```

## Reusing Files and Reproducing Analysis

To reproduce the analysis, we can simply run the code file either as a `system()` call or directly in R using `source()` (note this particular file begins with an `rm()` call so you may want to run it in a clean R session):

```R
system("Rscript chapter03.R")
source("chapter03.R")
```

Any well-produced set of analysis reproduction files, like this one, will easily run without error once the data and code are in-hand.



To archive your own reproducible analyses using Dataverse, see the ["Archiving Data" vignette](D-archiving.html).
