---
title: "Getting Started with the Phenocam API"
author: "Bijan Seyednasrollah"
date: "`r Sys.Date()`"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Getting Started with the Phenocam API}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---
```{r setup, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)
```


The `phenocamapi`[^*] is developed to simplify interacting with the 
<a href="https://phenocam.sr.unh.edu" target="_blank">PhenoCam network</a> 
dataset and perform data wrangling steps on PhenoCam sites' data and metaadata.


This tutorial will show you the basic commands for accessing PhenoCam data
through the PhenoCam API. The *phenocampapi* R package is developed and maintained by 
<a href="https://bnasr.github.io/" target="_blank">Bijan Seyednarollah</a>. 
The most recent release is available on GitHub (<a href="https://github.com/bnasr/phenocamapi" target="_blank">PhenocamAPI</a>). 
<a href="https://github.com/bnasr/phenocamapi/tree/master/vignettes" target ="_blank">Additional vignettes</a> 
can be found on how to merge external time-series (e.g. Flux data) with the 
PhenoCam time-series. 

We begin with several useful skills and tools for extracting PhenoCam data 
directly from the server:

- Exploring the PhenoCam metadata
- Filtering the dataset by site attributes
- Downloading PhenoCam time-series data
- Extracting the list of midday images
- Downloading midday images for a given time range

## Exploring PhenoCam metadata

Each PhenoCam site has specific metadata including but not limited to how a site 
is set up and where it is located, what vegetation type is visible from the 
camera, and its climate regime. Each PhenoCam may have zero to several Regions 
of Interest (ROIs) per vegetation type. The *phenocamapi* package is an 
interface to interact with the PhenoCam server to extract those data and 
process them in an R environment.

To explore the PhenoCam data, we'll use several packages for this tutorial. 

```{r libraries}

library(data.table)
library(phenocamapi)
library(jpeg)

```


We can obtain an up-to-date `data.frame` of the metadata of the entire PhenoCam 
network using the `get_phenos()` function. The returning value would be a 
`data.table` in order to simplify further data exploration.

```{r obtain-data, fig.height=5, fig.width=8, message=FALSE}

# obtaining the phenocam site metadata from the server as data.table
phenos <- get_phenos()

# checking out the first few sites
head(phenos$site)

# checking out the columns
colnames(phenos)
```

Now we have a better idea of the types of metadata that are available for the
Phenocams. 

### Remove null values

We may want to explore some of the patterns in the metadata before we jump into
specific locations. 

Let's look at Mean Annual Precipiation (MAP) and Mean Annual
Temperature (MAT) across the differnet field site and classify those by the
primary vegetation type (`primary_veg_type`) for each site. We can find out what the abbreviations
for the vegetation types mean from the following table:

| Abbreviation |	Description |
|----------|:-------------:|------:|
| AG |	agriculture |
| DB |	deciduous broadleaf |
| DN |	deciduous needleleaf |
| EB |	evergreen broadleaf |
| EN |	evergreen needleleaf |
| GR |	grassland |
| MX |	mixed vegetation (generally EN/DN, DB/EN, or DB/EB) |
| SH |	shrubs |
| TN |	tundra (includes sedges, lichens, mosses, etc.) |
| WT |	wetland |
| NV |	non-vegetated |
| RF |	reference panel | 	 
| XX |	unspecified |

To do this we'd first want to remove the sites where there is not data and then 
plot the data. 

```{r plot-MAT-MAP, message=FALSE, fig.height=8, fig.width=8}
# removing the sites with unkown MAT and MAP values
phenos <- phenos[!((MAT_worldclim == -9999)|(MAP_worldclim == -9999))]

# extracting the PhenoCam climate space based on the WorldClim dataset
# and plotting the sites across the climate space different vegetation type as different symbols and colors
phenos[primary_veg_type=='DB', plot(MAT_worldclim, MAP_worldclim, pch = 19, col = 'green', xlim = c(-5, 27), ylim = c(0, 4000))]
phenos[primary_veg_type=='DN', points(MAT_worldclim, MAP_worldclim, pch = 1, col = 'darkgreen')]
phenos[primary_veg_type=='EN', points(MAT_worldclim, MAP_worldclim, pch = 17, col = 'brown')]
phenos[primary_veg_type=='EB', points(MAT_worldclim, MAP_worldclim, pch = 25, col = 'orange')]
phenos[primary_veg_type=='AG', points(MAT_worldclim, MAP_worldclim, pch = 12, col = 'yellow')]
phenos[primary_veg_type=='SH', points(MAT_worldclim, MAP_worldclim, pch = 23, col = 'red')]

legend('topleft', legend = c('DB','DN', 'EN','EB','AG', 'SH'), 
       pch = c(19, 1, 17, 25, 12, 23), 
       col =  c('green', 'darkgreen', 'brown',  'orange',  'yellow',  'red' ))

```


### Filtering using attributes

Alternatively, we may want to only include Phenocams with certain attributes in
our datasets. For example, we may be interested only in sites with a co-located
flux tower. For this, we'd want to filter for those with a flux tower using the
`flux_sitenames` attribute in the metadata. 


```{r filter-flux, fig.height=5, fig.width=6.5, message=FALSE}
# store sites with flux_data available and the FLUX site name is specified
phenofluxsites <- phenos[flux_data==TRUE&!is.na(flux_sitenames)&flux_sitenames!='', 
                         .(PhenoCam=site, Flux=flux_sitenames)] # return as table 
#and specify which variables to retain

phenofluxsites <- phenofluxsites[Flux!='']

# see the first few rows
head(phenofluxsites)

```

We could further identify which of those Phenocams with a fluxtower and in 
decidious broadleaf forests (`primary_veg_type=='DB'`). 

```{r filter-flux-db, fig.height=5, fig.width=6.5, message=FALSE}

#list deciduous broadleaf sites with flux tower
DB.flux <- phenos[flux_data==TRUE&primary_veg_type=='DB', 
                  site]  # return just the site names as a list

# see the first few rows
head(DB.flux)
```


## PhenoCam time series

PhenoCam time series are extracted time series data obtained from ROI's for a 
given site. 

### Obtain ROIs
To download the phenological time series from the PhenoCam, we need to know the 
sitename, vegetation type and ROI ID. This information can be obtained from each
specific PhenoCam page on the 
<a href="https://phenocam.sr.unh.edu/webcam/gallery/" target="_blank">PhenoCam website</a> 
or by using the `get_rois()` function. 


```{r get-rois, fig.height=5, fig.width=6.5, message=FALSE}
# obtaining the list of all the available ROI's on the PhenoCam server
rois <- get_rois()

# view what information is returned
colnames(rois)

# view first few locations
head(rois$roi_name)

```

### Download time series

The `get_pheno_ts()` function can download a time series and return the result 
as a `data.table`. 
Let's work with the 
<a href="https://phenocam.sr.unh.edu/data/archive/dukehw/ROI/dukehw_DB_1000.html">Duke Forest Hardwood Stand (`dukehw`) PhenoCam</a> 
and specifically the ROI 
<a href="https://phenocam.sr.unh.edu/data/archive/dukehw/ROI/dukehw_DB_1000.html">`DB_1000`</a> 
we can run the following code. 

```{r, fig.height=5, fig.width=6.5, message=FALSE}
# list ROIs for dukehw
rois[site=='dukehw',]

# to obtain the DB 1000 from dukehw
dukehw_DB_1000 <- get_pheno_ts(site = 'dukehw', vegType = 'DB', roiID = 1000, type = '3day')

# what data are available
str(dukehw_DB_1000)

```

We now have a variety of data related to this ROI from the Hardwood Stand at Duke 
Forest. 

Green Chromatic Coordinate (GCC) is a measure of "greeness" of an area and is
widely used in Phenocam images as an indicator of the green pigment in vegetation.
Let's use this measure to look at changes in GCC over time at this site. Looking 
back at the available data, we have several options for GCC. `gcc90` is the 90th 
quantile of GCC in the pixels across the ROI (for more details, 
<a href="https://daac.ornl.gov/VEGETATION/guides/PhenoCam_V1.html" target="_blank"> PhenoCam v1 description</a>). 
We'll use this as it tracks the upper greeness values while not inlcuding many 
outliers.  

Before we can plot `gcc-90` we do need to fix our dates and convert them from 
Factors to Date to correctly plot. 

```{r plot-gcc90, fig.height=5, fig.width=8}
# date variable into date format
dukehw_DB_1000[,date:=as.Date(date)]

# plot gcc_90
dukehw_DB_1000[,plot(date, gcc_90, col = 'green', type = 'b')]
mtext('Duke Forest, Hardwood', font = 2)

```


## Download midday images

While PhenoCam sites may have many images in a given day, many simple analyses
can use just the midday image when the sun is most directly overhead the canopy. 
Therefore, extracting a list of midday images (only one image a day) can be useful. 


```{r midday-list, fig.height=5, fig.width=8, message=FALSE}

# obtaining midday_images for dukehw
duke_middays <- get_midday_list('dukehw')

# see the first few rows
head(duke_middays)

```

Now we have a list of all the midday images from this Phenocam. Let's download
them and plot 

```{r midday-download, fig.height=5, fig.width=8}
# download a file
destfile <- tempfile(fileext = '.jpg')

# download only the first available file
# modify the `[1]` to download other images
download.file(duke_middays[1], destfile = destfile, mode = 'wb')

# plot the image
img <- try(readJPEG(destfile))
if(class(img)!='try-error'){
  par(mar= c(0,0,0,0))
  plot(0:1,0:1, type='n', axes= FALSE, xlab= '', ylab = '')
  rasterImage(img, 0, 0, 1, 1)
}
```


### Download midday images for a given time range

Now we can access all the midday images and download them one at a time. However,
we frequently want all the images within a specific time range of interest. We'll
learn how to do that next. 

```{r midday-time-range, fig.height=6, fig.width=8, message=FALSE, eval=TRUE}

# open a temporary directory
tmp_dir <- tempdir()

# download a subset. Example dukehw 2017
download_midday_images(site = 'dukehw', # which site
                       y = 2017, # which year(s)
                       months = 1:12, # which month(s)
                       days = 15, # which days on month(s)
                       download_dir = tmp_dir) # where on your computer

# list of downloaded files
duke_middays_path <- dir(tmp_dir, pattern = 'dukehw*', full.names = TRUE)

head(duke_middays_path)

```

We can demonstrate the seasonality of Duke forest observed from the camera:
```{r plot-monthly-forest, fig.height=6, fig.width=8, message=FALSE, eval=TRUE}
n <- length(duke_middays_path)
par(mar= c(0,0,0,0), mfrow=c(4,3), oma=c(0,0,3,0))

for(i in 1:n){
  img <- readJPEG(duke_middays_path[i])
  plot(0:1,0:1, type='n', axes= FALSE, xlab= '', ylab = '')
  rasterImage(img, 0, 0, 1, 1)
  mtext(month.name[i], line = -2)
}
mtext('Seasonal variation of forest at Duke Hardwood Forest', font = 2, outer = TRUE)

```

The goal of this section was to show how to download a limited number of midday images from the PhenoCam server. However, more extensive datasets should be downloaded from the <a href="https://phenocam.sr.unh.edu/webcam/network/download/"> PhenoCam </a>.



[^*]: The R package is developed and maintained by <a href="https://bnasr.github.io/">Bijan Seyednarollah</a>. Most recent release is available from: https://github.com/bnasr/xROI.


