\documentclass[english]{article}
\usepackage{natbib}
\usepackage{url}
\usepackage{hyperref}
\hypersetup{
    colorlinks,
    citecolor=black,
    filecolor=black,
    linkcolor=black,
    urlcolor=black
}
\newcommand{\code}{\emph}
\newcommand{\pkg}{\textsl}
\newcommand{\proglang}{\textsl}
%% If the size of graphs is not set, the default is 6x6. 
%%  4x7 works well for graphs with a single panel, 
%%  but needs to be over-ridden for multi-panel graphs.
\SweaveOpts{eval=TRUE,echo=TRUE,results=hide,fig=FALSE,width=7.0,height=4}
%\SweaveOpts{eval=TRUE,echo=TRUE,results=verbatim,fig=TRUE}
%\VignetteIndexEntry{Time Series Database Interface (TSdbi) Guide and Illustrations}
\begin{document}
\begin{titlepage}
\begin{center}
{\huge Time Series Database Interface (TSdbi) Guide and Illustrations}\\[0.75cm]
{\Large Paul D. Gilbert}\\[0.2cm]
{\Large \today}\\[0.5cm]
\end{center}
\tableofcontents
\end{titlepage}
\begin{Scode}{echo=FALSE,results=hide}
options(continue="  ")
\end{Scode}
%%
\section{Introduction}
\label{sec:Intro}
%%
This vignette illustrates the various \proglang{R} \citep{R} \pkg{TSdbi} 
packages using time series 
data from several sources. The main purpose of \pkg{TSdbi} is to provide a
common \emph{API}, so simplicity of changing the data source is a primary
feature. 
The vignette also illustrates some simple time series
manipulation and plotting using packages \pkg{tframe} and \pkg{tfplot}.
While these do not need to be attached for data retrieval, functions in these
packages are used in several places in this vignette so they should be
attached to run examples in this vignette. 
This is done with
%%
\begin{Scode}{results=verbatim,fig=FALSE}
library("tframe")  
library("tfplot")  
\end{Scode}
%%
Package \pkg{tframePlus} is also used but will be attached when needed.

To generate this vignette requires most of the \pkg{TS*} packages, but 
users will only need one, or a few of these packages. For example, a time series
database can be built with several SQL database backends, 
but typically only one would be used. 
On the other hand, several packages pull data from 
various Internet sources, so several of these might be used to accommodate the
various specifics of the sources.

If package \pkg{TSdata} is installed on your system, it should be possible 
to view the pdf version of this guide 
with \emph{vignette("Guide", package="TSdata")}. Otherwise, consider 
getting the pdf file directly from CRAN at 
\url{http://cran.at.r-project.org/web/packages/TSdata/index.html} 
Many parts
of the vignette can be run by loading the appropriate packages, but some
examples use data that has been loaded into a local database 
and it will not be possible to reproduce those examples without the
underlying database.

Section \ref{sec:SimpleExamples} of this vignette illustrates the mechanism for
connecting to a data source. The connection contains all the source specific 
information so, once the connection is established, the syntax for retrieving
data is similar for different sources. The section illustrates packages that 
pull data from Internet sources. This currently includes \pkg{TSsdmx}, \pkg{TSjson}, 
and \pkg{TSmisc}. (Package \pkg{TSmisc} is an amalgamation of former packages 
\pkg{TSgetSymbol} \pkg{TShistQuote}, \pkg{TSxls}, \pkg{TSzip}, and some new connections.)
The section starts
with a table indicating which packages and connections might be used for various data
sources. These packages only get data, they do not support writing data to the database.
Finally, the section also illustrates the flexibility to return different representations
of time series objects.

Section \ref{sec:SQL} illustrates the SQL packages \pkg{TSPostgreSQL}, 
\pkg{TSMySQL}, \pkg{TSSQLite}, and \pkg{TSodbc}. These use a very standard SQL
table structure and syntax, so it should be possible to use other SQL backends.
The package \pkg{TSOracle} is available on R-forge at 
\url{http://tsdbi.r-forge.r-project.org/} but I currently do not have a server
in place to test it properly. (If anyone would be interested in doing this,
please contact me, pgilbert.ttv9z@ncf.ca.)
These packages get data and also support writing data to the database.
See Appendix ``B'' (separate vignette) and the vignette in package \pkg{TSdbi} 
for more explanation 
of the underlying database tables, and for bulk loading of
data into a database.
The section illustrates the packages by writing artificial data to the database. 

Section \ref{sec:MoreExamples} provides additional examples of \pkg{TSdbi}
functionality and of using and graphing series. 
Subsection \ref{sec:SQLexamples} illustrates fetching data from the web and 
loading it into a local database.
%%Subsection \ref{sec:etsExamples} provides examples of how one might use a
%%more extensively populated database that has been previously constructed.

Section \ref{sec:Comparing} illustrates package \pkg{TScompare} for comparing 
time series databases, and Section \ref{sec:Vintages} illustrates the use of
realtime vintages of data.

Section \ref{sec:JustData} illustrates some functions for exporting time series
data from \proglang{R} into .xls and .csv files. While regular \proglang{R}
users may not be too interested in this, it can be useful for colleagues that
are not yet regular users. The section is fairly self contained. (But will give
non-users some exposure to \proglang{R}, so they may
decide they do not actually need to export the data.)

Appendix ``A'' provides connection details specific to the different
database sources,
Appendix ``B'' provides more details about the structure of SQL
databases,
and Appendix ``C'' provides some examples of SQL queries that
may be useful for database maintenance.
Finally, Appendix ``D'' reproduces the README file for package \pkg{TSjson}, 
to give details regarding installation of Python helper utilities.


Many of the \pkg{TS*} packages are wrappers of other packages. The purpose is to
provide a common API for interfacing with time series databases, and an easy 
mechanism to specify the type of time series object that should be returned, for
example, a \code{ts} object or a \code{zoo} \citep{zoo} object. One consequence of providing
a common interface is that special strengths of some of the underlying packages
cannot always be used. If you really need some of these features then you will
need to go directly to the underlying package. However, if you limit your
reliance on these special features then you will be able to move from one
data source to another much more easily.

Loading a \pkg{TS*} package will also attach or load the namespace of required 
packages \pkg{TSdbi}, 
\pkg{DBI} \citep{DBI}, \pkg{methods} \citep{methods}, \pkg{tframePlus}, 
\pkg{zoo}, and then any underlying package that the specific \pkg{TS*} package uses.

%%
\section{Simple Examples of Time Series Data from the Internet}
\label{sec:SimpleExamples}
%%
This section uses packages to pull data
from the Internet. The general syntax of a connection is illustrated, and
some simple calculations and graphs are done to demonstrate how the data 
might be used.
However, the purpose of the \pkg{TS*} packages is to provide a common interface,
not to do all time series calculations and graphics. Once you have the data, 
you should be able to use whatever other \proglang{R} 
packages you like for your calculations and graphs.

The generic aspect of the interface API is accomplished by putting the
information specific to the underlying source into the ``connection''. Once the
connection is established, other aspects of using data are the same, so one
connection can be easily interchanged with another, and so your programs do not
need to be changed when the data source is changed. (But, of course, some
changes will be needed if the series identifiers of the variables change on
the different databases.) 

The following table indicates various data sources and the \pkg{TS*} connections 
and packages that can be used to extract data from them, \pkg{TSsdmx} is a 
wrapper for package \pkg{RJSDMX} \citep{RJSDMX} maintained by Attilio Mattiocco
at the Banca d'Italia. It uses SDMX/REST
to connect to several important sources for economic data. This API is probably the 
main future direction, and some of the other interfaces should be considered temporary,
pending the institutions getting SDMX/REST 2.1 in place. Package \pkg{RJSDMX} has an
\pkg{addProvider} function which can be used to add SDMX/REST 2.1 compliant servers.

%%%%%%%%%%%%%%%%%%%
\begin{table}
\begin{center}
\caption{Connections for various data sources.}
\label{tab:connections}
\begin{tabular}{@{}lclc@{}}
\\
\hline \\
Data source      &     source URL		 & Package&Connection \\
\\
\hline \\
ABS (Australia)  &\url{www.abs.gov.au}  	 & TSsdmx & sdmx \\
ECB       	 &\url{www.ecb.org}		 & TSsdmx & sdmx \\
EuroStat	 &\url{epp.eurostat.ec.europa.eu}& TSsdmx & sdmx \\
ILO	      	 &\url{www.ilo.org}		 & TSsdmx & sdmx \\
IMF	      	 &\url{www.imf.org}		 & TSsdmx & sdmx \\
INEGI (Mexico)	 &\url{www.inegi.org.mx}	 & TSsdmx & sdmx \\
ISTAT$^5$ (Italy)&\url{www.istat.it}		 & TSsdmx & sdmx \\
NBB   (Belgium)	 &\url{www.nbb.be}		 & TSsdmx & sdmx \\
OECD	      	 &\url{www.oecd.org}		 & TSsdmx & sdmx \\
UIS	      	 &\url{www.uis.unesco.org}	 & TSsdmx & sdmx \\
UN$^2$	      	 &\url{comtrade.un.org} 	 & TSsdmx & sdmx \\
Unesco     	 &\url{www.uis.unesco.org}	 & TSsdmx & sdmx \\
World Bank$^1$	 &\url{worldbank.org}		 & TSsdmx & sdmx \\
BIS$^3$	      	 &\url{www.bis.org}		 & TSsdmx & sdmx \\
FRED    	 &\url{research.stlouisfed.org}  & TSmisc & histQuote \\
FRED    	 &\url{research.stlouisfed.org}  & TSmisc & getSymbol \\
RBA (Australia)  &\url{www.rba.gov.au}  	 & TSmisc & xls \\
Statistics Canada&\url{www.statcan.gc.ca}	 & TSjson$^{4,6}$ & json \\
Bank of Canada	 &\url{www.bank-banque-canada.ca}& TSmisc2$^4$ & Quandl \\
Yahoo    	 &\url{quote.yahoo.com} 	 & TSmisc & getSymbol \\
Yahoo    	 &\url{quote.yahoo.com} 	 & TSmisc & histQuote \\
Oanda    	 &\url{www.oanda.com}		 & TSmisc & histQuote \\
PiTrading    	 &\url{pitrading.com}		 & TSmisc & zip \\
Quandl	  	 &\url{www.quandl.com}  	 & TSmisc2$^4$ & Quandl \\
Bloomberg 	 &\url{www.bloomberg.com}	 & TSbbg$^4$ & bbg \\
\\
\hline \\
\multicolumn{4}{l}{$^1$ beta} \\
\multicolumn{4}{l}{$^2$ planning} \\
\multicolumn{4}{l}{$^3$ requires an account} \\
\multicolumn{4}{l}{$^4$ not on CRAN, on R-forge \url{tsdbi.r-forge.r-project.org/}} \\
\multicolumn{4}{l}{$^5$ requires free registration at \url{sdmx.istat.it/registr_ut_sep/}} \\
\multicolumn{4}{l}{$^6$ using python mechanize,  unstable website API} \\
\\
\hline
\end{tabular}
\end{center}
\end{table}
%%%%%%%%%%%%%%%%%%%
The above URL is for the organization. The SDMX data web URL can be determined with \code{RJSDMX::sdmxHelp()} using the build command action to give the URL 
of an sdmx query.

The table is not comprehensive. In particular, Quandl, Bloomberg, Yahoo, Oanda, 
and some others, provide access to data from a large number of sources, especial market data.
%%
\subsection{\code{SDMX} Data}
%%
SDMX is emerging as a widely used standard for time series exchange. It is supported by several 
large organizations and has evolved over many years. Some of the ad hoc mechanisms 
in other \pkg{TS*} packages will be replaced as more organizations implement this standard.
It provides capabilites for transmitting large dataset, mixed frequency, and metada. 
These are well beyond the scope of what is used by the TSdbi API. 
The \pkg{RJSDMX} package provides
an interface to SDMX/REST 2.1 compliant databases. Similar to other \pkg{TS*} packages, the 
wrapper \pkg{TSsdmx} uses only some of the capabilities of the underlying package.  
Only reading of databases is supported.

Also, beware that data providers can change the availability and names of databases, both temporarily and permanently, and several are still in an experimental mode with respect to SDMX support. Examples below may sometimes
result in errors at the provider.

By default \pkg{RJSDMX} prints more output than \proglang{R} users will typically expect. 
This output can be supressed by settings in a configuration file, which has been done 
for this vignette. See Appendix ``A'' for more details.

\begin{Scode}{results=verbatim,fig=TRUE}
library("TSsdmx")  
oecd <- TSconnect("sdmx", dbname="OECD")

x <- TSget('QNA.CAN.B1_GE.CARSA.Q', oecd)

tfplot(x, title="Canada: GDP - expenditure approach", 
    subtitle="quarterly national accounts, national currency, current prices, SAAR",
    ylab="CDN dollars")

\end{Scode}

SDMX queries are roughly like database queries with the fields separated by dots.
They can use \code{*}, \code{|}, and \code{+} to return muliple series.
\code{TSget} provides the ability to rename series on retrieval when
\code{|}, and \code{+} are used, but not when \code{*} is used.

\begin{Scode}{results=verbatim,fig=TRUE}
x <- TSget('QNA.CAN+USA+MEX.B1_GE.CARSA.Q', 
         names=c("Canada", "United States", "Mexico"),   oecd)

tfOnePlot(ytoypc(x), title="Nominal GDP  - expenditure approach", 
         subtitle="quarterly national accounts, national currency, current prices, SAAR",
	 ylab="national currency", start=c(1975,1),
	 legend=seriesNames(x), legend.loc="topright")
\end{Scode}

Several different connections can be defined, and data retreived from them.
%%  some of these fail often, but not always with various provider errors
%%   names(getDimensions('EUROSTAT','ei_nama_q')) 
%%     getCodes('EUROSTAT', 'ei_nama_q', 'FREQ')
%%     getCodes('EUROSTAT', 'ei_nama_q', 'UNIT')
%%     getCodes('EUROSTAT', 'ei_nama_q', 'S_ADJ')
%%     getCodes('EUROSTAT', 'ei_nama_q', 'P_ADJ')
%%     getCodes('EUROSTAT', 'ei_nama_q', 'INDIC')$'NA-P72'#"Imports of services"
%%     getCodes('EUROSTAT', 'ei_nama_q', 'GEO')
%%     getCodes('EUROSTAT', 'ei_nama_q', '')
%%   verifyQuery('EUROSTAT',"ei_nama_q.Q.MIO-EUR.NSA.CP.NA-P72.IT")
%%  z <- TSget("ei_nama_q.Q.MIO-EUR.NSA.CP.NA-P72.IT", eurostat)
%%
%%   names(getDimensions('EUROSTAT','namq_gdp_c')) 
%%[1] "FREQ"     "S_ADJ"    "UNIT"     "INDIC_NA" "GEO"  
%%     getCodes('EUROSTAT', 'namq_gdp_c', 'INDIC_NA')
%%   verifyQuery('EUROSTAT',"namq_gdp_c.Q.NSA.MIO_EUR.B1GM.DE")
%% http://appsso.eurostat.ec.europa.eu/nui/show.do?dataset=namq_gdp_c&lang=en
%%  z <- TSget("namq_gdp_c.Q.NSA.MIO_EUR.B1GM.DE", eurostat)
%%  next is German employment
%%   verifyQuery('EUROSTAT',"lfsq_egan.Q.THS.T.Y15-64.TOTAL.DE")

\begin{Scode}{results=verbatim,fig=FALSE}
eurostat <- TSconnect("sdmx", dbname="EUROSTAT")
ecb <- TSconnect("sdmx", dbname="ECB")
abs <- TSconnect("sdmx", dbname="ABS")

 z <- TSget("ei_nama_q.Q.MIO-EUR.NSA.CP.NA-P72.IT", eurostat)
\end{Scode}

\code{TSget} allows for specifying start and end dates.

\begin{Scode}{results=verbatim,fig=FALSE}
z <- TSget('EXR.Q.USD.EUR.SP00.A', 
                start="2008-Q2", end="2014-Q3", ecb)
z <- TSget('EXR.Q.USD.EUR.SP00.A', 
                start=c(2008,2), end=c(2014,3), ecb)
\end{Scode}

The first request above uses SDMX syntax for \code{start} and \code{end}, 
which is passed to the query, and only data in the range is returned from 
the server. 
The second syntax uses an \proglang{R} convention which is difficult to translate 
into the query, because it is necessary to first determine the frequency of the 
data. For this query all the data is returned and the truncation is done 
in \proglang{R}. For \code{sdmx} connections, character string \code{start} 
and \code{end} specifications are assumed to be SDMX and are passed to the
underlying \pkg{RJSDMX} call and on to the server, otherwise truncation is 
done in \proglang{R} on the returned data.

A connection can be specified to be used as the default, so it does not need to
be specified each time:

\begin{Scode}{fig=FALSE}
options(TSconnection=abs)
z <- TSget("BOP.1.100.10.Q", start=c(1990, 1), end=c(2012, 2)) 
\end{Scode}

Several different connections will be used in this vignette, and so a default
will not be used. To unset the default
\begin{Scode}
options(TSconnection=NULL)
\end{Scode}

By default most \code{TSget} methods return a class \code{ts} series when
that is possible (annual, quarterly, monthly, or semi-annual data) and
\code{zoo} series otherwise. Thus the class of the series returned above is
\code{ts}, but \code{zoo} can be specified.
\begin{Scode}{results=verbatim,fig=FALSE}
class(z)
z <- TSget("BOP.1.100.10.Q", start=c(1990, 1), end=c(2012, 2), 
                 abs, TSrepresentation="zoo")
class(z)
\end{Scode}

A session default can also be set for this with
\begin{Scode}
options(TSrepresentation="zoo")
\end{Scode}
in which case all results will be return as \code{zoo} objects unless otherwise
specified. The session default is unset with
\begin{Scode}
options(TSrepresentation=NULL)
\end{Scode}
Beware that it does not make sense to set \code{ts} as the default, because it
is already the default for all series that can be represented as \code{ts}, and
will not work correctly for other series.
Other representations are possible. See the \code{TSget} help for 
more details. 
%%
\subsection{\code{histQuote} and \code{getSymbol}}
%%
\code{histQuote} and \code{getSymbol} in package \pkg{TSmisc} provide mechanisms to 
retrieve historical quote data from various sources. 
\code{histQuote} is a wrapper to \code{get.hist.quote} in package
\pkg{tseries} \citep{tseries}. A connection to Yahoo Finance is established, and data 
retrieved and plotted by

\begin{Scode}{results=verbatim,fig=TRUE}
library("TSmisc")  
yahoo <- TSconnect("histQuote", dbname="yahoo") 
x <- TSget("^gspc", quote = "Close", con=yahoo)
library("tfplot")  
tfplot(x)
\end{Scode}

Package \pkg{getSymbol} is a wrapper to \code{getSymbols} in package
\pkg{quantmod} \citep{quantmod}. A connection to Yahoo using this, and
retrieving the same data, can be done by

\begin{Scode}{results=verbatim,fig=TRUE}
yahoo <- TSconnect("getSymbol", dbname="yahoo") 
x <- TSget("^gspc", quote = "Close", con=yahoo)
tfplot(x)
\end{Scode}

Notice that the only difference is the name of
the driver provided when establishing the connection. After that, the code is the
same (and the data from the two connections should be the same, 
other than the difference in the default start date). This is the
coding approach one would typically follow, so that changing the data source is
easy. However, sometimes it is interesting to compared the same data from 
differrent sources, so here is an example where the connections are given 
different names, so the same data through two different connection
methods can be more easily compared:
\begin{Scode}{results=verbatim,fig=FALSE}
ya1 <- TSconnect("getSymbol", dbname="yahoo")
ya2 <- TSconnect("histQuote", dbname="yahoo")
ibmC1 <- TSget("ibm", ya1, quote = "Close", start="2011-01-03")
ibmC2 <- TSget("ibm", ya2, quote = "Close", start="2011-01-03")
\end{Scode}

In this example the underlying packages both return \code{zoo} time series
objects, but the encoding of the date index vectors  
are of different classes (\code{Date} vs \code{POSIXct}). This is usually 
not a problem
because one would usually work with one package or the other, but it does become
a problem when comparing the two objects returned by the different methods.
The time representation of \code{ibmC1} can be changed from \code{POSIXct} 
to \code{Date} by:
\begin{Scode}{results=verbatim,fig=FALSE}
tframe(ibmC1) <- as.Date(tframe(ibmC1))
\end{Scode}

The two series can then be plotted:
\begin{Scode}{results=verbatim,fig=TRUE}
tfplot(ibmC2, ibmC1,  
   ylab="IBM Close",
   title="IBM via getSymbol and histQuote",
   lastObs=TRUE, 
   legend=c("via histQuote (black)", "via getSymbol (red)"),
   source="Source: Yahoo")
\end{Scode}

Or the difference can be used to check equality:
\begin{Scode}{results=verbatim}
max(abs(ibmC2 - ibmC1))
\end{Scode}

(Note that this difference calculation does not catch a difference in length, 
which occurs if new data has been release on one connection and not the other. 
At some point Yahoo was releasing partial data early, and these connection are
correcting differently for this. So, at some times of day, the last available
data point is not the same on these two connections.)

A certain amount of meta data can be returned with the time series object 
and can be extracted with these utilities:
\begin{Scode}{results=verbatim,fig=FALSE}
TSdescription(x) 
TSdoc(x) 
TSlabel(x) 
TSsource(x) 
\end{Scode}
%%
%%
\subsection{\code{getSymbol} with FRED}

\pkg{getSymbol} can also be used to get data from the Federal Reserve 
Bank of St.Louis, as will be illustrated here. 
(Look at \url{http://research.stlouisfed.org/fred/} to find series identifiers.)

\begin{Scode}{fig=TRUE}
library("TSmisc")  
fred <- TSconnect("getSymbol", dbname="FRED") 
tfplot(TSget("M2", fred))
\end{Scode}

Additional information can be specified in \code{tfplot} and \code{tfOnePlot}:

\begin{Scode}{fig=TRUE}
tfOnePlot(percentChange(TSget("M2", fred), lag=52), 
    title = "Running commentary, blah, blah, blah", 
    subtitle="Broad Money (M2)",
    ylab= "y/y percent change*",
    source="Source: Federal Reserve Bank of St.Louis (M2)",
    footnoteLeft = "seasonally adjusted data",
    footnoteRight = "* approximated by 52 week growth",
    lastObs = TRUE )
\end{Scode}

It is also possible to return multiple series, but they should all be of the
same frequency. (The FRED series called M2 is a weekly series). 

\begin{Scode}{fig=TRUE,height=8}
x <- TSget(c("CPIAUCNS","M2SL"), fred)
tfplot(x, 
    title = "Running commentary, blah, blah, blah", 
    subtitle=c("Consumer Price Index for All Urban Consumers: All Items",
               "Broad Money"),
    ylab= c("Index 1982-84=100", "Billions of dollars"),
    source= c("Data Source: Federal Reserve Bank of St.Louis (CPIAUCNS)",
              "Data Source: Federal Reserve Bank of St.Louis (M2SL)"),
    footnoteLeft  = c("not seasonally adjusted",  "seasonally adjusted"),
    footnoteRight = paste("Extracted:", date()),
    lastObs = TRUE )
\end{Scode}


\begin{Scode}{results=verbatim,fig=FALSE}
TSdates(c("CPIAUCNS","M2SL"), fred) 
\end{Scode}

It is possible to specify the type of object to return:

\begin{Scode}
x <- TSget(c("CPIAUCNS","M2SL"), fred, TSrepresentation="zoo")
class(x) 
\end{Scode}

The following connects to yahoo and loads the ticker symbol for Ford. 
This is a multivariate time series with open, close, etc.
\begin{Scode}{fig=TRUE,height=8}
yahoo <- TSconnect("getSymbol", dbname="yahoo") 
x <- TSget("F", con=yahoo)
plot(x)
\end{Scode}

Most of the plots in this vignette are done with the utilities in the 
\pkg{tfplot} package, but the usual \code{plot} function, used above,
produces slightly different results that may be preferable in some situations.
Also, for some time series objects, the plot method has been much improved from
the default, so if you are using these objects you may find that \code{plot} 
provides attractive features. 

In the case of the ticker data above, \code{tfplot} displays graphs in verticle 
panels. However, six panels do not nicely fit on a printed page. 
The first three are displayed with:
\begin{Scode}{fig=TRUE,height=12}
tfplot(x,series=1:3)
\end{Scode}

It is possible to specify the number of graphs on an ouput screen
with \code{graphs.per.page}, for example, \code{tfplot(x, graphs.per.page=3)}. 
Set \code{par(ask=TRUE)} if 
you want to stop and prompt for <Return> between pages in the graphics output.

The \code{quote} argument to \code{TSget} can be used to specify that only a 
subset of the market data should be returned:
\begin{Scode}{fig=TRUE}
tfOnePlot(TSget("F", con=yahoo, quote=c("Open", "Close")),
     title="Ford from Yahoo;  Open (black); Close (red)",
     ylab="Price")
\end{Scode}
%%
%%
\subsection{\pkg{TSjson} with Statistics Canada}

Package \pkg{TSjson} provides a mechanism to extract data from websites and
pass it to \proglang{R} in JavaScript Object Notation (JSON) using package
\pkg{rjson} \citep{rjson}. Package \pkg{RJSONIO} \citep{RJSONIO} has previously
been used to bring the data into \proglang{R} but is not the current method.
\pkg{TSjson} uses 
\proglang{Python} code to mechanize clicking though web pages to get a
downloadable file. This really should be considered a temporary solution, until
the data provider implements a true API. The current version of \pkg{TSjson}
provides a connection to Statistics Canada's \url{http://www.statcan.gc.ca} 
Cansim database. (You should look at the Statistics Canada site to find series 
identifiers.)

The connection can be established in two different ways. The simplest requires
a system with \proglang{Python} installed, and \proglang{Python} modules 
\proglang{sys, json, mechanize, re, csv} and \proglang{urllib2}. This requires
Python 2 as not all modules are available for Python 3. Additional details 
are provided in a README file distributed with the \pkg{TSjson} package
and also copied in an appendix here. 

Checking that an adequate Python and modules are available on the system can
be done with
\begin{Scode}{eval=FALSE}
require('findpython')

cmdExists <- can_find_python_cmd(
    minimum_version = '2.6',
    maximum_version = '2.9',
    required_modules = c('sys', 're', 'urllib2', 'csv', 'mechanize', 'json')
    )

if (!cmdExists)  stop('adequate python was not found. ')
\end{Scode}
An adequate python would normally have been needed to install package
\pkg{TSjson} but system changes after the installation could break it.

Installing Python may be difficult in environments where users cannot easily
install software, but a second method using
a proxy server can be used. The proxy server needs \proglang{Python} and the
modules, as well as server software (e.g. \proglang{Web2Py}), but the client 
machine requires nothing special other than \proglang{R} and \pkg{TSjson}.
The proxy server can be anywhere on the Internet.

The following examples use the first method. More details on establishing
connections with the second method are provided in the appendix section
(separate vignette). First, establish a connection

Although the package \pkg{TSjson} usually works, it is not really stable because
the web interface is not a proper API and can change from time to time.
For this reason, \pkg{TSjson} is not on CRAN but available on R-forge
at \url{tsdbi.r-forge.r-project.org/}.
The following examples are not being evaluated because failures
prevent reliable automatic generation of the vignette. 
\begin{Scode}{eval=FALSE}
require("TSjson")
cansim <- TSconnect("json", dbname="cansim")
\end{Scode}

Now data can be retrieved and a plot generated by
%%begin{Scode}{fig=TRUE}
%% occassional problems
\begin{Scode}{eval=FALSE,fig=FALSE}
x <- TSget("v498086", cansim)
tfplot(x)
\end{Scode}

Meta data can also be retrieved:
\begin{Scode}{eval=FALSE,results=verbatim}
TSdescription("v498086", cansim)
TSdoc("v498086", cansim)
TSlabel("v498086", cansim)
TSsource("v498086", cansim)
\end{Scode}

A transformation of the data can be done, more detail added to the graph, and
a start date specified:
%% \begin{Scode}{eval=TRUE,fig=TRUE,width=7.0,height=8}
\begin{Scode}{eval=FALSE}
tfplot(ytoypc(x), start=c(1975,1),
   ylab="Year-to-Year Growth Rate",
   title="Canadian GDP", 
   source=paste("Statistics Canada ", seriesNames(x)),
   lastObs=TRUE)
\end{Scode}

The default settings of \pkg{tfplot} parameters usually work fairly well for 
interactive use but multi-series graphs may be squashed as above when pdf is
output. The next illustrates one way to achieve more specific control,
which will often be necessary for generating pdf documents.
The series can be retreived and plotted in one step:
%% \begin{Scode}{eval=TRUE,fig=TRUE,width=7.0,height=8}
\begin{Scode}{eval=FALSE}
oldpar <- par(omi=c(0.1,0.1,0.1,0.1),mar=c(3.1,4.1,0.6,0.1))
tfplot(ytoypc(TSget(c("v498086", "v498087"), cansim)))
par(oldpar)
\end{Scode}


Sometimes it is useful to check availability:
%% \begin{Scode}{eval=TRUE,results=verbatim}
\begin{Scode}{eval=FALSE}
TSdates(c("v498086", "v498087"), cansim)
\end{Scode}

The meta data can also be retrieved with the series, which will generally be
faster than retreiving it separately, if it is needed:
%% \begin{Scode}{eval=TRUE,fig=TRUE,results=verbatim}
\begin{Scode}{eval=FALSE}
resMorg <- TSget("V122746", cansim, TSdescription=TRUE, 
                   TSdoc=TRUE, TSlabel=TRUE)
TSdescription(resMorg)
TSdoc(resMorg)
TSlabel(resMorg)
TSseriesIDs(resMorg)
TSsource(resMorg)
seriesNames(resMorg) <- "Residential Mortgage Credit (SA)"

tfplot(ytoypc(resMorg), annualizedGrowth(resMorg),
   title=seriesNames(resMorg), 
   subtitle="year-to-year (black) and annualize monthly growth (red)",
   ylab="Growth Rate",
   source=paste("Bank of Canada, ", TSsource(x)),
   lastObs=TRUE)
\end{Scode}
%%
%%
\subsection{xls}
%%
The \code{xls} interface allows the 
use of spreadsheets as if they are a database. (This is a poor substitute 
for a real database, but is sometimes convenient.) 
\code{xls} uses \code{read.xls} in package \pkg{gdata} \citep{gdata}
\code{xls} does not support writing data to the spreadsheet (but to write 
time series data to a spreadsheet see \code{TSwriteXLS} in \pkg{tframePlus},
discussed in section \ref{sec:JustData}). The spreadsheet can be a 
remote file, which is retrieved when the connection is established.

The following retrieves a file from the Reserve Bank of Australia and maps 
the elements that are used: data, dates, identifiers, and series names. 
\begin{Scode}{results=verbatim}
library("TSmisc")  
rba <- TSconnect("xls", dbname=
      "http://www.rba.gov.au/statistics/tables/xls/d03hist.xls",
    map=list(ids  =list(i=11,     j="B:Q"), 
	           data =list(i=12:627, j="B:Q"), 
	           dates=list(i=12:627, j="A"),
                   names=list(i=4:7,    j="B:Q"), 
		   description = NULL,
		   tsrepresentation = function(data,dates){
		       ts(data,start=c(1959,7), frequency=12)}))
\end{Scode}

This also illustrates how \code{tsrepresentation} can be specified as an 
arbitrary function to set the returned time series object representation.
Note also that dates may display in the spreadsheet one way 
(e.g. 02-Jan-1989), show in the spreadsheet formula box in another way
(e.g. 1989-01-02), and import still differently 
(e.g. 02/Jan/1989). And the format could change when the file is saved,
possibly depending on what features are available in the spreadsheet program.
The imported format needs to be used when tsrepresentation
is specified. If in doubt, look at the connection slot \code{@dates} 
(\code{rba@dates} above).

Beware that data is read into \proglang{R} when the connection is established, so
changes in the spreadsheet will not be visible in \proglang{R} until a new connection
is established (in contrast to other \code{TS*} packages).

\begin{Scode}{results=verbatim,fig=TRUE}
x <- TSget("DMACN", rba)
require("tfplot")
tfplot(x)
\end{Scode}

\begin{Scode}{results=verbatim,fig=TRUE,height=8}
x <- TSget(c("DMAM1N", "DMAM3N"), rba)
tfplot(x)
TSdescription(x) 
\end{Scode}

\code{tfplot} treats each series in the first argument as a panel to be plotted.
It is possible to specify the number of graphs on each page of the output device
with the argument \code{graphs.per.page}. As previously illustrated, it is 
also possible to specify that a subset of the series should be selected.
(Also, as already illustrated above, the function \code{plot} displays the series 
somewhat differently than \code{tfplot}, and possibly differently depending on
the objects time series representation.)

\code{tfplot} takes additional time series objects as arguments. Series in the
first argument are plotted in separate panels. Series in subsequent time series
objects will be plotted respectively on the same panels as the first, 
so the number of series in each object must be the same.
\begin{Scode}{results=verbatim,fig=TRUE,height=8}
tfplot(TSget(c("DMAM1S", "DMAM3S", "DMABMS"), rba),
         TSget(c("DMAM1N", "DMAM3N", "DMABMN"), rba),
    ylab=c("DMAM1", "DMAM3", "DMABM"),
    title="Australian Monetary Aggregates")
\end{Scode}
%%
\subsection{zip}
%%
The \code{zip} interface allows the 
use of zipped files that can be read by \code{read.table} as if each file is a
database series (or group of series such as high, low, open, close, for a
stock). The \code{dbname} is a directory or \code{url}. 
\code{zip} does not support writing data to the database. 

The following retrieves zipped files 
from \url{http://pitrading.com/free\_market\_data.htm} which 
provides some end of day data free of charge. 
\begin{Scode}{results=verbatim,fig=TRUE,height=8}
library("TSmisc")  
pitr <- TSconnect("zip", dbname="http://pitrading.com/free_eod_data")
z <- TSget("INDU", pitr)
tfplot(z, series=c(1,4))
\end{Scode}

The following illustrates returning an \code{xts} \citep{xts} time series object.
\begin{Scode}{results=verbatim,fig=TRUE,height=8}
require("xts")
z <- TSget(c("EURUSD", "GBPUSD"), pitr, quote=c("Open","Close"),
       TSrepresentation=xts)
tfplot(z,
       title="EURUSD and GBPUSD open and closing values from pitrading", 
       start="1995-01-01",
       par=list(omi=c(0.1,0.3,0.1,0.1),mar=c(2.1,3.1,1.0,0.1)))
\end{Scode}

The default appearance of graphs can be changed (improved) by adjusting graphics 
device margins \code{omi} and \code{mar}. (They are set by the vector in 
order: bottom, left, top, right.) They can be set directly using \code{par()}
or passed to \code{tfplot()} as in this example.
The default behaviour of \code{tfplot()} is a compromise that
usually works reasonally well for both screen and printed output.
It is often useful to adjust these when generating \emph{pdf}
files for publication.
%%
%%
\section{SQL Time Series Databases}
\label{sec:SQL}
%%
This section gives several simple examples of putting series on and reading them
from a database. (If a large number of series are to be loaded into a database,
one would typically do this with a batch process using the database program's
utilities for loading data.)
The examples in this section will use \pkg{TSMySQL} but, other than the initial
connection, access will be similar for other SQL \pkg{TS*} packages. The syntax
for connecting with other packages, and other options for connecting with 
\pkg{TSMySQL}, are provided in Appendix ``A''.

The packages \pkg{TSPostgreSQL}, \pkg{TSMySQL}, 
\pkg{TSSQLite}, and \pkg{TSodbc}
use underlying packages \pkg{RPostgreSQL} \citep{RPostgreSQL}, 
\pkg{RMySQL} \citep{RMySQL}, \pkg{RSQLite} \citep{RSQLite}, and
\pkg{RODBC} \citep{RODBC}.
The \pkg{TS*} packages provide access to an SQL database with an underlying 
table structure that is set up to store time series data. 

The next lines of code do some preliminary setup of the database. This uses the
underlying database connection (\code{dbConnect}) rather than \code{TSconnect},
because \code{TSconnect} will not recognize the database until it has been
setup. Functon \code{removeTSdbTables()} is used first to remove any existing
tables, which would cause \code{createTSdbTables()} to fail.


WARNING: running this will overwrite the ``test'' database on your server.

\begin{Scode}
setup <- RMySQL::dbConnect(RMySQL::MySQL(), dbname="test")  
TSsql::removeTSdbTables(setup, yesIknowWhatIamDoing=TRUE)
TSsql::createTSdbTables(setup, index=FALSE)
DBI::dbListTables(setup) 
DBI::dbDisconnect(setup)
\end{Scode}
%%
%%
\subsection{Writing to SQL Databases}
%%
This subsection illustrates writing some simple artifical data to a database,
and reading it back.
This part of the vignette is generated using \pkg{TSMySQL}, but other backend SQL
servers work in a similar way. See Appendix ``A'' for details of establishing
other SQL database connections.


The first thing to do is to establish a \code{TSdbi} connection to the database:

\begin{Scode}{results=hide}
library("TSMySQL")
con <- TSconnect("MySQL", dbname="test") 
\end{Scode}

\code{TSconnect} uses \code{dbConnect} from the \pkg{DBI} package, but checks that the
database has expected tables, and checks for additional features. (It cannot
be used before the tables are created, as was done above.)

The follow illustrates the use of the \pkg{TSdbi} interface, which is 
common to all extension packages.

This puts a series called \code{vec}  on the database and then reads is back
 
\begin{Scode}
z <- ts(rnorm(10), start=c(1990,1), frequency=1)
seriesNames(z) <- "vec"
if(TSexists("vec", con)) TSdelete("vec", con) 
TSput( z, con)
z <- TSget("vec", con)
\end{Scode}

Note that the series name(s) and not the \proglang{R} variable name 
(in this case, \code{vec} not \code{z}) are used on the database. 
If the retrieved series is printed it is seen to be a ``ts'' time series with some extra
attributes.

\code{TSput} fails if the series already exists on the \code{con}, 
so the above example checks and deletes the series if it already exists. 
\code{TSreplace} does not fail if the series does
not yet exist, so examples below use it instead. 
\code{TSput}, \code{TSdelete}, \code{TSreplace}, and  \code{TSexists} all
return logical values \code{TRUE} or \code{FALSE}.

Several plots below show
original data and the data retrieved after it is written to the database. 
In the plot below, one is
added to the original data so that both lines are visible.

The \proglang{R} variable can contain multiple series of the same frequency. 
They are stored separately on the database.

\begin{Scode}{results=verbatim,fig=TRUE,height=8}
z <- ts(matrix(rnorm(200),100,2), start=c(1995,1), frequency=12)
 seriesNames(z) <- c("mat2c1", "mat2c2")
TSreplace(z, con)
tfplot(z+1, TSget(c("mat2c1","mat2c2"), con),
	  lty=c("solid", "dashed"), col=c("black", "red"))
\end{Scode}
 
The following extract information about the series from the database, although
not much information has been added for these examples. 
\begin{Scode}{results=verbatim}
TSmeta("mat2c1", con)
TSmeta("vec", con)
TSdates("vec", con)
TSdescription("vec", con)
TSdoc("vec", con)
TSlabel("vec", con)
\end{Scode}

Data documentation can be in three forms. A description specified by 
\code{TSdescription},
longer documentation specified by \code{TSdoc}, or a short label, typically
useful on a graph, specified by \code{TSlabel}. 
These can be added to the time
series object, in which case they will be written to the database when
\code{TSput} or \code{TSreplace} is used to put the series on the database. 
Alternatively, they
can be specified as arguments to \code{TSput} or \code{TSreplace}.
The description or documentation will be retrieved as part of the series object
with \code{TSget} only if this is specified with the logical arguments
\code{TSdescription} and \code{TSdoc}. They can also be retrieved directly from the
database with the functions \code{TSdescription} and \code{TSdoc}.

\begin{Scode}{results=verbatim}
z <- ts(matrix(rnorm(10),10,1), start=c(1990,1), frequency=1)
TSreplace(z, serIDs="Series1", con) 
zz <- TSget("Series1", con)
TSreplace(z, serIDs="Series1", con, 
    TSdescription="short rnorm series", 
    TSdoc="Series created as an example in the vignette.")
zz <- TSget("Series1", con, TSdescription=TRUE, TSdoc=TRUE)
start(zz)
end(zz)
TSdescription(zz)
TSdoc(zz)

TSdescription("Series1", con)
TSdoc("Series1", con)
\end{Scode}

The following examples use dates and times which are not handled by \code{ts},
so the \pkg{zoo} time representation is used. It is necessary to specify the
table where the data should be stored in cases where it is difficult to
determine the periodicity of the data. See Appendix ``B'' for details of the
specific tables.

\begin{Scode}{results=verbatim,fig=TRUE,height=8}
require("zoo") 

z <- zoo(matrix(rnorm(200),100,2), as.Date("1990-01-01") + 0:99)
 seriesNames(z) <- c("zooc1", "zooc2")
TSreplace(z, con, Table="D")
tfplot(z+1, TSget(c("zooc1","zooc2"), con),
	  lty=c("solid", "dashed"), col=c("black", "red"))
 
\end{Scode}

\begin{Scode}{results=verbatim}
z <- zoo(matrix(rnorm(200),100,2), as.Date("1990-01-01") + 0:99 * 7)
seriesNames(z) <- c("zooWc1", "zooWc2")
TSreplace(z, con, Table="W")
\end{Scode}

\begin{Scode}{results=hide}
dbDisconnect(con)
detach(package:TSMySQL)
\end{Scode}
%%
\section{More Examples}
\label{sec:MoreExamples}
%%
\subsection{Examples Using SQL Databases}
\label{sec:SQLexamples}
%%
This section illustrates fetching data from the web  and loading it into the
database. This would be a very slow way to load a database, but provides
examples of different kinds of time series data. The fetching is done with 
\code{histQuote}.
Fetching data can fail due to lack of an Internet connection or delays, which
will cause the generation of this vignette to fail. 

This part of the vignette is generated using \pkg{TSPostgreSQL}, but other backend SQL
servers work in a similar way. See Appendix ``A'' for details of establishing
other SQL database connections.

First set up the TS database structure on the database where data will be
saved (if it is not already in place)
\begin{Scode}{results=verbatim}
host	<- Sys.getenv("POSTGRES_HOST")
setup <-  DBI::dbConnect(RPostgreSQL::PostgreSQL(), dbname="test", host=host) 
TSsql::createTSdbTables(setup)
DBI::dbDisconnect(setup)
\end{Scode}

Then establish a TS connection to it:
\begin{Scode}
require("TSPostgreSQL")
con <-  TSconnect("PostgreSQL", dbname="test", host=host) 
\end{Scode}

Now connect to the web server and fetch data:

\begin{Scode}{results=verbatim}
require("TSmisc")
yahoo <- TSconnect("histQuote", dbname="yahoo") 
x <- TSget("^gspc", quote = "Close", con=yahoo)
\end{Scode}

Then write the data to the local server, specifying table B for business day 
data (using TSreplace in case the series
is already there from running this example previously):

\begin{Scode}{results=verbatim}
TSreplace(x,  serIDs="gspc", Table="B", con=con)
\end{Scode}

and check the saved version:

\begin{Scode}{results=verbatim,fig=TRUE}
TSrefperiod(TSget(serIDs="gspc", con=con)) 
TSdescription("gspc", con=con) 
TSdoc("gspc", con=con)
tfplot(TSget(serIDs="gspc", con=con))
\end{Scode}

\begin{Scode}{results=verbatim,fig=TRUE,height=8}
x <- TSget("ibm", quote = c("Close", "Vol"), con=yahoo)
TSreplace(x, serIDs=c("ibm.Cl", "ibm.Vol"),  con=con, Table="B",
	   TSdescription.=c("IBM Close","IBM  Volume"),
     	   TSdoc.= paste(c(
     	      "IBM  Close   retrieved on ",
     	      "IBM  Volume retrieved  on "), Sys.Date()))

z <- TSget(serIDs=c("ibm.Cl", "ibm.Vol"), 
	  TSdescription=TRUE, TSdoc=TRUE, con=con)
TSdescription(z)
TSdoc(z)
tfplot(z, xlab = TSdoc(z), title = TSdescription(z))
tfplot(z, title="IBM", start="2007-01-01")
\end{Scode}
%%
%% this is causing problems because around July 2012 Yahoo began frequently
%% having dupicate data on the final day (seems to be partial data for the 
%% next day?), which causes zoo the complain. Hopefully Yahoo bug will be fixed.
%%\begin{Scode}{results=verbatim,fig=TRUE,height=8}
%%x <- TSget(c("^ftse","^gspc"), con)
%%plot(x)
%%\end{Scode}
%%
%%\begin{Scode}{results=verbatim,fig=TRUE,height=8}
%%tfplot(x)
%%TSdescription(x) 
%%\end{Scode}
%%
%%\begin{Scode}{results=verbatim,fig=TRUE}
%%x <- TSget("ibm", con, quote = c("Close", "Vol"))
%%tfplot(x)
%%\end{Scode}
%%
%%\begin{Scode}{results=verbatim,fig=TRUE,height=8}
%%  tfplot(x, xlab = TSdescription(x))
%%\end{Scode}
%%
%%\begin{Scode}{results=verbatim,fig=TRUE}
%%tfplot(x, title="IBM", start="2007-01-01")
%%\end{Scode}
%%
%%\begin{Scode}{results=verbatim,fig=TRUE}
%%tfplot(x, title=TSdoc(x), xlab = TSlabel(x), start="2007-01-01")
%%\end{Scode}


Oanda has maximum of 500 days, so the start date is specified here so as to
not exceed that.

\begin{Scode}{results=verbatim}
Oanda <- TSconnect("histQuote", dbname="oanda") 
x <- TSget("EUR/USD", start=Sys.Date() - 495, con=Oanda)
TSreplace(x, serIDs="EUR/USD", Table="D", con=con)
\end{Scode}

Then check the saved version:

\begin{Scode}{results=verbatim,fig=TRUE}
z <- TSget(serIDs="EUR/USD",TSlabel=TRUE, 
	       TSdescription=TRUE,  con=con)
tfplot(z, title = TSdescription(z), ylab=TSlabel(z),
	 start="2007-03-01")
\end{Scode}

\begin{Scode}{results=hide}
dbDisconnect(con)
dbDisconnect(yahoo)
dbDisconnect(Oanda)
detach(package:TSPostgreSQL)
\end{Scode}
%%
%%
%%%%\subsection{``Inhouse'' Database Examples}
%%%%\label{sec:etsExamples}
%%%%%%
%%%%These examples use a database which has been called ``ets''. It has already been
%%%%populated with some series from various sources, so
%%%%the examples illustrate how a large ``inhouse'' database might be used. 
%%%%Typically a large database would be installed using
%%%%database scripts directly rather than from \proglang{R} with \code{TSput} or
%%%%\code{TSreplace}. 
%%%%
%%%%The following uses \pkg{TSMySQL} but, other than the initial connection,
%%%%everything should remain the same if another database is used. (See Appendix A
%%%%regarding the details of other connections.)
%%%%
%%%%First, establish a connection to the database, and make it the default.
%%%%\begin{Scode}{results=verbatim,fig=FALSE}
%%%%library("TSMySQL")
%%%%ets <-  TSconnect("MySQL", dbname="ets") 
%%%%options(TSconnection=ets)
%%%%\end{Scode}
%%%%
%%%%Often one needs to check if data is available or has been updated.
%%%%This defines a list of series ids, with more explanatory names, 
%%%%and checks availability:
%%%%\begin{Scode}{results=verbatim,fig=FALSE}
%%%%  EXCH.IDs <- matrix(c(
%%%%	    "M.SDR.CCUSMA02.ST",     "SDR/USD exchange rate",
%%%%	    "M.CAN.CCUSMA02.ST",     "CAN/USD exchange rate",
%%%%	    "M.MEX.CCUSMA02.ST",     "MEX/USD exchange rate",
%%%%	    "M.JPN.CCUSMA02.ST",     "JPN/USD exchange rate",
%%%%	    "M.EMU.CCUSMA02.ST",     "Euro/USD exchange rate",
%%%%	    "M.OTO.CCUSMA02.ST",     "OECD /USD exchange rate",
%%%%	    "M.G7M.CCUSMA02.ST",     "G7   /USD exchange rate",
%%%%	    "M.E15.CCUSMA02.ST",     "Euro 15. /USD exchange rate"
%%%%	    ), 2, 8)
%%%%
%%%%  TSdates(EXCH.IDs[1,])
%%%%\end{Scode}
%%%%
%%%%(The available data in this test data set is fairly old.)
%%%%
%%%%The following examples illustrate using metadata in the graph.
%%%%\begin{Scode}{results=verbatim,fig=TRUE}
%%%%  z <- TSget("V122646", TSdescription=TRUE, TSsource=TRUE, TSdoc=TRUE,
%%%%		  TSlabel=TRUE )
%%%%  tfplot(z, title=strsplit(TSdescription(z), ","))
%%%%\end{Scode}
%%%%
%%%%Some metadata fields in my example database are empty. If they were available,
%%%%this step would not be necssary. The missing meta data can be put into 
%%%%the returned object by:
%%%%\begin{Scode}{results=verbatim}
%%%%  TSsource(z) <- "Bank of Canada"
%%%%  TSlabel(z) <- "Short-Term Business Credit"
%%%%\end{Scode}
%%%%
%%%%With this, more of the meta data can be indicated on the graph:
%%%%\begin{Scode}{results=verbatim,fig=TRUE}
%%%%  tfplot(ytoypc(z), ylab="Year-over-Year Growth",
%%%%	title=TSlabel(z),
%%%%	source=paste(TSseriesIDs(z), "from", TSsource(z)),
%%%%	lastObs=TRUE, 
%%%%	footnote=TSdescription(z))
%%%%\end{Scode}
%%%%
%%%%%%\begin{Scode}{results=verbatim,fig=TRUE}
%%%%%%  z <- TSget(c("DSP500","SDSP500"), TSdescription=TRUE)
%%%%%%  tfplot(z, xlab=TSdescription(z))
%%%%%%\end{Scode}
%%%%  
%%%%\begin{Scode}
%%%%  dbDisconnect(ets)
%%%%  options(TSconnection=NULL)
%%%%\end{Scode}
%%%%
%%%%\begin{Scode}
%%%%  detach(package:TSMySQL)
%%%%\end{Scode}
%%
%%\section{Fancy Graphs with data from several sources}
%%\label{sec:Fancy}
%%
%% do this with Can and US using cansim and FRED
%%\begin{Scode}{results=verbatim,fig=TRUE}
%%  ETSgdp    <- annualizedGrowth(aggregate(TSget("V1992067", cansim),
%%                  nfrequency=4,FUN=mean), lag=4, names="GDP Y/Y Growth")
%%  tfplot(ETSgdp)
%%\end{Scode}
%%
%%
%%
\section{Comparing Time Series Databases}
\label{sec:Comparing}
%%
The purpose of package \pkg{TScompare} is to compare pairs of series on two database.
These series might have the same name, but for generality the main function, 
\code{TScompare}, is set up to use name pairs. The pairs to compare are 
indicated by a matrix of strings with two columns. (It would also be 
possible to compare pairs on the same database, which might make sense if
the names are different.) 

The connections are established using other \pkg{TSdbi} packages such as \pkg{TSMySQL}, 
\pkg{TSPostgreSQL}, etc. It will be necessary to establish two database connections, 
so it will also be necessary to load the database specific packages. 
Examples below use \code{histQuote},  \pkg{TSMySQL} 
and \pkg{TSSQLite}. 

\begin{Scode}
library("TScompare")  
library("TSmisc")  
library("TSMySQL")  
library("TSSQLite")  
\end{Scode}

First setup database tables that are used by TSdbi using a \code{dbConnect}
connection, after which a \code{TSconnect} connection can be used.
This uses the package \code{TSsql} which does not need to be attached for
most purposes, but is needed for the initial setup and for removing \code{TSdbi}
database tables (which destroys the structure \code{TSconnect} expects).
When this is re-done it insure the databases are empty: 
\begin{Scode}{results=hide}
setup <- RMySQL::dbConnect(RMySQL::MySQL(), dbname="test")  
TSsql::removeTSdbTables(setup, yesIknowWhatIamDoing=TRUE)
TSsql::createTSdbTables(setup, index=FALSE)
DBI::dbListTables(setup) 
DBI::dbDisconnect(setup)

setup <- RSQLite::dbConnect(RSQLite::SQLite(), dbname="test")  
TSsql::removeTSdbTables(setup, yesIknowWhatIamDoing=TRUE)
TSsql::createTSdbTables(setup, index=FALSE)
DBI::dbListTables(setup) 
DBI::dbDisconnect(setup)
\end{Scode}

Now TS connections to the databases are established.
\begin{Scode}{results=hide}
con1 <- TSconnect("MySQL", dbname="test")
con2 <- TSconnect("SQLite", dbname="test") 
\end{Scode} 

Next a connection to yahoo is used to get some series and write them to the 
local test database. \code{TSreplace} is used because \code{TSput} 
will fail if the series already exisits.

%% this is a problem (Aug, 2016). No longer supported for download by yahoo?
%% x <- TSget("^ftse", yahoo)
%% TSreplace(x, serIDs="ftse", Table="B", con=con1)
%% TSreplace(x, serIDs="ftse", Table="B", con=con2)

\begin{Scode}{results=verbatim}
yahoo <- TSconnect("histQuote", dbname="yahoo") 

x <- TSget("^gspc", yahoo)
TSreplace(x,  serIDs="gspc", Table="B", con=con1)
TSreplace(x,  serIDs="gspc", Table="B", con=con2)

x <- TSget("ibm", con=yahoo, quote = c("Close", "Vol"))
TSreplace(x, serIDs=c("ibmClose", "ibmVol"), Table="B", con=con1)
TSreplace(x, serIDs=c("ibmC",	  "ibmV"),   Table="B", con=con2)
\end{Scode}

Now to do a comparison:
\begin{Scode}{results=verbatim}
ids <- AllIds(con1)
ids
\end{Scode}

If the second database has the same names then ids can be made into a 
matrix with identical columns. 

\begin{Scode}{results=verbatim}
ids <- cbind(ids, ids)

eq   <- TScompare(ids, con1, con2, na.rm=FALSE)
summary(eq)

eqrm <- TScompare(ids, con1, con2, na.rm=TRUE)
summary(eqrm)
\end{Scode}

Since names are not identical the above indicates discrepancies, which 
are resolves by indicating the corresponding name pairs:
%% ids <- matrix(c("ftse","gspc","ibmClose", "ibmVol",
%% 		"ftse","gspc","ibmC", "ibmV"),4,2)
\begin{Scode}{results=verbatim}
ids <- matrix(c("gspc","ibmClose", "ibmVol",
		"gspc","ibmC", "ibmV"),3,2)

ids
eq   <- TScompare(ids, con1, con2, na.rm=FALSE)
summary(eq)

eqrm <- TScompare(ids, con1, con2, na.rm=TRUE)
summary(eqrm)
\end{Scode}

While it may not be necessary to \code{detach} packages, the following prevents
warnings later about objects being masked:
\begin{Scode}
dbDisconnect(con1)
dbDisconnect(con2)
dbDisconnect(yahoo)
detach(package:TSMySQL)
detach(package:TSSQLite)
\end{Scode}
%%
%%
%%
\section{Vintages of Realtime Data}
\label{sec:Vintages}
%%
Examples in the section have been disabled pending availability of a different 
dataset.

Data vintages, or ``realtime data'' are snapshots of data that was available
at different points in time. The most obvious feature of earlier snapshots is
that the series end earlier. However, the reason for retaining vintages is that
data is often revised, so, for some observations,
earlier vintages have different data. Typically most revisions happen for the
most recent periods, but this is often the data of most interest for forcasting
and policy decisions. Thus, the revision records are valuable for understanding the
implications of differences between early and revised releases of the data.
A simple mechanism for accessing vintages of data is available in several
\pkg{TS*} packages. This is illustrated here with an SQL database that has been
set up with vintage support.

First establish a connection to th database and get
a vector of the available vintages:
\begin{Scode}{eval=FALSE,results=verbatim,fig=FALSE}
require("TSMySQL") 
require("tfplot") 
ets <- TSconnect("MySQL",dbname="etsv") 
v  <- TSvintages(ets)
\end{Scode}

The following checks if a certain variable (Consumer Credit -- V122707) is
available in the different vintages. (V numbers replaced B numbers circa 2003, 
so the V numbers do not exist in older vintages. This could be supported by
implementing aliases, but that has not been done here.)
\begin{Scode}{eval=FALSE,results=verbatim,fig=FALSE}
ve <- TSexists("V122707", vintage=v, con=ets)
\end{Scode}

%A temporary problem with the db loading of some vintages is fixed by
\begin{Scode}{eval=FALSE,results=hide,fig=FALSE}
ve[224:length(ve)] <- FALSE 
\end{Scode}

The vintages can them be retrieved and plotted by
\begin{Scode}{eval=FALSE,results=verbatim,fig=TRUE}
CC <- TSget(serIDs="V122707", con=ets, vintage=v[ve])

tfOnePlot(ytoypc(CC), start=c(2000,1), 
   ylab="Consumer Credit (V122707) y/y Growth",
   title=paste("Vintages", v[ve][1], "to",  v[ve][189]),
   lastObs=TRUE, source="Source: Bank of Canada")
\end{Scode}

With package \pkg{TSfame} vintages are supported if the vintages are 
stored in files
with names like ``etsmfacansim\_20110513.db''. Then the vintages can be accessed 
as follows:

\begin{Scode}{eval=FALSE}
dbs <- paste("ets /path/to/etsmfacansim_", c(
	   "20110513.db", "20060526.db", "20110520.db"), sep="")
names(dbs) <- c("2011-05-13", "2006-05-26", "2011-05-20")
           
conetsV <-TSconnect("fame", dbname=dbs, "read", current="2011-05-13")

z <- TSget("V122646", con=conetsV, vintage=c("2011-05-13", "2006-05-26"))
dbDisconnect(conetsV)
\end{Scode}

(The above example should work, but beware that I am no longer testing it
because I no longer have Fame access.)

The package \pkg{googleVis} can be used to produce a plot that is very useful
for examining vintage more closely, and finding outliers and other 
data problems. The names are used in the legend of this next plot, 
so the series names are specified in the argument to \code{ytoypc}. 
(Otherwise they get reset to indicate the year-to-year calculation, which
makes the legend messy to read.)

\begin{Scode}{eval=FALSE,fig=FALSE}
require("googleVis") 
tfVisPlot(ytoypc(CC, names=seriesNames(CC)), start=c(2006,1),
    options=list(title="Vintages of Consumer Credit (V122707) y/y Growth"))
\end{Scode}

This will produce a graph in your web browser. It is not reproduced here.
(And beware that it may not be very fast.)
Pointing your mouse at the legend of this plot will highlight the corresponding
vintage, and
pointing at the graph will give information about the source of a data point.
\begin{Scode}{eval=FALSE,fig=FALSE}
dbDisconnect(ets)
detach(package:TSMySQL)
\end{Scode}
%%
%%
%%
\section{Just Want Data in a xls or csv File}
\label{sec:JustData}
%%
Occasionally one may need to get data into a program other than \proglang{R}. 
Or, perhaps you have a friend that does not want to use \proglang{R} but would
like easy access to data. This section describes two utilities for putting time
series data obtained with \code{TSdbi} connections into .xls and .csv files. 
Since it is often straight forward to downlaod an xls or csv file from a web
site in the first place, the advantage of this will usually be only in the case
where you need to repeatedly get the data. In that case, the process can be 
automated with these tools. 

The starting point will be to get some data. Previous sections illustrate
several possibilities. For example purposes here use

\begin{Scode}{fig=FALSE}
library("TSsdmx") 
 
# RJSDMX::sdmxHelp()  # can be useful for finding series identifiers, etc

oecd <- TSconnect("sdmx", dbname="OECD")

x <- TSget('QNA.CAN|MEX|USA|.B1_GE.CARSA.Q', oecd)

\end{Scode}

%% cansim <- TSconnect("json", dbname="cansim")
%% x <- TSget(c("V36415","V122707","V122746"),  con=cansim)
%% seriesNames(x) <- c("Total", "Consumer", "ResMort")
%% library("tfplot")
%% tfplot(x, title="Canadian Household Debt")

While this is optional, it might be useful to look at the data to see if it is
really what you expect:

\begin{Scode}{fig=TRUE,width=7.0,height=12}
library("tframePlus")
library("tfplot")

tfplot(x, title="Nominal GDP at PPP? (relative to US)", 
         subtitle=paste(c("Canada","Mexico","US"),"quarterly national accounts, SAAR"),
	 ylab=rep("guess the units",3))
\end{Scode}

If the data in one series starts at a different time then \code{z} will be filled
with \code{NA} for missing points. 
It is possible to trim the \code{NA} data from the beginning with 

\begin{Scode}{results=verbatim,fig=FALSE}
start(x)
x <- trimNA(x, endNAs=FALSE)
start(x)
\end{Scode}

By default, \code{trimNA} will trim all series if any one of them is \code{NA},
so trimming the end will not be what you want if you are interested in the 
most recent data. Setting \code{endNAs=FALSE)} overrides the default.

It is also possible to trim the data to a specified subsample with

\begin{Scode}{fig=FALSE}
x <- tfwindow(x, start=c(1995,1))
\end{Scode}

Now write the data to an .xls file using \code{TSwriteXLS} which uses 
\code{WriteXLS} \citep{WriteXLS} but automatically adds time information:

\begin{Scode}{fig=FALSE}
library("WriteXLS")
tofile <- tempfile(fileext = ".xls")
TSwriteXLS(x, FileName=tofile)
unlink(tofile) 
\end{Scode}

In this case a temporary file is written, then removed with \code{unlink}, so
that scratch files do not accumulate from building this vignette, but you would
typically use a more meaningful name, and not remove the file. It is also
possible to write multiple sheets in the file. For more details on this
see the help

\begin{Scode}{eval=FALSE}
?TSwriteXLS
\end{Scode}

The dates are written in some different formats in the .xls file, since this is
often convenient for calculations and graphing in a spreadsheet program.
Beware that .xls files have size limitations that you might encounter if you try
to put large amounts of data into the file. Consider using .csv files in that
case.

To write the data to a .csv file

\begin{Scode}{fig=FALSE}
tofile <- tempfile(fileext = ".csv")
TSwriteCSV(x, FileName=tofile)
unlink(tofile) 
\end{Scode}
%%
%%
%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\bibliographystyle{apalike}
%%abbrv    apalike  plain    unsrt alpha ieeetr siam newapa
\setlength{\bibsep}{3pt}
\bibliography{Guide}

\end{document}
