\documentclass[english]{article}
%\VignetteIndexEntry{dse Guide}
\newcommand{\proglang}{\emph}
\newcommand{\pkg}{\emph}
\newcommand{\code}{\emph}
\title{Brief User's Guide: Dynamic Systems Estimation (DSE)}
\author{Paul Gilbert}
\date{September, 2009. Copyright 1993-2009, Bank of Canada.}
\begin{document}
\maketitle
The user of this software has the right to use, reproduce and distribute
it. The Bank of Canada makes no warranties with respect to the software
or its fitness for any particular purpose. The software is distributed
by the Bank of Canada solely on an "as is" basis.
By using the software, the user agrees to accept the entire risk of using
this software.

The software documented here is available on the
the Comprehensive R Archive Network (CRAN)
<http://cran.r-project.org>. Please
check for new versions.

This Guide is generated automatically using the \proglang{R}
Sweave utilities (see F. Leisch, R News v2/3, Dec. 2002, p 28-31), so the
examples should all work. The text and examples are included
in the distributed package subdirectory inst/doc/dse-guide.Stex. Please check 
that file if there is any doubt about the example code. The output from some of
the examples is shown but, to conserve paper, much of the output is not shown.
It is intended that users should work through the examples and see the output
themselves.
  
I regularly use the code  with \proglang{R} on Linux and 
sometimes on Windows. There is an extensive set of tests which is run
on all R test platforms for packages distributed on CRAN. Please report any errors
you find. In the past, the code has also worked with \proglang{Splus} 3.3 on 
Solaris, but I no longer run this. 
There are known problems with \proglang{Splus} since version 5.

Caveat: This software is the by-product of ongoing research. It is
not a commercial product. Limited effort is put into maintaining the
documentation (but the R tools do automatically check that all functions 
and their arguments are documented in the help system, and all examples work). 
This guide may have references to functions which do not yet
work and/or have not been distributed, and the documentation may not
correspond to the current capabilities of the functions (but please report 
these problems if you find them). While the
software does many standard time-series things, it is really intended
for doing some non-standard things. The main difference between \pkg{dse}
and most widely available software is that \pkg{dse} is
designed for working with multivariate time series and 
for studying estimation techniques and forecasting models.

Constructive suggestions and comments are welcomed by the package maintainer.
\pagebreak
\SweaveOpts{eval=TRUE,echo=TRUE,results=hide,fig=FALSE,keep.source=TRUE}
\begin{Scode}{echo=FALSE,results=hide}
 options(continue="  ")
 options(width=65)
\end{Scode}

\section{Introduction to \pkg{dse}}

\section{Getting Started}

\section{General Outline of \pkg{dse} Objects and Methods}

\section{Defining a \code{TSdata} Structure}

\section{ARMA and State-Space \code{TSmodel}s}

\section{VAR and VARX \code{TSmodel}s}

\section{Model Estimation}

\section{Forecasting, Etc}

\section{Evaluation of Forecasting Models}

\section{Adding New \code{TSmodel} Classes}

\section{Adding New \code{TSdata} Classes}

\section{Mini-Reference}

\section*{Related Packages (not in this guide)}

setRNG, tframe, EvalEst, CDNmoney, tsfa, TSdbi

\pagebreak
\setcounter{section}{0}

\section{Introduction to \pkg{dse}}

\pkg{dse} was originally
designed with linear, time-invariant auto-regressive moving-average
(ARMA) models and state-space (SS) models in mind. These remain the
most well developed models and provide the basis for
the examples in this guide. 

In order to provide examples, implemented estimation
techniques and methods for converting among various representations
of time series models are used in this guide. (However, it is possible to use dse
structure and add other estimation techniques.) 
Many functions for the usual diagnostics which
are preformed with time series data and models are included in the package.
Additional information on specific functions is available through
the help facility. For details of some of the underlying theory of
ARMA and SS model equivalence and examples of some of the capabilities
of the \pkg{dse} packages see Gilbert (1993)
\footnote{P.D. Gilbert, 1993. 
''State Space and ARMA Models: An Overview of the Equivalence'', 
Bank of Canada working paper 93--4.
Available at http://www.bankofcanada.ca/1993/03/publications/research/working-paper-199/}. 
For examples where \pkg{dse} is used
to evaluate estimation methods see Gilbert (1995)
\footnote{P.D. Gilbert, 1995. 
''Combining VAR Estimation and State Space Model Reduction for 
Simple Good Predictions'', 
\textit{J. of Forecasting: Special Issue on VAR Modelling}, 14, 229--250.}.
Examples of the
use of several functions are illustrated in the files in the demo
subdirectories. (In \proglang{R} see \code{demo(package="dse")} )

\section{Getting Started}

These packages works with recent versions of the \proglang{R} 
language (Ihaka and Gentleman, 1996)
\footnote{R. Ihaka and R. Gentleman, 1996.
''R: A Language for Data Analysis and Graphics'',
\textit{Journal of Computational and Graphical Statistics}, 5(3), 299--314.} 
available at <http://cran.r-project.org>. 
\code{Italics} will be used to
indicate functions and objects, and () is frequently
added to function names to help distinguish them as such. Anything
entered after a \# is a comment in \proglang{R}.
Most examples in this guide show only the user input,
not the computer output.

If \pkg{dse} is not installed on your system, please use the usual \proglang{R}
package installation procedures.
Once \proglang{R} is started the \code{dse} packages must be made available.

\begin{Scode}
library("dse")
\end{Scode}

The code from the vignette that generates this guide can 
be loaded into an editor with \code{edit(vignette("Guide", package="dse"))}.
This uses the default editor, which can be changed using \code{options()}.

Several data sets are included with \pkg{dse} and will be used
in examples in this guide. The names of the data sets 
can be listed with

\begin{Scode}{eval=FALSE}
data(package="dse")
\end{Scode}
They are made available by

\begin{Scode}
data(eg1.DSE.data, package="dse")  
data(egJofF.1dec93.data, package="dse")  
\end{Scode}

The \pkg{dse} package requires \pkg{tframe}. It and other required packages  
will be loaded automatically. Some functions (in particular,
\code{tfplot}) are part of the \pkg{tframe} package.

Descriptions of functions and objects are available in the R help 
system once the packages are installed.

\section{General Outline of \pkg{dse} Objects and Methods}

\pkg{dse} implements three main classes of objects: \code{TSdata}, \code{TSmodel},
and \code{TSestModel}. These are respectively, representations of data, models,
and models with data and estimation information.


\code{TSdata} is an object which contains a (multivariate) time series object
called output and optionally another called input. Methods for defining
the general version of this class of object are described in the next
section and more details are provided in the help for \code{TSdata}. Input
and output correspond to what are often labelled $x$ and $y$ in econometrics
and time series discussions of ARMA models. These are sometimes called
exogenous and endogenous variables, though those terms are often not
correct for these models. Statistically, output is the variable which
is modelled and input is the conditioning data. From a practical and
computational point of view, the model forecasts output data and input
data must always be supplied. In particular, to forecasts multiple
periods into the future requires supplying input data for the future
so that the model can calculate outputs. The terms input
and output are commonly used in the engineering literature, and often
correspond to a control variable and the output from a physical system.
However, the causal interpretation in this context is not always appropriate
for other uses of time series models. In addition, even when a causal
direction is known or assumed, it is not always desirable to define
the exogenous variable as an input. If the model is to give forecasts
into the future then it may be better to define exogenous variables
as outputs and let the model forecast them, unless better forecasts
of the exogenous variables are available from other sources. One context
in which an input variable is important is to examine policy scenarios.
In this context the policy variable is defined as the input and forecasts
are produced conditioned on different assumptions about the policy.


\code{TSmodel} objects are models which are arranged to use \code{TSdata}. These
objects always have another specific class indicating the type of
model. The ARMA and SS constructor methods for ARMA \code{TSmodel}s and state-space
\code{TSmodel}s are described in a section below. Other specific classes
of \code{TSmodel}s can be defined and many of the methods in \pkg{dse}
will work with these new models, as long as they use \code{TSdata} and have
a few important methods implemented. More details on defining other
classes of models are given in another section of this guide. Details
on the representation of models are provided in the help for \code{TSmodel}
and the help for specific model constructors.


\code{TSestModel} objects are objects which contain \code{TSdata}, a \code{TSmodel}, and
some statistical information generated by \code{l(model, data)}. The \code{l()}
method originally meant likelihood, but the method returns the one-step-ahead
predictions and other information based on those predictions. Methods
for studying one-step-ahead model forecasts extract the predictions
from these objects. Other methods treat \code{TSestModel} objects as a simple
way to group together a model and data. For example, methods for studying
multi-step forecasts need to generate the forecasts, so they do not
use the predictions in the \code{TSestModel} object. More detail about 
\code{TSestModel} objects is available in the help system.

The default method for \code{TSdata()} constructs a \code{TSdata}
 object, as will
be described in the next section. The generic methods \code{\code{TSmodel}()} and
\code{TSdata()} can also be used to extract the \code{TSmodel} or \code{TSdata} object
from another object (such as a \code{TSestModel}).

The functions in \pkg{dse} can be used by starting with data and
estimating a model, or by starting with a model and producing simulated
data. The next section on \code{TSdata} starts with data, but it would be equally
possible to start with models as described in the sections on
ARMA and State-Space \code{\code{TSmodel}s}.

\section{Defining a \code{TSdata} Structure}
This section describes how to construct a \code{TSdata} structure
if you have other data you would like to use. Some installations may
have an online database and it may be possible to connect directly
to this data. See the \pkg{TSdbi} package regarding some
possibilities for doing this.

For many people the situation will be that the data is in some ASCII
file. This can be loaded into session variables with a number of standard
R functions, the most useful of which are probably \code{scan()}
and \code{read.table()}. Following is an example which reads data
from an ASCII file called ''eg1.dat'' and puts it in the variable
called \code{eg1.DSE.data} (which is also one of the available data
sets). The file is in the \pkg{dse} package directory \code{otherdata}. The
file has five columns of numbers and 364 rows. The first column just
enumerates the rows and is discarded.

\begin{Scode}
fileName <- system.file("otherdata", "eg1.dat", package="dse")
eg1.DSE.data <- t(matrix(scan(fileName),5, 364))[, 2:5]
\end{Scode}
This matrix can be used to form a \code{TSdata} object by

\begin{Scode}
eg1.DSE.data <- TSdata(input= eg1.DSE.data[,1,drop = F],
                      output= eg1.DSE.data[, 2:4, drop = F])
\end{Scode}
The matrix and the resulting \code{TSdata} object do not have a good
time scale associated with points. A better time scale can be added by

\begin{Scode}
eg1.DSE.data <-tframed(eg1.DSE.data, 
          list(start=c(1961,3), frequency=12)) 
\end{Scode}

There are several different possibilities for representing time in
\proglang{R} objects. The most common is the \code{ts} object, which is applied
in the above default \code{tframed} method to both input and output. 
Either \code{tframed} or \code{ts} can also be used directly on the matrix 
before the \code{TSdata} object
is formed. The methods from the \pkg{tframe} package are used
extensively in the \pkg{dse} package because they extend to other time 
representations in addition
to \code{ts}, and provide a mechanism for extending methods to other objects
like \code{TSdata} and \code{TSmodel}s.

Names can be given to the series with

\begin{Scode}
seriesNamesInput(eg1.DSE.data) <- "R90"
seriesNamesOutput(eg1.DSE.data) <- c("M1","GDPl2", "CPI") 
\end{Scode}

Setting the series names is not necessary but many functions can
use the names if they are available. (This overlaps somewhat with
dimnames, but is the preferred method in \pkg{dse} as it extends
to data which is not a matrix.) The \code{TSdata} object with elements
input and output is the structure which the functions in \pkg{dse}
expect. More details on this structure are available in the help for
\code{TSdata}. The input and output elements can be defined in a number of
different ways and new representations can be fairly easily added.

Once data is available a model can be estimated:

\begin{Scode}
model1 <- estVARXls(eg1.DSE.data)
model2 <- estSSMittnik(eg1.DSE.data, n=4) 
# or model2 <- estSSMittnik(eg1.DSE.data) prompts for state dimension
\end{Scode}
(Note: these models are not the same as those reported in Gilbert,1993.
In that paper a variant of \code{estVARXar} was used.) The scales
of the different series in \code{eg1.DSE.data} are very different, with the
result that the covariance matrix of the residuals from the estimation
is nearly singular. This is detected during the calculation of residual
statistics. Statistics are then calculated using only the non-degenerate
subspace and a warning message is printed. A better model might be
obtained if the data were scaled differently.

Information about the estimated models can be displayed, for example:

\begin{Scode}
summary(model1)
summary(model2)
model1
model2
stability(model1)
stability(model2)
informationTests(model1, model2)
\end{Scode}
Typing the name of an object in \proglang{R} results in the object being printed.

The function \code{tfplot} produces separate graphs for each series.

\begin{Scode}{fig=TRUE,width=8,height=8}
tfplot(model1)
\end{Scode}

Note that initial conditions have been set to zero, but the effect
of this dies out quickly. (Also note that the graph size and text labels 
may be different depending on your \code{par} settings for
graphical parameters -- it is difficult to get this good for the automatically
produced vignette, but generally it is possible to produce very good quality 
graphs in \proglang{R}.)

Other examples that display plots:

\begin{Scode}
tfplot(model2)
tfplot(eg1.DSE.data)
checkResiduals(model1)
checkResiduals(model2) 
\end{Scode}

\section{ARMA and State Space \code{TSmodel}s}
Specifying ARMA and SS models is described below, but first their
definition is outlined. The linear time-invariant ARMA representation is

\begin{equation}
A(L)y_{t}=B(L)e_{t}+C(L)u_{t}
\end{equation}
where $y_{t}$ is a $p$ dimensional vector of observed output variables,
$u_{t}$ is an $m$ dimensional vector of input variables, $e_{t}$ is a $p$ dimensional
unobserved disturbance vector process and $A$, $B$ and $C$ are matrices
of the appropriate dimension in the lag (back shift) operator $L$. VAR
models can be thought of as a special case of ARMA models with $B(L)=I$.
ARIMA models are also a special case of ARMA models.

A linear time-invariant state space representation in innovations
form is given by

\begin{eqnarray}
  z_{t} = & Fz_{t-1}  + Gu_{t}  + Ke_{t-1} \nonumber\\
  y_{t} = & Hz_{t}    + e_{t}   \nonumber
\end{eqnarray}
where $z_{t}$ is the unobserved underlying $n$ dimensional state vector,
$F$ is the state transition matrix, $G$, the input matrix, $H$, the output
matrix, and $K$, the Kalman gain. The first equation is commonly referred to as
the \emph{state transition equation} and the second as 
the \emph{measurement equation}.

\pkg{dse} also has some limited
capabilities to work with the more general non-innovations form

\begin{eqnarray}
  z_{t} = & Fz_{t-1}  + Gu_{t}  + Qn_{t} \nonumber\\
  y_{t} = & Hz_{t}    + Re_{t}  \nonumber
\end{eqnarray}
where $n_{t}$ is the system noise, $Q$, the system noise matrix, and $R$ the
output (measurement) noise matrix.

Note that the time convention implies that the input variable
$u_{t}$ can influence the state $z_{t}$ and then the output variable $y_{t}$ 
in the same time period. This convention is not always used in time-series 
models. It is important with economic data, especially at annual frequencies,
that the input can influence the output in the same period.
Another convention that is often used is to have the output in the measurement
equation depend on the state in the previous time period. Then, to achieve the
same objective, it is necessary to include the input $u_{t}$ in the measurement
equation as well. A different convention will also result in slightly 
different algrebra converting between ARMA and state-space models.

Models are specified by setting up the arrays that define the model
and grouping them into a \code{TSmodel} object. Here is an example ARMA model
with two series, a second order AR polynomial, a first order MA polynomial
and no exogenous variable:

\begin{Scode}{results=verbatim}{
AR <- array(c(1, .5, .3, 0, .2, .1, 0, .2, .05, 1, .5, .3),
            c(3,2,2)) 
MA <- array(c(1, .2, 0, .1, 0, 0, 1, .3), c(2,2,2)) 
arma <- ARMA(A=AR, B=MA, C=NULL) 
rm(AR, MA) # these can be removed from the environment as 
           #they are no longer needed 
arma 
stability(arma) 
data.arma.sim <- simulate(arma, start=c(1920,1), freq=1) 
arma <- l(arma, data.arma.sim) 
summary(arma) 
tfplot(data.arma.sim) 
tfplot(arma) 
\end{Scode}

Note that arrays are filled in the order of their dimensions, which
may not be what you expect. The function \code{l()} evaluates the
model with the simulated data. Functions generally use default values
for some arguments. For example, the length of the simulation and
the covariance of the noise can be specified. The above example uses
the default values. If \code{start} and \code{freq}, of some other way to
determine the time frame is not provided, then it is not set, with the result
that plots may be for a matrix rather than a time series, that is, points
instead of line graphs.
See the help on \code{simulate} for more details. In
the example above, \code{arma} is initially assigned an object of class \code{TSmodel},
but it is then re-assigned the value returned by \code{l()}, which
is an object of class \code{TSestModel}. Also, many functions work with different
classes of objects, and do different things depending on the class
of the argument. The function \code{tfplot()} works with objects of
class \code{TSdata} and \code{TSestModel} and also with time series matrices.

Here is an example of a state space model:

\begin{Scode}
f <- array(c(.5, .3, .2, .4), c(2,2)) #Note: do not use capital
                                 #F (=FALSE) as a variable name
h <- array(c(1, 0, 0, 1), c(2,2))
k <- array(c(.5, .3, .2, .4), c(2,2))
ss <- SS(F=f, H=h, K=k) #F is argument name not variable name
print(ss)
stability(ss)
data.ss.sim <- simulate(ss, start=c(1920,1), freq=1)
ss <- l(ss, data.ss.sim)
summary(ss)
tfplot(ss) 
\end{Scode}

Data which has been generated with \code{simulate} is a \code{TSdata} object and
can be used with estimation routines. This provides a convenient way
to generate data for estimation algorithms, but remember that estimation
will not necessarily get back to the model you start with, since there
are equivalent representations (see Gilbert, 1993). However, a good
estimate will get close to the likelihood and predictions of the original
model.

Here is an example of changing between state space and ARMA representations
using the models defined in the previous example:

\begin{Scode}
ss.from.arma <- l(toSS(arma), data.arma.sim)
arma.from.ss <- l(toARMA(ss), data.ss.sim)
summary(ss.from.arma)
summary(arma)
summary(arma.from.ss)
summary(ss)
stability(arma)
stability(ss.from.arma) 
\end{Scode}

The function \code{roots()} is used by \code{stability()} and can
be used by itself to return the roots but not evaluate their 
magnitude 
\footnote{By default the roots of an ARMA model are calculated by converting
the model to state space form, for reasons explained in Gilbert 
(2000, ''A note on the computation of time series model roots'', 
\textit{Applied Economics Letters}, 7, 423--424).
By specifying by.poly=T the method can be changed to use an expansion
of the polynomial determinant.}. When their arguments are \code{TSmodel}s 
the functions \code{toSS()} and
\code{toARMA()} return objects of class \code{TSmodel} which are
not assigned to a variable in the above example, but used in the evaluation
of \code{l()}. The models are returned as part of the \code{TSestModel} returned
by \code{l()}.

For state space models there is often interest in the underlying state series.
These can be extracted from an estimated model with the function \code{state}.

\begin{Scode}
tfplot(state(ss))
\end{Scode}

For an innovations form model the state is defined as an expectation given past
information, so the Kalman filter estimates the state exactly. For an
non-innovations form model the filter and smoother give slightly different
estimates. (These are often called one-sided and two-sided filters in the
economics literature.) An innovations form model would usually be specified 
based on some additional information about the structure of the system,
typically a physical understanding of the system in engineering, or some
theory in economics. In the absence of this, an arbitary technique is to use a
Cholesky decomposition to convert an innovations form model to an
non-innovations form model.

The filter values are automatically returned by
\code{l()} but, because of the additional time and space requirements, the
smoother values are not. The smoother is run separately by the function 
\code{smoother()}. 

\begin{Scode}
ssc <- toSSChol(ss) 

ssc <- smoother(ssc)
\end{Scode}

\begin{Scode}
tfplot(state(ssc, filter=TRUE))

tfplot(state(ssc, smoother=TRUE))
\end{Scode}

These can be compared more easily with

\begin{Scode}
tfplot(state(ssc, smoother=TRUE), state(ssc, filter=TRUE))
\end{Scode}

The term \code{state estimate} is well established, but these should not
be confused with model parameter estimates. The error in the model parameter
estimates converges to zero as the length of the series increases to infinity 
(with good estimators and assuming estimation assumptions are satisfied). State
estimation errors never converge to zero, and some authors prefer the term
\code{state prediction} because of this. The state tracking error can also be
extracted from an non-innovations form model.

\section{VAR and VARX \code{TSmodel}s}

Vector auto-regressive models (VAR) and vector auto-regressive models with
exogenous inputs (VARX) models are special cases of ARMA models
covered in the last section. (If you did not notice, please go back and 
re-read the previous section.) For the moment, this section is only here because
of the number of time I get asked if \pkg{dse} can do VARs. Sometime I might add
more special case examples here.

\section{Model Estimation}
The example data \code{eg1.DSE.data} and \code{egJofF.1dec93.data}
are available with \pkg{dse} and are used in examples in this section.

To estimate an AR model with the default number of lags:

\begin{Scode}
model.eg1.ls <- estVARXls(trimNA(eg1.DSE.data)) 
\end{Scode}

In this example trimNA removes NA padding from the ends of the data,
since the estimation method cannot handle missing values. This padding
may not be present, depending on how the data was retrieved. This data is
highly correlated and highly parameterized models result in a degenerate
covariance matrix. When this happens a warning is produced in this and 
other examples.

It is also possible to select a subsample of the data:

\begin{Scode}
subsample.data <- tfwindow(eg1.DSE.data, start=c(1972,1), 
   end=c(1992,12), warn=FALSE) 
\end{Scode}

This creates a new variable with data starting in January 1972 and
ending in December 1992. The R function \code{window} also usually works,
however the function \code{tfwindow} is typically used in \pkg{dse}
and this guide because of some programming advantages. The argument
\code{warn=FALSE} prevents some warning messages from being printed. For
example, when the specified \code{start} or \code{end} date corresponds to the start or end
date of the data, then the default \code{warn=TRUE} results in a warning that
the sample has not been truncated.

Various functions can be applied to the estimation result

\begin{Scode}
summary(model.eg1.ls)
print(model.eg1.ls)
tfplot(model.eg1.ls)
checkResiduals(model.eg1.ls) 
\end{Scode}

Other estimation techniques are available

\begin{Scode}
model.eg1.ar <- estVARXar(trimNA(eg1.DSE.data))
model.eg1.ss <- estSSfromVARX(trimNA(eg1.DSE.data))
model.eg1.bft <- bft(trimNA(eg1.DSE.data))
model.eg1.mle <- estMaxLik(estVARXls(trimNA(eg1.DSE.data),
                           max.lag=1)) # see note below 
\end{Scode}

\code{tfplot} can put multiple similar objects on a plot.

\begin{Scode}
tfplot(model.eg1.ls, model.eg1.ar)
tfplot(model.eg1.ls, model.eg1.ar, start=c(1990,1))
\end{Scode}

Most of the estimation techniques have several optional 
parameters which control the
estimation. Consult the help for the individual functions.
\code{estMaxLik} extracts data from a \code{TSestModel} and uses the model
structure and initial parameter values for the estimation. (Note:
Maximum likelihood estimation can be very slow and may not converge
in the default number of iterations. It also tends to over fit unless
used with care, so that out-of-sample performance is not good. I do
not generally recommend it, although it does offer possibilities for
constraining the structure in specific ways (e.g. fixing some model
matrix entries to zero or one). You might consider comparing mle to
other estimation techniques using functions discussed in the following
sections and in the package \pkg{EvalEst}.) In the above \code{estMaxLik} example a smaller (one lag) 
model is used. Be prepared for the estimation to take some time when models
have a large number of parameters.

An important point to note is that the one-step-ahead predictions
and related statistics returned by these estimation techniques are
calculated by evaluating \code{l(model, data)} as the final step after the
model has been estimated. This can give different results than might
be expected using the estimation residuals, particularly with respect
to initial condition effects. (For stable models initial condition
effects should not be too important. If they are an important factor
check the documentation for specific models regarding the specification
of initial conditions.)

Also remember when estimating a model that, if you want to predict
future values of a variable, it will need to be an output in the \code{TSdata}
object.

For the next example a four variable subset of the data 
in \code{egJofF.1dec93.data} will be used. This subset is extracted by

\begin{Scode}
eg4.DSE.data<- egJofF.1dec93.data
outputData(eg4.DSE.data) <- outputData(eg4.DSE.data, 
                                        series=c(1,2,6,7)) 
\end{Scode}
which selects the 1st, 2nd, 6th, and 7th series of the output data.
The following uses the currently preferred automatic estimation procedure:

\begin{Scode}
model.eg4.bb <- estBlackBox(trimNA(eg4.DSE.data), max.lag=3) 
\end{Scode}
An optional argument \code{verbose=F} will make the function print
much less detail about the steps of the procedure. The optional argument,
\code{max.lag=3}, specifies the maximum lag which should be considered.
The default \code{max.lag=12} may take a very long time for models
with several variables. \code{estBlackBox} currently uses \code{estBlackBox}4,
also known as \code{bft(..., standardize=T)} which is called the brute
force technique in Gilbert (1995).

The traditional model information criteria tests can be performed
to compare models:

\begin{Scode}
informationTests(model.eg1.ar, model.eg1.ss) 
\end{Scode}
An arbitrary number of models can be supplied. The generated table
lists several information criteria. For state space models the calculations
are done with both the number of parameters (the number of unfixed
entries in the model arrays) and the theoretical parameter space dimension.
See Gilbert (1993, 1995) for a more extensive discussion of this subject.

Note that converting among representations produces input-output
equivalent models, so that predictions, prediction errors, and any
statistics calculated from these, will be the same for the models.
However, different estimation techniques produce different models
with different predictions. So, \code{estVARXls(data)} 
and \code{toSS(estVARXls(data))}
will produce equivalent models and \code{estSSMittnik(data)} and
\code{toARMA(estSSMittnik(data))} will produce equivalent models,
but the first two will not be equivalent to the second two.

\section{Forecasting, Etc.}

The \code{TSestModel} object returned by estimation is a \code{TSmodel}
with \code{TSdata}
and some estimation information. To use different data, the new data
needs to be in a variable which is a \code{TSdata} object. For example, suppose
a model is estimated by

\begin{Scode}
eg4.DSE.model <- estVARXls(eg4.DSE.data) 
\end{Scode}
and suppose new data becomes available. Data might come directly from a database
(see the \code{TSdbi} package for an example). For the following demonstration 
purposes \code{new.data} is generated with

\begin{Scode}
new.data <- TSdata(
   input=ts(rbind(inputData(eg4.DSE.data), matrix(0.1,10,1)),
       start = start(eg4.DSE.data), 
       frequency = frequency(eg4.DSE.data)),
   output = ts(rbind(outputData(eg4.DSE.data), matrix(0.3,5,4)),
       start = start(eg4.DSE.data), 
       frequency = frequency(eg4.DSE.data))) 
\end{Scode}

This simply appends ten observations of $0.1$ onto the input and five
observations of $0.3$ onto the outputs. The function \code{ts} assigns time
series attributes which are taken from \code{eg4.DSE.data}. The model
can be evaluated with the new data by

\begin{Scode}
z <- l(TSmodel(eg4.DSE.model), trimNA(new.data)) 
\end{Scode}
Recall that \code{TSmodel()} extracts the \code{TSmodel} from the
\code{TSestModel}.
\code{trimNA} on a \code{TSdata} object removes NAs from the ends
and truncates both input and output to the same sub-sample. \code{l()}
does not easily give forecasts beyond the period where all data is
available. (Optional arguments can be used to achieve this, but the
function \code{forecast} is more convenient.)

Forecasts are conditioned on input so it must be supplied for periods
for which forecasts are to be calculated. (That is, input is not forecast
by the model.) When more data is available for input than for output,
as in \code{new.data} generated above, then \code{forecast()} will use input
data and produce a forecast of output.

\begin{Scode}
z <- forecast(TSmodel(eg4.DSE.model), new.data) 
\end{Scode}
The input data can also be specified as a separate argument. For
example, the same result will be achieved with

\begin{Scode}
z <- forecast(TSmodel(eg4.DSE.model), trimNA(new.data),
              conditioning.inputs = inputData(new.data)) 
\end{Scode}
The \code{conditioning.inputs} override input in the \code{TSdata} supplied in
the second argument to the function.

To see plots of the forecasts use

\begin{Scode}{fig=TRUE,width=8,height=12}
tfplot(z, start=c(1990,6)) 
\end{Scode}

Sometimes a forecast for input data comes from another source, perhaps
another model. Rather than construct the \code{conditioning.inputs}
as described above, another way to combine this forecast with the
historical input data is to use the argument \code{conditioning.inputs.forecasts}:

\begin{Scode}
z <- forecast(eg4.DSE.model, 
              conditioning.inputs.forecasts = matrix(0.5,6,1)) 
\end{Scode}
This would use the input data from \code{eg4.DSE.model} and append
$6$ periods of $0.5$ to it.

Some generic functions which work with the structure returned by
\code{forecast}:

\begin{Scode}
summary(z)

print(z)

tfplot(z)

tfplot(z, start=c(1990,1)) 
\end{Scode}
If you actually want the numbers from the forecast they can be extracted
with

\begin{Scode}
forecasts(z)[[1]]
\end{Scode}
The \code{[[1]]} indicates the first forecast (in this example
there is only one, but the same structures are used for other purposes
discussed below. To see a subset of the data use \code{tfwindow}:

\begin{Scode}
tfwindow(forecasts(z)[[1]], start=c(1994,1), warn=FALSE) 
\end{Scode}
This prints values starting in the first period of 1994.

The horizon for the forecast is determined by the available input
data (\code{conditioning.inputs} or \code{conditioning.inputs.forecasts}).
If neither of these are supplied then the argument \code{horizon},
which has a default value of $36$, is used to replicate the last period
of data to the indicated horizon. For models with no input variables
the argument \code{horizon} controls the length of the forecast.

\section{Evaluating Forecasting Models}
How well does the model do at forecasting? The first thing to check
is that model forecasts actually track the data more or less. The
generic function \code{tfplot()} works with results from the following functions.
Recall that the function \code{l()} applies a \code{TSmodel} to \code{TSdata} and returns
a \code{TSestModel} which includes one-step ahead forecasts. It can be used
with any \code{TSmodel} and \code{TSdata} of corresponding dimension. So

\begin{Scode}
z <- l(TSmodel(eg4.DSE.model), new.data)
\end{Scode}
applies the previously estimated model to the new data, and

\begin{Scode}
tfplot(z)
\end{Scode}
would plot the one-step ahead forecasts. The function \code{forecast} discussed
in the previous section calculates multi-step ahead forecasts from
the end of the data. For evaluating forecasting models it is more
useful to calculate forecasts within the sample of available data.
This is for two reasons. First, the forecast can be compared against
the actual outcome. Second, if the model has an input then the forecast
is conditioned on it. If data is available then the actual input data
can be used. (But beware that this is not a true test of the model's
ability to forecast if the whole sample has been used to estimate
the model.) There are two methods to calculate multi-step ahead forecasts
within the data sample. \code{featherForecasts} produces multiple period
ahead forecasts beginning at specified periods. The name comes from
the fact that the graph sometimes looks like a feather (although it
will not if the forecasts are good).

\begin{Scode}
z <- featherForecasts(TSmodel(eg4.DSE.model), new.data)
tfplot(z)
\end{Scode}
In the example above the forecasts begin by default every tenth period.
In the following example the forecasts begin at periods 20, 50, 60,
70 and 80 and forecast for 150 periods.

\begin{Scode}
z <- featherForecasts(TSmodel(eg4.DSE.model), new.data, 
 from.periods = c(20, 50, 60, 70, 80), horizon=150)
\end{Scode}
The plot looks like this:

\begin{Scode}{fig=TRUE,width=8,height=12}
tfplot(z)
\end{Scode}

The second method, \code{horizonForecasts}, produces forecasts from
every period for specified horizons.

\begin{Scode}
z <- horizonForecasts(TSmodel(eg4.DSE.model), new.data, 
                      horizons = c(1,3,6))
\end{Scode}
produces forecasts 1, 3 and 6 steps ahead. The plot looks like this:

\begin{Scode}{fig=TRUE,width=8,height=12}
tfplot(z)
\end{Scode}

The result is aligned so that the forecast for a particular period
is plotted against the actual outcome for that period. Thus, in the
last example, the plot will show the data for each period along with
the forecast produced from 1, 3, and 6 periods prior. This plot is
particularly useful for illustrating when models do well and when
they do not. A common experience with economic data is that models
do well during periods of expansion and contraction, but miss the
turning points. The forecast covariance, to be discussed next, averages
over all periods. It is quite possible that a model can indicate turning
points well but not do so well on average, and thus be overlooked
if only forecast covariance is considered. It is always useful to
keep in mind the intended use of the model.

The numbers which generate the above plot can be extracted from the
result of \code{horizonForecasts} with \code{forecasts()}. This gives an array with
the first dimension corresponding to the horizons and the time frame
aligned to correspond to the data. So \code{forecasts(z)[2,30,]} from
the above example will be the prediction made for the 30th period
from 3 periods previous (the second element indicated in \code{horizons}
is 3) and \code{forecasts(z)[3,30,]} will be the prediction made for
the 30th period from 6 periods previous (\code{horizons[3]} is 6). Remember
that these forecasts are conditioned on the supplied input data, which
means that the output variables here are forecast 1, 3 and 6 periods
ahead, but true, not forecasted, input data is used.

If the forecasts look reasonable then examine the forecast errors
more systematically. The following calculates the forecast covariances
at different horizons. 

\begin{Scode}
fc <- forecastCov(TSmodel(eg4.DSE.model), data = eg4.DSE.data)
tfplot(fc)
tfplot(forecastCov(TSmodel(eg4.DSE.model), data = eg4.DSE.data, 
                   horizons = 1:4)) 
\end{Scode}
The last example calculates for horizons from 1 to 4 rather than
the default 1 to 12. To see how the model forecasts relative to a
zero forecast and a trend forecast:

\begin{Scode}
fc <- forecastCov(TSmodel(eg4.DSE.model), data = eg4.DSE.data, 
                  zero = T, trend =T )
tfplot(fc)
\end{Scode}
This is a very useful check (and often very humbling).

You can also do out-of-sample forecast covariance analysis. This is
discussed in the \pkg{EvalEst} package vignette.

There is not yet implemented in \pkg{dse} any measure of forecast errors
which can be compared across models - inevitably the covariance of
the error is smaller for less variable series and is also affected
by scaling of the series. This may just mean that the series is easier
to predict or has a different scale, not that the forecast equation
is more brilliant. MAPE may be implemented sometime.

\section{Adding New \code{TSmodel} Classes}
\pkg{dse} uses object oriented methods for studying new estimation techniques and
other kinds of time series models. Methods were implemented for studying
Troll (Intex Solutions, Inc.) models and some neural
net architectures have also been explored. These different
model objects and estimation methods were implemented for research purposes. 
Users are encouraged to
consider specific representations used in this guide as examples in
the context of \pkg{dse}'s broader objectives.

Models used in the package are of class \code{TSmodel} with secondary
classes to indicate specific types of models. The distributed package
supports subclass \code{ARMA} and \code{SS}. 
The main methods which will be necessary for a new class of models
''xxx'' are \code{print.xxx}, \code{is.xxx}, \code{l.xxx}, \code{simulate.xxx}, 
\code{seriesNamesInput.xxx}, \code{seriesNamesOutput.xxx}, 
\code{checkConsistentDimensions.xx}x, and \code{MonteCarloSimulations.xxx}.
Also, the method \code{to.xxx} is useful for converting models from existing
classes to this new class where possible. Models should inherit from
\code{TSmodel}.

\section{Adding New \code{TSdata} Classes }
Data used by functions in this package are objects of class \code{TSdata}.
The default methods assume that this is a list with an element output
and optionally an element input, each of which is a (multivariate)
time series object. New classes of time series can be defined and
the \pkg{dse} package should work as long as the methods describe in the
\pkg{tframe} package are implemented for the new time series class. This
usually will not require any changes to \code{TSdata} methods (or anything
else in the \pkg{dse} package). 

\pagebreak
\section{Appendix I: Mini-Reference}
Following is a short list of some of the functions. The online help
contains more details on all functions, while the guides for each
package contain more complete descriptions.

\paragraph{OBJECTS}

\begin{itemize}
\item  \code{ARMA} - define an ARMA \code{TSmodel} 
\item  \code{SS} - define a state-space \code{TSmodel} 
\item \code{TSdata} - an input/output time series data structure 
\item \code{TSestModel} - a \code{TSmodel} estimated with \code{TSdata} 
\end{itemize}

\paragraph{MODEL INFORMATION}

\begin{itemize}
\item  \code{print} - display model arrays 
\item  \code{summary} - summary information about a model 
\item  \code{tfplot} - plot data or model predictions. 
\end{itemize}

\paragraph{MODEL PROPERTIES}

\begin{itemize}
\item \code{McMillan.degree} - calculate the McMillan degree of a model 
\item \code{roots} - calculate the roots of a model 
\item \code{stability} - check stability of model 
\end{itemize}

\paragraph{MODEL CONVERSION}

\begin{itemize}
\item \code{to.SS} - convert to an equivalent state space innovations representation 
\item \code{to.ARMA} - convert to an ARMA representation 
\end{itemize}

\paragraph{SIMULATION, ONE-STEP PREDICTIONS \& RELATED STATISTICS}

\begin{itemize}
\item \code{simulate} - Simulate a model to generate artificial data. 
\item \code{l} - evaluate a \code{TSmodel} with \code{TSdata} and return a \code{TSestModel} object 
\item \code{smoother} - calculate smoothed state for a state space model. 
\item \code{check.residuals} - distribution, autocorrelation and partial autocorrelation
of residuals 
\item \code{information.tests} - print model selection criteria 
\end{itemize}

\paragraph{MODEL ESTIMATION \& REDUCTION}

\begin{itemize}
\item \code{est.VARX.ls} - estimate VAR model with exogenous variable using OLS 
\item \code{est.VARX.ar} - estimate VAR model with exogenous variable using autocorrelations 
\item \code{est.SS.from.VARX} - estimate a VARX model and convert to state space 
\item \code{est.SS.Mittnik} - estimate state space model using Mittnik's markov
parameter technique 
\item \code{estMaxLik} - Maximum likelihood estimation of models. 
\item \code{est.black.box} - estimate and find the best reduced model 
\item \code{bft} - estimate and find the best reduced model by techniques
in Gilbert (1995), also referred to as est.black.box4
\item \code{reduction.Mittnik} - nested-balanced state space model reduction by
svd of Hankel generated from a model 
\end{itemize}

\paragraph{FORECAST AND FORECAST EVALUATION}

\begin{itemize}
\item \code{forecast} - generate a forecast from given model and data. 
\item \code{featherForecasts} - forecast from specified periods 
\item \code{horizonsForecasts} - forecast specified periods ahead 
\item \code{forecastCov} - calculate covariance of multi-period ahead forecasts 
\end{itemize}
\end{document}
