---
title: "Introduction to AutoML using lares"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Introduction to AutoML using lares}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(
  collapse = TRUE, comment = "#>", results = "hide",
  fig.width = 7.1, fig.height = 5.5, dpi = 70,
  error = TRUE, purl = FALSE
)
h2o::h2o.init() # Just to be sure
```

The `lares` package has multiple families of functions to help the analyst or data scientist achieve quality robust analysis without the need of much coding. One of the most complex but valuable functions we have is `h2o_automl`, which semi-automatically runs the whole pipeline of a Machine Learning model given a dataset and some customizable parameters. **AutoML** enables you to train high-quality models specific to your needs and accelerate the research and development process.

**HELP**: Before getting to the code, I recommend checking `h2o_automl`'s full documentation [here](https://laresbernardo.github.io/lares/reference/h2o_automl.html) or within your R session by running `?lares::h2o_automl`. In it you'll find a brief description of all the parameters you can set into the function to get exactly what you need and control how it behaves.

## Pipeline

In short, these are some of the things that happen on its backend:

![Mapping `h2o_automl`](https://raw.githubusercontent.com/laresbernardo/lares/master/man/figures/automl_map.png){width=100%}

1. Input a dataframe `df` and choose which one is the independent variable (`y`) you'd like to predict. You may set/change the `seed` argument to guarantee reproducibility of your results.

2. The function decides if it's a classification (categorical) or regression (continuous) model looking at the independent variable's (`y`) class and number of unique values, which can be control with the `thresh` parameter.

3. The dataframe will be split in two: test and train datasets. The proportion of this split can be control with the `split` argument. This can be replicated with the `msplit()` function.

4. You could also `center` and `scale` your numerical values before you continue, use the `no_outliers` to exclude some outliers, and/or `impute` missing values with `MICE`. If it's a classification model, the function can balance (under-sample) your training data. You can control this behavior with the `balance` argument. Until here, you can replicate the whole process with the `model_preprocess()` function.

5. Runs `h2o::h2o.automl(...)` to train multiple models and generate a leaderboard with the top (`max_models` or `max_time`) models trained, sorted by their performance. You can also customize some additional arguments such as `nfolds` for k-fold cross-validations, `exclude_algos` and `include_algos` to exclude or include some algorithms, and any other additional argument you wish to pass to the mother function.

6. The best model given the default performance metric (which can be changed with `stopping_metric` parameter) evaluated with cross-validation (customize it with `nfolds`), will be selected to continue. You can also use the function `h2o_selectmodel()` to select another model and recalculate/plot everything again using this alternate model.

7. Performance metrics and plots will be calculated and rendered given the test predictions and test actual values (which were NOT passed to the models as inputs to be trained with). That way, your model's performance metrics shouldn't be biased. You can replicate these calculations with the `model_metrics()` function.

8. A list with all the inputs, leaderboard results, best selected model, performance metrics, and plots. You can either (play) see the results on console or export them using the `export_results()` function.

## Load the library

Now, let's (install and) load the library, the data, and dig in:
```{r load, eval=TRUE}
# install.packages("lares")
library(lares)

# The data we'll use is the Titanic dataset
data(dft)
df <- subset(dft, select = -c(Ticket, PassengerId, Cabin))
```

**NOTE**: I'll randomly set some parameters on each example to give visibility on some of the arguments you can set to your models. Be sure to also check all the print, warnings, and messages shown throughout the process as they may have relevant information regarding your inputs and the backend operations.

## Modeling examples

Let's have a look at three specific examples: **classification models (binary and multiple categories) and a regression model**. Also, let's see how we can export our models and put them to work on any environment.

### Classification: Binary

Let's begin with a binary (TRUE/FALSE) model to predict if each passenger `Survived`:

```{r class_2, results="show"}
r <- h2o_automl(df, y = Survived, max_models = 1, impute = FALSE, target = "TRUE")
```

Let's take a look at the plots generated into a single dashboard:

```{r class_2_print, results="show"}
plot(r)
```

We also have several calculations for our model's performance that may come useful such as a confusion matrix, gain and lift by percentile, area under the curve (AUC), accuracy (ACC), recall or true positive rate (TPR), cross-validation metrics, exact thresholds to maximize each metric, and others:

```{r class_2_metrics, results="show"}
r$metrics
```

The same goes for the plots generated for these metrics. We have the gains and response plots on test data-set, confusion matrix, and ROC curves.

```{r class_2_metrics_plots, results="show"}
r$plots$metrics
```

For all models, regardless of their type (classification or regression), you can check the importance of each variable as well:

```{r class_2_importance, results="show"}
head(r$importance)

r$plots$importance
```

### Classification: Multi-Categorical

Now, let's run a multi-categorical (+2 labels) model to predict `Pclass` of each passenger:

```{r class_3, results="show"}
r <- h2o_automl(df, Pclass, ignore = c("Fare", "Cabin"), max_time = 30, plots = FALSE)
```

Let's take a look at the plots generated into a single dashboard:

```{r class_3_results, results="show"}
plot(r)
```

### Regression

Finally, a regression model with continuous values to predict `Fare` payed by passenger:

```{r regression, results="show"}
r <- h2o_automl(df, y = "Fare", ignore = "Pclass", exclude_algos = NULL, quiet = TRUE)
print(r)
```

Let's take a look at the plots generated into a single dashboard:

```{r regression_plots, results="show"}
plot(r)
```

## Export models and results

Once you have you model trained and picked, you can export the model and it's results, so you can put it to work in a production environment (doesn't have to be R). There is a function that does all that for you: `export_results()`. Simply pass your `h2o_automl` list object into this function and that's it! You can select which formats will be exported using the `which` argument. Currently we support: `txt`, `csv`, `rds`, `binary`, `mojo` [best format for production], and `plots`. There are also 2 quick options (`dev` and `production`) to export some or all the files. Lastly, you can set a custom `subdir` to gather everything into a new sub-directory; I'd recommend using the model's name or any other convention that helps you know which one's which.

## Import and use your models

If you'd like to re-use your exported models to predict new datasets, you have several options:

- `h2o_predict_MOJO()` *[recommended]*: This function lets the user predict using `h2o`'s `.zip` file containing the MOJO files. These files are also the ones used when putting the model into production on any other environment. Also, MOJO let's you change `h2o`'s versions without issues
- `h2o_predict_binary()`: This function lets the user predict using the h2o binary file. The `h2o` version/build must match for it to work.
- `h2o_predict_model()`: This function lets the user run predictions from a `H2O Model Object` same as you'd use the `predict` base function. Will probably only work in your current session as you must have the actual trained object to use it.

## Addittional Posts

- DataScience+: [Visualizations for Classification Models Results](https://datascienceplus.com/machine-learning-results-one-plot-to-rule-them-all/)
- DataScience+: [Visualizations for Regression Models Results](https://datascienceplus.com/machine-learning-results-in-r-one-plot-to-rule-them-all-part-2-regression-models/)

```{r h2o_shutdown, echo=FALSE, include=FALSE, message=FALSE}
h2o::h2o.shutdown(prompt = FALSE)
Sys.sleep(2)
```
