Anaconda Enterprise: Documentation

# Backing up and restoring AEN

- Document purpose
- Important notes
- Server component steps
  - Backup
    - Mongo database
    - AEN Server config files (including License file)
    - Nginx config (if needed)
    - SSL certificates (if needed)
  - Restore
    - Reinstall AEN-Server
    - Restore Mongo database
    - AEN Server config files (including License file)
    - Nginx config (if needed)
    - SSL certificates (if needed)
    - Restart server
- Gateway component steps
  - Backup
    - Config files
    - Custom .condarc file (if needed)
    - SSL certificates (if needed)
  - Restore
    - Reinstall AEN-Gateway
    - Config files
    - Custom .condarc file (if needed)
    - SSL certificates (if needed)
    - Restart gateway
- Compute component steps
  - Backup
    - Config files
    - Custom Changes (rare)
    - Create user list
    - Project files
    - Full Anaconda (option 1)
    - Partial Anaconda (option 2)
  - Restore
    - Reinstall AEN-Compute
    - Config files
    - Custom changes (rare)
    - Create users
    - Project files
    - Full Anaconda (option 1)
    - Partial Anaconda (option 2)
    - Custom environments (if needed)
    - Restart compute node

## Document purpose

This document lays out the steps to backup and restore Anaconda Enterprise Notebooks (AEN) for Disaster Recovery. It is not intended to provide High Availability. Each of the components (Server, Gateway and Compute) has its own instructions and each may be done individually as needed. The steps primarily involve creating tar files of important configuration files and data.

This document is written for a system administrator who is comfortable with basic Linux command line navigation and usage.

To migrate to a new cluster, use these backup and restore instructions to back up the system from the old cluster and restore it to the new cluster.

## Important notes

Review the Concepts page to become familiar with the different components and how they work together.

Root or sudo access is required for some commands.

CAUTION: All commands MUST be run by `$AEN_SRVC_ACCT` (the account used to run AEN) except for those commands explicitly indicated to run as root or sudo. If the commands are not run by the correct user, the installation will not work, and a full uninstallation and reinstallation will be required!

These instructions assume that the fully qualified domain name (FQDN) has not changed for any of the component nodes. If any of the FQDNs are not the same, additional steps will be needed.

## Server component steps

### Backup

#### Mongo database

This will create a single tar file called `aen_mongo_backup.tar` that includes only the database named “wakari” that is used by AEN. It also generates a log of the database backup.

NOTE: These commands must be run by `$AEN_SRVC_ACCT`.

    mongodump -db wakari -o aen_main >> mongo_backup.log
    tar -cvf aen_mongo_backup.tar aen_main

#### AEN Server config files (including License file)

Create a tar file of all of the configuration files, including any license files.

NOTE: This command must be run by `$AEN_SRVC_ACCT`.

    tar -cvf aen_server_config.tar -C /opt/wakari/ wakari-server/etc/wakari/

#### Nginx config (if needed)

Make a copy of the nginx configuration file if it has been customized. The default configuration for the AEN server is a symlink.

NOTE: This command must be run by `$AEN_SRVC_ACCT`.

    /etc/nginx/conf.d/www.enterprise.conf -> /opt/wakari/wakari-server/etc/nginx/conf.d/www.enterprise.conf

#### SSL certificates (if needed)

Make a copy of the SSL certificates files (certfiles) for the server, including the key file, and a copy of the certfile for the gateway, which is needed for verification if using self-signed or private CA signed certs.

### Restore

#### Reinstall AEN-Server

See the instructions for installing the current version of AEN-Server.

It is not necessary to upload the license, because it will be restored with the config files.

NOTE: The new installation will generate a new password for the local `$AEN_SRVC_ACCT` account.

#### Restore Mongo database

This assumes that mongo was reinstalled as part of the reinstallation of the server component. Untar the mongo database and restore it.

NOTE: These commands must be run by `$AEN_SRVC_ACCT`.

    tar -xvf aen_mongo_backup.tar
    mongorestore --drop aen_main

NOTE: The `--drop` option resets the `$AEN_SRVC_ACCT` user password and restores the database to the exact state it was in at the time of backup. Please see the MongoDB documentation for more information about mongorestore options for Mongo 2.6.

NOTE: AEN uses Mongo 2.6 by default. If you are using a different version, consult the documentation for your version.

#### AEN Server config files (including License file)

Untar the tar file of all of the configuration files, including any license files.

NOTE: This command must be run by `$AEN_SRVC_ACCT`.

    tar -xvf aen_server_config.tar -C /opt/wakari/

Make sure the files are in `/opt/wakari/wakari-server/etc/wakari/` and are owned by the `$AEN_SRVC_ACCT`.

#### Nginx config (if needed)

Make sure any modifications to the nginx configuration are either in `/etc/nginx/conf.d` or in `/opt/wakari/wakari-server/etc/nginx/conf.d/` with a proper symlink.

NOTE: This command must be run by `$AEN_SRVC_ACCT`.

    /etc/nginx/conf.d/www.enterprise.conf -> /opt/wakari/wakari-server/etc/nginx/conf.d/www.enterprise.conf

#### SSL certificates (if needed)

Move any SSL certificate files to the locations indicated in the config files.

#### Restart server

Restart the server application.

NOTE: This command must be run as root or with sudo.

    service wakari-server restart

## Gateway component steps

### Backup

#### Config files

Create a tar file of all of the configuration files.

NOTE: This command must be run by `$AEN_SRVC_ACCT`.

    tar -cvf aen_gateway_config.tar -C /opt/wakari/ wakari-gateway/etc/wakari/

#### Custom .condarc file (if needed)

Make a copy of any `/opt/wakari/miniconda/.condarc` if it has been modified.

#### SSL certificates (if needed)

Make a copy of SSL certificate files for the gateway (including the key file) and the certfile for the server (needed for verification if using self-signed or private CA signed certs).

### Restore

#### Reinstall AEN-Gateway

##### Setting variables and changing permissions

NOTE: These commands must be run by `$AEN_SRVC_ACCT`.

Run:

    export AEN_SERVER=<FQDN HOSTNAME OR IP ADDRESS> # Use the real FQDN
    export AEN_GATEWAY_PORT=8089
    export AEN_GATEWAY=<FQDN HOSTNAME OR IP ADDRESS>  # will be needed shortly
    chmod a+x aen-*.sh                # Set installer to be executable

NOTE: Change <FQDN HOSTNAME OR IP ADDRESS> to the actual fully qualified domain hostname or IP address.

NOTE: You must perform the entire procedure before closing the terminal to ensure the variable export persists. If the terminal is closed before successful installation, export the variables to continue with the installation.

##### Running the AEN gateway installer

Run:

    sudo -E ./aen-gateway-4.2.1-Linux-x86_64.sh -w $AEN_SERVER
    <license text>
    ...
    ...

    PREFIX=/opt/wakari/wakari-gateway
    Logging to /tmp/wakari_gateway.log
    ...
    ...
    Checking server name
    Please restart the Gateway after running the following command
    to connect this Gateway to the AEN Server
    ...

#### Config files

Untar the configuration files.

NOTE: This command must be run by `$AEN_SRVC_ACCT`.

    tar -xvf aen_gateway_config.tar -C /opt/wakari

Verify that the files are in `/opt/wakari/wakari-gateway/etc/wakari/` and are owned by the `$AEN_SRVC_ACCT`.

#### Custom .condarc file (if needed)

Move the custom .condarc file to `/opt/wakari/miniconda/.condarc`.

#### SSL certificates (if needed)

Move any SSL certificate files to the locations indicated in the config files.

#### Restart gateway

Restart the gateway application.

NOTE: This command must be run as root or with sudo.

    service wakari-gateway restart

## Compute component steps

### Backup

#### Config files

Create a tar file of all of the configuration files.

NOTE: This command must be run by `$AEN_SRVC_ACCT`.

    tar -cvf aen_compute_config.tar -C /opt/wakari/ wakari-compute/etc/wakari

#### Custom Changes (rare)

Manually backup any custom changes that were applied to the code. One change might be additional files in the skeleton folder:

`/opt/wakari/wakari-compute/lib/node_modules/wakari-compute-launcher/skeleton`

#### Create user list

AEN uses POSIX access control lists (ACLs) for project sharing, so the backup must preserve the ACL information. This is done with a script that creates a file named `users.lst` containing a list of all users that have access to projects on a given compute node. Download and run the script.

NOTE: These commands must be run by `$AEN_SRVC_ACCT`.

    wget https://s3.amazonaws.com/continuum-airgap/misc/wk-compute-get-acl-users.py
    chmod 755 wk-compute-get-acl-users.py
    ./wk-compute-get-acl-users.py

#### Project files

Create a tar of the projects directory with ACLs enabled. The default projects base location is `/projects`.

NOTE: This command must be run as root or with sudo.

    tar --acls -cpvf projects.tar -C <projects base location>/*

#### Full Anaconda (option 1)

If any changes have been made to the default Anaconda installation (additional packages installed or packages removed), it is necessary to backup the entire Anaconda installation.

NOTE: This command must be run by `$AEN_SRVC_ACCT`.

    tar -cvf aen_anaconda.tar -C /opt/wakari/anaconda/*

If no changes have been made to the default installation of Anaconda, you may just backup the `.condarc` file and any custom environments.

#### Partial Anaconda (option 2)

##### Custom `.condarc` file

Make a copy of `/opt/wakari/anaconda/.condarc`.

##### Custom environments (if needed)

Create a tar file of any custom shared environments.

NOTE: This command must be run by `$AEN_SRVC_ACCT`.

    tar -cvf aen_compute_envs.tar -C /opt/wakari/ anaconda/envs

NOTE: If no custom shared environments have been created, the `envs` folder will not be present.

### Restore

#### Reinstall AEN-Compute

##### Setting variables and changing permissions

NOTE: These commands must be run by `$AEN_SRVC_ACCT`.

Run:

    export AEN_SERVER=<FQDN HOSTNAME OR IP ADDRESS> # Use the real FQDN
    chmod a+x aen-*.sh                # Set installer to be executable

NOTE: Change <FQDN HOSTNAME OR IP ADDRESS> to the actual fully qualified domain hostname or IP address.

NOTE: You must perform the entire procedure before closing the terminal to ensure the variable export persists.

##### Running the AEN compute installer

Run:

    sudo -E ./aen-compute-4.2.1-Linux-x86_64.sh -w $AEN_SERVER
    ...
    ...
    PREFIX=/opt/wakari/wakari-compute
    Logging to /tmp/wakari_compute.log
    Checking server name
    ...
    ...
    Initial clone of root environment...
    Starting Wakari daemons...
    installation finished.
    Do you wish the installer to prepend the wakari-compute install location
    to PATH in your /root/.bashrc ? [yes|no]
    [no] >>> yes

#### Config files

Untar the config files.

NOTE: This command must be run by `$AEN_SRVC_ACCT`.

    tar -xvf aen_compute_config.tar -C /opt/wakari

NOTE: Verify that they are located in `/opt/wakari/wakari-compute/etc/wakari` and are owned by the `$AEN_SRVC_ACCT`.

#### Custom changes (rare)

Manually restore any custom changes you saved in the backup section. If there are changes in the skeleton directory, these files must be world readable or projects will refuse to start.

#### Create users

NOTE: Only create users with these instructions if your Linux machine is not bound to LDAP.

In order for the ACLs to be set properly on restore, all users that have permissions to the files must be available on the machine. Ask your system administrator for the proper way to do this for your system, such as using the “useradd” tool. A list of users that are needed was created in the backup process as a file named `users.lst`.

A process similar to the following `useradd` example will be suitable for most Linux systems.

NOTE: This command must be run by `$AEN_SRVC_ACCT`.

    xargs -0 -n 1 useradd --user-group < users.lst

#### Project files

Create the projects directory in the location specified in projectRoot in `wk-compute-launcher-config.json`.

NOTE: By default this directory is `/projects`.

Then untar the projects directory with ACLs.

NOTE: This command must be run as root or with sudo:

    tar --acls -xpvf projects.tar -C <projects base location>

#### Full Anaconda (option 1)

If you did a full backup of the full Anaconda installation, untar this file to `/opt/wakari/anaconda`.

NOTE: This command must be run by `$AEN_SRVC_ACCT`.

    tar -xvf aen_anaconda.tar -C /opt/wakari

#### Partial Anaconda (option 2)

Restore the custom `.condarc` file.

If you did a partial backup of the Anaconda installation, move the copy of the `.condarc` file to `/opt/wakari/anaconda/.condarc`.

#### Custom environments (if needed)

Untar any custom environments that were created to `/opt/wakari/anaconda/envs`.

NOTE: This command must be run by `$AEN_SRVC_ACCT`.

    tar -xvf aen_compute_envs.tar -C /opt/wakari

#### Restart compute node

Restart the compute-launcher application.

NOTE: This command must be run as root or with sudo.

    service wakari-compute restart