Anaconda Enterprise: Documentation

# Upgrading AEN (AEN 4.1.3)

- Before you upgrade
- Upgrading the AEN server node
- Upgrading the AEN gateway node
- Upgrading AEN compute nodes
- After upgrading

CAUTION: These instructions are for upgrading AEN to the current version 4.1.3 from 4.1.2 ONLY. Each version must be upgraded iteratively from the previous version. Do not skip versions.

Upgrade instructions for previous versions:

- AEN 4.1.2 upgrade instructions
- AEN 4.1.1 upgrade instructions.
- AEN 4.1.0 upgrade instructions.
- AEN 4.0.0 upgrade instructions.

For upgrades from versions before those listed above, please contact your enterprise support representative.

NOTE: Named Service Account functionality is available with AEN 4.0.0+ for new installations only. It is not available for upgraded installations. Contact your enterprise support representative for more information.

An AEN platform update requires that each instance of the 3 node types be upgraded individually:

- AEN Server
- AEN Gateway
- AEN Compute

The upgrade process requires that all AEN service instances be stopped, upgraded, and then restarted.

NOTE: Any commands that call for the root user can also be done using sudo.

If you encounter any difficulty during the upgrade process, see Troubleshooting which provides guidance on:

- processes
- configuration files
- log files
- ports

If you are unable to resolve an installation or upgrade problem, please contact your enterprise support representative.

## Before you upgrade

CAUTION: Make a tested backup of your installation before starting the upgrade. Upgrading to a higher version of AEN is not reversible. Any errors during the upgrade procedure may result in partial or complete data loss and require restoring data from backups.

CAUTION: Terminate all AEN applications and stop all projects before starting the upgrade process.

Before upgrading each service on each host:

1. Suspend the services on each of the nodes:

       sudo service wakari-server stop
       sudo service wakari-gateway stop
       sudo service wakari-compute stop

2. Set the AEN Functional ID (“NFI”) and AEN Functional Group (“NFG”) to the NFI and NFG of the current installation:

       export AEN_SRVC_ACCT="wakari"
       export AEN_SRVC_GRP="wakari"

   NOTE: The default NFI is wakari, but aen_admin or any other name may be used instead.

   For more information on NFI and NFG, see the installation instructions.

3. Install `wget`:

       yum install wget

## Upgrading the AEN server node

NOTE: If you are using LDAP-based authentication, back up the `/opt/wakari/wakari-server/etc/wakari/wk-server-config.json` configuration file. After the server has been upgraded, copy that file back into the same location as before the upgrade.

Complete the following steps on the server host:

1. Stop the Elasticsearch service:

       sudo service elasticsearch stop

2. Remove any previous index:

       sudo rm -rf /var/lib/elasticsearch/*

   NOTE: You can choose to keep the old index, but if you detect any issues with the search capabilities after the upgrade, you will need to run the following to start with a clean index:

       sudo service wakari-server stop
       sudo service elasticsearch stop
       sudo rm -rf /var/lib/elasticsearch/*
       sudo service elasticsearch start
       sudo service wakari-server start

3. Upgrade the server:

       pushd /tmp
       wget http://j.mp/aen-server-update-4_1_3

       sudo -E -u $AEN_SRVC_ACCT /opt/wakari/miniconda/bin/conda install \
                       -p /opt/wakari/wakari-server          \
                       --file aen-server-update-4_1_3

       sudo -E -u $AEN_SRVC_ACCT /opt/wakari/miniconda/bin/conda install \
                        -p /opt/wakari/wakari-server          \
                        --no-deps                             \
                        wakari-enterprise-server-conf-update=2.0.5
       popd

4. Start Elasticsearch:

       sudo service elasticsearch start

   Or, if you do not want to use the search features, edit your server’s `/opt/wakari/wakari-server/etc/wakari/config.json` file by adding the line `"SEARCH_ENABLED": false`.

5. Restart the NGINX server:

   AEN server version 4.1.3 uses Unix sockets for communication with NGINX. Restart NGINX to load this new configuration:

       sudo nginx -s stop
       sudo nginx

6. Start the server:

       sudo service wakari-server start

7. Check that the server is running properly:

       sudo service wakari-server status

8. In AEN 4.1.3 port 5000 is no longer used. It has been replaced by a unix socket at the location `unix:/tmp/wakari-server.sock`. The installation will make this change automatically. If you see NGINX errors, please check the configuration at `/opt/wakari/wakari-server/etc/nginx/conf.d/www.enterprise.conf:18`.

9. Connect to AEN server using your web browser with the correct protocol (http or https), hostname and port number.

## Upgrading the AEN gateway node

Complete the following steps on each gateway host:

1. Upgrade the gateway:

       pushd /tmp
       wget http://j.mp/aen-gateway-update-4_1_3

       sudo -E -u $AEN_SRVC_ACCT /opt/wakari/miniconda/bin/conda install \
                       -p /opt/wakari/wakari-gateway         \
                       --file aen-gateway-update-4_1_3

       sudo -E -u $AEN_SRVC_ACCT /opt/wakari/miniconda/bin/conda install \
                       -p /opt/wakari/wakari-gateway         \
                       --no-deps                             \
                       wakari-enterprise-gateway-conf-update=2.0.5
       popd

2. Start the gateway:

       sudo service wakari-gateway start

3. Check that the gateway is running properly:

       sudo service wakari-gateway status

4. Connect to the gateway using your web browser with the correct http/https, hostname and port number.

## Upgrading AEN compute nodes

Complete the following steps on each host where an AEN compute service is running:

1. Check for any `wakari-indexer` processes running:

       ps aux | grep wakari-indexer

   NOTE: If you stopped all the projects, you will not see any wakari-indexer processes running.

   Terminate any remaining `wakari-indexer` processes:

       sudo killall wakari-indexer

   NOTE: The processes killed with `killall` are run by the $AEN_SRVC_ACCT user, so they can be killed as root with `sudo killall` or killed as the $AEN_SRVC_ACCT user with `sudo -u $AEN_SRVC_ACCT killall`. Example commands show the `sudo killall` option.

2. Check for any AEN applications processes running—Workbench, Viewer, Terminal or Notebook:

       ps aux | grep wk-app-gateone
       ps aux | grep wk-app-workbench
       ps aux | grep wk-app-viewer
       ps aux | grep wk-app-terminal
       ps aux | grep jupyter-notebook

   NOTE: If you stopped all the projects, you will not see any AEN app processes running.

   Terminate any remaining AEN application processes by running one or more of the following:

       sudo killall wk-app-gateone
       sudo killall wk-app-workbench
       sudo killall wk-app-viewer
       sudo killall wk-app-terminal
       sudo killall jupyter-notebook

3. Verify the contents of `/opt/wakari/anaconda/.condarc`. Modify it to contain the following entries, and possibly others if you customized the `.condarc` file. You must remove `auto_update_conda: false` so that conda will upgrade.

   NOTE: Modify the file as the AEN_SRVC_ACCT user (or be sure to keep the same ownership).

       channels:
         - https://conda.anaconda.org/t/<TOKEN>/anaconda-nb-extensions
         - r
         - https://conda.anaconda.org/wakari
         - defaults

       create_default_packages:
         - anaconda-client
         - ipykernel

   NOTE: Contact your enterprise support representative to get your token for the Anaconda channel referenced above. Replace `<TOKEN>` with the actual token from your enterprise support representative.

4. Upgrade conda in the root environment:

       sudo -E -u $AEN_SRVC_ACCT /opt/wakari/anaconda/bin/conda install \
                       -p /opt/wakari/anaconda conda=4.3.24

5. After the successful conda upgrade, in the `/opt/wakari/anaconda/.condarc` file, add the `auto_update_conda: false` that you removed earlier.

   NOTE: You must disable `auto_update_conda` to prevent AEN errors. Check to be sure that this step is complete and correct before continuing.

6. The accelerate package was removed from AEN 4.1.3. To avoid error messages, it must be removed before upgrading Anaconda:

       sudo -E -u $AEN_SRVC_ACCT /opt/wakari/anaconda/bin/conda remove \
                       -p /opt/wakari/anaconda accelerate

7. Upgrade Anaconda in the root environment:

       pushd /tmp
       wget http://j.mp/aen-anaconda-update-4_1_3

       sudo -E -u $AEN_SRVC_ACCT /opt/wakari/anaconda/bin/conda install \
                       -p /opt/wakari/anaconda              \
                       --file aen-anaconda-update-4_1_3

       popd

8. Initialize the root environment to prime the package cache:

       sudo -E -u $AEN_SRVC_ACCT /opt/wakari/anaconda/bin/conda create \
                       -p /opt/wakari/testenv \
                       --clone root

9. Test the offline cloning step:

       sudo -E -u $AEN_SRVC_ACCT /opt/wakari/anaconda/bin/conda create \
                       -p /opt/wakari/testenvoffline \
                       --clone root --offline

10. Remove the test environments:

        sudo rm -rf /opt/wakari/testenv
        sudo rm -rf /opt/wakari/testenvoffline

11. Upgrade each compute service:

        pushd /tmp
        wget http://j.mp/aen-compute-update-4_1_3

        sudo -E -u $AEN_SRVC_ACCT /opt/wakari/anaconda/bin/conda install \
                        -p /opt/wakari/wakari-compute        \
                        --file aen-compute-update-4_1_3

        sudo -E -u $AEN_SRVC_ACCT /opt/wakari/anaconda/bin/conda install \
                        --no-deps                            \
                        -p /opt/wakari/wakari-compute        \
                        wakari-enterprise-compute-conf-update=2.0.10
        popd

    NOTE: For AEN 4.1.0 and above, `wakari-app-gateone` is deprecated and no longer supported.

    NOTE: When upgrading the wakari-compute environment, you may see ImportError warnings with some nbextensions. As long as the Validating message is OK, the ImportError warnings are harmless—a consequence of the post-link presence on those packages.

12. Install necessary dependencies:

    NOTE: Skip this step if you already have these dependencies installed from previous installations.

        sudo yum groupinstall "X Window System" -y
        sudo yum install git -y

    NOTE: If you don’t want to install the whole X Window System, you must install the following packages to have R plotting support:

        sudo yum install -y libXrender libXext libXdmcp libSM libICE libXt \
        dejavu-sans-fonts dejavu-serif-fonts dejavu-fonts-common \
        fontpackages-filesystem

13. Start the compute service:

        sudo service wakari-compute start

14. Verify the compute service is running properly:

        sudo service wakari-compute status

15. Repeat this upgrade procedure for all compute nodes in your Data Center.

## After upgrading

1. Restart the projects and start using AEN applications.

2. If you have a customized default environment, you may choose to upgrade it depending on the needs of your users.

   NOTE: Upgrading the default environment at `/opt/wakari/anaconda/envs/default` does NOT automatically upgrade the default environment in the users pre-existing projects. For pre-existing projects, the upgrade, if requested, should be done on a per-user basis.

3. If you did not stop all your projects before upgrading, then the first time you start an application you will see an error page requesting that you restart the application.

4. Restart the application to complete the upgrade.

5. If you still see old applications or icons after restart, reload the page to reset the browser cache.

- project_permissions.py (AEN 4.1.3)
- repair_team_members.py (AEN 4.1.3)