Upgrading Anaconda Enterprise Notebooks¶
Summary¶
These instructions are for upgrading Anaconda Enterprise Notebooks to the current version 4.1 from 4.0 ONLY.
For help upgrading from a version older than 4.0, see the AEN 4.0 upgrade instructions, and contact your enterprise support representative for help.
NOTE: The Named Service Account functionality is available with Anaconda Enterprise Notebooks 4.0.0 and above for new installations only, and is not available for upgraded installations.
An upgrade to the Anaconda Enterprise Notebooks Platform requires all instances of the three service categories to be upgraded individually:
- AEN Server
- AEN Gateway
- AEN Compute
A typical installation of Anaconda Enterprise Notebooks likely consists of several Compute instances running on independent Compute Nodes, and possibly multiple Gateways if there is more than one Data Center tied into Anaconda Enterprise Notebooks. More details can be found in the Overview.
The upgrade process requires that all Anaconda Enterprise Notebooks service instances be stopped, upgraded, and then restarted. These instructions describe how to perform this process.
Note: Any commands that call for the root
user can also be done
using sudo
.
If you encounter any difficulty during the process, please refer to the Troubleshooting Guide which provides guidance on:
- processes
- configuration files
- log files
- ports
In the event you are unable to resolve an installation or upgrade problem please contact your enterprise support representative.
Pre-Flight Check¶
IMPORTANT: You must have a tested backup of your installation before starting the upgrade.
IMPORTANT: We recommend that you terminate all AEN apps and stop all projects before starting the upgrade process.
Before upgrading each service on each host, you should perform the following operation:
Suspend the services on each of the nodes with the following commands:
sudo service wakari-server stop sudo service wakari-gateway stop sudo service wakari-compute stop
Set the AEN Functional ID (NFI) and AEN Functional Group (NFG), which are described in the installation instructions.
The NFI is the username of the AEN Service Account which is used to run all AEN services and is also the username of the AEN Admin account.
The NFG is the Linux group that includes the AEN Service Account, so all files and directories that have the owner NFI also have the group NFG.
When upgrading AEN, set the NFI and NFG to the NFI and NFG of the current installation.
The default NFI is “wakari”, and “aen_admin” or any other name may be used instead.
Set the NFI and NFG with these commands, replacing “aen_admin” if you are using another name:
export AEN_SRVC_ACCT="aen_admin" export AEN_SRVC_GRP="aen_admin"
AEN Server¶
NOTE: If you are using LDAP-based authentication, back up the configuration file:
/opt/wakari/wakari-server/etc/wakari/wk-server-config.json
After the Server has been upgraded, copy that file back into its location from before the upgrade.
The following operations are all done on the Server host.
If you are taking advantage of the search capabilities provided by ElasticSearch and a Java Runtime Environment (JRE 8):
Stop the ElasticSearch service:
sudo service elasticsearch stop
Remove any previous index:
sudo rm -rf /var/lib/elasticsearch/*
NOTE: You can choose to keep the old index but if you detect any issues with the search capabilities after the upgrade, you will need to perform the following steps to start with a clean index:
sudo service wakari-server stop
sudo service elasticsearch stop
sudo rm -rf /var/lib/elasticsearch/*
sudo service elasticsearch start
sudo service wakari-server start
Upgrade the Server:
pushd /tmp wget http://j.mp/aen-server-update-latest sudo -u $AEN_SRVC_ACCT /opt/wakari/miniconda/bin/conda install \ -p /opt/wakari/wakari-server \ --file aen-server-update-latest sudo -u $AEN_SRVC_ACCT /opt/wakari/miniconda/bin/conda install \ -p /opt/wakari/wakari-server \ --no-deps \ wakari-enterprise-server-conf-update popd
Start ElasticSearch:
sudo service elasticsearch start
Or, if you do not want to use the search features, edit your server’s
config.json
:/opt/wakari/wakari-server/etc/wakari/config.json
adding the line item:
"SEARCH_ENABLED": false
.Start the Server:
sudo service wakari-server start
Check that the Server is running properly:
sudo service wakari-server status
Finally, use your web browser to connect to AEN Server, using the correct protocol (http or https), hostname, and port number.
AEN Gateway¶
The following operations are all done on each Gateway host.
Upgrade the Gateway:
pushd /tmp wget http://j.mp/aen-gateway-update-latest sudo -u $AEN_SRVC_ACCT /opt/wakari/miniconda/bin/conda install \ -p /opt/wakari/wakari-gateway \ --file aen-gateway-update-latest sudo -u $AEN_SRVC_ACCT /opt/wakari/miniconda/bin/conda install \ -p /opt/wakari/wakari-gateway \ --no-deps \ wakari-enterprise-gateway-conf-update popd
Start the Gateway:
sudo service wakari-gateway start
Check that the Gateway is running properly:
sudo service wakari-gateway status
Finally, use your web browser to connect to the Gateway, using the correct protocol (http or https), hostname, and port number.
AEN Compute¶
The following operations are all done on each host where an AEN Compute service is running.
Check for any
wakari-indexer
processes running:ps aux | grep wakari-indexer
NOTE: If you stopped all the projects, you should not see any wakari-indexer processes running.
Terminate any remaining
wakari-indexer
processes:killall wakari-indexer
Check for any AEN apps (gateone, workbench, viewer, terminal, notebook) processes running:
ps aux | grep wk-app-gateone ps aux | grep wk-app-workbench ps aux | grep wk-app-viewer ps aux | grep wk-app-terminal ps aux | grep notebookwrapper
NOTE: If you stopped all the projects, you should not see any AEN app processes running.
Terminate any remaining AEN apps processes that you find:
killall wk-app-gateone killall wk-app-workbench killall wk-app-viewer killall wk-app-terminal killall notebookwrapper
Verify the contents of
/opt/wakari/anaconda/.condarc
. Modify it properly to contain the following entries (and possibly others if you customized the .condarc file).NOTE: Please modify the file as the AEN_SRVC_ACCT user (or be sure to keep the same ownership).
channels: - https://conda.anaconda.org/t/``<TOKEN>``/anaconda-nb-extensions - r - https://conda.anaconda.org/wakari - defaults create_default_packages: - anaconda-client - ipykernel auto_update_conda: false
NOTE: Contact our sales representatives to get your
<TOKEN>
for the Anaconda channel referenced above.Remove any pinning to specific conda versions from Anaconda Enterprise Notebooks 4.0.0:
sudo rm /opt/wakari/anaconda/conda-meta/pinned
As root, modify the ownership of the package cache:
chown -R $AEN_SRVC_ACCT:$AEN_SRVC_GRP /opt/wakari/anaconda/pkgs
Upgrade the Compute service.
NOTE: From AEN 4.1.0 onward, wakari-app-gateone is deprecated and no longer supported, so we remove that package with conda here.
pushd /tmp wget http://j.mp/aen-compute-update-latest wget http://j.mp/aen-anaconda-update-latest sudo -u $AEN_SRVC_ACCT /opt/wakari/anaconda/bin/conda install \ -p /opt/wakari/wakari-compute \ --file aen-compute-update-latest sudo -u $AEN_SRVC_ACCT /opt/wakari/anaconda/bin/conda remove \ -p /opt/wakari/wakari-compute \ wakari-app-gateone sudo -u $AEN_SRVC_ACCT /opt/wakari/anaconda/bin/conda install \ --no-deps \ -p /opt/wakari/wakari-compute \ wakari-enterprise-compute-conf-update sudo -u $AEN_SRVC_ACCT /opt/wakari/anaconda/bin/conda install \ -p /opt/wakari/anaconda conda=4.2.9 sudo -u $AEN_SRVC_ACCT /opt/wakari/anaconda/bin/conda remove \ -p /opt/wakari/anaconda \ spyder ipython-we pandoc keyring \ ecdsa util-linux sphinx_rtd_theme \ mklfft numbapro numbapro_cudalib \ ncurses nlopt paramiko \ wakari-client conda-manager conda-api\ conda_analysis_package sudo -u $AEN_SRVC_ACCT /opt/wakari/anaconda/bin/conda install \ -p /opt/wakari/anaconda \ --file aen-anaconda-update-latest popd
NOTE: When upgrading the wakari-compute environment nbextensions you may see ImportError warnings. As long as the Validating message is OK, the ImportError warnings are harmless.
NOTE: If the package ncurses produces a CondaOSError, remove this file:
sudo rm /opt/wakari/anaconda/lib/terminfo
After removing the terminfo file, run the last conda install command again to complete the anaconda upgrade.
Initialize the root environment to prime the package cache:
sudo -u $AEN_SRVC_ACCT /opt/wakari/anaconda/bin/conda create \ -p /opt/wakari/testenv \ --clone root
and test the offline cloning step with:
sudo -u $AEN_SRVC_ACCT /opt/wakari/anaconda/bin/conda create \ -p /opt/wakari/testenvoffline \ --clone root --offline
finally remove those test environments with:
sudo rm -rf /opt/wakari/testenv sudo rm -rf /opt/wakari/testenvoffline
Run the upgrade_410 script to make your old projects compatible with the 4.1.0 codebase:
pushd /tmp wget http://j.mp/upgrade_410 chmod a+x upgrade_410 ./upgrade_410 popd
Start the Compute service:
sudo service wakari-compute start
Verify the Compute service is running properly:
sudo service wakari-compute status
Post-Flight Check¶
- You can now start the projects and start using the AEN apps.
- If you have a customized default environment, you may choose to upgrade it depending on the needs of your users.
NOTE: Upgrading the default
environment at
/opt/wakari/anaconda/envs/default
does not automatically upgrade the
default
environment in the users pre-existing projects. For
pre-existing projects the upgrade (if it is requested) should be done on a
per-user basis.
Project Permissions¶
Since version 0.10.0 Anaconda Enterprise Notebook uses POSIX.1e access control lists to manage shared content, rather than plain POSIX user groups. If you are upgrading from an AEN version earlier than 0.5.0, please execute the project_permissions.py script on every Compute node:
/opt/wakari/wakari-compute/bin/python project_permissions.py /projects
NOTE: In some deployments, the project directory may be different
from /projects
.
Then, on the AEN Server host, run repair_team_members.py:
/opt/wakari/wakari-server/bin/python repair_team_members.py
These scripts will read all the Project and Team information from the AEN database, and re-add individuals to all the Project on all Compute nodes.