Upgrading AEN (AEN 4.1.3)
=========================

.. raw:: html

    <div class="contents local topic" id="contents">
    <ul class="simple">
    <li><a class="reference internal" href="#before-you-upgrade" id="id1">Before you upgrade</a></li>
    <li><a class="reference internal" href="#upgrading-the-aen-server-node" id="id2">Upgrading the AEN server node</a></li>
    <li><a class="reference internal" href="#upgrading-the-aen-gateway-node" id="id3">Upgrading the AEN gateway node</a></li>
    <li><a class="reference internal" href="#upgrading-aen-compute-nodes" id="id4">Upgrading AEN compute nodes</a></li>
    <li><a class="reference internal" href="#after-upgrading" id="id5">After upgrading</a></li>
    </ul>
    </div>
    <p>CAUTION: These instructions are for upgrading AEN to the
    current version 4.1.3 from 4.1.2 ONLY. Each version must be
    upgraded iteratively from the previous version. Do not skip versions.</p>
    <p>Upgrade instructions for previous versions:</p>
    <ul class="simple">
    <li><a class="reference internal" href="../../../4.1.2/install/installation_update.html"><span class="doc">AEN 4.1.2 upgrade instructions</span></a></li>
    <li><a class="reference internal" href="../../../4.1.1/install/installation_update.html"><span class="doc">AEN 4.1.1 upgrade instructions</span></a>.</li>
    <li><a class="reference internal" href="../../../4.1.0/install/installation_update.html"><span class="doc">AEN 4.1.0 upgrade instructions</span></a>.</li>
    <li><a class="reference internal" href="../../../4.0/install/installation_update.html"><span class="doc">AEN 4.0.0 upgrade instructions</span></a>.</li>
    </ul>
    <p>For upgrades from versions before those listed above, please
    contact your enterprise support representative.</p>
    <p>NOTE: Named Service Account functionality is available with AEN 4.0.0+ for new
    installations only. It is not available for upgraded installations. Contact
    your enterprise support representative for more information.</p>
    <p>An AEN platform update requires that each instance of the 3 node
    types be upgraded individually:</p>
    <ul class="simple">
    <li>AEN Server</li>
    <li>AEN Gateway</li>
    <li>AEN Compute</li>
    </ul>
    <p>The upgrade process requires that all AEN service instances be
    stopped, upgraded, and then restarted.</p>
    <p>NOTE: Any commands that call for the root user can also be done
    using sudo.</p>
    <p>If you encounter any difficulty during the upgrade process, see
    <a class="reference internal" href="../troubleshooting.html"><span class="doc">Troubleshooting</span></a> which provides guidance on:</p>
    <ul class="simple">
    <li>processes</li>
    <li>configuration files</li>
    <li>log files</li>
    <li>ports</li>
    </ul>
    <p>If you are unable to resolve an installation or upgrade
    problem, please contact your enterprise support representative.</p>
    <div class="section" id="before-you-upgrade">
    <span id="aen-installation-update-preflight"></span><h2><a class="toc-backref" href="#id1">Before you upgrade</a><a class="headerlink" href="#before-you-upgrade" title="Permalink to this headline">¶</a></h2>
    <p>CAUTION: Make a tested backup of your installation before starting the upgrade.
    Upgrading to a higher version of AEN is not reversible. Any errors during the
    upgrade procedure may result in partial or complete data loss and require
    restoring data from backups.</p>
    <p>CAUTION: Terminate all AEN applications and stop all projects
    before starting the upgrade process.</p>
    <p>Before upgrading each service on each host:</p>
    <ol class="arabic">
    <li><p class="first">Suspend the services on each of the nodes:</p>
    <div class="highlight-bash"><div class="highlight"><pre><span></span>sudo service wakari-server stop
    sudo service wakari-gateway stop
    sudo service wakari-compute stop
    </pre></div>
    </div>
    </li>
    <li><p class="first">Set the AEN Functional ID (&#8220;NFI&#8221;) and AEN Functional Group (&#8220;NFG&#8221;)
    to the NFI and NFG of the current installation:</p>
    <div class="highlight-bash"><div class="highlight"><pre><span></span><span class="nb">export</span> <span class="nv">AEN_SRVC_ACCT</span><span class="o">=</span><span class="s2">&quot;wakari&quot;</span>
    <span class="nb">export</span> <span class="nv">AEN_SRVC_GRP</span><span class="o">=</span><span class="s2">&quot;wakari&quot;</span>
    </pre></div>
    </div>
    <p>NOTE: The default NFI is wakari, but aen_admin or any other
    name may be used instead.</p>
    <p>For more information on NFI and NFG, see the
    <a class="reference internal" href="prepare-for-install.html#aen-functional-id"><span class="std std-ref">installation instructions</span></a>.</p>
    </li>
    <li><p class="first">Install <code class="docutils literal"><span class="pre">wget</span></code>:</p>
    <div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">yum</span> <span class="n">install</span> <span class="n">wget</span>
    </pre></div>
    </div>
    </li>
    </ol>
    </div>
    <div class="section" id="upgrading-the-aen-server-node">
    <span id="aen-server"></span><h2><a class="toc-backref" href="#id2">Upgrading the AEN server node</a><a class="headerlink" href="#upgrading-the-aen-server-node" title="Permalink to this headline">¶</a></h2>
    <p>NOTE: If you are using LDAP-based authentication, back up the
    <code class="docutils literal"><span class="pre">/opt/wakari/wakari-server/etc/wakari/wk-server-config.json</span></code>
    configuration file. After the server has been upgraded, copy that
    file back into the same location as before the upgrade.</p>
    <p>Complete the following steps on the server host:</p>
    <ol class="arabic">
    <li><p class="first">Stop the Elasticsearch service:</p>
    <div class="highlight-bash"><div class="highlight"><pre><span></span>sudo service elasticsearch stop
    </pre></div>
    </div>
    </li>
    <li><p class="first">Remove any previous index:</p>
    <div class="highlight-bash"><div class="highlight"><pre><span></span>sudo rm -rf /var/lib/elasticsearch/*
    </pre></div>
    </div>
    <p>NOTE: You can choose to keep the old index, but if you detect
    any issues with the search capabilities after the upgrade, you
    will need to run the following to start with a clean index:</p>
    <div class="highlight-bash"><div class="highlight"><pre><span></span>sudo service wakari-server stop
    sudo service elasticsearch stop
    sudo rm -rf /var/lib/elasticsearch/*
    sudo service elasticsearch start
    sudo service wakari-server start
    </pre></div>
    </div>
    </li>
    <li><p class="first">Upgrade the server:</p>
    <div class="highlight-bash"><div class="highlight"><pre><span></span><span class="nb">pushd</span> /tmp
    wget http://j.mp/aen-server-update-4_1_3

    sudo -E -u <span class="nv">$AEN_SRVC_ACCT</span> /opt/wakari/miniconda/bin/conda install <span class="se">\</span>
                    -p /opt/wakari/wakari-server          <span class="se">\</span>
                    --file aen-server-update-4_1_3

    sudo -E -u <span class="nv">$AEN_SRVC_ACCT</span> /opt/wakari/miniconda/bin/conda install <span class="se">\</span>
                     -p /opt/wakari/wakari-server          <span class="se">\</span>
                     --no-deps                             <span class="se">\</span>
                     wakari-enterprise-server-conf-update<span class="o">=</span><span class="m">2</span>.0.5
    <span class="nb">popd</span>
    </pre></div>
    </div>
    </li>
    <li><p class="first">Start Elasticsearch:</p>
    <div class="highlight-bash"><div class="highlight"><pre><span></span>sudo service elasticsearch start
    </pre></div>
    </div>
    <p>Or, if you do not want to use the search features, edit your
    server&#8217;s <code class="docutils literal"><span class="pre">/opt/wakari/wakari-server/etc/wakari/config.json</span></code>
    file by adding the line <code class="docutils literal"><span class="pre">&quot;SEARCH_ENABLED&quot;:</span> <span class="pre">false</span></code>.</p>
    </li>
    <li><p class="first">Restart the <em>NGINX</em> server:</p>
    <p>AEN server version 4.1.3 uses Unix sockets for communication with NGINX.
    Restart NGINX to load this new configuration:</p>
    <div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">sudo</span> <span class="n">nginx</span> <span class="o">-</span><span class="n">s</span> <span class="n">stop</span>
    <span class="n">sudo</span> <span class="n">nginx</span>
    </pre></div>
    </div>
    </li>
    <li><p class="first">Start the server:</p>
    <div class="highlight-bash"><div class="highlight"><pre><span></span>sudo service wakari-server start
    </pre></div>
    </div>
    </li>
    <li><p class="first">Check that the server is running properly:</p>
    <div class="highlight-bash"><div class="highlight"><pre><span></span>sudo service wakari-server status
    </pre></div>
    </div>
    </li>
    <li><p class="first">In AEN 4.1.3 port 5000 is no longer used. It has been replaced by a unix socket
    at the location <code class="docutils literal"><span class="pre">unix:/tmp/wakari-server.sock</span></code>. The installation will make
    this change automatically. If you see NGINX errors, please check the configuration
    at <code class="docutils literal"><span class="pre">/opt/wakari/wakari-server/etc/nginx/conf.d/www.enterprise.conf:18</span></code>.</p>
    </li>
    <li><p class="first">Connect to AEN server using your web browser with the
    correct protocol (http or https), hostname and port number.</p>
    </li>
    </ol>
    </div>
    <div class="section" id="upgrading-the-aen-gateway-node">
    <span id="aen-gw"></span><h2><a class="toc-backref" href="#id3">Upgrading the AEN gateway node</a><a class="headerlink" href="#upgrading-the-aen-gateway-node" title="Permalink to this headline">¶</a></h2>
    <p>Complete the following steps on each gateway host:</p>
    <ol class="arabic">
    <li><p class="first">Upgrade the gateway:</p>
    <div class="highlight-bash"><div class="highlight"><pre><span></span><span class="nb">pushd</span> /tmp
    wget http://j.mp/aen-gateway-update-4_1_3

    sudo -E -u <span class="nv">$AEN_SRVC_ACCT</span> /opt/wakari/miniconda/bin/conda install <span class="se">\</span>
                    -p /opt/wakari/wakari-gateway         <span class="se">\</span>
                    --file aen-gateway-update-4_1_3

    sudo -E -u <span class="nv">$AEN_SRVC_ACCT</span> /opt/wakari/miniconda/bin/conda install <span class="se">\</span>
                    -p /opt/wakari/wakari-gateway         <span class="se">\</span>
                    --no-deps                             <span class="se">\</span>
                    wakari-enterprise-gateway-conf-update<span class="o">=</span><span class="m">2</span>.0.5
    <span class="nb">popd</span>
    </pre></div>
    </div>
    </li>
    <li><p class="first">Start the gateway:</p>
    <div class="highlight-bash"><div class="highlight"><pre><span></span>sudo service wakari-gateway start
    </pre></div>
    </div>
    </li>
    <li><p class="first">Check that the gateway is running properly:</p>
    <div class="highlight-bash"><div class="highlight"><pre><span></span>sudo service wakari-gateway status
    </pre></div>
    </div>
    </li>
    <li><p class="first">Connect to the gateway using your web browser with the correct
    http/https, hostname and port number.</p>
    </li>
    </ol>
    </div>
    <div class="section" id="upgrading-aen-compute-nodes">
    <span id="aen-compute"></span><h2><a class="toc-backref" href="#id4">Upgrading AEN compute nodes</a><a class="headerlink" href="#upgrading-aen-compute-nodes" title="Permalink to this headline">¶</a></h2>
    <p>Complete the following steps on each host where an AEN compute
    service is running:</p>
    <ol class="arabic">
    <li><p class="first">Check for any <code class="docutils literal"><span class="pre">wakari-indexer</span></code> processes running:</p>
    <div class="highlight-bash"><div class="highlight"><pre><span></span>ps aux <span class="p">|</span> grep wakari-indexer
    </pre></div>
    </div>
    <p>NOTE: If you stopped all the projects, you will not see any
    wakari-indexer processes running.</p>
    <p>Terminate any remaining <code class="docutils literal"><span class="pre">wakari-indexer</span></code> processes:</p>
    <div class="highlight-bash"><div class="highlight"><pre><span></span>sudo killall wakari-indexer
    </pre></div>
    </div>
    <p>NOTE: The processes killed with <code class="docutils literal"><span class="pre">killall</span></code> are run by the
    $AEN_SRVC_ACCT user, so they can be killed as root with
    <code class="docutils literal"><span class="pre">sudo</span> <span class="pre">killall</span></code> or killed as the $AEN_SRVC_ACCT user with
    <code class="docutils literal"><span class="pre">sudo</span> <span class="pre">-u</span> <span class="pre">$AEN_SRVC_ACCT</span> <span class="pre">killall</span></code>. Example commands show
    the <code class="docutils literal"><span class="pre">sudo</span> <span class="pre">killall</span></code> option.</p>
    </li>
    <li><p class="first">Check for any AEN applications processes running&#8212;Workbench,
    Viewer, Terminal or Notebook:</p>
    <div class="highlight-bash"><div class="highlight"><pre><span></span>ps aux <span class="p">|</span> grep wk-app-gateone
    ps aux <span class="p">|</span> grep wk-app-workbench
    ps aux <span class="p">|</span> grep wk-app-viewer
    ps aux <span class="p">|</span> grep wk-app-terminal
    ps aux <span class="p">|</span> grep jupyter-notebook
    </pre></div>
    </div>
    <p>NOTE: If you stopped all the projects, you will not see any
    AEN app processes running.</p>
    <p>Terminate any remaining AEN application processes by
    running one or more of the following:</p>
    <div class="highlight-bash"><div class="highlight"><pre><span></span>sudo killall wk-app-gateone
    sudo killall wk-app-workbench
    sudo killall wk-app-viewer
    sudo killall wk-app-terminal
    sudo killall jupyter-notebook
    </pre></div>
    </div>
    </li>
    <li><p class="first">Verify the contents of <code class="docutils literal"><span class="pre">/opt/wakari/anaconda/.condarc</span></code>.
    Modify it to contain the following entries, and possibly
    others if you customized the <code class="docutils literal"><span class="pre">.condarc</span></code> file. You must remove
    <code class="docutils literal"><span class="pre">auto_update_conda:</span> <span class="pre">false</span></code> so that conda will upgrade.</p>
    <p>NOTE: Modify the file as the AEN_SRVC_ACCT user (or be sure to
    keep the same ownership).</p>
    <div class="highlight-yaml"><div class="highlight"><pre><span></span><span class="l l-Scalar l-Scalar-Plain">channels</span><span class="p p-Indicator">:</span>
      <span class="p p-Indicator">-</span> <span class="l l-Scalar l-Scalar-Plain">https://conda.anaconda.org/t/&lt;TOKEN&gt;/anaconda-nb-extensions</span>
      <span class="p p-Indicator">-</span> <span class="l l-Scalar l-Scalar-Plain">r</span>
      <span class="p p-Indicator">-</span> <span class="l l-Scalar l-Scalar-Plain">https://conda.anaconda.org/wakari</span>
      <span class="p p-Indicator">-</span> <span class="l l-Scalar l-Scalar-Plain">defaults</span>

    <span class="l l-Scalar l-Scalar-Plain">create_default_packages</span><span class="p p-Indicator">:</span>
      <span class="p p-Indicator">-</span> <span class="l l-Scalar l-Scalar-Plain">anaconda-client</span>
      <span class="p p-Indicator">-</span> <span class="l l-Scalar l-Scalar-Plain">ipykernel</span>
    </pre></div>
    </div>
    <p>NOTE: Contact your enterprise support representative to get
    your token for the Anaconda channel referenced above. Replace
    <code class="docutils literal"><span class="pre">&lt;TOKEN&gt;</span></code> with the actual token from your enterprise support
    representative.</p>
    </li>
    <li><p class="first">Upgrade conda in the root environment:</p>
    <div class="highlight-bash"><div class="highlight"><pre><span></span>sudo -E -u <span class="nv">$AEN_SRVC_ACCT</span> /opt/wakari/anaconda/bin/conda install <span class="se">\</span>
                    -p /opt/wakari/anaconda <span class="nv">conda</span><span class="o">=</span><span class="m">4</span>.3.24
    </pre></div>
    </div>
    </li>
    <li><p class="first">After the successful conda upgrade, in the
    <code class="docutils literal"><span class="pre">/opt/wakari/anaconda/.condarc</span></code> file, add the
    <code class="docutils literal"><span class="pre">auto_update_conda:</span> <span class="pre">false</span></code> that you removed earlier.</p>
    <p>NOTE: You <em>must</em> disable <code class="docutils literal"><span class="pre">auto_update_conda</span></code> to prevent AEN errors. Check
    to be sure that this step is complete and correct before continuing.</p>
    </li>
    <li><p class="first">The <em>accelerate</em> package was removed from AEN 4.1.3. To avoid error
    messages, it must be removed before upgrading <em>Anaconda</em>:</p>
    <div class="highlight-default"><div class="highlight"><pre><span></span>sudo -E -u $AEN_SRVC_ACCT /opt/wakari/anaconda/bin/conda remove \
                    -p /opt/wakari/anaconda accelerate
    </pre></div>
    </div>
    </li>
    <li><p class="first">Upgrade Anaconda in the root environment:</p>
    <div class="highlight-bash"><div class="highlight"><pre><span></span><span class="nb">pushd</span> /tmp
    wget http://j.mp/aen-anaconda-update-4_1_3

    sudo -E -u <span class="nv">$AEN_SRVC_ACCT</span> /opt/wakari/anaconda/bin/conda install <span class="se">\</span>
                    -p /opt/wakari/anaconda              <span class="se">\</span>
                    --file aen-anaconda-update-4_1_3

    <span class="nb">popd</span>
    </pre></div>
    </div>
    </li>
    <li><p class="first">Initialize the root environment to prime the package cache:</p>
    <div class="highlight-bash"><div class="highlight"><pre><span></span>sudo -E -u <span class="nv">$AEN_SRVC_ACCT</span> /opt/wakari/anaconda/bin/conda create <span class="se">\</span>
                    -p /opt/wakari/testenv <span class="se">\</span>
                    --clone root
    </pre></div>
    </div>
    </li>
    <li><p class="first">Test the offline cloning step:</p>
    <div class="highlight-bash"><div class="highlight"><pre><span></span>sudo -E -u <span class="nv">$AEN_SRVC_ACCT</span> /opt/wakari/anaconda/bin/conda create <span class="se">\</span>
                    -p /opt/wakari/testenvoffline <span class="se">\</span>
                    --clone root --offline
    </pre></div>
    </div>
    </li>
    <li><p class="first">Remove the test environments:</p>
    <div class="highlight-bash"><div class="highlight"><pre><span></span>sudo rm -rf /opt/wakari/testenv
    sudo rm -rf /opt/wakari/testenvoffline
    </pre></div>
    </div>
    </li>
    <li><p class="first">Upgrade each compute service:</p>
    <div class="highlight-bash"><div class="highlight"><pre><span></span><span class="nb">pushd</span> /tmp
    wget http://j.mp/aen-compute-update-4_1_3

    sudo -E -u <span class="nv">$AEN_SRVC_ACCT</span> /opt/wakari/anaconda/bin/conda install <span class="se">\</span>
                    -p /opt/wakari/wakari-compute        <span class="se">\</span>
                    --file aen-compute-update-4_1_3

    sudo -E -u <span class="nv">$AEN_SRVC_ACCT</span> /opt/wakari/anaconda/bin/conda install <span class="se">\</span>
                    --no-deps                            <span class="se">\</span>
                    -p /opt/wakari/wakari-compute        <span class="se">\</span>
                    wakari-enterprise-compute-conf-update<span class="o">=</span><span class="m">2</span>.0.10
    <span class="nb">popd</span>
    </pre></div>
    </div>
    <p>NOTE: For AEN 4.1.0 and above, <code class="docutils literal"><span class="pre">wakari-app-gateone</span></code> is deprecated and no
    longer supported.</p>
    <p>NOTE: When upgrading the wakari-compute environment, you may
    see ImportError warnings with some nbextensions. As long as
    the Validating message is OK, the ImportError warnings are
    harmless&#8212;a consequence of the post-link presence on those
    packages.</p>
    </li>
    <li><p class="first">Install necessary dependencies:</p>
    <p>NOTE: Skip this step if you already have these dependencies
    installed from previous installations.</p>
    <div class="highlight-bash"><div class="highlight"><pre><span></span>sudo yum groupinstall <span class="s2">&quot;X Window System&quot;</span> -y
    sudo yum install git -y
    </pre></div>
    </div>
    <p>NOTE: If you don&#8217;t want to install the whole X Window System,
    you must install the following packages to have R plotting
    support:</p>
    <div class="highlight-bash"><div class="highlight"><pre><span></span>sudo yum install -y libXrender libXext libXdmcp libSM libICE libXt <span class="se">\</span>
    dejavu-sans-fonts dejavu-serif-fonts dejavu-fonts-common <span class="se">\</span>
    fontpackages-filesystem
    </pre></div>
    </div>
    </li>
    <li><p class="first">Start the compute service:</p>
    <div class="highlight-bash"><div class="highlight"><pre><span></span>sudo service wakari-compute start
    </pre></div>
    </div>
    </li>
    <li><p class="first">Verify the compute service is running properly:</p>
    <div class="highlight-bash"><div class="highlight"><pre><span></span>sudo service wakari-compute status
    </pre></div>
    </div>
    </li>
    <li><p class="first">Repeat this upgrade procedure for all compute nodes in your Data Center.</p>
    </li>
    </ol>
    </div>
    <div class="section" id="after-upgrading">
    <h2><a class="toc-backref" href="#id5">After upgrading</a><a class="headerlink" href="#after-upgrading" title="Permalink to this headline">¶</a></h2>
    <ol class="arabic">
    <li><p class="first">Restart the projects and start using AEN applications.</p>
    </li>
    <li><p class="first">If you have a <a class="reference internal" href="config/set-up-default-project-env.html"><span class="doc">customized default environment</span></a>, you may choose to
    upgrade it depending on the needs of your users.</p>
    <p>NOTE: Upgrading the default environment at
    <code class="docutils literal"><span class="pre">/opt/wakari/anaconda/envs/default</span></code> does NOT automatically
    upgrade the default environment in the users pre-existing
    projects. For pre-existing projects, the upgrade, if
    requested, should be done on a per-user basis.</p>
    </li>
    <li><p class="first">If you did not stop all your projects before upgrading, then
    the first time you start an application you will see an error
    page requesting that you restart the application.</p>
    </li>
    <li><p class="first">Restart the application to complete the upgrade.</p>
    </li>
    <li><p class="first">If you still see old applications or icons after restart, reload
    the page to reset the browser cache.</p>
    </li>
    </ol>
    <div class="toctree-wrapper compound">
    </div>
    </div>
