User Guide

Overview

Using Anaconda Scale with Anaconda Adam, you can create and manage conda packages, including Python and R, in multiple conda environments across your nodes. You can prepend adam scale to a subset of familiar conda commands to install and manage conda packages and environments across a cluster.

$ adam scale -n cluster conda --help
Usage: adam conda [OPTIONS] COMMAND [ARGS]...

Options:
  -h, --help          Show this message and exit.

Commands:
  create   Create conda environment
  info     Get information about conda installation
  install  Install package(s)
  list     List conda packages
  remove   Remove package(s)
  update   Update package(s)

By default, the root conda environment is installed in /opt/continuum/anaconda on all of the cluster nodes, which can be configured for a different location. You can manage multiple conda environments across the same cluster (e.g., Python 2, Python 3, and R) without affecting the framework installation of Python on the cluster nodes.

Install Conda Packages

You can install conda packages on the cluster nodes using adam scale conda install.

For example, to install numpy and pandas and their dependencies into the root conda environment on a cluster named cluster:

$ adam scale -n cluster conda install numpy pandas
All nodes (x4) response:
{
  "actions": {
    "LINK": [
      "mkl-11.3.3-0 /opt/continuum/anaconda/pkgs 1",
      "openssl-1.0.2h-1 /opt/continuum/anaconda/pkgs 1",
      "sqlite-3.13.0-0 /opt/continuum/anaconda/pkgs 1",
      "zlib-1.2.8-3 /opt/continuum/anaconda/pkgs 1",
      "python-2.7.11-5 /opt/continuum/anaconda/pkgs 1",
      "conda-env-2.5.1-py27_0 /opt/continuum/anaconda/pkgs 1",
      "numpy-1.11.0-py27_2 /opt/continuum/anaconda/pkgs 1",
      "pycosat-0.6.1-py27_1 /opt/continuum/anaconda/pkgs 1",
      "pytz-2016.4-py27_0 /opt/continuum/anaconda/pkgs 1",
      "pyyaml-3.11-py27_4 /opt/continuum/anaconda/pkgs 1",
      "requests-2.10.0-py27_0 /opt/continuum/anaconda/pkgs 1",
      "ruamel_yaml-0.11.7-py27_0 /opt/continuum/anaconda/pkgs 1",
      "setuptools-23.0.0-py27_0 /opt/continuum/anaconda/pkgs 1",
      "six-1.10.0-py27_0 /opt/continuum/anaconda/pkgs 1",
      "wheel-0.29.0-py27_0 /opt/continuum/anaconda/pkgs 1",
      "conda-4.1.4-py27_0 /opt/continuum/anaconda/pkgs 1",
      "pip-8.1.2-py27_0 /opt/continuum/anaconda/pkgs 1",
      "python-dateutil-2.5.3-py27_0 /opt/continuum/anaconda/pkgs 1",
      "pandas-0.18.1-np111py27_0 /opt/continuum/anaconda/pkgs 1"
    ],
    "PREFIX": "/opt/continuum/anaconda",
  },
  "success": true
}

Create Conda Environments

You can create multiple conda environments on the cluster nodes using adam scale conda create.

For example, to create a conda environment called py3 with Python 3 on a cluster named cluster:

$ adam scale -n cluster conda create -n py3 python=3
All nodes (x4) response:
{
  "actions": {
    "LINK": [
      "openssl-1.0.2h-1 1 None",
      "readline-6.2-2 1 None",
      "sqlite-3.13.0-0 1 None",
      "tk-8.5.18-0 1 None",
      "xz-5.2.2-0 1 None",
      "zlib-1.2.8-3 1 None",
      "python-3.5.1-5 1 None",
      "setuptools-23.0.0-py35_0 1 None",
      "wheel-0.29.0-py35_0 1 None",
      "pip-8.1.2-py35_0 1 None"
    ],
    "PREFIX": "/opt/continuum/anaconda/envs/py3",
    "SYMLINK_CONDA": [
      "/opt/continuum/anaconda"
    ],
  },
  "success": true
}

Update Conda Packages

You can update conda packages on the cluster nodes using adam scale conda update.

For example, to update pandas and its dependencies in the root conda environment on a cluster named cluster:

$ adam scale -n cluster conda update pandas
All nodes (x4) response:
{
  "actions": {
    "LINK": [
      "pandas-0.18.1-np111py27_0 1 None"
    ],
    "PREFIX": "/opt/continuum/anaconda",
    "SYMLINK_CONDA": [
      "/opt/continuum/anaconda"
    ],
    "UNLINK": [
      "pandas-0.18.0-np111py27_0"
    ],
  },
  "success": true
}

Remove Conda Packages or Environments

You can remove conda packages on the cluster nodes using adam scale conda remove.

For example, to remove pandas from the root conda environment on a cluster named cluster:

$ adam scale -n cluster conda remove pandas
All nodes (x4) response:
{
  "actions": {
    "PREFIX": "/opt/continuum/anaconda",
    "UNLINK": [
      "pandas-0.18.1-np111py27_0"
    ],
  },
  "success": true
}

You can also remove conda environments on the cluster nodes using adam scale conda remove.

For example, to remove an environment called py3 on a cluster named cluster:

$ adam scale -n cluster conda remove -n py3 --all
All nodes (x4) response:
{
  "actions": {
    "PREFIX": "/opt/continuum/anaconda/envs/py3",
    "UNLINK": [
      "openssl-1.0.2j-0",
      "pip-8.1.2-py35_0",
      "python-3.5.2-0",
      "readline-6.2-2",
      "setuptools-27.2.0-py35_0",
      "sqlite-3.13.0-0",
      "tk-8.5.18-0",
      "wheel-0.29.0-py35_0",
      "xz-5.2.2-0",
      "zlib-1.2.8-3"
    ]
  },
  "success": true
}

List Conda Packages

You can list conda packages on the cluster nodes using adam scale conda list.

For example, to list conda packages in the root conda environment on a cluster named cluster:

$ adam scale -n cluster conda list
All nodes (x4) response:
+-----------------+---------+--------+
| Package         | Version | Build  |
+-----------------+---------+--------+
| numpy           | 1.11.0  | py27_2 |
| conda-env       | 2.5.1   | py27_0 |
| tk              | 8.5.18  | 0      |
| pip             | 8.1.2   | py27_0 |
| readline        | 6.2     | 2      |
| setuptools      | 23.0.0  | py27_0 |
| openssl         | 1.0.2h  | 1      |
| pycosat         | 0.6.1   | py27_1 |
| python-dateutil | 2.5.3   | py27_0 |
| sqlite          | 3.13.0  | 0      |
| pytz            | 2016.4  | py27_0 |
| mkl             | 11.3.3  | 0      |
| yaml            | 0.1.6   | 0      |
| ruamel_yaml     | 0.11.7  | py27_0 |
| zlib            | 1.2.8   | 3      |
| pyyaml          | 3.11    | py27_4 |
| requests        | 2.10.0  | py27_0 |
| wheel           | 0.29.0  | py27_0 |
| python          | 2.7.11  | 5      |
| conda           | 4.1.4   | py27_0 |
| pycrypto        | 2.6.1   | py27_0 |
| six             | 1.10.0  | py27_0 |
+-----------------+---------+--------+

Get Conda Information

You can view information about the conda installation on the cluster nodes using adam scale conda info.

For example, to view information about the root conda environment on a cluster named cluster:

$ adam scale -n cluster conda info
All nodes (x4) response:
Current conda install:

             platform : linux-64
        conda version : 4.1.4
    conda-env version : 2.5.1
  conda-build version : not installed
       python version : 2.7.11.final.0
     requests version : 2.10.0
     root environment : /opt/continuum/anaconda  (writable)
  default environment : /opt/continuum/anaconda
     envs directories : /opt/continuum/anaconda/envs
        package cache : /opt/continuum/anaconda/pkgs
         channel URLs : https://repo.continuum.io/pkgs/free/linux-64/
                        https://repo.continuum.io/pkgs/free/noarch/
                        https://repo.continuum.io/pkgs/pro/linux-64/
                        https://repo.continuum.io/pkgs/pro/noarch/
          config file : None
         offline mode : False
    is foreign system : False

Or to view a list of conda environments on the cluster nodes:

$ adam scale -n cluster conda info -e
All nodes (x4) response:
# conda environments:
#
py27                     /opt/continuum/anaconda/envs/py27
py35                     /opt/continuum/anaconda/envs/py35
r                        /opt/continuum/anaconda/envs/r
root                  *  /opt/continuum/anaconda

Managing Conda with Environment Definition Files

You can also manage conda environments on the cluster nodes using adam scale env:

$ adam scale -n cluster env
Usage: adam scale env [OPTIONS] COMMAND [ARGS]...

Options:
  -h, --help  Show this message and exit.

Commands:
  create  Create conda environment
  list    List conda environments
  remove  Remove conda environment
  update  Update conda environment

This functionality uses conda-env and conda environment environment definition (YAML) files.

For example, the following environment.yml file describes a conda environment named my-env with Python 2.7 and additional packages:

name: my-env
channels:
- defaults
dependencies:
- python=2.7.*
- bokeh
- dask
- distributed
- pandas

You can use adam scale env create to create a conda environment based on the above conda environment definition file (environment.yml):

$ adam scale -n cluster env create -f environment.yml
All nodes (x4) response:
{
  "actions": {
    "EXTRACT": [
      "msgpack-python-0.4.8-py27_0",
      "numpy-1.11.2-py27_0",
      "psutil-5.0.0-py27_0",
      "pytz-2016.7-py27_0",
      "tblib-1.3.0-py27_0",
      "python-dateutil-2.6.0-py27_0",
      "zict-0.1.0-py27_0",
      "pandas-0.19.1-np111py27_0",
      "tornado-4.4.2-py27_0",
      "bokeh-0.12.3-py27_0",
      "dask-0.12.0-py27_0",
      "distributed-1.14.1-py27_0"
    ],
    "FETCH": [
      "msgpack-python-0.4.8-py27_0",
      "numpy-1.11.2-py27_0",
      "psutil-5.0.0-py27_0",
      "pytz-2016.7-py27_0",
      "tblib-1.3.0-py27_0",
      "python-dateutil-2.6.0-py27_0",
      "zict-0.1.0-py27_0",
      "pandas-0.19.1-np111py27_0",
      "tornado-4.4.2-py27_0",
      "bokeh-0.12.3-py27_0",
      "dask-0.12.0-py27_0",
      "distributed-1.14.1-py27_0"
    ],
    "LINK": [
      "mkl-11.3.3-0 1",
      "openssl-1.0.2j-0 1",
      "readline-6.2-2 1",
      "sqlite-3.13.0-0 1",
      "tk-8.5.18-0 1",
      "yaml-0.1.6-0 1",
      "zlib-1.2.8-3 1",
      "python-2.7.12-1 1",
      "backports-1.0-py27_0 1",
      "backports_abc-0.4-py27_0 1",
      "click-6.6-py27_0 1",
      "cloudpickle-0.2.1-py27_0 1",
      "futures-3.0.5-py27_0 1",
      "heapdict-1.0.0-py27_1 1",
      "locket-0.2.0-py27_1 1",
      "markupsafe-0.23-py27_2 1",
      "msgpack-python-0.4.8-py27_0 1",
      "numpy-1.11.2-py27_0 1",
      "psutil-5.0.0-py27_0 1",
      "pytz-2016.7-py27_0 1",
      "pyyaml-3.12-py27_0 1",
      "requests-2.11.1-py27_0 1",
      "setuptools-27.2.0-py27_0 1",
      "six-1.10.0-py27_0 1",
      "tblib-1.3.0-py27_0 1",
      "toolz-0.8.0-py27_0 1",
      "wheel-0.29.0-py27_0 1",
      "chest-0.2.3-py27_0 1",
      "jinja2-2.8-py27_1 1",
      "partd-0.3.6-py27_0 1",
      "pip-9.0.1-py27_0 1",
      "python-dateutil-2.6.0-py27_0 1",
      "singledispatch-3.4.0.3-py27_0 1",
      "ssl_match_hostname-3.4.0.2-py27_1 1",
      "zict-0.1.0-py27_0 1",
      "pandas-0.19.1-np111py27_0 1",
      "tornado-4.4.2-py27_0 1",
      "bokeh-0.12.3-py27_0 1",
      "dask-0.12.0-py27_0 1",
      "distributed-1.14.1-py27_0 1"
    ],
    "PREFIX": "/opt/continuum/anaconda/envs/my-env",
    "SYMLINK_CONDA": [
      "/opt/continuum/anaconda"
    ],
    "op_order": [
      "RM_FETCHED",
      "FETCH",
      "RM_EXTRACTED",
      "EXTRACT",
      "UNLINK",
      "LINK",
      "SYMLINK_CONDA"
    ]
  },
  "success": true
}

Note that adam scale env create is only compatible with conda dependencies and will ignore any pip or other non-conda dependencies that are specified in the conda environment definition file.

You can use adam scale env list to list conda environments on the cluster:

$ adam scale -n cluster env list
All nodes (x4) response:
# conda environments:
#
my-env                   /opt/continuum/anaconda/envs/my-env
root                  *  /opt/continuum/anaconda

You can use adam scale env update to update packages in an existing conda environment based on a conda environment definition file:

$ adam scale -n cluster env update -f environment.yml
    All nodes (x4) response:
    {
      "message": "All requested packages already installed.",
      "success": true
    }

You can use adam scale env remove to remove a conda environment from the cluster:

$ adam scale -n cluster env remove -n my-env
All nodes (x4) response:
{
  "actions": {
    "PREFIX": "/opt/continuum/anaconda/envs/my-env",
    "UNLINK": [
      "mkl-11.3.3-0",
      "numpy-1.11.2-py27_0",
      "openssl-1.0.2j-0",
      "pandas-0.18.0-np111py27_0",
      "pip-9.0.1-py27_0",
      "python-2.7.12-1",
      "python-dateutil-2.6.0-py27_0",
      "pytz-2016.7-py27_0",
      "readline-6.2-2",
      "setuptools-27.2.0-py27_0",
      "six-1.10.0-py27_0",
      "sqlite-3.13.0-0",
      "tk-8.5.18-0",
      "wheel-0.29.0-py27_0",
      "zlib-1.2.8-3"
    ]
  },
  "success": true
}