User Guide¶
Overview¶
Using Anaconda Scale with Anaconda Adam, you can create and manage conda
packages, including Python and R, in multiple conda environments across your
nodes. You can prepend adam scale
to a subset of familiar conda
commands
to install and manage conda packages and environments across a cluster.
$ adam scale -n cluster conda --help
Usage: adam conda [OPTIONS] COMMAND [ARGS]...
Options:
-h, --help Show this message and exit.
Commands:
create Create conda environment
info Get information about conda installation
install Install package(s)
list List conda packages
remove Remove package(s)
update Update package(s)
By default, the root conda environment is installed in
/opt/continuum/anaconda
on all of the cluster nodes, which can be configured
for a different location. You can manage multiple conda environments across the
same cluster (e.g., Python 2, Python 3, and R) without affecting the framework
installation of Python on the cluster nodes.
Install Conda Packages¶
You can install conda packages on the cluster nodes using
adam scale conda install
.
For example, to install numpy
and pandas
and their dependencies into the
root conda environment on a cluster named cluster
:
$ adam scale -n cluster conda install numpy pandas
All nodes (x4) response:
{
"actions": {
"LINK": [
"mkl-11.3.3-0 /opt/continuum/anaconda/pkgs 1",
"openssl-1.0.2h-1 /opt/continuum/anaconda/pkgs 1",
"sqlite-3.13.0-0 /opt/continuum/anaconda/pkgs 1",
"zlib-1.2.8-3 /opt/continuum/anaconda/pkgs 1",
"python-2.7.11-5 /opt/continuum/anaconda/pkgs 1",
"conda-env-2.5.1-py27_0 /opt/continuum/anaconda/pkgs 1",
"numpy-1.11.0-py27_2 /opt/continuum/anaconda/pkgs 1",
"pycosat-0.6.1-py27_1 /opt/continuum/anaconda/pkgs 1",
"pytz-2016.4-py27_0 /opt/continuum/anaconda/pkgs 1",
"pyyaml-3.11-py27_4 /opt/continuum/anaconda/pkgs 1",
"requests-2.10.0-py27_0 /opt/continuum/anaconda/pkgs 1",
"ruamel_yaml-0.11.7-py27_0 /opt/continuum/anaconda/pkgs 1",
"setuptools-23.0.0-py27_0 /opt/continuum/anaconda/pkgs 1",
"six-1.10.0-py27_0 /opt/continuum/anaconda/pkgs 1",
"wheel-0.29.0-py27_0 /opt/continuum/anaconda/pkgs 1",
"conda-4.1.4-py27_0 /opt/continuum/anaconda/pkgs 1",
"pip-8.1.2-py27_0 /opt/continuum/anaconda/pkgs 1",
"python-dateutil-2.5.3-py27_0 /opt/continuum/anaconda/pkgs 1",
"pandas-0.18.1-np111py27_0 /opt/continuum/anaconda/pkgs 1"
],
"PREFIX": "/opt/continuum/anaconda",
},
"success": true
}
Create Conda Environments¶
You can create multiple conda environments on the cluster nodes using
adam scale conda create
.
For example, to create a conda environment called py3
with Python 3 on a
cluster named cluster
:
$ adam scale -n cluster conda create -n py3 python=3
All nodes (x4) response:
{
"actions": {
"LINK": [
"openssl-1.0.2h-1 1 None",
"readline-6.2-2 1 None",
"sqlite-3.13.0-0 1 None",
"tk-8.5.18-0 1 None",
"xz-5.2.2-0 1 None",
"zlib-1.2.8-3 1 None",
"python-3.5.1-5 1 None",
"setuptools-23.0.0-py35_0 1 None",
"wheel-0.29.0-py35_0 1 None",
"pip-8.1.2-py35_0 1 None"
],
"PREFIX": "/opt/continuum/anaconda/envs/py3",
"SYMLINK_CONDA": [
"/opt/continuum/anaconda"
],
},
"success": true
}
Update Conda Packages¶
You can update conda packages on the cluster nodes using
adam scale conda update
.
For example, to update pandas
and its dependencies in the root conda
environment on a cluster named cluster
:
$ adam scale -n cluster conda update pandas
All nodes (x4) response:
{
"actions": {
"LINK": [
"pandas-0.18.1-np111py27_0 1 None"
],
"PREFIX": "/opt/continuum/anaconda",
"SYMLINK_CONDA": [
"/opt/continuum/anaconda"
],
"UNLINK": [
"pandas-0.18.0-np111py27_0"
],
},
"success": true
}
Remove Conda Packages or Environments¶
You can remove conda packages on the cluster nodes using
adam scale conda remove
.
For example, to remove pandas
from the root conda environment on a cluster
named cluster
:
$ adam scale -n cluster conda remove pandas
All nodes (x4) response:
{
"actions": {
"PREFIX": "/opt/continuum/anaconda",
"UNLINK": [
"pandas-0.18.1-np111py27_0"
],
},
"success": true
}
You can also remove conda environments on the cluster nodes using
adam scale conda remove
.
For example, to remove an environment called py3
on a cluster named
cluster
:
$ adam scale -n cluster conda remove -n py3 --all
All nodes (x4) response:
{
"actions": {
"PREFIX": "/opt/continuum/anaconda/envs/py3",
"UNLINK": [
"openssl-1.0.2j-0",
"pip-8.1.2-py35_0",
"python-3.5.2-0",
"readline-6.2-2",
"setuptools-27.2.0-py35_0",
"sqlite-3.13.0-0",
"tk-8.5.18-0",
"wheel-0.29.0-py35_0",
"xz-5.2.2-0",
"zlib-1.2.8-3"
]
},
"success": true
}
List Conda Packages¶
You can list conda packages on the cluster nodes using
adam scale conda list
.
For example, to list conda packages in the root conda environment on a cluster
named cluster
:
$ adam scale -n cluster conda list
All nodes (x4) response:
+-----------------+---------+--------+
| Package | Version | Build |
+-----------------+---------+--------+
| numpy | 1.11.0 | py27_2 |
| conda-env | 2.5.1 | py27_0 |
| tk | 8.5.18 | 0 |
| pip | 8.1.2 | py27_0 |
| readline | 6.2 | 2 |
| setuptools | 23.0.0 | py27_0 |
| openssl | 1.0.2h | 1 |
| pycosat | 0.6.1 | py27_1 |
| python-dateutil | 2.5.3 | py27_0 |
| sqlite | 3.13.0 | 0 |
| pytz | 2016.4 | py27_0 |
| mkl | 11.3.3 | 0 |
| yaml | 0.1.6 | 0 |
| ruamel_yaml | 0.11.7 | py27_0 |
| zlib | 1.2.8 | 3 |
| pyyaml | 3.11 | py27_4 |
| requests | 2.10.0 | py27_0 |
| wheel | 0.29.0 | py27_0 |
| python | 2.7.11 | 5 |
| conda | 4.1.4 | py27_0 |
| pycrypto | 2.6.1 | py27_0 |
| six | 1.10.0 | py27_0 |
+-----------------+---------+--------+
Get Conda Information¶
You can view information about the conda installation on the cluster nodes using
adam scale conda info
.
For example, to view information about the root conda environment on a cluster
named cluster
:
$ adam scale -n cluster conda info
All nodes (x4) response:
Current conda install:
platform : linux-64
conda version : 4.1.4
conda-env version : 2.5.1
conda-build version : not installed
python version : 2.7.11.final.0
requests version : 2.10.0
root environment : /opt/continuum/anaconda (writable)
default environment : /opt/continuum/anaconda
envs directories : /opt/continuum/anaconda/envs
package cache : /opt/continuum/anaconda/pkgs
channel URLs : https://repo.continuum.io/pkgs/free/linux-64/
https://repo.continuum.io/pkgs/free/noarch/
https://repo.continuum.io/pkgs/pro/linux-64/
https://repo.continuum.io/pkgs/pro/noarch/
config file : None
offline mode : False
is foreign system : False
Or to view a list of conda environments on the cluster nodes:
$ adam scale -n cluster conda info -e
All nodes (x4) response:
# conda environments:
#
py27 /opt/continuum/anaconda/envs/py27
py35 /opt/continuum/anaconda/envs/py35
r /opt/continuum/anaconda/envs/r
root * /opt/continuum/anaconda
Managing Conda with Environment Definition Files¶
You can also manage conda environments on the cluster nodes using
adam scale env
:
$ adam scale -n cluster env
Usage: adam scale env [OPTIONS] COMMAND [ARGS]...
Options:
-h, --help Show this message and exit.
Commands:
create Create conda environment
list List conda environments
remove Remove conda environment
update Update conda environment
This functionality uses conda-env and conda environment environment definition (YAML) files.
For example, the following environment.yml
file describes a conda
environment named my-env
with Python 2.7 and additional packages:
name: my-env
channels:
- defaults
dependencies:
- python=2.7.*
- bokeh
- dask
- distributed
- pandas
You can use adam scale env create
to create a conda environment based on the
above conda environment definition file (environment.yml
):
$ adam scale -n cluster env create -f environment.yml
All nodes (x4) response:
{
"actions": {
"EXTRACT": [
"msgpack-python-0.4.8-py27_0",
"numpy-1.11.2-py27_0",
"psutil-5.0.0-py27_0",
"pytz-2016.7-py27_0",
"tblib-1.3.0-py27_0",
"python-dateutil-2.6.0-py27_0",
"zict-0.1.0-py27_0",
"pandas-0.19.1-np111py27_0",
"tornado-4.4.2-py27_0",
"bokeh-0.12.3-py27_0",
"dask-0.12.0-py27_0",
"distributed-1.14.1-py27_0"
],
"FETCH": [
"msgpack-python-0.4.8-py27_0",
"numpy-1.11.2-py27_0",
"psutil-5.0.0-py27_0",
"pytz-2016.7-py27_0",
"tblib-1.3.0-py27_0",
"python-dateutil-2.6.0-py27_0",
"zict-0.1.0-py27_0",
"pandas-0.19.1-np111py27_0",
"tornado-4.4.2-py27_0",
"bokeh-0.12.3-py27_0",
"dask-0.12.0-py27_0",
"distributed-1.14.1-py27_0"
],
"LINK": [
"mkl-11.3.3-0 1",
"openssl-1.0.2j-0 1",
"readline-6.2-2 1",
"sqlite-3.13.0-0 1",
"tk-8.5.18-0 1",
"yaml-0.1.6-0 1",
"zlib-1.2.8-3 1",
"python-2.7.12-1 1",
"backports-1.0-py27_0 1",
"backports_abc-0.4-py27_0 1",
"click-6.6-py27_0 1",
"cloudpickle-0.2.1-py27_0 1",
"futures-3.0.5-py27_0 1",
"heapdict-1.0.0-py27_1 1",
"locket-0.2.0-py27_1 1",
"markupsafe-0.23-py27_2 1",
"msgpack-python-0.4.8-py27_0 1",
"numpy-1.11.2-py27_0 1",
"psutil-5.0.0-py27_0 1",
"pytz-2016.7-py27_0 1",
"pyyaml-3.12-py27_0 1",
"requests-2.11.1-py27_0 1",
"setuptools-27.2.0-py27_0 1",
"six-1.10.0-py27_0 1",
"tblib-1.3.0-py27_0 1",
"toolz-0.8.0-py27_0 1",
"wheel-0.29.0-py27_0 1",
"chest-0.2.3-py27_0 1",
"jinja2-2.8-py27_1 1",
"partd-0.3.6-py27_0 1",
"pip-9.0.1-py27_0 1",
"python-dateutil-2.6.0-py27_0 1",
"singledispatch-3.4.0.3-py27_0 1",
"ssl_match_hostname-3.4.0.2-py27_1 1",
"zict-0.1.0-py27_0 1",
"pandas-0.19.1-np111py27_0 1",
"tornado-4.4.2-py27_0 1",
"bokeh-0.12.3-py27_0 1",
"dask-0.12.0-py27_0 1",
"distributed-1.14.1-py27_0 1"
],
"PREFIX": "/opt/continuum/anaconda/envs/my-env",
"SYMLINK_CONDA": [
"/opt/continuum/anaconda"
],
"op_order": [
"RM_FETCHED",
"FETCH",
"RM_EXTRACTED",
"EXTRACT",
"UNLINK",
"LINK",
"SYMLINK_CONDA"
]
},
"success": true
}
Note that adam scale env create
is only compatible with conda dependencies
and will ignore any pip
or other non-conda dependencies that are specified
in the conda environment definition file.
You can use adam scale env list
to list conda environments on the cluster:
$ adam scale -n cluster env list
All nodes (x4) response:
# conda environments:
#
my-env /opt/continuum/anaconda/envs/my-env
root * /opt/continuum/anaconda
You can use adam scale env update
to update packages in an existing conda
environment based on a conda environment definition file:
$ adam scale -n cluster env update -f environment.yml
All nodes (x4) response:
{
"message": "All requested packages already installed.",
"success": true
}
You can use adam scale env remove
to remove a conda environment from the
cluster:
$ adam scale -n cluster env remove -n my-env
All nodes (x4) response:
{
"actions": {
"PREFIX": "/opt/continuum/anaconda/envs/my-env",
"UNLINK": [
"mkl-11.3.3-0",
"numpy-1.11.2-py27_0",
"openssl-1.0.2j-0",
"pandas-0.18.0-np111py27_0",
"pip-9.0.1-py27_0",
"python-2.7.12-1",
"python-dateutil-2.6.0-py27_0",
"pytz-2016.7-py27_0",
"readline-6.2-2",
"setuptools-27.2.0-py27_0",
"six-1.10.0-py27_0",
"sqlite-3.13.0-0",
"tk-8.5.18-0",
"wheel-0.29.0-py27_0",
"zlib-1.2.8-3"
]
},
"success": true
}