User Guide¶
- Overview
- Manage Anaconda on the Cluster
- Install and Manage Platform Components
- Remote Commands
- SSH to Nodes
- Version and License Information
- List Cluster Information
- Creating, Provisioning, and Removing Clusters
- Adding Nodes to an Existing Cluster
- Additional Provision Commands (Advanced)
- Execute Salt Commands (Advanced)
Overview¶
Typing the adam
command shows the different subcommands that can be used:
$ adam
Usage: adam [OPTIONS] COMMAND [ARGS]...
Options:
--version Show the version and exit.
-l, --log-level [info|debug|error]
Logging level [default: debug]
-h, --help Show this message and exit.
Commands:
accelerate Anaconda Accelerate options
cmd Execute a command on the nodes
describe Describe an Adam cluster or profile
ec2 EC2 options
enterprise-notebooks Anaconda Enterprise Notebooks options
info Display Adam version, system, and license information
list List Adam clusters and profiles
mosaic Anaconda Mosaic options
provision Provision options
remove Remove a cluster
repository Anaconda Repository options
salt Execute a Salt module
scale Anaconda Scale options
ssh SSH to one of the nodes (0-based index)
up Create a cluster from a profile
You can view more information about a subcommand using the -h
or --help
option, for example, adam up --help
.
More detail on the available subcommands is shown in the sections below.
Manage Anaconda on the Cluster¶
Using Anaconda Scale, you can create and manage conda packages, including Python and R, in multiple conda environments across your nodes. Refer to the Anaconda Scale documentation for more information about installing and managing Anaconda across a cluster.
Install and Manage Platform Components¶
adam accelerate
adam enterprise-notebooks
adam mosaic
adam repository
adam scale
For more information on installing and managing platform components, refer to the Getting Started documentation.
Remote Commands¶
You can execute remote commands on all of the cluster nodes using adam cmd
:
$ adam cmd --help
Usage: adam cmd [OPTIONS] COMMAND
Options:
-n, --cluster TEXT Cluster name [required]
-t, --target TEXT Target nodes [default: *]
-h, --help Show this message and exit.
For example, to run the date
command on all nodes:
$ adam cmd -n my-cluster date
192.168.1.1: Wed Jun 29 03:08:10 UTC 2016
192.168.1.2: Wed Jun 29 03:08:10 UTC 2016
192.168.1.3: Wed Jun 29 03:08:10 UTC 2016
192.168.1.4: Wed Jun 29 03:08:10 UTC 2016
SSH to Nodes¶
You can SSH to the cluster nodes using adam ssh
:
$ adam ssh -n my-cluster
Last login: Wed Jun 29 03:08:49 2016 from 172.18.0.1
[root@192.168.1.1 ~]#
By default, you will connect to the head node. You can optionally include the node to SSH to, where node 0 is the head node, node 1 is the first compute node, and so on:
$ adam ssh -n my-cluster 2
Last login: Wed Jun 29 03:10:14 2016 from 172.18.0.1
[root@192.168.1.3 ~]#
Version and License Information¶
You can view version and license information using adam info
:
$ adam info
Adam version: 4.1.0
Plugins:
adam_enterprise_notebooks: 4.1.0
adam_repository: 4.1.0
adam_accelerate: 4.1.0
adam_mosaic: 4.1.0
adam_notebook: 4.1.0
adam_dask: 4.1.0
Platform: Darwin-15.6.0-x86_64
Darwin Kernel Version 15.6.0: Thu Jun 23 18:25:34 PDT 2016; root:xnu-3248.60.10~1/RELEASE_X86_64
Python version: 3.5.1 |Continuum Analytics, Inc.| (default, Dec 7 2015, 11:24:55)
[GCC 4.2.1 (Apple Inc. build 5577)]
Processor: i386
Byte-ordering: little
License information:
Number of nodes currently in use: 14
Number of licensed nodes: 16
Number of managed clusters: 2
Valid platform component licenses:
1: Anaconda Repository Enterprise. End date: 2017-05-24
2: Anaconda Cluster. End date: 2017-05-24
3: Wakari Enterprise. End date: 2017-05-24
4: mkl-optimizations. End date: 2017-05-24
5: iopro. End date: 2017-05-24
6: accelerate. End date: 2017-05-24
List Cluster Information¶
You can view a list of running clusters using adam list
:
$ adam list
my-cluster
second-cluster
third-cluster
You can view a list of cluster profiles using adam list profiles
:
$ adam list profiles
my-cluster-profile
second-cluster-profile
third-cluster-profile
You can view more detailed information about a cluster or profile using
adam describe
. For example, to view information about the cluster named
my-cluster
:
$ adam describe -n my-cluster
conda:
anaconda_hash: md5=42dac45eee5e58f05f37399adda45e85
anaconda_url: https://repo.continuum.io/miniconda/Miniconda2-4.0.5-Linux-x86_64.sh#adam-ci
enabled: true
rootdir: /opt/continuum
name: my-cluster
nodes:
- host: ci_head_1
keypair: ~/.ssh/my_private_key
port: 22
username: root
- host: ci_compute_1
keypair: ~/.ssh/my_private_key
port: 22
username: root
Creating, Provisioning, and Removing Clusters¶
You can create, provision, and remove clusters using adam up
,
adam provision
, and adam remove
.
To create a new cluster from a profile:
$ adam up -n my-cluster my-cluster
To (re)provision an existing cluster:
$ adam provision -n my-cluster
To remove an existing bare-metal cluster:
$ adam remove -n my-cluster
Are you sure you want to remove the cluster my-cluster? [y/N]: y
To destroy an existing cloud-based cluster:
$ adam ec2 -n my-cluster destory
Are you sure you want to destroy the cluster my-cluster? [y/N]: y
Adding Nodes to an Existing Cluster¶
The following steps describe how to add additional nodes to an existing cluster in Anaconda Adam.
To add additional nodes to a cluster managed by Anaconda Adam, you can modify an existing cluster profile by specifying additional node(s). For example, if you want to add the following nodes:
node10.company.com
node11.company.com
node12.company.com
to a cluster profile located at ~/.continuum/adam/profile.d/my-cluster.yaml
with the following settings:
name: my-cluster
provider: bare
bare:
username: centos
# password: password # Optional
port: 22
keypair: ~/.ssh/my-private-key
nodes:
- host: node1.company.com
- host: node2.company.com
- host: node3.company.com
- host: node4.company.com
Assuming the same authentication mode and credentials for the additional nodes,
you can add a host
line for each node to the cluster profile:
name: my-cluster
provider: bare
bare:
username: centos
# password: password # Optional
port: 22
keypair: ~/.ssh/my-private-key
nodes:
- host: node1.company.com
- host: node2.company.com
- host: node3.company.com
- host: node4.company.com
- host: node10.company.com
- host: node11.company.com
- host: node12.company.com
Then, you can re-run the adam up
command to reprovision the cluster:
$ adam up -n my-cluster my-cluster
The adam up
command will prompt to overwrite the cluster definition file
located at ~/.continuum/adam/cluster.d/my-cluster.yaml
to include the newly
defined nodes. During this reprovision process, Anaconda Adam will not
reinstall on the previously defined nodes.
Additional Provision Commands (Advanced)¶
By default, adam provision
will install check SSH connectivity, create a
default user, install Salt, sync formulas, and provision enabled plugins and
security settings that are defined in the cluster profile.
If you only need to run a portion of the adam provision
process, you can
view the available subcommands:
$ adam provision --help
Usage: adam provision [OPTIONS] COMMAND [ARGS]...
Execute all the provisioning steps for a cluster
See subcommands help for more info. Usage: adam provision -n
Options:
-n, --cluster TEXT Cluster name [required]
--flush-iptables Flush iptables [default: False]
--selinux-permissive Set SELinux to permissive [default: False]
--selinux-context Set SELinux contexts [default: False]
--plugins Install enabled plugins on provision [default: False]
-h, --help Show this message and exit.
Commands:
anaconda-user Create the default anaconda user in all the nodes
check-ssh Check SSH connection to the nodes
flush-iptables Flush iptables
plugins Provision enabled plugins
roles Set roles on the cluster nodes
salt Provision Salt
security-settings Execute the security settings
selinux-context Set SELinux contexts
selinux-permissive Set SELinux to permissive
sync Sync formulas and settings
For example, to check SSH connectivity to all of the nodes in an existing cluster:
$ adam provision -n cluster check-ssh
Checking SSH connection to nodes
2016-10-03 22:10:49,327 - adam.models - DEBUG - Checking ssh connection for 54.88.20.164
2016-10-03 22:10:50,905 - adam.models - DEBUG - SSH connection to 54.88.20.164: OK
2016-10-03 22:10:51,008 - adam.models - DEBUG - Checking ssh connection for 52.90.168.130
2016-10-03 22:10:51,996 - adam.models - DEBUG - SSH connection to 52.90.168.130: OK
+---------------+---------------+
| Node IP | SSH Available |
+---------------+---------------+
| 54.88.20.164 | True |
| 52.90.168.130 | True |
+---------------+---------------+
| All nodes | True |
+---------------+---------------+
Execute Salt Commands (Advanced)¶
You can execute Salt modules using adam salt
.
For example, to execute the test.ping
Salt module:
$ adam salt -n my-cluster '*' test.ping
192.168.1.1: true
192.168.1.2: true
192.168.1.3: true
192.168.1.4: true