.. _mirroring-anaconda-repository:

================================
Mirroring an Anaconda repository
================================

.. contents::
   :local:
   :depth: 1

NOTE: This documentation is for versions of Anaconda Repository 2.26
and newer. For versions between 2.30 and 2.26, see :ref:`2.23
documentation <mirrors-anaconda-repository-2.23>`. For
older versions, see :ref:`2.20 documentation
<mirrors-sync-configuration-2.2>`.

Before you start
================

You need to have already installed and configured your
Repository instance. Due to the size of
Repository, it is important that you have configured a file
storage location with sufficient disk space. If necessary, see
the :ref:`requirements for the file storage location
<repo-hardware-reqs>`.

A full Anaconda mirror requires approximately 90 GB.

You will also need to install ``cas-mirror`` as it is the recommended
mirroring tool.

NOTE: Both ``anaconda-mirror`` and ``anaconda-server-sync-conda``
have been deprecated and will not be updated any further.

Mirroring all packages
======================

You can mirror some or all of the contents of the
`Anaconda repository
<https://docs.anaconda.com/anaconda/pkg-docs.html>`_ using the
``cas-sync-api-v4`` command::

   $ cas-sync-api-v4 --help
   usage: cas-sync-api-v4 [-h] [-f FILENAME] [--config] [--version]
                          [-l LOG_LEVEL] [-v]

   Updates an Anaconda repository instance

   optional arguments:
     -h, --help            show this help message and exit
     -f FILENAME, --file FILENAME
                           Configuration file location (Defaults to:
                           /home/abarto/.cas-mirror or /etc/cas-mirror)
     --config, --show-config
                           Show running configuration and exit
     --version             Print version and exit
     -l LOG_LEVEL, --log-level LOG_LEVEL
                           Set the log level (CRITICAL, ERROR, WARNING, INFO,
                           DEBUG)Default: INFO.
     -v, --verbose         Shorthand for --log-level DEBUG

This tools mirrors all of the packages from the Anaconda repository default
channels into the ``anaconda`` user account. It leverages the functionality
exposed by the ``anaconda-client`` package to import the remote packages
into an existing site.

#. Associate the URL of your Anaconda repository with a site using the
   ``anaconda`` command::

       anaconda config --set sites.myrepo.url "http://your-anaconda-repo/"

   NOTE: Replace ``your-anaconda-repo`` with the URL to your installation
   of Repository.

#. Create a configuration file ``sync.yaml`` that tells ``cas-sync-api-v4``
   which site to use::

       dest_site: myrepo

#. Check that the configuration is valid with the ``--config`` parameter::

       $ cas-sync-api-v4 -f sync.yaml --config
       path: /home/ec2-user/sync.yaml
       remote_url: https://repo.continuum.io/
       mirror_dir: /opt/cas-mirror
       platforms: ['osx-32', 'osx-64', 'win-32', 'win-64', 'linux-32', 'linux-64', 'linux-armv6l', 'linux-armv7l', 'linux-ppc64le']
       fetch_installers: True
       repodata_source: False
       dir_names: ['archive']
       server_log_dir: None
       server_port: None
       dest_site: myrepo
       dest_channel: anaconda
       verify_checksum: False
       delta: False
       delta_dir: None
       log_dir: None
       log_level: 20
       python_versions: []
       pkg_list: []
       license_blacklist: []
       blacklist: []
       whitelist: []
       channels:
           - https://repo.continuum.io/pkgs/main/
           - https://repo.continuum.io/pkgs/free/
           - https://repo.continuum.io/pkgs/pro/

   With this configuration, ``cas-sync-api-v4`` mirrors the contents of all the
   default channels into the ``anaconda`` account of the ``myrepo`` site.

#. Run ``cas-sync-api-v4``::

       cas-sync-api-v4 -f sync.yaml

#. Verify the mirroring by opening a browser and loading this URL::

       http://your-anaconda-repo/anaconda/

   NOTE: Replace ``your-anaconda-repo`` with the URL to your installation
   of Repository.

Mirroring some packages
=======================

Alternately, you may not want to mirror all packages. To mirror a
subset of the total repository, specify which platforms you want to
include, or use the whitelist, blacklist or license_blacklist
functionality to control which packages are mirrored, by copying the
default configuration file
``$PREFIX/etc/anaconda-server/mirror/anaconda.yaml`` to
``$PREFIX/etc/anaconda-server/mirror/anaconda-custom.yaml``.

For Repository 2.27 or newer, there are sample ``yaml`` config files
located at: ``$PREFIX/etc/anaconda-server/mirror``. ``PREFIX`` is
the install location of Repository, which by default is
``~anaconda-server/repo/etc/anaconda-server/mirror``.

This command mirrors the repository according to the settings in the
configuration file ``anaconda-custom.yaml``::

  cas-sync-api-v4 -f anaconda-custom.yaml

For more information, see :doc:`customize-mirror`.

Offline mirroring
=================

Offline mirroring is done by using both ``cas-sync`` and
``cas-sync-api-v4``. First download all the packages
onto a host with Internet access. The ``cas-mirror`` package
is independent of Anaconda repository, so it can be installed
on any machine.

EXAMPLE:

To download the packages, create a configuration file named ``export.yaml``::

   mirror_dir: /opt/mirror/export/
   platforms:
     - linux-64
     - win-64
   python_versions:
     - 2.7
     - 3.6
   fetch_installers: false
   pkg_list:
     - ca-certificates
     - certifi
     - libedit
     - libffi
     - libgcc-ng
     - libstdcxx-ng
     - ncurses
     - openssl
     - pip
     - python
     - readline
     - setuptools
     - sqlite
     - tk
     - wheel
     - xz
     - zlib

This example downloads only a subset of the default channels.

Run ``cas-sync``::

    cas-sync -f export.yaml

After it finishes, a conda repository for each platform is created
in the directory ``/opt/mirror/export/pkgs``. We can now take
the contents of the directory ``/opt/mirror/export/`` to the
air-gapped environment.

To import the packages, create a config file named ``import.yaml``::

   dest_site: mysite
   dest_channel: anaconda
   channels:
     - file:///opt/mirror/export/pkgs/
   platforms:
     - linux-64
     - win-64
   python_versions:
     - 2.7
     - 3.6

Make sure these requirements are true:

* The files mirrored onto the connected box have been put in
  the ``/opt/mirror/export`` directory.

* An anaconda site named ``mysite`` is properly configured and
  the user has logged into it using the ``anaconda`` command.

* An "anaconda" account exists in the "mysite" site and the
  logged in user has access to it.

After these requirements are true, run ``cas-sync-api-v4``::

    cas-sync-api-v4 -f import.yaml


Mirroring with the deprecated anaconda-mirror command
=====================================================

The ``anaconda-mirror`` command is deprecated but still in use at some
installations.

It is used with the command ``anaconda-mirror sync``, or with a
configuration file such as ``anaconda-custom.yaml`` with the command
``anaconda-mirror --config-file anaconda-custom sync``.

Resetting packages
------------------

Use the ``--reset`` option to reset the previously mirrored
packages::

  anaconda-mirror --config-file anaconda-custom sync --reset

This resets the "last sync" time for the repository, so
``anaconda-mirror`` requests all packages, not just those changed or
added since the last sync. As the requests are processed,
``anaconda-mirror`` still automatically downloads only those files
that differ from the files currently in the repository.

Exporting a mirror
------------------

To generate a mirror archive::

  anaconda-mirror export mirror.tar

This command dumps the packages, according to the
:doc:`configured settings <customize-mirror>`, into the file
``mirror.tar``.

This mirror can be used in an air gapped environment.

Importing a mirror
------------------

To mirror the Anaconda repository in an air gapped environment,
point ``anaconda-mirror`` to the exported mirror archive.

Mount the USB drive and then run::

  anaconda-mirror import $USB/mirror.tar

This command mirrors the contents of the local Anaconda repository to
your Repository installation under the username "anaconda."

Filtering
---------

If you want to update the filters on your mirror---for example, to
exclude additional licenses---running ``anaconda-mirror sync`` again
retrieves new packages that match this filter, but it does not remove
existing packages that no longer match the filter.

To see which packages no longer match your filter::

  anaconda-mirror clean --dry-run

To remove these packages from your mirror::

  anaconda-mirror clean

.. _mirror-other-channels:

Mirroring additional channels
=============================

If mirroring from an air gap archive, the channel in the following
configuration points to a local directory to which the archive is
expanded.

In addition, if a platform-specific archive is downloaded, then the
config file needs the platforms section. The examples in the
following sections assume
``x64-repo-mirrors-\*.tar <airgap-archive-mirrors>`` is expanded to
``$MIRRORS_ARCHIVE``.

Similarly, for an online system, the channel points to Anaconda Cloud.
The platforms are optional and limit the mirrored conda packages to
the specified platforms.

Mirroring R channel
-------------------

#. Create the ``yaml`` config file.

   EXAMPLE: The following is a config to mirror from an air gap
   archive containing only x64 packages::

     cat $PREFIX/etc/anaconda-server/mirror/r.yaml

     channels:
       - file://$MIRRORS_ARCHIVE/r/pkgs

     # The platforms should correspond to the platforms contained in
     # the archive. Omit if the archive contains conda packages for all platforms.
     platforms:
       - linux-64
       - osx-64
       - win-64



   EXAMPLE: The following is for an online system::

     cat $PREFIX/etc/anaconda-server/mirror/r.yaml

     channels:
       - https://conda.anaconda.org/r

#. Mirror the packages to r-channel::

     anaconda-server-sync-conda --mirror-config \
         $PREFIX/etc/anaconda-server/mirror/r.yaml --account=r-channel




Mirroring Wakari channel for AEN
--------------------------------

#. Create the ``yaml`` config file.

   EXAMPLE: The following is a config to mirror from an air gap
   archive containing only x64 packages::

     cat $PREFIX/etc/anaconda-server/mirror/wakari.yaml

     channels:
       - file://$MIRRORS_ARCHIVE/wakari/pkgs

     # The platforms should correspond with the platforms contained in
     # the archive. Omit if the archive contains conda packages for all platforms.
     platforms:
       - linux-64
       - osx-64
       - win-64



   EXAMPLE: The following is for an online system::

     cat $PREFIX/etc/anaconda-server/mirror/wakari.yaml

     channels:
       - https://conda.anaconda.org/t/<TOKEN>/anaconda-nb-extensions
       - https://conda.anaconda.org/wakari



   NOTE: Replace ``<TOKEN>`` with the token for the
   anaconda-nb-extensions channel that you should have received
   along with your Repository license.

#. Mirror the packages to the Wakari channel::

    anaconda-server-sync-conda --mirror-config \
        $PREFIX/etc/anaconda-server/mirror/r.yaml --account=wakari



Mirroring anaconda-adam channel for cluster management
------------------------------------------------------

#. Create the ``yaml`` config file.

   EXAMPLE: The following is a config to mirror from an air gap
   archive containing only x64 packages::

     cat $PREFIX/etc/anaconda-server/mirror/anaconda-adam.yaml

     channels:
       - file://$MIRRORS_ARCHIVE/anaconda-adam/pkgs

     # The platforms should correspond with the platforms contained in
     # the archive. Omit if the archive contains conda packages for all
     # platforms.
     platforms:
           * linux-64
           * osx-64
           * win-64



   EXAMPLE: The following is for an online system::

     cat $PREFIX/etc/anaconda-server/mirror/anaconda-adam.yaml

     channels:
       - https://conda.anaconda.org/anaconda-adam

#. Mirror the packages to anaconda-adam channel::

     anaconda-server-sync-conda --mirror-config \
         $PREFIX/etc/anaconda-server/mirror/r.yaml --account=anaconda-adam



Configuring conda
=================

Having created the mirror, you still need to configure conda to
search for packages here rather than on the default Anaconda
repository. You can do that by editing your ``~/.condarc`` file to add
the appropriate channel::

  channels:
      - http://<anaconda.repo.ipaddress>:<port>/conda/anaconda/

NOTE: Replace ``<anaconda.repo.ipaddress>`` with the URL to your
installation of Repository.

NOTE: This configuration change can be made at the user level or via
an `administrative <http://conda.pydata.org/docs/admin.html>`_ conda
file, to force all internal users to use your local Anaconda mirror
rather than querying the Anaconda repository.

NOTE: Users can download Anaconda installers that are
pre-configured to search your Repository from
``http://<anaconda.repository.addr>/downloads``. To learn how to
generate these installers, see :doc:`../customize-installers`.
