Home 9 Resources 9 Software 9 AI software

PSC created environments for AI applications

PSC has built some environments which provide a rich, unified, Anaconda-based environment for AI, Machine Learning, and Big Data applications. Each environment includes several popular AI/ML/BD packages, selected to work together well.

The AI environments are built for the GPU nodes on Bridges-2. Be sure to use one of the GPU partitions. See the Bridges-2 User  Guide for information on Bridges-2 partitions and how to choose one to use.

See also:

  • the python documentation, for a description of the ways to use python on Bridges-2
  • the Anaconda modules, for information on creating, editing and storing anaconda environments on Bridges-2

Using the AI environments on Bridges-2

Typing module spider AI will list the available AI environments.

module spider AI

----------------------------------------------------------------------------
  AI:
----------------------------------------------------------------------------
    Description:
      TensorFlow 2.10.0 AI development environment

     Versions:
        AI/anaconda2-tf1.2019.10
        AI/anaconda3-tf1.2020.11
        AI/anaconda3-tf2.2020.11
        AI/pytorch_22.07-1.12-py3
        AI/pytorch_23.02-1.13.1-py3
        AI/tensorflow_22.07-2.8-py3
        AI/tensorflow_23.02-2.10.0-py3

 

Note that AI/anaconda2 environments use python2, while AI/anaconda3 environments use python3.

 

For additional help, type module help AI/package-version.

 module help AI/tensorflow_23.02-2.10.0-py3

---------- Module Specific Help for "AI/tensorflow_23.02-2.10.0-py3" -----------
TensorFlow 2.10.0
-----------------------------

Description
-----------
The modulefile AI/tensorflow_23.02-2.10.0-py3 provides a TensorFlow 2.10.0 devel
opment environment for Artificial Intelligence(AI)/Machine Learning(ML)/Big Data
(BD) on top of Python 3.

Module contents
---------------
Several popular libraries are included in this environment, such as:
    bokeh, matplotlib, mkl, numba, numpy, pandas, pillow, scikit-learn, theano,
tensorflow.

To check the full list of available packages in this environment, first activate
 it and then run the command
    conda list

* bokeh                     3.0.3
* cudnn                     8.2.1
.
.
.

See what the PSC defined AI environment contains

To see the full list of software included in a given environment, first load the module and activate the environment. Then type

conda list

Customize the PSC defined AI environment

If you need software that is not in the pre-built environment, you can create a new environment by cloning the PSC defined one and then customizing it. First load the module and activate the PSC defined environment, as above, then  clone it with

conda create --name your-new-environment-name --clone $AI_ENV

Then you can activate the new environment and proceed with your customization.

Example

In this example, the user installs the h5py package in a new environment they are creating. Use the following commands.

Note:

  • The conda list command shows what packages are currently installed. Check to see if what you need is already available.  The conda list command also shows the version number of the installed packages.
  • The conda create command clones $AI_ENV to create a new environment.  This can take a long time, so ask for an hour of time with the interact command.
  • Here, the new environment is named clone-env-1, and is stored in the user's ocean directory.  The --prefix flag names the full path to the where the environment will be stored. You can name the environment anything you like and store it in any directory you like.
interact -gpu -t 01:00:00
 module load AI    # loads the default AI module
 source activate $AI_ENV
 conda list 
 conda create --name clone-env-1 --clone $AI_ENV
 conda activate clone-env-1
 conda install h5py

The conda install command will install the newest version of the package. If you want to install a version of the package not available in the public installations use the --revision option to the conda install command.