Installation

Installation¶

The following will show how to install both SmartSim and SmartRedis

Prerequisites¶

The base prerequisites to install SmartSim and SmartRedis are:

Python 3.7-3.9

Pip

Cmake 3.13.x (or later)

C compiler

C++ compiler

GNU Make > 4.0

git

git-lfs

For most developer systems, many of these packages will already be installed.

GCC 5-9 is recommended. There are known bugs with GCC >= 10.

Git LFS can be installed through conda install git-lfs

Be sure to reference the installation notes for helpful information regarding various system types before installation.

Supported Versions¶

Supported System for Pre-built Wheels¶
Platform	CPU	GPU	Python Versions
MacOS	x86_64	Not supported	3.7 - 3.9
Linux	x86_64	Nvidia	3.7 - 3.9

Note

Windows is not supported and there are currently no plans to support Windows.

SmartSim supports multiple machine learning libraries through the use of RedisAI. The following libraries are supported.

Supported ML Libraries¶
Library	Versions	Python Versions	Built By Default
PyTorch	1.7	3.7 - 3.9	Yes
Tensorflow / Keras	2.5.2	3.7 - 3.9	Yes
ONNX	1.9	3.7 - 3.9	No

TensorFlow 2.0 and Keras are supported through graph freezing.

ScikitLearn and Spark models are supported by SmartSim as well through the use of the ONNX runtime.

SmartSim¶

There are two stages for the installation of SmartSim.

pip install SmartSim Python package

Build SmartSim using the smart command line tool installed by the pip package.

Step 1: Install Python Package¶

Activate a new virtual environment:

conda activate <env name>

and install SmartSim from PyPi with the following command

pip install smartsim

If you would like SmartSim to also install Machine Learning libraries that can be used outside SmartSim to build SmartSim-compatible models, you can request their installation through the ml flag as follows:

pip install smartsim[ml]
# add ray extra if you would like to use ray with SmartSim as well
pip install smartsim[ml,ray]
# or if using ZSH
pip install smartsim\[ml\]
pip install smartsim\[ml,ray\]

At this point, SmartSim is installed and can be used for more basic features. If you want to use the machine learning features of SmartSim, you will need to install the ML backends in the section below.

Step 2: Build SmartSim¶

Use the smart cli tool to install the machine learning backends that are built into the Orchestrator database. smart is installed during the pip installation of SmartSim and may only be available while your virtual environment is active.

To see all the installation options:

smart

Note

If the smart tool is not found. Look for it in places like ~/.local/bin and other bin locations and add it to your $PATH

CPU Install¶

To install the default ML backends for CPU, run

# Optionally, setup toolchain and build settings to be used. ex for GCC

export CC=gcc
export CXX=g++
export NO_CHECKS=1 # skip build checks

# run one of the following
smart build --device cpu          # install PT and TF for cpu
smart build --device cpu --onnx   # install all backends (PT, TF, ONNX) on gpu

By default, smart will install PyTorch and TensorFlow backends for use in SmartSim.

Note

If a re-build is needed for any reason, smart clean will remove all of the previous installs for the ML backends and smart clobber will remove all pre-built dependencies as well as the ML backends.

GPU Install¶

To install the database ML backends for GPU, set the following environment variables if CUDNN is not in your LD_LIBRARY_PATH or default loader locations.

CUDNN_INCLUDE_DIR - path to directory containing cudnn.h

CUDNN_LIBRARY - path to directory containing libcudnn.so

For example, for bash do:

export CUDNN_LIBRARY=/lus/cuda/lib64/
export CUDNN_INCLUDE_DIR=/lus/cuda/include/
export LD_LIBRARY_PATH=$CUDNN_LIBRARY:$LD_LIBRARY_PATH

# run one of the following
smart build --device gpu          # install PT and TF for gpu
smart build --device gpu --onnx   # install all backends (PT, TF, ONNX) on gpu

Note

Currently, SmartSim is solely compatible with NVIDIA GPUs on Linux systems and CUDA >= 11 is required to build.

SmartRedis¶

There are implementations of the SmartRedis client in 4 languages: Python, C++, C and Fortran. The Python client is installed through pip and the compiled clients can be built as a static or shared library through cmake.

SmartRedis Python supports the same architectures for pre-built wheels that SmartSim does.

Supported Systems for Pre-built Wheels¶
Platform	Python Versions
MacOS	3.7 - 3.9
Linux	3.7 - 3.9

The Python client for SmartRedis is installed through pip as follows:

First, activate your Python virtual environment:

conda activate <env name>

Install SmartRedis Python client from PyPI:

pip install smartredis

Developers installing the Python client from PyPI can install the SmartRedis client with additional dependencies for testing and documentation with:

pip install smartredis[dev]
# or if using ZSH
pip install smartredis\[dev\]

Now, when inside your virtual environment, you should be able to import the Client from smartredis as follows:

Python 3.7.7 (default, May  7 2020, 21:25:33)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from smartredis import Client
>>>

Build SmartRedis Library (C++, C, Fortran)¶

Clone the SmartRedis repository and checkout the most recent release:

git clone https://github.com/CrayLabs/SmartRedis.git --depth=1 --branch v0.3.1 smartredis

Note that the release tarball can also be used instead of cloning the git repository, but the preferred method is a repository clone.

To build the SmartRedis library for the C++, C, and Fortran, make sure to be in the top level directory of smartredis-0.3.1.

make lib

The SmartRedis library will be installed in smartredis-0.3.1/install/lib/ and the SmartRedis header files will be installed in smartredis-0.3.1/install/include/. The library installation can be used to easily include SmartRedis capabilities in C++, C, and Fortran applications. For example, the CMake instructions below illustrate how to compile a C or C++ application with SmartRedis.

project(Example)

cmake_minimum_required(VERSION 3.13)

set(CMAKE_CXX_STANDARD 17)

find_library(sr_lib smartredis
             PATHS path/to/smartredis/install/lib
             NO_DEFAULT_PATH REQUIRED
)

include_directories(SYSTEM
    /usr/local/include
    path/to/smartredis/install/include
)

# Build executables

add_executable(example
    example.cpp
)
target_link_libraries(example
    ${sr_lib}
)

Compiling a Fortran application with the SmartRedis library is very similar to the instructions above. The only difference is that the Fortran SmartRedis client source files currently need to be included in the compilation. An example CMake file is shown below for a Fortran application.

project(Example)

cmake_minimum_required(VERSION 3.13)

enable_language(Fortran)

set(CMAKE_CXX_STANDARD 17)
set(CMAKE_C_STANDARD 99)

set(ftn_client_src
    path/to/smartredis/src/fortran/fortran_c_interop.F90
    path/to/smartredis/src/fortran/dataset.F90
    path/to/smartredis/src/fortran/client.F90
)

find_library(sr_lib smartredis
             PATHS path/to/smartredis/install/lib
             NO_DEFAULT_PATH REQUIRED
)

include_directories(SYSTEM
    /usr/local/include
    path/to/smartredis/install/include
)

add_executable(example
    example.F90
        ${ftn_client_src}
)

target_link_libraries(example
    ${sr_lib}
)

From Source¶

This section will be geared towards contributors who want to install SmartSim and SmartRedis from source. If you are installing from source for other reasons, follow the steps below but use the distribution provided hosted on GitHub or PyPi.

Install SmartSim from Source¶

First, clone SmartSim.

git clone https://github.com/CrayLabs/SmartSim smartsim

And then install SmartSim with pip in editable mode. This way, SmartSim is installed in your virtual environment and available in PYTHONPATH, but the source remains at the site of the clone instead of in site-packages.

cd smartsim
pip install -e .[dev,ml]   # for bash users
pip install -e .\[dev,ml\] # for zsh users

Use the now installed smart cli to install the machine learning runtimes.

# run one of the following
smart build -v --device cpu          # verbose install cpu
smart build -v --device gpu          # verbose install gpu
smart build -v --device gpu --onnx   # install all backends (PT, TF, ONNX) on gpu

Install SmartRedis from Source¶

Developer Makefile¶

SmartRedis has a Makefile that automates the build and install process. The Makefile is shown below, and in the following sections, the process for building and installing the SmartRedis clients from source will be described.

SmartRedis Makefile help

help                           - display this makefile's help information

Build
-------
deps                           - Make SmartRedis dependencies
lib                            - Build SmartRedis clients into a static library
test-deps                      - Make SmartRedis testing dependencies
test-deps-gpu                  - Make SmartRedis GPU testing dependencies
build-tests                    - build all tests (C, C++, Fortran)
build-test-cpp                 - build the C++ tests
build-test-c                   - build the C tests
build-test-fortran             - build the Fortran tests
build-examples                 - build all examples (serial, parallel)
build-example-serial           - buld serial examples
build-example-parallel         - build parallel examples (requires MPI)
clean-deps                     - remove third-party deps
clean                          - remove builds, pyc files, .gitignore rules
clobber                        - clean, remove deps, builds, (be careful)

Style
-------
style                          - Sort imports and format with black
check-style                    - check code style compliance
format                         - perform code style format
check-format                   - check code format compliance
sort-imports                   - apply import sort ordering
check-sort-imports             - check imports are sorted
check-lint                     - run static analysis checks

Documentation
-------
docs                           - generate project documentation
cov                            - generate html coverage report for Python client

Test
-------
test                           - Build and run all tests (C, C++, Fortran, Python)
test-verbose                   - Build and run all tests [verbosely]
test-c                         - Build and run all C tests
test-cpp                       - Build and run all C++ tests
test-py                        - run python tests
test-fortran                   - run fortran tests
testpy-cov                     - run python tests with coverage

Clone SmartRedis¶

First, clone the SmartRedis repo:

git clone https://github.com/CrayLabs/SmartRedis smartredis
cd smartredis

Building the Python Client from Source¶

After cloning the repository, the Python client can be installed from source with:

pip install .

If installing SmartRedis from source for development, it is recommended that the Python client be installed with the -e and [dev]:

pip install -e .[dev]
# or if using ZSH
pip install -e .\[dev\]

Now, when inside your virtual environment, you should be able to import the Client from smartredis as follows

Python 3.7.7 (default, May  7 2020, 21:25:33)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from smartredis import Client
>>>

Building SmartRedis Static Library from Source¶

Assuming the above steps have already been done, you are now ready to build SmartRedis as a static library.

A static library of the SmartRedis C++, C, and Fortran clients can be built with the command:

make lib

The SmartRedis library will be installed in smartredis/install/lib/ and the SmartRedis header files will be installed in smartredis/install/include/. The library installation can be used to easily include SmartRedis capabilities in C++, C, and Fortran applications. For example, the CMake instructions below illustrate how to compile a C or C++ application with SmartRedis.

project(Example)

cmake_minimum_required(VERSION 3.13)

set(CMAKE_CXX_STANDARD 17)

find_library(sr_lib smartredis
             PATHS path/to/smartredis/install/lib
             NO_DEFAULT_PATH REQUIRED
)

include_directories(SYSTEM
    /usr/local/include
    path/to/smartredis/install/include
)

# Build executables

add_executable(example
    example.cpp
)
target_link_libraries(example
    ${sr_lib}
)

project(Example)

cmake_minimum_required(VERSION 3.13)

enable_language(Fortran)

set(CMAKE_CXX_STANDARD 17)
set(CMAKE_C_STANDARD 99)

set(ftn_client_src
    path/to/smartredis/src/fortran/fortran_c_interop.F90
    path/to/smartredis/src/fortran/dataset.F90
    path/to/smartredis/src/fortran/client.F90
)

find_library(sr_lib smartredis
             PATHS path/to/smartredis/install/lib
             NO_DEFAULT_PATH REQUIRED
)

include_directories(SYSTEM
    /usr/local/include
    path/to/smartredis/install/include
)

add_executable(example
    example.F90
        ${ftn_client_src}
)

target_link_libraries(example
    ${sr_lib}
)

Building the Documentation¶

Users can optionally build documentation of SmartSim through make docs or make docks. make docs requires the user to install the documentation build dependencies, whereas a`make docks`` only requires docker. make docks is the recommended method for building the documentation locally, due to ease of use.

With docker¶

Note

To build the full documentation with make docks, users need to install docker so that docker is available on the command line.

# From top level smartsim git repository directory
make docks

Once the documentation has successfully built, users can open the main documents page from docs/develop/index.html.

Without docker¶

Note

To build the full documentation via make docs, users need to install doxygen 1.9.1. For Mac OS users, doxygen can be installed through brew install doxygen

# From top level smartsim git repository directory
git clone https://github.com/CrayLabs/SmartRedis.git
make docs

Once the documentation has successfully built, users can open the main documents page from doc/_build/html/index.html

Installation Notes for Specific System Types¶

The following describes installation details for various system types that SmartSim may be used on.

SmartSim on MacOS¶

We recommend users and contributors install brew for managing installed packages. For contributors, the following brew packages can be helpful

openmpi for building and running parallel SmartRedis examples
doxygen for building the documention
cmake for building SmartSim and SmartRedis from source

For Mac OS users, the version of make that comes with the Mac command line tools is often 3.81 which needs to be updated to install SmartSim. Users can brew install make to get make > 4.0 but brew will install it as gmake. An easy way around this is to do alias make=gmake.

SmartSim on Ubuntu or Linux Workstations¶

When building SmartSim for Linux systems where the user has root access, many of the needed packages can be installed through apt.

If you have a CUDA enabled GPU and want to run SmartSim on GPU on a Linux system you have root access to, you can install CUDA through apt with the cuda package.

In addition, cuDNN can be installed through the libcudnn and libcudnn-dev pacakges.

If you run into trouble compiling the machine learning runtimes on Ubuntu because of cuDNN, it might be due to this issue

SmartSim on Cray XC, CS, and EX¶

If on a Cray system, be sure to set the correct toolchain. SmartSim is tested on PrgEnv-GNU and PrgEnv-Cray modules.

Note

If on a Cray, please note that the intel and PGI compiler toolchains are currently not supported by SmartSim.

Before installing the machine learning runtimes with the smart cli tool, be sure to set the CRAYPE_LINK_TYPE to dynamic

export CRAYPE_LINK_TYPE=dynamic

Keep in mind, the libraries installed above need to be accessible by SmartSim at runtime. If using a networked file system (NFS), make sure to install these somewhere reachable from head, MOM, and compute nodes (network mounted).

CUDA and CUDNN on Cray¶

Usually cudatoolkit is available as a module and CUDA is installed in /usr/local/cuda. In this case, prior to installation run

module load cudatoolkit

If cuDNN libraries and includes are not installed as a module or otherwise, you can install them with conda.

# Install CUDA requirements
conda install cudatoolkit cudnn
export CUDNN_LIBRARY=/path/to/miniconda/pkgs/cudnn-x.x.x-cudax.x_x/lib
export CUDNN_INCLUDE_DIR=/path/to/miniconda/pkgs/cudnn-x.x.x-cudax.x_x/include

Be sure to get the cuDNN version that matches your CUDA installation. The package names usually specify the versions.

SmartSim on Summit at OLCF¶

Since SmartSim does not have a built PowerPC build, the build steps for an IBM system are slightly different than other systems.

Luckily for us, a conda channel with all relevant packages is maintained as part of the OpenCE initiative. Users can follow these instructions to get a working SmartSim build with PyTorch and TensorFlow for GPU on Summit. Note that SmartSim and SmartRedis will be downloaded to the working directory from which these instructions are executed.

# setup Python and build environment
export ENV_NAME=smartsim-0.4.1
git clone https://github.com/CrayLabs/SmartRedis.git smartredis
git clone https://github.com/CrayLabs/SmartSim.git smartsim
conda config --prepend channels https://ftp.osuosl.org/pub/open-ce/1.4.1/
conda create --name $ENV_NAME -y  python=3.9 \
                                  git-lfs \
                                  cmake \
                                  make \
                                  cudnn=8.1.1_11.2 \
                                  cudatoolkit=11.2.2 \
                                  tensorflow=2.6.2 \
                                  libtensorflow=2.6.2 \
                                  pytorch=1.9.0 \
                                  torchvision=0.10.0
conda activate $ENV_NAME
export CC=$(which gcc)
export CXX=$(which g++)
export LDFLAGS="$LDFLAGS -pthread"
export CUDNN_LIBRARY=/ccs/home/$USER/.conda/envs/$ENV_NAME/lib/
export CUDNN_INCLUDE_DIR=/ccs/home/$USER/.conda/envs/$ENV_NAME/include/
module load cuda/11.4.2
export LD_LIBRARY_PATH=$CUDNN_LIBRARY:$LD_LIBRARY_PATH:/ccs/home/$USER/.conda/envs/$ENV_NAME/lib/python3.9/site-packages/torch/lib
module load gcc/9.3.0
module unload xalt
# clone SmartRedis and build
pushd smartredis
make lib && pip install .
popd

# clone SmartSim and build
pushd smartsim
pip install .

# install PyTorch and TensorFlow backend for the Orchestrator database.
export Torch_DIR=/ccs/home/$USER/.conda/envs/$ENV_NAME/lib/python3.9/site-packages/torch/share/cmake/Torch/
export CFLAGS="$CFLAGS -I/ccs/home/$USER/.conda/envs/$ENV_NAME/lib/python3.9/site-packages/tensorflow/include"
export SMARTSIM_REDISAI=1.2.5
export Tensorflow_BUILD_DIR=/ccs/home/$USER/.conda/envs/$ENV_NAME/lib/python3.9/site-packages/tensorflow/
smart build --device=gpu --torch_dir $Torch_DIR --libtensorflow_dir $Tensorflow_BUILD_DIR -v

# Show LD_LIBRARY_PATH for future reference
echo "SmartSim installation is complete, LD_LIBRARY_PATH=$LD_LIBRARY_PATH"

When executing SmartSim, if you want to use the PyTorch and TensorFlow backends in the orchestrator, you will need to set up the same environment used at build time:

module load cuda/11.4.2
export CUDNN_LIBRARY=/ccs/home/$USER/.conda/envs/$ENV_NAME/lib/
export LD_LIBRARY_PATH=/ccs/home/$USER/.conda/envs/smartsim/lib/python3.8/site-packages/torch/lib/:$LD_LIBRARY_PATH:$CUDNN_LIBRARY
module load gcc/9.3.0
module unload xalt

SmartSim on Cheyenne at NCAR¶

Since SmartSim does not currently support the Message Passing Toolkit (MPT), Cheyenne users of SmartSim will need to utilize OpenMPI.

The following module commands were utilized to run the examples:

$ module purge
$ module load ncarenv/1.3 gnu/8.3.0 ncarcompilers/0.5.0 netcdf/4.7.4 openmpi/4.0.5

With this environment loaded, users will need to build and install both SmartSim and SmartRedis through pip. Usually we recommend users installing or loading miniconda and using the pip that comes with that installation.

$ pip install smartsim
$ smart build --device cpu  #(Since Cheyenne does not have GPUs)

To make the SmartRedis library (C, C++, Fortran clients), follow these steps with the same environment loaded.

# clone SmartRedis and build
$ git clone https://github.com/SmartRedis.git smartredis
$ cd smartredis
$ make lib

Introduction

Community

SmartSim 0.4.1 documentation

Installation

Contents

Installation¶

Prerequisites¶

Supported Versions¶

SmartSim¶

Step 1: Install Python Package¶

Step 2: Build SmartSim¶

CPU Install¶

GPU Install¶

SmartRedis¶

Build SmartRedis Library (C++, C, Fortran)¶

From Source¶

Install SmartSim from Source¶

Install SmartRedis from Source¶

Developer Makefile¶

Clone SmartRedis¶

Building the Python Client from Source¶

Building SmartRedis Static Library from Source¶

Building the Documentation¶

With docker¶

Without docker¶

Installation Notes for Specific System Types¶

SmartSim on MacOS¶

SmartSim on Ubuntu or Linux Workstations¶

SmartSim on Cray XC, CS, and EX¶

CUDA and CUDNN on Cray¶

SmartSim on Summit at OLCF¶

SmartSim on Cheyenne at NCAR¶