Developer

Developer#

This section details common practices and tips for contributors to SmartSim and SmartRedis.

Building the Documentation#

Users can optionally build documentation of SmartSim through make docs or make docks. make docs requires the user to install the documentation build dependencies, whereas make docks only requires docker. make docks is the recommended method for building the documentation locally, due to ease of use.

With docker#

Note

To build the full documentation with make docks, users need to install docker so that docker is available on the command line.

# From top level smartsim git repository directory
make docks

Once the documentation has successfully built, users can open the main documents page from docs/develop/index.html.

Without docker#

Note

To build the full documentation via make docs, users need to install doxygen 1.9.1. For Mac OS users, doxygen can be installed through brew install doxygen

# From top level smartsim git repository directory
git clone https://github.com/CrayLabs/SmartRedis.git smartredis
make docs

Once the documentation has successfully built, users can open the main documents page from doc/_build/html/index.html

Testing SmartSim#

Note

This section describes how to run the SmartSim (infrastructure library) test suite. For testing SmartRedis, see below

SmartSim utilizes Pytest for running its test suite. In the top level of SmartSim, users can run multiple testing commands with the developer Makefile

Test
-------
test                       - Build and run all tests
test-verbose               - Build and run all tests [verbosely]
test-cov                   - run python tests with coverage

Note

You must have the extra dev dependencies installed in your python environment to execute tests. Install smartsim with dev extension pip install -e .[dev]

Local#

There are two levels of testing in SmartSim. The first runs by default and does not launch any jobs out onto a system through a workload manager like Slurm.

If any of the above commands are used, the test suite will run the “light” test suite by default.

PBSPro, Slurm, LSF#

To run the full test suite, users will have to be on a system with one of the above workload managers. Additionally, users will need to obtain an allocation of at least 3 nodes.

# for slurm (with srun)
salloc -N 3 -A account --exclusive -t 00:10:00

# for PBSPro (with aprun)
qsub -l select=3 -l place=scatter -l walltime=00:10:00 -q queue

# for LSF (with jsrun)
bsub -Is -W 00:30 -nnodes 3 -P project $SHELL

Values for queue, account, or project should be substituted appropriately.

Once in an iterative allocation, users will need to set the test launcher environment variable: SMARTSIM_TEST_LAUNCHER to one of the following values

  • slurm

  • pbs

  • lsf

  • local

If tests have to run on an account or project, the environment variable SMARTSIM_TEST_ACCOUNT can be set.

Testing SmartRedis#

To build and run all tests on the local host, run the following command in the top level of the smartredis repository:

make test

Note

The tests require:

  • GCC >= 5

  • CMake >= 3.13

Since these are usually system libraries, we do not install them for the user.

You can also run tests for individual clients as follows:

make test-c         # run C tests
make test-fortran   # run Fortran tests. Implicitly, SR_FORTRAN=ON
make test-cpp       # run all C++ tests
make unit-test-cpp  # run unit tests for C++
make test-py        # run Python tests. Implicitly, SR_PYTHON=ON
make testpy-cov     # run python tests with coverage. Implicitly, SR_PYTHON=ON SR_BUILD=COVERAGE
make testcpp-cpv    # run cpp unit tests with coverage. Implicitly, SR_BUILD=COVERAGE

Customizing the Tests#

Several Make variables can adjust the manner in which tests are run:

  • SR_BUILD: change the way that the SmartRedis library is built. (supported: Release, Debug, Coverage; default for testing is Debug)

  • SR_FORTRAN: enable Fortran language build and testing (default is OFF)

  • SR_PYTHON: enable Python language build and testing (default is OFF)

  • SR_TEST_PORT: change the base port for Redis communication (default is 6379)

  • SR_TEST_NODES: change the number of Redis shards used for testing (default is 3)

  • SR_TEST_REDIS_MODE: change the type(s) of Redis servers used for testing. Supported is Clustered, Standalone, UDS; default is Clustered)

  • SR_TEST_REDISAI_VER: change the version of RedisAI used for testing. (Default is v1.2.3; the parameter corresponds the the RedisAI gitHub branch name for the release)

  • SR_TEST_DEVICE: change the type of device to test against. (Supported is cpu, gpu; default is cpu)

  • SR_TEST_PYTEST_FLAGS: tweak flags sent to pytest when executing tests (default is -vv -s)

These variables are all orthogonal. For example, to run tests for all languages against a standalone Redis server, execute the following command:

make test SR_FORTRAN=ON SR_PYTHON=ON SR_TEST_REDIS_MODE=Standalone

Similarly, it is possible to run the tests against each type of Redis server in sequence (all tests against a standalone Redis server, then all tests against a Clustered server, then all tests against a standalone server with a Unix domain socket connection) via the following command:

make test SR_FORTRAN=ON SR_PYTHON=ON SR_TEST_REDIS_MODE=All

Note

Unix domain socket connections are not supported on MacOS. If the SmartRedis test system detects that it is running on MacOS, it will automatically skip UDS testing.

Git Workflow#

Setup#

  • Fork the SmartSim (SmartRedis) repository

  • The origin remote should be set to your fork for pull and push

  • Set upstream as the main repository and set upstream push remote to no_push

  • Follow installation instructions

Pull Requests#

Please check the following before submitting a pull request to the SmartSim repository

  1. Your feature is on a new branch off master.

  2. You are merging the feature branch from your fork into the main repository.

  3. All unnecessary whitespace has been purged from your code.

  4. For Python code changes, Black and isort have been applied to format code and sort imports.

  5. Pylint errors have been minimized as much as possible.

  6. All your code has been appropriately documented.

  7. The PR description is clear and concise.

  8. You have requested a review.

Merging#

When merging, there are a few guidelines to follow

  • Wrap all merge messages to 70 characters per line.

Python Guidelines#

For the most part, the conventions are to follow PEP8 that is supplied by pylint. However, there are a few things to specifically mention.

  • Underscores should precede methods not meant to be used outside a class

  • All methods should have docstrings (with some exceptions)

  • Variable names should accurately capture what values it is storing without being overly verbose

  • No bad words

  • Use Black and isort frequently

  • Utilize conftest.py for creating pytest fixtures

SmartSim Python Style Guide Do’s and Don’ts:

  • DON’T use global variables or the global keyword unless necessary

  • DON’T over comment code when it reads like English

  • DO use underscores on methods that should not be used outside a class

  • DO use comprehensions

  • DON’T write functions for more than one purpose

  • DON’T allow functions to return more than one type

  • DON’T use protected member variables outside a class

  • DON’T use single letter variable names

  • DO use entire words in names (i.e. get_function not get_func)

  • DON’T use wildcard imports (i.e. from package import *) unless __all__ is defined

  • DO use snake_case when naming functions, methods, and variables

  • DO use PascalCase when naming Classes

  • DO use the logging module

Editor Suggestions#

The editor that we suggest developers use is VSCode. Below are some extensions that could make the process of developing on SmartSim a bit easier.

  • GitLens, for viewing changes in the git history

  • Remote SSH, for connecting to clusters and supercomputers

  • PyLance, Language Server

  • Python indent, for correcting python indents

  • reStructuredText, for writing documentation

  • Strict Whitespace, for ensuring no whitespace left in code

  • Python Docstring Generator, for writing docstring quickly

  • C/C++, for client development

  • Settings Sync, for syncing settings across remote servers

Contents