Developer#
This section details common practices and tips for contributors to SmartSim and SmartRedis.
Building the Documentation#
Users can optionally build documentation of SmartSim through make docs
or
make docks
. make docs
requires the user to install the documentation
build dependencies, whereas make docks
only requires docker. make docks
is the recommended method for building the documentation locally, due to ease of
use.
With docker#
Note
To build the full documentation with make docks
, users need to install
docker so that docker
is available
on the command line.
# From top level smartsim git repository directory
make docks
Once the documentation has successfully built, users can open the main documents
page from docs/develop/index.html
.
Without docker#
Note
To build the full documentation via make docs
, users need to install
doxygen 1.9.1
. For Mac OS users, doxygen can be installed through brew
install doxygen
# From top level smartsim git repository directory
git clone https://github.com/CrayLabs/SmartRedis.git smartredis
make docs
Once the documentation has successfully built, users can open the main documents
page from doc/_build/html/index.html
Testing SmartSim#
Note
This section describes how to run the SmartSim (infrastructure library) test suite. For testing SmartRedis, see below
SmartSim utilizes Pytest
for running its test suite. In the top level of
SmartSim, users can run multiple testing commands with the developer Makefile
Test
-------
test - Build and run all tests
test-verbose - Build and run all tests [verbosely]
test-cov - run python tests with coverage
Note
You must have the extra dev dependencies installed in your python
environment to execute tests. Install smartsim with dev
extension
pip install -e .[dev]
Local#
There are two levels of testing in SmartSim. The first runs by default and does not launch any jobs out onto a system through a workload manager like Slurm.
If any of the above commands are used, the test suite will run the “light” test suite by default.
PBSPro, Slurm, LSF#
To run the full test suite, users will have to be on a system with one of the above workload managers. Additionally, users will need to obtain an allocation of at least 3 nodes.
# for slurm (with srun)
salloc -N 3 -A account --exclusive -t 00:10:00
# for PBSPro (with aprun)
qsub -l select=3 -l place=scatter -l walltime=00:10:00 -q queue
# for LSF (with jsrun)
bsub -Is -W 00:30 -nnodes 3 -P project $SHELL
Values for queue, account, or project should be substituted appropriately.
Once in an iterative allocation, users will need to set the test launcher
environment variable: SMARTSIM_TEST_LAUNCHER
to one of the following values
slurm
pbs
lsf
local
If tests have to run on an account or project, the environment variable
SMARTSIM_TEST_ACCOUNT
can be set.
Testing SmartRedis#
To build and run all tests on the local host, run the following command in the top level of the smartredis repository:
make test
Note
- The tests require:
GCC >= 5
CMake >= 3.13
Since these are usually system libraries, we do not install them for the user.
You can also run tests for individual clients as follows:
make test-c # run C tests
make test-fortran # run Fortran tests. Implicitly, SR_FORTRAN=ON
make test-cpp # run all C++ tests
make unit-test-cpp # run unit tests for C++
make test-py # run Python tests. Implicitly, SR_PYTHON=ON
make testpy-cov # run python tests with coverage. Implicitly, SR_PYTHON=ON SR_BUILD=COVERAGE
make testcpp-cpv # run cpp unit tests with coverage. Implicitly, SR_BUILD=COVERAGE
Customizing the Tests#
- Several Make variables can adjust the manner in which tests are run:
SR_BUILD: change the way that the SmartRedis library is built. (supported: Release, Debug, Coverage; default for testing is Debug)
SR_FORTRAN: enable Fortran language build and testing (default is OFF)
SR_PYTHON: enable Python language build and testing (default is OFF)
SR_TEST_PORT: change the base port for Redis communication (default is 6379)
SR_TEST_NODES: change the number of Redis shards used for testing (default is 3)
SR_TEST_REDIS_MODE: change the type(s) of Redis servers used for testing. Supported is Clustered, Standalone, UDS; default is Clustered)
SR_TEST_REDISAI_VER: change the version of RedisAI used for testing. (Default is v1.2.3; the parameter corresponds the the RedisAI gitHub branch name for the release)
SR_TEST_DEVICE: change the type of device to test against. (Supported is cpu, gpu; default is cpu)
SR_TEST_PYTEST_FLAGS: tweak flags sent to pytest when executing tests (default is -vv -s)
These variables are all orthogonal. For example, to run tests for all languages against a standalone Redis server, execute the following command:
make test SR_FORTRAN=ON SR_PYTHON=ON SR_TEST_REDIS_MODE=Standalone
Similarly, it is possible to run the tests against each type of Redis server in sequence (all tests against a standalone Redis server, then all tests against a Clustered server, then all tests against a standalone server with a Unix domain socket connection) via the following command:
make test SR_FORTRAN=ON SR_PYTHON=ON SR_TEST_REDIS_MODE=All
Note
Unix domain socket connections are not supported on MacOS. If the SmartRedis test system detects that it is running on MacOS, it will automatically skip UDS testing.
Git Workflow#
Setup#
Fork the SmartSim (SmartRedis) repository
The origin remote should be set to your fork for pull and push
Set upstream as the main repository and set upstream push remote to
no_push
Follow installation instructions
Pull Requests#
Please check the following before submitting a pull request to the SmartSim repository
Your feature is on a new branch off master.
You are merging the feature branch from your fork into the main repository.
All unnecessary whitespace has been purged from your code.
For Python code changes, Black and isort have been applied to format code and sort imports.
Pylint errors have been minimized as much as possible.
All your code has been appropriately documented.
The PR description is clear and concise.
You have requested a review.
Merging#
When merging, there are a few guidelines to follow
Wrap all merge messages to 70 characters per line.
Python Guidelines#
For the most part, the conventions are to follow PEP8 that is supplied by pylint. However, there are a few things to specifically mention.
Underscores should precede methods not meant to be used outside a class
All methods should have docstrings (with some exceptions)
Variable names should accurately capture what values it is storing without being overly verbose
No bad words
Use Black and isort frequently
Utilize
conftest.py
for creating pytest fixtures
SmartSim Python Style Guide Do’s and Don’ts:
DON’T use global variables or the global keyword unless necessary
DON’T over comment code when it reads like English
DO use underscores on methods that should not be used outside a class
DO use comprehensions
DON’T write functions for more than one purpose
DON’T allow functions to return more than one type
DON’T use protected member variables outside a class
DON’T use single letter variable names
DO use entire words in names (i.e. get_function not get_func)
DON’T use wildcard imports (i.e.
from package import *
) unless__all__
is definedDO use snake_case when naming functions, methods, and variables
DO use PascalCase when naming Classes
DO use the logging module
Editor Suggestions#
The editor that we suggest developers use is VSCode. Below are some extensions that could make the process of developing on SmartSim a bit easier.
GitLens, for viewing changes in the git history
Remote SSH, for connecting to clusters and supercomputers
PyLance, Language Server
Python indent, for correcting python indents
reStructuredText, for writing documentation
Strict Whitespace, for ensuring no whitespace left in code
Python Docstring Generator, for writing docstring quickly
C/C++, for client development
Settings Sync, for syncing settings across remote servers