// Using Prism

Prism is equipped with several powerful servers built specifically for accelerating AI/ML/DL workloads with GPUs. This cutting edge platform is easy to access and the preinstalled software/libraries provide foundational tools that enable scientists to maximize their workflows.

• Environment
  • System Specifications
  • Operating System and Software
• Access
• Tools
  • Ganglia
  • JupyterHub
  • Conda
  • Modules
• SLURM
  • SLURM Interface Methods
  • Configuration Flags
  • Examples & Documentation
  • Useful SLURM Commands
• High Speed Data Storage
• Converting workflows to ARM architecture
  • Non AI/ML Python Environment
  • AI/ML Python environment
• Using containers on Prism
  • Podman
  • Singularity/Apptainer
  • Troubleshooting
  • AI/ML workflows on Jupyter Hub
  • Custom Kernels in Grace Hopper Jupyter Hub

Environment

Access

Access to the Prism GPU Cluster is provided with your NCCS user account. Prism Jupyter Hub is available via NAMS request. You may connect by logging in to adapt.nccs.nasa.gov or adaptlogin.nccs.nasa.gov, then ssh to gpulogin1. Once you are connected to the login node, you will need to use SLURM in order to access the Prism GPU resources.

For more information on SLURM, see the SLURM section below.

For more information on access and login, see NCCS Account Setup.

Tools

Ganglia
To see gpu utilization on every node, go to: Prism Ganglia

All nodes have four GPUs, you need to select "gpuX_util" for GPUs 0, 1, 2 and 3. The DGX provides access to GPUs 4-7, as well.

JupyterHub
See here for information on accessing and using the current JupyterHubs on ADAPT.

Conda
Click here to read our nugget about configuring Conda.

Conda environments have been used to install the Python machine learning frameworks. These environments can be accessed by loading the Conda module with:

To activate an environment, run 'conda activate <ENV>' on the environment of your choice once the module is loaded.

Users can inspect the complete list of packages and versions installed within an environment by running: $ conda list

Users can also inspect other available environments by running: $ conda env list

Users may also create Conda environments in their home directory. This will allow users to maintain the environment on their own.

For more information regarding Conda usage in ADAPT, see our Instructional Video and Tech Talk slides.

If you are experiencing issues with Conda, and/or if additional package installation support is needed, please contact NCCS support.

Modules
Users are recommended to load both the Conda and NVIDIA modules in Prism using 'module load nvidia' and 'module load Conda'; use 'module spider' for more options. For more information on how to load modules, see the Tips & Info for New NCCS ADAPT Users Tech Talk slides under "Modules".

SLURM

SLURM allows for more efficient resource allocation, fairer sharing, and easier management of the resources on the GPU nodes. There are three main ways to interface with SLURM on the GPU nodes:

1. 'sbatch': Submit a batch script to SLURM. Create a job script that can be submitted to the queue and call multiple tasks from within.


2. 'srun': Specify resources for running a single command or execute a job step.


3. 'salloc': Run interactively on allocated resources, or run a step by step job.

All three of these mechanisms share most, but not all, of the standard SLURM configuration flags. Some of the most useful are as follows:

Examples & Links to Documentation:
Here are some examples to get you started. If you desire more advanced configuration to optimize your jobs, reference the documentation (linked below) for each command to see the complete list of available flags and usage options.

SBATCH: 'sbatch job.sh'
#!/bin/bash
#SBATCH -G5 -t 60 -n5 -N1 -J myBatchScriptSLURMJob --export=ALL
module load Conda
conda activate <ENV>
#Run the same task
#Run tasks in parallel with '&' and 'wait'
srun -G3 -n1 python my_program.py1 &
srun -G2 -n1 python my_program2.py & wait
#Run tasks sequentially without '&'
srun -G5 -n1 python my_program3.py
srun -G5 -n1 python my_program4.py

SBATCH: 'sbatch job with partition.sh'
#!/bin/bash
#SBATCH -G5 -t 60 -n5 -N1 -J myBatchScriptSLURMJob --export=ALL -p grace
module load Conda
conda activate <ENV>
#Run the same task
#Run tasks in parallel with '&' and 'wait'
srun -G3 -n1 python my_program.py1 &
srun -G2 -n1 python my_program2.py & wait
#Run tasks sequentially without '&'
srun -G5 -n1 python my_program3.py
srun -G5 -n1 python my_program4.py

SRUN:
srun -G2 -t 60 -n1 --mem-per-cpu=100 -J myOneLineSLURMJob python myScript.py

SALLOC:
salloc -G1 -t 60 -n1 -c6 –mem-per-cpu=1028 --nodelist=gpu001 -J myInteractiveSLURMJob

SALLOC for the DGX (NVIDIA A100s):
salloc -G2 -p dgx

SALLOC for the Grace Hopper (Nvidia H100s):
salloc -G1 -p grace

SALLOC for the Grace Grace (CPU only):
salloc -p grace-cpuonly

Running salloc will give you an interactive shell with access to specified resources. This is similar to sshing into one of the nodes, however, the resources that you can use will be limited to those that are requested through your allocation.

Learn more about SLURM on ADAPT

High Speed Data Storage

Should your workload be experiencing limitations as a result of file I/O, each system has several TB of high speed NVME based storage available for use. This storage is located at "/lscratch". Run the command "mkdir /lscratch/$(whoami)" to create a directory owned by you. This storage is local to the node, and is not shared with other nodes in the cluster. Note that this space is temporary, and data stored there is subject to deletion at the completion of your job.

Converting workflows to ARM architecture

1. Non AI/ML Python Environment
   1. Python environments compiled in x86_64 architecture will need to be recompiled for aarch64 (ARM) architecture.
   2. Load the NCCS minforge/mamba module
      module load miniforge
   3. First export the configuration of the original x86 conda environment.
      conda env export -n (environment name) --no-builds > environment.yaml

Ex: conda env export --no-builds -n torch_roth > environment.yaml
   4. Attempt to create a new environment on ARM based machine.

      We are using mamba for this step as it has better version translation capabilities. Using the conda command will result in a failed conversion.

      mamba env create -n (new environment name) -f (Dump file just created)

Ex: mamba env create -n arm_torch2 -f environment.yaml

      This command may fail if packages installed on an x86_64 environment are not available for ARM (aarch64). In this case the package that could not be installed will need to be removed from the dump file created earlier.

      If you have Pytorch installed and intend to use it, you will want to remove torch torchaudio and torchvision from the list of packages to install. This will allow a new install of Pytorch that will work properly with the Grace Hopper nodes.

   5. After the install successfully completes, you can load your environment.
   6. If you need Pytorch, please continue on to the next section for instructions on installing Pytorch.
2. AI/ML Python environment.
   1. Pytorch
      1. A Pytorch package that is compatible with Grace Hopper can be installed in your conda environment from the pytorch site:
         [bsroth@gh004 ~]$ conda create --name torch
[bsroth@gh004 ~]$ conda activate torch
(torch) [bsroth@gh004 ~]$ conda install pip

An example version:
pip install torch==2.7.1 torchvision==0.22.1 torchaudio==2.7.1 --index-url https://download.pytorch.org/whl/cu129

You can alter the versions of torch,torchvision and torchaudio as require. In addition the 'cu129' directory refers to the cuda version 12.9. This can be altered as well to the appropriate cuda version.
      2. Note if a list of available versions are needed, visit the url for the cuda version you are attempting to use. In the example above if you go to https://download.pytorch.org/whl/cu129 you can find directories for each application and find available versions listed within.
   2. Tensorflow
      1. At this point in time Tensorflow for Grace Hopper is only available via container from the Nvidia container repository Link Here. You can download this container form Nvidia and customize as needed. Different versions are available via the tags tab.

Using containers on Prism

Podman

Singularity/Apptainer

Troubleshooting

1. Podman
   1. Container does not download properly, shows "diff: operation no supported" error at end of download.
      1. Our clustered storage system does not support the layered filesystem used by Podman/Docker. Typically the user profile script will fix this on login to any of the Prism GPU nodes.
      2. The error typically looks as below:
         Error: copying system image from manifest list: writing blob: adding layer with blob "sha256:b91d8878f844c327b4ff924d4973661a399f10256ed50ac7c640b30c5894166b"/""/"sha256:238e596eb101589755d14f2ad4a1979baa274147028bba76728b1b0a069c00e0": creating read-only layer with ID "238e596eb101589755d14f2ad4a1979baa274147028bba76728b1b0a069c00e0": lsetxattr /home/bsroth/.local/share/containers/storage/overlay/238e596eb101589755d14f2ad4a1979baa274147028bba76728b1b0a069c00e0/diff: operation not supported
      3. To solve this issue check the following:
         1. Make sure /lscratch/$USER exists
         2. Remove /home/$USER.config/containers directory if it exists
         3. Log out of the node and back into the node to run the profile script again.
         4. Ensure that /home/$USER/.config/containers/storage.conf exists with contents:
            [root@gh004 containers]# cat storage.conf
[storage]
driver = "overlay"
runroot = "/run/user/578207330"
graphroot = "/lscratch/bsroth"

Note the UID in runroot will be different and the user id should match your user id.
         5. Remove the directory in /home/$USER/.local/containers or rename the directory, then repeat the previous steps.
2. Singularity/Podman
   1. Docker container conversion to .sif file freezes or has errors.
      1. Typically when errors appear similar to the following, the environment will need to be set to use a local file system as opposed to a shared file system (/home or /explore/nobackup)
         2025/10/09 10:18:11 warn xattr{usr/share/doc/cuda-runtime-9-0/NVIDIA-CUDA.jpg} ignoring ENOTSUP on setxattr "user.rootlesscontainers"
2025/10/09 10:18:11 warn xattr{/explore/nobackup/people/bsroth/.nccstmp/build-temp-3844063746/rootfs/usr/share/doc/cuda-runtime-9-0/NVIDIA-CUDA.jpg} destination filesystem does not support xattrs, further warnings will be suppressed
      2. To solve this the following environment variables will need to be set, making sure that the directory you are indicating exists:
         export SINGULARITY_TMPDIR=/lscratch/$USER/
export SINGULARITY_CACHEDIR=/lscratch/$USER/
export TMPDIR=/lscratch/$USER/

AI/ML workflows on Jupyter Hub for Grace Hopper

Custom Kernels in Grace Hopper Jupyter Hub

If you desire additional python or other packages installed inside of the Tensorflow or Pytorch containers you can user the NCCS containers as a base to build a custom container to load into Jupyter Hub.

1. Existing containers available to users can be found at /app/jupyter/kernels.
2. Create a apptainer/singularity build file:

   Nvidia containers are Debian based and use apt for package management.

   Bootstrap: localimage
From: (path to base image)

# Optional for metadata and documentation
%labels
    Image prism-nvidia-pytorch
    Tag-name tag-value
%help
    Help message

%post
    (additional bash commands to install needed packages)


Example:
Bootstrap: localimage
From: /app/jupyter/kernels/pytorch/pytorch_aarch64.sif

# Optional for metadata and documentation
%labels
    Image prism-nvidia-pytorch
    Tag-name tag-value
%help
    Help message

%post
    pip install mypackage
3. Save this file.
4. Build the new container image:
   singularity build (container file to create).sif (path to build file)

example:
singularity build mycontainer.sif mybuildfile.txt
5. Once the container is built create a directory for the custom kernel configuration file in ~/.local/share/jupyter/kernels
   mkdir ~/.local/share/jupyter/kernels/<kernelname>

Example:

mkdir ~/.local/share/jupyter/kernels/mykernel
6. Create the custom kernel "kernel.json" file in the path created in previous step.
Template:

{
    "argv": [
        "apptainer",
        "exec",
        "--nv",
        "-B",
        "/app,/home,/explore/nobackup/people,/explore/nobackup/projects,{connection_file}",
        "(path to custom container)",
        "python",
        "-m",
        "ipykernel_launcher",
        "-f",
        "{connection_file}"
    ],
    "display_name": "<kernel display name>",
    "language": "python",
    "metadata": {
        "debugger": true
    }
}
Example:

{
    "argv": [
        "apptainer",
        "exec",
        "--nv",
        "-B",
        "/app,/home,/explore/nobackup/people,/explore/nobackup/projects,{connection_file}",
        "/home/bsroth/test.sif",
        "python",
        "-m",
        "ipykernel_launcher",
        "-f",
        "{connection_file}"
    ],
    "display_name": "bsroth pytorch",
    "language": "python",
    "metadata": {
        "debugger": true
    }
}
7. Start a Jupyter Hub session and verify the custom kernel appears in the kernel pull down menu and works as expected.