// Multiple Jobs per Node
Background
The number of CPUs (aka "cores") per node among Discover's processor architectures has continued to increase over time, with current Milan processors having 128 cores. Skylake and Cascade Lake nodes offer 40 and 46 cores per node respectively.
Many NCCS users have legitimate use cases for significantly lower core counts per job (particularly for various pre- and post-processing steps, among others). Ideally, to enhance throughput, two or more such jobs could be packed onto a single node to run simultaneously. Slurm possesses a feature (called ExclusiveUser) that facilitates this mode of operation; only jobs running under your user ID will be able to share nodes.
Running Multiple Jobs per Node
The NCCS has established the new "packable" partition, in which users can run multiple small-core-count jobs per node, to increase overall throughput.
Customized to Your Needs
Since each packable workflow is unique, the NCCS Discover Team offers (and requires) 1-on-1 consultation with users and groups that wish to take advantage of this mode of operation, to customize a solution for your workflow.
In addition to establishing your job(s) as good candidates for the packable partition, we will also determine whether the Slurm "job array" feature should be leveraged, to enable you to, with a single sbatch command, submit a large set of jobs that vary only by the input and output data files they process. It will often be the case that job arrays can and should be used in the packable partition.
Onboarding
Once the Discover Team determines the job scheduling best practices relevant to your workflow, we'll establish a custom quality of service ("QoS") to handle your packable workload, and make that partition available to you. We'll monitor your workload initially to make sure it's working properly for you, and is non-disruptive with respect to filesystems load, etc.
First Step
If you're interested in exploring this option–and we encourage you to–please open a ticket by sending an email to support@nccs.nasa.gov, and include the phrase "packable candidate" in your subject.
