Issue to be resolved
Where is the best place to write temporary (scratch) data on Discover?
Solution
Discover has multiple scratch filesystems: $LOCAL_TMPDIR, $TMPDIR, $TSE_TMPDIR.
Each of these environment variables are created during login sessions for interactive use as well as each individual SLURM batch job.
$LOCAL_TMPDIR is node specific so any data stored there can only be accessed from the particular node where it is placed. $TMPDIR is accessible from any node and has a default quota of 5TB and 300k inodes.
To increase the I/O performance of jobs that read and write temporary files, there is a fast scratch filesystem using NVMe (flash) technology.
Access this storage by using $TSE_TMPDIR where you might otherwise use $TMPDIR. Each user has a default quota of 250 GB and 200,000 inodes on $TSE_TMPDIR. If you have a job that needs more than the default allocation, please email us at support@nccs.nasa.gov.
The data stored in the TMPDIR environments are not backed up and are deleted at the end of your job or login session because these resources are very limited.
Do not specify $ARCHIVE for temporary storage. It is not part of the local Discover filesystems. It is a tape library mounted over the network and most users do not have access because it is a deprecated service.
Review additional information on Discover Filesystem and Storage.