Policeme provides utility to monitor/record memory used by processes running on each compute node of a slurm job. The program generates information in XML format files which can then be post-processed, using supplied python script, to generate PNG format files.
The latest version of policeme 1.0.7 is located on the Discover cluster in the following directory:
This directory contains the executable policeme.exe and a scripts directory. The scripts directory contains csh script that can be used to set module environment and a python script that is used to process XML files to generate PNG format plot-image files.
The utility policeme.exe should be run in background mode and should be the first running process within a slurm job. The remaining job running instructions should be place after the policeme.exe line.
To run a policeme process in the background in a slurm script, place following after the #SBATCH statement lines and before any other processes or program executable commands:
The command above will execute policeme and create a log directory. The directory will be created with a name that will be string equivalent of the slurm jobID, which is automatically generated after the job script executes. The directory is created in your current working directory.
or, to place the directory at a specific location, use:
/usr/local/other/policeme/v1.0.7/policeme.exe -d fullPath &
The command above will execute policeme and place the log directory which string equivalent of Slurm jobID, and place the directory in the fullPath mentioned in the command.
Every execution of the policeme program creates a directory with name string equivalent of slurm jobID. The directory contains the output files, which are log and XML files.
The following options are available for policeme:
|-j, --jobid=JobID||The Slurm Job Id (only submitted mode )|
|-r, --samplerate=SampleRate||The sample rate in seconds|
|-s, --swapratio=SwapRatio||The swap ratio limit. Default value is set to 0.6, and max value can be set to 1.0.|
|-d, --logdir=LogDir||The directory where the logs are kept (This includes XML and log files for each node of the slurm job)|
|-q, --quietmode=QuietMode||If the memory logs are not required specify 1 (Quiet Mode), if they are required specify 0|
|-v, --version||Display the version number|
|-c, --config||Print the configuration mode|
As mentioned above, run the policeme process before any other program that requires memory monitoring. Other programs can be run normally once policeme has begun running in the background using the examples mentioned above.
Displaying the Results from Policeme
Use /usr/local/other/policeme/v1.0.7/scripts/plotXMLResults.py to process XML files.
In the following example, assume that the policeme.exe is running using the following commands:
#SBATCH . . . .
/usr/local/other/policeme/v1.0.7/policeme.exe -d $NOBACKUP/myPoliceOut &
Note: Please ensure the directory mentioned in the command pre-exists, as the -d flag is being utilized in the example above. If -d option is used, ensure the directory, following this option, exists.
Within this directory, a new slurm jobID directory will be created. The new directory will contain log files, which are text files and XML format files. The XML format files can be processed, to form PNG format files, using the following python script:
For more help on using the python script, run:
$ /usr/local/other/policeme/v1.0.7/scripts/plotXMLResults.py --help
The python script can be run using:
$ /usr/local/other/policeme/v1.0.7/scripts/plotXMLResults.py -j <JobID> -d <LogDir>
where <LogDir> is the prefix path for the log directory, and <JobID> is the slurm job ID.
Once the python script has executed, the output image in PNG format files will be created and placed in the jobID directory. Use the "display" command to display PNG image files.