// Monitoring Data on MSS 

How the Data Management Facility (DMF) works

Any file written to /archive (the cache filesystem) gets written to tape, even if the file is small, with one exception: empty files (files with zero lengths) are not written to tape. There are no guarantees any particular tape will store a particular file. If you have a thousand files, they can be spread over a thousand tapes. This can also create a thousand tape mounts (if you recall them), which is a mechanical process, and therefore takes a long time. Because of that process, and the quota limit on the number of files you are allowed, it is better to have a few large files over many small ones.

As you determine your actual storage usage, to effectively manage data holdings within assigned quotas, you should keep the following in mind. Normal Linux commands like "du" only show what is used on the disk subsystem. They do not reflect what is offline on tape. Below there are DM aware commands that account for files on tape, as well as MSS Report commands.

Monitor using Quota, MSS Reports, and DM Commands

The quota command allows you to monitor your Dirac $HOME space and inode consumption. However, the quota command cannot read tape data; so, MSS Reports and DM Commands are best used to monitor data on $ARCHIVE tapes.

On Dirac, there is a space quota for groups and a limit on the number of files for individual users. Use the "quota" command (instead of "showquota" on Discover) to see individual user /home quotas:

dirac $ quota -ls userid The [-l] attribute will not query the NFS filesystems and show only your local quota. The [-s] attribute will display your quota with units (MB, GB, etc.).

Disk quotas for user userid (uid 1230):

dirac $ quota -ls userid
Disk quotas for user userid (uid 1230):
     Filesystem   space   quota   limit   grace   files   quota   limit   grace
/dev/cxvm/is5660_home
                    40K    500M    600M             11    250k    400k
/dev/cxvm/sfa_cache05
                    266M    0K     0K                63    250k    400k

In the first entry, the quota for this /home directory is 500MB with a hard limit of 600MB. You can exceed your quota if you are under the hard limit, however, any process that attempts to create enough data to exceed it is immediately terminated.

User Usage Reports

Since the quota command does not account for data that is only on tape, use the /local/nccs/bin/mssuserreport command to see all your data holdings.

$ /local/nccs/bin/mssuserreport

Your groups: group1 group2
Your default group: group1

Invalid groups being used:
Usage Report (GBs:# of Files) for user userid(1230)

            sfa_cache01    ...    sfa_cache05    ...    sfa_cache09        Totals
group1         0.0:0        ...        3.7:59     ...         0.0:0           3.7:59
Totals         3.7 (GBs)         59 (# of Files)

Your /archive tape usage is printed on the left in GBs, and your /archive inode consumption is printed on the right. To compare this utilization to your group(s) quota, use the group report command.

Group Usage Reports

Group data quotas were enabled on Dirac so users can monitor the space available on /archive tapes, which was granted to each group's principal investigator in the SBU allocation process. Use the /local/nccs/bin/mssgroupreport command to see who is storing data in the group:

$ /local/nccs/bin/mssgroupreport

Your groups are: group1
Usage Report (GBs:# of Files) for group group1(groupid)
                 sfa_cache01     ...     sfa_cache04    ...     sfa_cache09
userid            0.0:0                121789.8:255367             0.0:0
Totals          121789.8 (GBs)              255367 (# of Files)
Quota            124789.8
Percentage of Quota              97.6

The group report command will print a list of groups you are a member of, then it will print a usage report for each of those groups sorted by their members.

Files managed under the Data Management Facility (DMF) can be in one of several states:

State Description
UNM are in the process of moving from tape to disk, also known as "unmigrating".
REG are not managed by DMF. Directories fall in this category.
OFL are offline, they are on tape only.
MIG are in the process of migrating from disk to tape.
DUL are in dual state, that is, they are available on disk and tape.

DM Commands

Command Description Usage
dmls The dmls command on dirac lets you know the state of a file. $ dmls -l $ARCHIVE/filename
dmdu The dmdu command gives you a reasonable estimate of how much space is used in a directory by all files, those on disk and offline on tape. In contrast, du accounts for files only on disk and has no idea about the tapes. $ cd $ARCHIVE
$ dmdu -h
dmfind The base linux find command will find all files whether they are offline or not, because the directory structure itself is always on disk, only the actual files are migrated to tape. Therefore, we suggest using the DMF aware dmfind command, which works similarly to the find command, but can tell the difference between DMF states. $ dmfind $ARCHIVE -type f -state OFL
# this command prints all files that are offline
dmget The dmget command is used to recall a file from tape to the disk cache. This could take a while. After completion the state of affected file changes from offline (OFL) to dual (DUL). Copying offline files to your $NOBACKUP is allowed, but you will endure the additional wait time caused by the implicit unmigration process to complete. $ dmget filename.tar
dmtag The dmtag command is used to check or set the sitetag of a file. More information $ dmtag -t 2 list-of-archive-files

If you need to recall more than a few hundred files at a time, consider asking us to organize your list by tape, so that you can recall the list in groups without overloading the tape system. Also, please be cognizant of the amount of data you recall at once. If you need to recall multiple terabytes of data, please request our help.



Monitor Project Storage using MSS Commands

Use MSS scripts to monitor storage for yourself and your team:

The following scripts allow users to better manage their and their team's storage:

Script Name Location Purpose Description
groupaccessreport /local/nccs/bin/groupaccessreport Provides a snapshot of the group’s data holding based on last access. The header numbers are days (e.g. 30 is data accessed in the last 30 days). The first data line is number of files, and the second is amount of data in GBs. If no options are provided, it outputs the list from your default group. If you provided “-g groupname”, the list from groupname is provided, as long as you are a member.
groupchange /local/nccs/bin/groupchange.pl Provides a report on a requested group showing the change in usage for each of the group’s members Options are [-g groupname] [-s Start Date] [-e End Date]. Date format is YYYYMMDD. If the group is not provided, your default group is used. If either date is not specified, the current date is used.
groupgrowthreport /local/nccs/bin/groupgrowthreport Provides a daily total in GBs of data storage under a group. If no options are provided, the list from your default group is output. If you provided “-g groupname”, the list from groupname is output provided you are a member of that group.
useraccessreport /local/nccs/bin/useraccessreport Provides a snapshot of the user’s data holding based on last access. The header numbers are days (e.g. 30 is data accessed in the last 30 days). The first data line is number of fines, and the second is amount of data in GBs.
usergrowthreport /local/nccs/bin/usergrowthreport Provides a daily total in GBs of data storage under the user running this command.