Attention
The U.S. government is closed. However, because the information this website provides is necessary to protect lives and property this site will be updated and maintained during the federal government shutdown.
For the latest forecasts and critical weather information, visit weather.gov.
To learn more, visit commerce.gov/news/blog.
Data Storage
Each RDHPCS system has multiple options for data storage. Each user has multiple user-affiliated storage spaces, and each project has multiple project-affiliated storage spaces where data can be shared for collaboration. Below we give an overview and explain where each storage area is mounted.
Summary of Storage Areas
The storage area to use in any given situation depends upon the activity you wish to carry out. Storage areas are either user-centric or project-centric, and are further divided by the underlying storage type (e.g., Network File System (NFS), High Performance Storage System (HPSS), Lustre, IBM Spectrum Scale). Each storage type has a different intended use as described below.
On each system, each user has a User Home area, and may have a User Archive area. Each project has a Project Home area, Work areas, and Archive areas. The different storage areas are summarized in the list and table below.
User Home: Long-term data for routine access that is unrelated to a project. It is mounted on compute nodes as read/write. We strongly recommend that users launch and run jobs from one of the Work file systems due to its larger storage capacity and superior performance.
Project Home: Long-term project data for routine access that’s shared with other project members. It is mounted on compute as read/write. We strongly recommend that users launch and run jobs from one of the Work file systems due to its larger storage capacity and superior performance.
Member Work: Short-term user data for fast, batch-job access that is not shared with other project members.
Project Work: Short-term project data for fast, batch-job access that’s shared with other project members.
World Work: Short-term project data for fast, batch-job access that’s shared with users outside your project.
Member Archive: Long-term project data for archival access that is not shared with other project members.
Project Archive: Long-term project data for archival access that’s shared with other project members.
World Archive: Long-term project data for archival access that’s shared with users outside your project.
User-Shared Applications: User-managed applications shared with all users on the system.
Area |
Path |
Type |
Permissions |
Quota |
Backups |
Purged |
Retention |
On Compute Nodes |
---|---|---|---|---|---|---|---|---|
User Home |
|
NFS |
User set |
50 GB |
Yes |
No |
90 days |
Read/Write |
Project Home |
|
NFS |
2770 |
Project Based |
Yes |
No |
90 days |
Read/Write |
Member Work |
|
GPFS |
User set |
Project Based |
No |
No |
N/A |
Read/Write |
Project Work |
|
GPFS |
2770 |
Project Based |
No |
No |
N/A |
Read/Write |
Project World Work |
|
GPFS |
2775 |
Project Based |
No |
No |
N/A |
Read/Write |
User-Shared Applications |
|
NFS |
0755 |
N/A |
No |
No |
N/A |
Read/Write |
Area |
Path |
Type |
Permissions |
Quota |
Backups |
Purged |
Retention |
On Compute Nodes |
---|---|---|---|---|---|---|---|---|
User Home |
|
NFS |
User set |
10 GB |
Yes |
No |
90 days |
Read/Write |
Member Work |
|
Lustre |
Project set |
Project Based |
No |
No |
N/A |
Read/Write |
User-Shared Applications |
|
NFS |
0755 |
Project Based |
No |
No |
N/A |
Read/Write |
Area |
Path |
Type |
Permissions |
Quota |
Backups |
Purged |
Retention |
On Compute Nodes |
---|---|---|---|---|---|---|---|---|
User Home |
|
NFS |
User set |
10 GB |
Yes |
No |
90 days |
Read/Write |
Member Work |
|
Lustre |
Project set |
Project Based |
No |
No |
N/A |
Read/Write |
User-Shared Applications |
|
NFS |
0755 |
Project Based |
No |
No |
N/A |
Read/Write |
Area |
Path |
Type |
Permissions |
Quota |
Backups |
Purged |
Retention |
On Compute Nodes |
---|---|---|---|---|---|---|---|---|
User Home |
|
NFS |
User set |
10 GB |
Yes |
No |
90 days |
Read/Write |
Member Work |
|
Lustre |
Project set |
Project Based |
No |
No |
N/A |
Read/Write |
User-Shared Applications |
|
NFS |
0755 |
N/A |
No |
No |
N/A |
Read/Write |
Area |
Path |
Type |
Permissions |
Quota |
Backups |
Purged |
Retention |
On Compute Nodes |
---|---|---|---|---|---|---|---|---|
User Home |
|
NFS |
User set |
5 GB |
Yes |
No |
90 days |
Read/Write |
Member Work |
|
Lustre |
Project set |
60 TB |
No |
60 days |
N/A |
Read/Write |
Member Work |
|
Lustre |
Project set |
20 TB |
No |
14 days |
N/A |
Read/Write |
User-Shared Applications |
|
N/A |
N/A |
N/A |
N/A |
N/A |
N/A |
N/A |
Area |
Path |
Type |
Permissions |
Quota |
Backups |
Purged |
Retention |
AN/PP Nodes |
---|---|---|---|---|---|---|---|---|
User Home |
|
NFS |
User set |
10 GB |
Yes |
No |
90 days |
Read/Write |
User Work |
|
NFS |
User set |
10 GB |
Yes |
No |
90 days |
Read/Write |
Member Work |
|
CXFS |
User set |
Project Based |
No |
Yes |
N/A |
Read/Write |
Member Work |
|
NFS |
User set |
Project Based |
No |
No |
N/A |
Read/Write |
Member Work |
|
NFS |
User set |
Project Based |
No |
Yes |
N/A |
Read/Write |
Member Work |
|
Local |
User set |
Project Based |
No |
No |
N/A |
Read/Write |
Member Work |
|
NFS |
User set |
Project Based |
No |
No |
N/A |
Read/Write |
Member Archive |
|
NFS |
User set |
Project Based |
No |
No |
N/A |
Read/Write |
Important
Files within “Work” directories (i.e., Member Work, Project Work, World Work) are not backed up and are purged on a regular basis according to the time frames listed above.
Notes on User-Centric Data Storage
User Home Directories (NFS)
The environment variable $HOME
will always point to your current home
directory. It is recommended, where possible, that you use this variable to
reference your home directory. In cases in which using $HOME
is not
feasible, it is recommended that you use /home/$USER
(for ursa, hera, jet,
mercury, and pan) and ncrc/home/$USER
for gaea.
Users should note that since this is an NFS-mounted filesystem, its performance will not be as high as other file systems.
User Home Quotas
Quotas are enforced on user home directories. To request an increased quota,
contact the Help Desk. To view your current quota and usage, use the command
quota
on Gaea, Ursa, Hera, Jet, and Niagara; and homeuse
on Pan:
$ quota -Qs
Disk quotas for user userid (uid 12345):
Filesystem space quota limit grace files quota limit grace
ncrc-svm1.ncrc.gov:/ncrc/home2
9228M 51200M 51200M 101k 4295m 4295m
$ quota -Qs
Disk quotas for user userid (uid 12345):
Filesystem space quota limit grace files quota limit grace
10.181.1.1:/home
4147M 0K 5120M 2112 0 0
$ quota -Qs
Disk quotas for user userid (uid 12345):
Filesystem space quota limit grace files quota limit grace
10.181.1.1:/home
4147M 0K 5120M 2112 0 0
$ quota -Qs
Disk quotas for user userid (uid 12345):
Filesystem space quota limit grace files quota limit grace
10.181.1.1:/home
4147M 0K 5120M 2112 0 0
$ quota -Qs
Disk quotas for user userid (uid 12345):
Filesystem space quota limit grace files quota limit grace
10.181.1.2:/home_mercury
544K 0K 5120M 23 0 0
$ homeuse
/home & /nbhome usage - 2024.01.16 10:01
GROUP USERNAME FILESYS FILES GB QUOTA USE%
grp userid - /home 447,121 29.80 40 74%
grp userid - /nbhome 113,115 5.34 10 53%
User Home Permissions
The default permissions for user home directories is shown in the Filesystem Summary Table. Users have the ability to change permissions on their home directories, although it is recommended that permissions be set to as restrictive as possible (without interfering with your work).
User Home Backups
If you accidentally delete files from your home directory, you may be able to
retrieve them. Online backups are performed at regular intervals. Hourly
backups for the past 24 hours, daily backups for the last 7 days, and
once-weekly backups are available. It is possible that the deleted files are
available in one of those backups. The backup directories are named
hourly.*
, daily.*
, and weekly.*
where *
is the date/time stamp
of backup creation. For example, hourly.2020-01-01-0905
is an hourly backup
made on January 1st, 2020 at 9:05 AM.
The backups are accessed via the .snapshot
subdirectory. Note that ls
alone (or even ls -a
) will not show the .snapshot
subdirectory exists,
though ls .snapshot
will show its contents. The .snapshot
feature is
available in any subdirectory of your home directory and will show the online
backups available for that subdirectory.
To retrieve a backup, simply copy it into your desired destination with the
cp
command.
User Archive Directories (PAN Only)
The GFDL Archive provides long-term storage for the large amounts of data created on the NOAA compute systems. The mass storage facility consists of tape and disk storage components, servers, and the Data Migration Facility (DMF) software. After data is uploaded, it persists on disk for some period of time. The length of its life on disk is determined by how full the disk caches become.
User archive areas on HPSS are intended for storage of data not immediately needed in either User Home directories (NFS) or User Work directories (GPFS or Lustre). Where available, User Archive directories should not be used to store project-related data. Rather, Project Archive directories should be used for project data.
User Archive Access
Only GFDL users are given a personal archive space. Users are granted HPSS access if they are members of projects with Project Archive areas. GFDL users can transfer data to HPSS from any RDHPCS system using the Princeton DTN, or the Princeton Globus end point.
User Archive Accounting
The GFDL director allocates tape storage to each GFDL group. A group leader may
also set allocations for individuals in the group. These allocations, and the
percent used, are shown by the local archrpt
command.
For information on usage and best practices for HPSS, please see the GFDL archive page.
Notes on Project-Centric Data Storage
Project directories provide members of a project with a common place to store code, data, and other files related to their project.
Project Home Directories (NFS)
On some RDHPCS systems, projects are provided with a Project Home storage area in the NFS-mounted filesystem. This area is intended for storage of data, code, and other files that are of interest to all members of a project. Since Project Home is an NFS-mounted filesystem, its performance will not be as high as other file systems.
Note
Data files stored in the project home area on Gaea should only be small files (<100MB). Larger files should be stored in the project work area.
Project Home Path, Quota, and Permissions
The path, quota, and permissions for Project Home directories are summarized in the Filesystem Summary Table.
Quotas are enforced on Project Home directories. To check a Project Home
directory’s usage on gaea, run df -h /ncrc/proj/[projid]
(where
[projid]
is the project ID). Note, however, that permission settings on
some subdirectories may prevent you from accessing them, and in that case you
will not be able to obtain the correct usage. If this is the case, contact
help@olcf.ornl.gov for the usage information.
Project Home directories are root-owned and are associated with the project’s Unix group. Default permissions are set such that only members of the project can access the directory, and project members are not able to change permissions of the top-level directory.
Project Home Backups
If you accidentally delete files from your project home directory, you may be
able to retrieve them. Online backups are performed at regular intervals.
Hourly backups for the past 24 hours, daily backups for the last 7 days, and
once-weekly backups are available. It is possible that the deleted files are
available in one of those backups. The backup directories are named
hourly.*
, daily.*
, and weekly.*
where *
is the date/time stamp
of backup creation. For example, hourly.2020-01-01-0905
is an hourly backup
made on January 1st, 2020 at 9:05 AM.
The backups are accessed via the .snapshot
subdirectory. Note that ls
alone (or even ls -a
) will not show the .snapshot
subdirectory exists,
though ls .snapshot
will show its contents. The .snapshot
feature is
available in any subdirectory of your project home directory and will show the
online backups available for that subdirectory.
To retrieve a backup, simply copy it into your desired destination with the
cp
command.
Project Work Areas
Project Work Areas to Facilitate Collaboration (Gaea)
To facilitate collaboration among researchers, RDHPCS systems provide distinct types of project-centric work storage areas. Each directory should be used for storing files generated and used by computationally-intensive HPC jobs related to a project.
The difference between the three storage areas lies in the accessibility of the data to project members and to researchers outside of the project. Member Work directories are accessible only by an individual project member by default. Project Work directories are accessible by all project members. World Work directories are potentially readable by any user on the system.
Permissions
UNIX Permissions on each project-centric work storage area differ according to the area’s intended collaborative use. Under this setup, the process of sharing data with other researchers amounts to simply ensuring that the data resides in the proper work directory.
Member Work Directory:
700
Project Work Directory:
770
World Work Directory:
775
For example, if you have data that must be restricted only to yourself, keep them in your Member Work directory for that project (and leave the default permissions unchanged). If you have data that you intend to share with researchers within your project, keep them in the project’s Project Work directory. If you have data that you intend to share with researchers outside of a project, keep them in the project’s World Work directory.
Backups
Member Work, Project Work, and World Work directories are not backed up. Project members are responsible for backing up these files, either to Project Archive areas (HPSS) or to an off-site location.
Project Archive Directories
Projects may be allocated project-specific archival space on the High Performance Storage System (HPSS) or on the GFDL archive. Each project is given a quota. If a higher quota is needed, contact the appropriate help desk.
Permissions
UNIX Permissions on each project-centric archive storage area differ according to the area’s intended collaborative use. Under this setup, the process of sharing data with other researchers amounts to simply ensuring that the data resides in the proper archive directory.
Member Archive Directory:
700
Project Archive Directory:
770
World Archive Directory:
775
For example, if you have data that must be restricted only to yourself, keep them in your Member Archive directory for that project (and leave the default permissions unchanged). If you have data that you intend to share with researchers within your project, keep them in the project’s Project Archive directory. If you have data that you intend to share with researchers outside of a project, keep them in the project’s World Archive directory.
Project Archive Access
Project Archive directories stored on HPSS may only be accessed via utilities called HSI and HTAR. For more information on using HSI or HTAR, see the NESCC HPSS page.
Project Archive directories stored on GFDL archive can be accessed from Pan, the GFDL workstations, and using Globus.
NESCC HPSS
The centralized, long-term data archive system at National Environmental Security Computing Center (NESCC) is based on IBM’s High Performance Storage System (HPSS). The NESCC HPSS environment includes 22 petabytes of front-end disk cache, five Oracle SL8500 enterprise tape libraries, three Spectra Logic TFinity tape libraries, and 148 tape drives. Total available capacity is 430 PB. HPSS is accessible from WCOSS2, Hera, Mercury, Jet, and Gaea.
Users should keep the following things in mind when using the HPSS system:
The HPSS system is well suited for storing large volumes of data.
Users should avoid transferring many small files (in the megabyte range or smaller) to HPSS since the process of moving numerous individual small files to and from tape is inefficient. Please tar up small files into one large tar file before storing data into HPSS, or use HTAR.
All data stored in HPSS is single copy. Deleted data cannot be recovered.
HPSS is not accessible from compute nodes. Access is available via Hera/Mercury/Jet front-end nodes (FEs), Gaea Data Transfer Nodes (DTNs), and WCOSS2 transfer nodes.
Batch jobs that require access to HPSS must be submitted to the correct systems service or transfer queues. Look for queue or partition names that contain “dtn”.
For questions regarding the HPSS system, email rdhpcs.hpss.help@noaa.gov.
Gaining Access to use HPSS
New HPSS User Requests
Access to an HPSS project must be requested. A HPSS user must be a current
user of a NOAA HPC compute resource (RDHPC or WCOSS) to have access to HPSS.
When you are a member of a project on a compute resource you are not
automatically added to the companion HPSS project (if there is one). Being
added to a HPSS project that you are already a member of on a NOAA compute
resource is done without PI or Portfolio Manager approval, but both are
notified that you are being added. If you are not a member of the project on a
NOAA compute resource then PI or Portfolio manager approval is required before
you will be added. To start the process please send an email to
rdhpcs.hpss.help@noaa.gov with the following subject line USERNAME requests
access to HPSS - PORTFOLIO/PROJECT
, replacing USERNAME
and
PORTFOLIO/PROJECT
with your username and project request. If PI or
Portfolio Manager approval is required, the email should come from them.
All requests must have the following information:
User Name
Requested project(s) - See NESCC HPSS Data Structure for available HPSS projects
System HPSS access is needed from (Hera/Jet/Mercury/Gaea/WCOSS)
Adding New Projects to HPSS
Projects on a NOAA compute resource are not given access to the HPSS until requested by the Portfolio Manager (PfM). The PfM also approves the maximum time retention directory (pool) that a project is allowed to use on HPSS. All lesser time pools will also be available. To add a project to the HPSS the Portfolio Manager should submit a help request with the following information:
Project name
Associated users
Maximum time retention pool
To remove a project from the HPSS the PfM should submit a help request with the project name and data disposition directions. Requests are reviewed and approved by the HPSS Resource Manager and sent to the HPSS system administrator for implementation.
Note
Requests for adding users and projects to HPSS is NOT supported in AIM, but instead the request/review/implementation/notification process is handled by the HPSS help ticket system.
Email: rdhpcs.hpss.help@noaa.gov.
NESCC HPSS Data Structure
HPSS data at NESCC is organized by portfolio, project and retention period,
with a directory structure of PORTFOLIO/PROJECT/RETENTION
. Each retention
period (1-5 year & permanent) is set up as a separate file family, e.g.,
1year
, 2year
, 3year
, 4year
, 5year
, and permanent
. This
means that all data for a retention period is stored on the same tapes.
Projects live under the appropriate portfolio and have been assigned access to
specific retention directories. Project users have access to write data to any
of their projects’ retention directories. Data within the same retention
directory can be moved to other projects within the same retention directory.
If data needs to be moved to another retention period (ex: /1year
->
/2year
) it must be copied.
The structure has the following syntax:
/PORTFOLIO/PROJECT/RETENTION
Examples are:
/BMC/wrf-chem/2year
/NCEPDEV/emc-meso/5year
Note
Please be sure to store the data you write to HPSS in the appropriate retention directory, and in the correct project if you belong to multiple projects. This will avoid movement of data once it is stored on tape.
Portfolios Using HPSS
Portfolios with projects currently storing data in HPSS are NCEPPROD, NCEPDEV, BMC, HFIP, CPO, NAGAPE, NOS and SYSADMIN
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Data Retention
Retention based storage is the HPSS archive policy in Fairmont, to better manage data growth. Six retention storage pools (1-5year and permanent) were created. Each retention period is setup as a separate file family. This means all data for a retention period is stored on the same tapes. All HPSS projects were then configured to write to one or more of these pools. Data in these pools expires based upon the retention pool it was written in and would be deleted upon expiration. All files in the HPSS archive have been assigned an expiration date based on the file create time and the retention period it was written to. Upon expiration files will be deleted from the HPSS archive.
Expired Data Deletion Process
User Notification
Users will be notified of expired data via posted lists and email. These
notifications will take place on or before the first day of the month following
the data’s expiration. For example, data that has an expiration date between
October 1 and October 31 2016 will have its notification posted on or before
November 1, 2016. The expired file list is located on HPSS in
/Expired_Data_Lists/expired.YYYY-MM.txt
. All HPSS users have read access to
this file and can retrieve it for review. The file is easily searchable by HPSS
username. For each file included in the expired list the file owner, file
group, filename/path, and expire date are shown. For example:
root system /1year/SYSADMIN/nesccmgmt/test_file-1G-11 Jul-6-2016.
Email notification will also be sent to all users who have data listed in this file. It is the user’s responsibility to regularly check the posted list for expired files they own. Once deleted these files cannot be recovered.
Expired Data - Deletions
The following table maps out when future deletions will take place.
Expire Date |
Notification Date |
Delete Date |
---|---|---|
Dec 1 - Dec 31 |
January 1 |
February 1 |
Jan 1 - Jan 31 |
February 1 |
March 1 |
Feb 1 - Feb 28 |
March 1 |
April 1 |
Mar 1 - Mar 31 |
April 1 |
May 1 |
Apr 1 - Apr 30 |
May 1 |
June 1 |
May 1 - May 31 |
June 1 |
July 1 |
Jun 1 - June 30 |
July 1 |
August 1 |
Jul 1 - Jul 31 |
August 1 |
September 1 |
Aug 1 - Aug 30 |
September 1 |
October 1 |
Sept 1 - Sept 30 |
October 1 |
November 1 |
Oct 1 - Oct 31 |
November 1 |
December 1 |
Nov 1 - Nov 30 |
December 1 |
January 1 |
File Size Guidelines
Archiving files to HPSS is a much different process than writing files to disk storage. Please be aware that the size of the files you write to HPSS can impact the performance and efficiency of the system.
Preferred file size range
File sizes in the gigabyte range are preferred for storing in HPSS. A few files of hundreds of gigabytes each make the most efficient use of the system.
Considerations for very large files
Files larger than 3 TB WILL FAIL TO STORE IN HPSS. Transferring files that are 1 TB or larger increases the risk of poor system performance as well as the risk (although small) of losing a file that contains a large amount of data. We recommend storing files that are 1 TB or smaller.
Avoid small files
Avoid transferring many small files — those in the megabyte range or smaller. The process of moving numerous individual files to and from tape is inefficient. It can become very time consuming and result in slowing the system for all users.
When you need to store many small files, use one of these two approaches:
Use htar to transfer them together as a single archive file.
Use an archiving utility, e.g.
tar
, on the source system to bundle the member files and then transfer the resulting archive file withhsi put
or orhsi cput
.
Please contact the HPSS helpdesk if you need help determining appropriate file sizes for your specific workload.
Data Recovery Policy
Occasionally an archive tape is damaged or otherwise becomes partially unreadable. When that happens, the local RDHPCS staff works with the manufacturer to troubleshoot the problem and take steps to attempt to recover the missing data. Very rarely, even with these efforts, we are unable to recover the missing files. The user will then be informed of the files we cannot recover.
In that case, the user has one further option. There are a number of outside recovery services which will make further attempts at recovery for a fee. Some charge a flat fee, some charge more if they are able to recover than if they are unable to recover. If the user wishes to sign up for such a service and pay the fee, RDHPCS will handle the logistics of shipping and other coordination with the recovery service.
Getting Started
HPSS is only accessible from WCOSS, Theia, Jet and Gaea Data Transfer Nodes (DTNs). Batch jobs should be used to access HPSS and need be submitted to the respective systems service or transfer queues. Look for queue names that contain “dtn”.
Modules have been created on each system to provide the proper user environment and tools to access HPSS from these systems. These modules are not loaded by default and will need to be loaded before you can use any of the HPSS commands. To add the HPSS tools to your environment, use the following module command:
module load hpss
module use /usw/hpss/modulefiles
module load hsi
HTAR
HTAR allows the creation of archive files directly in HPSS without the need to do an intermediate step of first creating the archive (tar) file on local disk storage before copying the archive file to HPSS. In addition, HTAR creates a separate index file, which contains the names and locations of all of the member files in the archive file. The index file allows individual files and directories in the archive to be randomly retrieved without the need to read through the archive file.
Note
Limitations
HTAR has the following limitations:
File size: An individual file within the tar file may not be larger than 68 GB.
Directory paths: The directory path of any file may not exceed 154 characters in length.
File names: File names may not exceed 99 characters in length.
HTAR Cookbook
Creating an HTAR Archive File Examples
To create a new archive file files.tar
that contains file1
and
file2
in the HPSS at /SYSADMIN/nesccmgmt/1year/testuser/work
:
htar -cvf /SYSADMIN/nesccmgmt/1year/testuser/work/files.tar file1 file2
To create a new archive file time.tar
that contains all files that match
the glob pattern time*
in the HPSS directory
/SYSADMIN/nesccmgmt/1year/testuser/work
:
$ htar -cvf /SYSADMIN/nesccmgmt/1year/testuser/work/time.tar time*
Retrieving an HTAR Archive File Examples
To extract file1
and file2
from the archive files.tar
located in
the HPSS directory /SYSADMIN/nesccmgmt/1year/testuser/work
:
$ htar -xvf /SYSADMIN/nesccmgmt/1year/testuser/work/files.tar ./file1 ./file2
To extract all files from the archive files.tar
located in the HPSS
directory /SYSADMIN/nesccmgmt/1year/testuser/work
:
$ htar -xvf /SYSADMIN/nesccmgmt/1year/testuser/work/files.tar
List Files in an HTAR Archive File Example
To list the names of files in the archive files.tar
located in the HPSS
directory /SYSADMIN/nesccmgmt/1year/testuser/work
:
$ htar -tvf /SYSADMIN/nesccmgmt/1year/testuser/work/files.tar
Recreating a HTAR Index File Example
This operation is used either to reconstruct an index for tar files whose index file is unavailable (e.g., accidentally deleted), or for tar files that were not originally created by HTAR.
$ htar -Xvf /SYSADMIN/nesccmgmt/1year/testuser/work/files.tar
HSI
HSI is an FTP-like interface to the HPSS. HSI is most useful for file and
directory manipulation. HSI supports wild cards for local and HPSS pathname
pattern matching, and provides recursion for many commands, including the
ability to store, retrieve, and list entire directory tress, or change
permissions on entire trees. Some HSI operations, such as cp
and mkdir
resemble their Linux and UNIX counterparts.
For example:
hsi ls
lists the contents of a directoryhsi cp
copies files within the HPSShsi rm
permanently removes a filehsi mkdir
creates a directoryhsi rmdir
deletes a directoryhsi mv
moves files within the HPSS directory structure
HSI Basic Usage
HSI can accept input several different way.
Interactive Command
When using the interactive command form, enter the HSI operations.
$ hsi
[connecting to hpsscore1.fairmont.rdhpcs.noaa.gov/1217]
******************************************************************
* Welcome to the NESCC High Performance Storage System *
* *
* Current HPSS version: 7.4.3 Patch 2 *
* *
* *
* Please Submit Helpdesk Request to *
* rdhpcs.hpss.help@noaa.gov *
* *
* Announcements: *
******************************************************************
Username: User.ID UID: 1234 Acct: 1234(1234) Copies: 1 Firewall: off [hsi.5.0.2.p5 Mon Sep 12 15:22:37 UTC 2016]
[hpsscore1]/PORTFOLIO-> mkdir foo
[hpsscore1]/PORTFOLIO-> cd foo
[hpsscore1]/PORTFOLIO/foo-> put hpss_file
Single line execution
Enclose the HSI operations in quotes, separated with the semicolon (;) character.
hsi "mkdir foo; cd foo; put hpss_file"
Using commands from a File
Use the HSI in
operation to read HSI operations from a file
$ cat command_file
mkdir foo
cd foo
put hpss_file
$ hsi in command_file
Using a Heredoc
Similar to using operations contained in a file, the shell’s heredoc feature can be used to pass to HSI the operations. This method is useful in a batch job script.
In this example, we get a file from HPSS, hpss_file
, and place it in a new
directory foo on the local system.
$ hsi <<EOF
lmkdir foo
lcd foo
get local_file : hpss_file
EOF
Note
The HSI get
and put
operations use a different syntax than FTP to
identify the local file name. The HSI syntax uses a :
(colon character)
to separate the local pathname from the HPSS pathname.
Caution
The mv
, put
, and get
HSI operations can overwrite data at their
targets without warning. This is a problem if you mistakenly remove or
overwrite data, because it cannot be recovered. To help prevent inadvertently
overwriting your HPSS files with these commands, establish directory
permissions carefully.
HSI Cookbook
Moving Files/Directories in HPSS
To move a directory or file to a new location in HPSS:
$ hsi mv /1year/PORTFOLIO/old/location /1year/PORTFOLIO/new/location
Please note that the mv
operation will only work for files/directories
stored in the same retention directory. If you need to move data between
retention directories you must use cp. Please contact the HPSS helpdesk for
steps on doing this efficiently.
Writing Files to HPSS
To put the file local_file
into the HPSS directory
/BMC/testproj/myid/work
$ hsi put /full_local/path/local_file : /BMC/testproj/myid/work/local_file
Retrieve a File from HPSS
In this example, we will To get the HPSS file hpss_file
located in the HPSS
directory /BMC/testproj/myid/work
.
To place hpss_file
in your current directory:
$ hsi get /BMC/testproj/myid/work/hpss_file
To place hpss_file
in the local directory /full_local/path
with the
name new_name
:
$ hsi get /full_local/path/new_name : /BMC/testproj/myid/work/hpss_file
Retrieve a File from HPSS and Preserve the Modification Time
$ hsi get -p /BMC/testproj/myid/work/hpss_file
Listing the Contents of an HPSS Directory
To list the contents of the directory /BMC/testproj
$ hsi ls /BMC/testproj
The ls
operation has other useful options. Using the -N
option will
list fill file information, along with the full path to the file.
[core]/-> ls -N /BMC/testproj
-rw------- 1 User.ID grp 54727283200 Mar 20 2016 /BMC/testproj/hpss_file1.tar
-rw------- 1 User.ID grp 5408 Mar 20 2016 /BMC/testproj/hpss_file1.tar.idx
-rw------- 1 User.ID grp 54727283200 Mar 20 2016 /BMC/testproj/hpss_file2.tar
-rw------- 1 User.ID grp 5408 Mar 20 2016 /BMC/testproj/hpss_file2.tar.idx
The -V
option will list the tape volume information for a file (PV List is
the tape volume):
[core]/-> ls -V /BMC/testproj/hpss_file1.tar
/BMC/testproj:
-rw------- 1 User.ID grp 5 1234 TAPE 54727283200 Mar 20 2016 hpss_file1.tar
Storage VV Stripe
Level Count Width Bytes at Level
----------------------------------------------------------------------------
1 (tape) 1 1 54727283200
VV[ 0]: Object ID: 8c0772a0-8552-11e4-af76-0002559ae41b
ServerDep: 7d72478a-bb87-11d6-9419-0002559ae41b
Pos: 121+0 PV List: N0998300
File Expiration Commands
The HSI operations expls
and expfind
are used to show and find the
expiration date of data stored in HPSS. Each operation has the -h
option
to display the usage information.
Operation expls Help
$ hsi "expls -h"
Usage expls [-?] [-A] [-R] [-v] [path ...]
-? : display this usage
-A : display absolute pathnames
-R : [standard option]recursively list hash entries for files in the specified path(s)
-v : verbose listing mode
Operation expfind Help
$ hsi "expfind -h"
Usage: expfind[ete] [-?] [-A] [-b beginTime] [-d days] [-e endTime] [-R] [path ...]
-? : display this usage
-A : display absolute pathname for files
-b : specify beginning time in range
-d : find file that will be expiring in specified number of days from today
-e : specify ending time in range
-R : [standard option]recursively delete expiration time for the specified path(s)
Note: If -b is not specified, then files whose expiration time is <= endTime are listed
If -e is not specified, then files whose expiration time is>= beginTime are listed
If neither -b nor -e is specified, all expired files in the path(s) are listed
based on the time at which the command is started
Times are of the form YYYY-MM-DD[-hh:mm:ss]
hours/mins/seconds are optional and default to 00:00:00 if not specified
List the Expiration Date of a File
$ hsi "expls /1year/BMC/testproj/file.20160712"
Wed Jul 12 15:57:35 2017 /1year/BMC/testproj/file.20160712
Find Files that Expired On or Before a Certain Date
$ hsi "expfind -e 2016-08-30"
Expiring: /bench1/gyre.tar (Wed Jan 20 22:16:58 2016) Owner: User.Id [1234] Group: grp [1234]
Expiring: /bench1/HSUBSYS1.0.hpssdb.NODE0000.CATN0000.20150605013019.001 (Sat Jun 18 13:32:36 2016) Owner: root [0] Group: system [0]
Expiring: /bench1/HSUBSYS1.0.hpssdb.NODE0000.CATN0000.20150606013020.001 (Sat Jun 18 15:41:39 2016) Owner: root [0] Group: system [0]
Expiring: /bench1/htar_thiea_baseline.tar (Thu Jan 28 20:58:11 2016) Owner: User.Id [1234] Group: grp [1234]
Expiring: /bench1/htar_thiea_baseline.tar.idx (Thu Jan 28 20:58:11 2016) Owner: User.Id [1234] Group: grp [1234]
Sample HPSS Batch Job
The following is a sample script that shows how to transfer data to HPSS via a batch job:
#!/bin/bash
#SBATCH --ntasks=1
#SBATCH --time=0:30:00
#SBATCH --account=<ENTER A VALID PROJECT HERE>
# Use the proper partition name.
# Jet, Hera, Mercury use the 'service' partition
# Gaea is 'dtn_f5_f6', with optional --constraint=f5 or --constraint=f6
# to route to a node which has that file system mounted
#SBATCH --partition=<USE THE CORRECT PARTITION, SEE ABOVE>
#SBATCH --qos windfall
#SBATCH --job-name=hpss-test
# Initialize the module environment, load the appropriate module for a given HPCS
source $MODULESHOME/init/bash
domainname=$(perl -T -e "use Net::Domain(hostdomain); print hostdomain")
if [[ $domainname =~ boulder|fairmont ]]; then
module load hpss
elif [[ $domainname =~ ncrc ]]; then
module use /usw/hpss/modulefiles
module load hsi
fi
set -x
hpssdir=${hpssdir:-/1year/PORTFOLIO/project/User.Id} # XXXX: Location of your file in HPSS
tarfile=${tarfile:-hpss_file.tar} # XXXX: Name of the tar file in HPSS
dirsave=${dirsave:-/path/to/save/directory} # XXXX: Location of data you want to write to HPSS
cd $SLURM_SUBMIT_DIR
# Check if the tarfile index exists. If it does, assume that
# the data for the corresponding directory has already been
# tarred and saved.
hsi "ls -l ${hpssdir}/${tarfile}.idx"
tar_file_exists=$?
if [ $tar_file_exists -eq 0 ]
then
echo "File $tarfile already saved."
exit
fi
# htar is used to create the archive, -P creates
# the directory path if it does not already exist,
# and an index file is also made.
htar -P -cvf ${hpssdir}/$tarfile $dirsave
err=$?
if [ $err -ne 0 ]
then
echo "File $tarfile was not successfully created."
exit 3
fi
Note
The HSMS is not an infinite resource. Quotas will be enabled over time to prevent uncontrolled use. Only save what you need to save. Consider the cost of time and compute resources to regenerate data from the original input files. That is often cheaper than storing the data long term.
HPSS Help
For additional questions, please email: rdhpcs.hpss.help@noaa.gov.
GFDL Archive
The GFDL archive is a long-term archive system based on HPE’s Data Management Framework (DMF). The GFDL archive includes 13.1 petabytes of front-end disk cache, 5 tape libraries, and 160+ tape drives. Total available capacity is 360 PB. The GFDL archive is accessible from Pan and other GFDL systems.
Users should keep the following things in mind when using the GFDL archive system:
The GFDL archive is well suited for storing large volumes of data.
Users should avoid storing small files (in the megabyte range or smaller) to the archive since the process of moving numerous individual small files to and from tape is inefficient. Please tar up small files into one large tar file before storing data into the archive.
All data stored in the archive is single copy. Deleted data cannot be recovered.
The archive file system should not be used for direct file access. Users should copy files to one of the other Pan file systems prior to working with the file.
For questions regarding the HPSS system, email oar.gfdl.help@noaa.gov.
Gaining Access to use the GFDL Archive
All users with a GFDL account will have a personal archive directory.
GFDL Archive Data Structure
The users archive area is located at /archive/$USER
. The
/archive/$USER
is a symbolic link to /arch[0-8bcfgh]/$USER
. These
arch[0-8bcfgh]
directories are used to better distribute the load on the
front-end cache drives.
All files are ultimately stored on tapes housed in the tape libraries, access to those files is using the Linux file system structure on the front-end cache. When a file is accessed, DMF will automatically recall the file from tape and place it on the front-end cache disk.
Data Retention
At this time files stored in the GFDL archive do not have a set retention policy. GFDL divisions and users can decide how long to keep files in the archive. However to ensure adequate space is available for all, uses should remove data that is no longer needed.
Data Recovery Policy
Occasionally an archive tape is damaged or otherwise becomes partially unreadable. When that happens, the local RDHPCS staff works with the manufacturer to troubleshoot the problem and take steps to attempt to recover the missing data. Very rarely, even with these efforts, we are unable to recover the missing files. The user will then be informed of the files we cannot recover.
In that case, the user has one further option. There are a number of outside recovery services which will make further attempts at recovery for a fee. Some charge a flat fee, some charge more if they are able to recover than if they are unable to recover. If the user wishes to sign up for such a service and pay the fee, RDHPCS will handle the logistics of shipping and other coordination with the recovery service.
Getting Started
The GFDL archive is accessible from all Pan nodes, including all DTNs, and via a Globus endpoint. The GFDL archive is also available as read-only from the GFDL workstations.
DMF has a few utilities to help manage files stored on in the tape library and
their residency on the front-end disk cache. The list, with a brief
description is in the table below. See man <command>
for more information.
Commands |
Description |
---|---|
|
Recall files from tape to the front-end cache |
|
Write files to tape |
|
Show pending dmgets with wait times |
|
Show file state (disk/tape residence) |
Warning
Files should be copied from /archive
to the one of the other Pan file
systems, e.g., /vftmp
fast-scratch filesystem, before use.
Allocation and Quota
The GFDL director allocates tape storage to each GFDL group. A group leader may
also set allocations for individuals in the group. These allocations, and the
percent used, are shown by the local archrpt
command:
archrpt -s archrpt -r <group>
These allocations are enforced administratively. For details, see archrpt
-h
.
Finding Files
To facility the ability for users to search for files in the GFDL archive, a
database of file in the archive is updated daily. To search for files in the
archive, supply a pattern to the dmlocate
command:
$ dmlocate <pattern>
Refer to man dmlocate
for more information.
GFDL Archive Help
For additional questions, please email: oar.gfdl.help@noaa.gov.