Data Storage and Transfers

Each RDHPCS system has multiple options for data storage. Each user has multiple user-affiliated storage spaces, and each project has multiple project-affiliated storage spaces where data can be shared for collaboration. Below we give an overview and explain where each storage area is mounted.

• Summary of Storage Areas
  • Notes on User-Centric Data Storage
  • Notes on Project-Centric Data Storage
• NESCC HPSS
  • Gaining Access to use HPSS
  • NESCC HPSS Data Structure
  • Data Retention
  • File Size Guidelines
  • Data Recovery Policy
  • Getting Started
  • HPSS Help
• GFDL Archive
  • Gaining Access to use the GFDL Archive
  • GFDL Archive Data Structure
  • Data Retention
  • Data Recovery Policy
  • Getting Started
  • GFDL Archive Help
• Data Transfer Overview
  • Data Transfer Methods
  • Requests for Firewall Exceptions
  • Firewall Exception Terms
• Transferring Data
  • Globus Connect
  • Trusted Data Transfer Nodes (DTN)
  • Untrusted Data Transfer Nodes (UDTN)
  • Transfer and Syntax Examples
  • Firewall Modification Requests for DTNs
  • Unattended Data Transfers or Password-less Transfers to/from RDHPCs Systems
  • Using a Pre-Established SSH Port Tunnel
• Globus Online Data Transfer
  • Overview
  • RDHPCS Globus Collection Summary
  • NOAA RDHPCS Globus Endpoint Types
  • Globus Command Line Interface (CLI)
  • Transferring Data to and from Your Computer
• Globus Example
  • What you need to have on hand
  • What you need to do
  • RDHPCS Object Stores in the Cloud
  • External S3 Bucket Connectors
  • Publicly accessible buckets, no keys required
  • Non-public, secret keys required
• Migrating Data Between Local File Systems
  • General Guidelines
  • Suggested Tools
  • A sample batch script to transfer data
  • Known Issues