Logo
  • Accounts, Projects, and Access Tokens
    • Applying for a user account
    • Request access to RDHPCS projects
    • Requesting access to MSU-HPC systems
      • General Access Requirements
      • Foreign Nationals
    • Account Activity Requirements: Suspension, Deactivation, Reactivation
      • Deactivated Accounts
    • Request a New Allocation or Project
    • Common Access Card (CAC)
    • CAC/PIV SSH Key Enrollment
    • Yubikey
      • Configuring Yubikey for the NOAA RDHPCS
      • Yubikey Registration
      • Yubico Authenticator Setup Instructions for Windows/Mac
      • Yubico Authenticator Setup Instructions for Linux
      • Testing your Yubikey Registration for NOAA RDHPCS
      • Troubleshooting RDHPCS Yubikey Authentication
    • Role Accounts
      • Requesting a new role account
      • Requesting changes to a role account
      • Accessing a role account from your user account
      • Accessing a role account for Data Transfers or Bastion SSH Login
      • X Applications With role accounts
      • Using CRON or SCRON
  • Connecting
    • Connecting for the first time
    • Bastion Hostnames
    • Secure Shell (SSH) Access
    • Common Access Card (CAC) SSH Login
      • Linux
      • Mac OS
      • Windows
        • Install PuTTY-CAC
        • Configure your Session in PuTTY-CAC
    • Yubikey SSH Login
    • Selecting a Node
    • Tectia
      • Tectia Initial Setup procedure
      • Install the Tectia Client
      • Configure the Tectia Client
    • Port tunnels
    • X11 forwarding
    • Reference
    • X11 Graphics
    • SSH Port Tunnels
    • Web based ParallelWorks Access
  • Getting Help
    • Submitting a Good Help Request
      • Use a Good Subject
      • Provide Detailed Description of the Problem
      • Provide Job Information
      • Describe How to Reproduce the Problem
      • Only Report One Problem Per Help Ticket
      • Follow up With Additional Information or Questions
    • Required Information for Specific Types of Help
      • Basic Ticket Information
      • File System Problems
      • Compilation Problems
      • Job Submission Problems
      • Job Completion Problems
      • Providing a Reproducer
      • Reporting Data Transfer Issues
    • Recording a Terminal Session
      • Using script to Record a Terminal Session
      • Submitting a Recording to the Help Desk
    • GFDL ServiceDesk
      • Submitting a Ticket through ServiceDesk
    • Managing OTRS Help Tickets
      • Reply to a Ticket
      • Search for a Ticket
      • Create a New Ticket
  • Systems
    • Cloud User Guide
      • Cloud Processing
        • Getting Help
      • Parallel Works
        • Using ACTIVATE
        • Using Parallel Works to access on-prem HPC Systems
        • Access the head node from within Parallel Works [PW]
        • Running a Jupyter workflow on a Slurm compute node
        • Accessing nodes in a cluster
        • Running VS Code from a Compute Node
      • Cloud Projects
        • Create/request a new project
      • Best Practices
      • Storage Types and Storage Costs
        • Lustre file system
        • Bucket/Block blob storage
        • Contrib file system
      • Costing
        • Cost Calculator
      • Errors
        • Warning messages from the on-prem system about exceeding quota
      • Usage Reports
      • Cloud Presentations
      • Knowledge Base from Asked Questions
        • General Issues
        • Storage functionalities
        • Clusters and snapshots
        • Data Transfer
        • Configuration Questions
        • AWS Lustre explained
        • Azure Lustre explained
        • Best practice in resource configuration
        • Slurm
        • Errors
        • Miscellaneous
    • Gaea User Guide
      • System Overview
        • Node types
        • Compute nodes
        • Login nodes
        • Data transfer nodes
        • System interconnect
        • File systems
        • Operating system
      • Connecting
      • Data and storage
        • NFS file systems
        • GPFS file systems
        • Move data to and from Gaea
      • Programming environment
        • Environment Modules
        • Cray Modules (CPE)
        • Compilers
        • MPI
      • Compiling
        • Compilers
        • Dynamic linking
      • Running jobs
        • Login vs Compute Nodes
        • Slurm
      • Software
      • Debugging
        • Linaro DDT
        • GDB
        • GDB4HPC
        • Valgrind4hpc
      • Profiling
        • HPE Performance Analysis Tools
      • Tips and tricks
        • GPFS (F5) Performance
      • Known issues
        • Open issues
    • Hera User Guide
      • System Overview
        • System Configuration
      • Hera Partitions
      • File System Usage
      • Applications and Libraries
        • Using Miniforge3 and Python on Hera
        • MATLAB
        • Using IDL on Hera
        • Using ImageMagick on Hera
        • Using R on Hera
        • Libraries
      • Using Modules
      • Using MPI
        • Loading the MPI module
        • Using PGI and mvapich2
        • Tuning MPI (TBD)
        • Profiling an MPI application with Intel MPI
      • Debugging Codes
        • Debugging Intel MPI Applications
        • Application Debuggers
        • Invoking DDT on Hera with Intel IMPI
      • Profiling Codes
        • Linaro Forge
        • TAU
      • Managing Contrib Projects
    • Jet User Guide
      • Jet Data Disposition
    • Mercury User Guide
      • System Overview
      • Interactive Logins
      • Data Transfer
      • Per User Data Management on Mercury
      • Software Stack
      • CRON Services
      • HPSS Access
      • Getting Help
    • MSU-HPC User Guide
      • Introduction
        • MSU’s Official HPC Documentation
      • General Information
        • Logging In
      • Running Jobs on MSU-HPC Systems
        • Submitting a Job
        • Specifying a Partition
        • Monitoring Jobs
        • Getting Information about your Projects
      • MSU-HPC System Configuration
        • File Systems
        • Orion Compute System
        • Hercules Compute System
      • Account Management
        • Overview
        • MSU Account Management Policies
        • Managing Project and Role Account Members
        • NOAA Portfolio, Project, and User Management on MSU-HPC
        • Getting An Account
        • Account Renewal
        • Managing Portfolios, Projects and Allocation
        • Role Accounts
      • Help, Policies, Best Practices, Issues
        • MSU-HPC Help Requests
        • Policies and Best Practices
        • Protecting Restricted Data
      • MSU FAQ
    • PPAN User Guide
      • Login to Analysis
        • ssh setup for GFDL Workstations
        • C-shell Setup
        • C-shell keybinds for analysis
        • xterm and gnome-terminal
        • Logging into the PP nodes
      • File Systems
        • /home and /nbhome
        • /archive
        • /vftmp & $TMPDIR
        • /work
        • /ptmp
        • /net, /net2, /net3
      • Batch Software
        • Access
        • Local Commands
        • Gotchas
      • Podman
      • Analysis Software
        • Using Modules
        • netcdf Library
        • MATLAB Licenses
      • About Archrpt
        • Report Option [-r]
        • Show Archive Report By Specified Group [view]
        • Show Archive Report By Specified User [view]
        • Show Archive Report By Specified Group and Sort By Files [view] [sort]
        • Show Archive Report By Specified Group and Sort By Bytes [view] [sort]
        • Show Archive Report By Specified Date [date]
        • Summary Option [-s]
        • Show Archive Summary
        • Show Archive Summary and Sort By Files [sort]
        • Show Archive Summary and Sort By Bytes [sort]
        • Show Archive Summary By Date [date]
      • Group Quotas
      • User Quotas
        • Info
        • Configuration
      • Enforcing Quotas
      • Analysis Hosts
      • Post-Processing Hosts
        • Smartsites
        • Log Session ID
        • Supported Filesystems
        • Helpful Hints
        • Caveats
        • Using the –batch option on Gaea
    • Ursa User Guide
      • System Overview
      • System Configuration
        • Partitions
        • Node Sharing
        • Front Ends and Service Partition
      • Using GPU Resources on Ursa
        • Using H100 GPU Resources With a GPU allocation
        • Using H100 GPU Resources Without a GPU allocation
      • Using the Exploratory GPU Resources
      • Software Stack
      • HPFS File Systems
        • Usage/Quota information for /scratch[3,4] file systems
      • Cron and Scrontab Services
      • Getting Started with Ursa
      • Getting Help
  • Data Storage and Transfers
    • Data Storage
      • Summary of Storage Areas
        • Notes on User-Centric Data Storage
        • User Home Quotas
        • User Home Permissions
        • User Home Backups
        • User Archive Access
        • User Archive Accounting
        • Project Home Path, Quota, and Permissions
        • Project Home Backups
        • Project Work Areas to Facilitate Collaboration (Gaea)
        • Permissions
        • Backups
        • Permissions
        • Project Archive Access
      • NESCC HPSS
        • Gaining Access to use HPSS
        • New HPSS User Requests
        • Adding New Projects to HPSS
        • Portfolios Using HPSS
        • Expired Data Deletion Process
        • User Notification
        • Expired Data - Deletions
        • HTAR
        • HTAR Cookbook
        • HSI
        • HSI Basic Usage
        • HSI Cookbook
        • File Expiration Commands
        • Sample HPSS Batch Job
      • GFDL Archive
        • Gaining Access to use the GFDL Archive
        • Allocation and Quota
        • Finding Files
    • Data Transfers
      • Overview
        • Data Transfer Methods
        • Requests for Firewall Exceptions
        • Firewall Exception Terms
      • Transferring Data
        • Globus Connect
        • Trusted Data Transfer Nodes (DTN)
        • Untrusted Data Transfer Nodes (UDTN)
        • Transfer and Syntax Examples
        • Firewall Modification Requests for DTNs
        • Unattended Data Transfers or Password-less Transfers to/from RDHPCs Systems
        • Using a Pre-Established SSH Port Tunnel
      • Globus Online Data Transfer
        • Overview
        • RDHPCS Globus Collection Summary
        • NOAA RDHPCS Globus Endpoint Types
        • Globus Command Line Interface (CLI)
        • Transferring Data to and from Your Computer
        • Data Sharing with External Collaborators
        • Globus Example
        • Using Globus Online Data Transfer
        • NOAA RDHPCS Globus Endpoint Types
        • NOAA RDHPCS UDTN’s (Globus Untrusted Endpoint)
        • NOAA RDHPCS Object Stores in the Cloud
        • Accessing Cloud Endpoints in our environment
        • Globus Command Line Interface (CLI)
        • Transferring Data to and from Your Computer
        • GFDL Institutional Data Portal
        • Migrating Data Between Local File Systems
        • Known Issues
  • Policies and Best Practices
    • System Usage
      • Login Node Usage
    • Cron and scrontab usage
      • Cron/Scrontab Dos
      • Cron/Scrontab Don’ts
    • Allocations
      • Request an Increase in Allocations
      • Adding a Project to an Allocation
      • Cloud Computing Allocations
    • Quotas
      • Requesting Additional Storage for a Project
    • File System Usage Practices and Policies
      • High Performance File System (HPFS - Scratch)
      • General Parallel File System (GPFS)
      • /data_untrusted
      • HFS
      • Long Term Data Archive Storage
    • Filesystem Backup and Data Retention
      • Recover Recently Deleted Files from /home
      • HPSS (Data Retention)
        • Expired Data Deletion Process
        • Data Recovery Policy
    • Data Disposition
      • HPFS (Scratch) Data
      • Home File System (HFS) Data
      • Protecting Restricted Data
    • Managing Packages in /contrib
      • Overview of contrib Packages
      • Responsibilities of a contrib Package Maintainer
      • contrib Packages Guidelines
      • contrib Package Maintainer Requests
      • Managing a contrib Package
      • Maintaining “Metadata” for contrib Packages
      • contrib Package Directory Naming Conventions
    • Queue Policy
      • Overview
      • Specifying a Quality of Service (QOS)
      • Changing QOS’s
      • Hera and Ursa QOS
      • Gaea
      • General Recommendations
      • Priorities Between QOS
      • Debug & Batch QOS
  • Software
    • Modules
      • View Active Modules
      • Finding Modules
      • Load Modules
      • Module Collections
      • Adding Additional Module Paths
      • Using Modules in Batch Jobs
      • Command Summary
    • Python
      • Overview
      • Python Guides
        • Conda Basics
        • Installing Miniforge
        • Jupyter on RDHPCS Systems
      • Module Usage
        • Python
        • Conda
      • Python and Conda Environments
        • Conda Environments
      • Running Python
        • RDHPCS compute nodes
      • Python best practices
        • Cloning the base environment using conda
        • Environment locations (storage)
        • Adding known conda environment locations
        • Keep your Python and Conda caches trimmed
        • Deactivate your environments before running batch jobs
        • Unbuffered input
      • Additional Resources
    • Workflow Software
      • Cron and Slurm crontab
        • Cron
        • Viewing currently running crontab processes
        • Slurm Crontab
      • Flexible Modeling System Runtime Environment
      • Rocoto
        • Rocoto on RDHPCS Systems
        • Rocoto Documentation
        • Rocoto Help
        • Rocoto Best Practices
    • Debugging with Forge DDT
      • DDT remote connection
      • Debugging an MPI process
        • First time configuration
        • Submit a debug job
    • Compilers
      • Using the Intel compiler with Intel MPI
        • Recommended Intel Compiler Options for Optimization
      • Compiling for reproducible results
        • Recommended Intel compiler options for debugging
        • Thread-safe compilation
        • Other potentially useful Intel compiler options
      • Using the NVIDIA compilers
        • Documentation on NVIDIA compiler options
        • NVIDIA compiler options for optimization
        • NVIDIA compiler options for debugging
    • Gitlab Server
      • Web Access
        • Tunnel Setup
        • Configure User Authentication for Command Line
        • Gitlab Project
      • Git Usage
        • GitLab CLI (glab)
      • GitLab CI/CD Pipelines
        • Example .gitlab-ci.yml Pipeline
      • GitLab Runners
        • Installing GitLab Runner on RDHPCS systems
        • Register a Gitlab Runner with Shell Executor
        • Gitlab Runner Execution
        • Runner Tags in a Gitlab Pipeline
        • Shell Executor Best Practices
        • Custom Slurm Executor
      • Container Registry
        • Configuration on RDHPCS system
        • Registry Login
        • Image pull and push
    • Containers
      • Introduction
        • Background
        • Supported RDHPCS Container Solutions
      • Singularity Containers
        • Container User Identity
        • Differences Between SingularityCE and Apptainer
        • Usage Considerations and Security Responsibilities
      • Obtaining and Building Container Images
        • Pull and Convert Docker Images to Singularity Format
        • Build from a Docker Container
        • Build a Writable Sandbox Container
        • Build a Writable Sandbox for Development (Fakeroot)
        • Build an Image from the Existing Sandbox
        • Build a Container from a Definition File
        • Build/Pull a Container for Different Architecture
      • Runtime Model
        • Running Containers
        • Interactive Shell in Containers
        • Executing Applications in Containers
        • Environment Variable Propagation
        • Bind Mounting Host Directories Into a Container
      • MPI Integration Models
        • Hybrid MPI Model (Host + Container MPI)
        • Host MPI Only (Bind) Model
        • Fully Containerized MPI Model
      • Container Execution Workflows
        • Compiling an Application With a Container
        • Running a Serial Application
        • Running an MPI Application
      • Container help, questions, and guidance
    • VSCode
      • Setting up VSCode for use on RDHPCS systems
        • Set up the install directory
        • Set up Port Forwarding on your Local client
        • Create a session with the tunnel and keep this session open
        • Test to make sure the tunnel is working
        • Login with your VSCode client
  • Slurm
    • Overview
      • Running a Job
        • Batch Scripts
        • Interactive Jobs
        • Common sbatch Options
        • Specifying Partitions and QOS
      • Determine and specify a memory limit for your jobs
        • Specify a processor layout for your job (uniform layout example)
        • Using report-mem utility in batch jobs
        • Using report-mem utility on a job that is currently running
        • Determining the amount of memory used by a process
        • Slurm Environment Variables
        • State Codes
        • Job Reason Codes
        • Job Dependencies
        • Srun
      • Monitoring Jobs
        • Show Pending and Running Jobs
        • Show Completed Jobs
        • Getting Details About a Job
      • Priority and Fairshare
        • Understanding Slurm Fairshare
        • Fairshare Priority Factor
        • Fairshare Definitions
        • Fairshare Reporting
        • Priority Reporting
      • Getting Information About Your Projects
        • sfairshare
        • saccount_params
      • Generating Reports
        • Sreport
        • Shpcrpt
      • References
    • Advanced Slurm Usage
      • Running Parallel srun Job Steps in the Background
        • Overview
        • Prerequisites
        • Core Concepts
        • Recommended Flag Combination
        • Launching Steps in Parallel
        • Discovering the Allocated CPU Count
        • Submitting via sbatch
        • Confirming Parallel Execution
        • Interpreting --cpu-bind=verbose Output
        • Detecting Genuine Core Collisions
        • Deliberate Core Sharing
        • Why a Plain Command Instead of MPI
        • Summary
    • Gaea Batch Job Overview
      • Compiling
      • Running
      • Staging/Combining
      • Transferring Data to/from Gaea
      • Allocation
      • Running a Simple Job
      • Running the Script
        • Once the job is submitted
        • Once the job is Finished
  • Frequently Asked Questions
    • Accounts
      • How Do I Get an RDHPCS Account?
      • PW login is getting a “Invalid username or password” error.
      • My RSA Token is locked
      • I forgot my passphrase, how do I reset it?
      • How do I use X11 application with shared user account (role account)?
    • Jobs
      • My job hasn’t started and I have been waiting a long time. What is wrong?
      • My job hasn’t started and it is in a reservation, what is wrong?
      • What is the meaning of the exit code?
      • All my multi-node MPI jobs are timing out, even simple jobs! What is wrong?
      • My multi-node jobs fail on mpirun/mpiexec.
    • User Issues
      • How do I change my default login shell?
      • How can I recover recently deleted files from /home?
      • Why am I not able to ssh between nodes, it is asking me for a password!
      • How can I recover files that I accidentally deleted from my project space?
      • Can you please install the xyz python package(s)?
      • Why are my jobs failing intermittently?
      • Why am I getting these errors? I am using hpc-stack for NCEPLIBS
      • I am using spack-stack and getting some errors
      • When is my .bashrc executed? When would it be ignored?
      • I got the message “REMOTE HOST IDENTIFICATION HAS CHANGED!”. What should I do?
      • Where can I find “Operational Data” from WCOSS2 on Hera?
      • My jobs using NCL are no longer working
      • Compile WRF on Hera with Rocky OS
      • How do I enable x11 forwarding using PowerShell on a Windows system?
    • Port Tunnels
      • How do I set up an ssh port tunnel?
      • SSH Port Tunnel For PuTTy Windows Systems
      • How to transfer small files to/from an RDHPCS system?
      • I can no longer transfer files via the port tunnel, please help!
    • Known Issues
      • Intel MPI Collective Algorithms issue
    • Recent User-Facing Changes
      • Jan 22, 2025: DTNs for Ursa are now available
    • RDHPCS Office Hours
  • Glossary
  • Contributing to these docs
    • Overview
      • Submitting suggestions
      • Contributing changes via the browser
    • Set up your authoring environment
    • Build and validate
    • Edit and submit
    • GitHub guidelines
    • Resources
NOAA RDHPCS User Documentation
  • Search

NOAA Home
Science. Service. Stewardship.
  • NOAA High Performance Computing Systems
  • Protecting Your Privacy
  • FOIA
  • Information Quality
  • Accessibility
  • Guidance
  • Budget & Performance
  • Disclaimer
  • EEO
  • No-Fear Act
  • USA.gov
  • Ready.gov
  • HPCS Webmaster
RDHPCS Events
RDHPCS Events
Stay connected to NOAA
NOAA on Twitter NOAA on Facebook NOAA on Instagram NOAA on YouTube