NESE - Notice history

All systems operational



Status page for The Northeast Storage Exchange (NESE)

Please scroll down to see details on any incidents or maintenance notices.


GETTING HELP
Email: help@nese.mghpcc.org
NESE Documentation: https://nese.mghpcc.org/support/

NESE Disk Storage - Operational

100% - uptime
Jun 2026 · 100.0%Jul · 100.0%Aug · 100.0%

Previous page

Next page

Jun 2026
Jul 2026
Aug 2026

NESE Tape Storage - Operational

99% - uptime
Jun 2026 · 100.0%Jul · 98.01%Aug · 100.0%

Previous page

Next page

Jun 2026
Jul 2026
Aug 2026

NESE Globus Service - Operational

100% - uptime
Jun 2026 · 100.0%Jul · 100.0%Aug · 100.0%

Previous page

Next page

Jun 2026
Jul 2026
Aug 2026

Notice history

Aug 2026

NESE Software Upgrade (Downtime)
  • Planned
    August 17, 2026 at 4:55 PMUTC
    Planned
    August 17, 2026 at 4:55 PMUTC

    The NESE team will be performing a software upgrade on Monday, August 17th starting at around 9:00 am (EST). The upgrade will not require downtime of uploads or downloads of files already recalled from disk. The transfer of files to and from tape will be suspended for one or more short intervals during preparation and execution of the upgrade. All work is planned to be completed by 12:00 Noon (EST), and we expect full service to be available uninterrupted once we are done.

  • Completed
    August 17, 2026 at 4:00 PMUTC
    Completed
    August 17, 2026 at 4:00 PMUTC
    Maintenance has completed successfully
  • In progress
    August 17, 2026 at 1:00 PMUTC
    In progress
    August 17, 2026 at 1:00 PMUTC
    Maintenance is now in progress

Jul 2026

NESE Tape Servicd Down Due to Tape Library Faults
  • Resolved
    UTC
    Resolved

    Vendor technicians have applied temporary repairs, and NESE Tape service is fully available.

    The immediate cause of the tape library stopping all tape movements, which was the reason for suspending tape recalls, was a mechanical failure involving a tape drive and the "robot arm" which moves tapes. Essentially, neither one would let go first.

    The vendor technicians released the drive, tape, and robot arm from this "deadly embrace" and removed the stuck "gripper" from the arm assembly. Technicians will return later to complete the repair, and no downtime is expected to be needed for completion.

    Since the tape library was returned to service, I have observed successful recalls from tape. We will continue working with the vendor to find root causes and apply fixes to make the NESE Tape service more robust.

  • Investigating
    UTC
    Investigating

    Apologies for the false sense of resolution, colleagues. This is my first time running an incident in InStatus and I appear to have clicked unwisely.

  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Investigating
    UTC
    Investigating

    NESE Tape service may have been impaired or unavailable since approximately 6:30 PM EDT on 7/21/2026. The service has been administratively shut down since 9:45 AM on 7/22/2026. The specific symptoms are that the tape library is failing to move and load tape cartridges and the library cannot be reached by the software that manages migating files between tape and disk.

    The vendor, IBM, has been engaged at the highest supported priority, "Severity One." IBM routinely has techicians trained on the TS4500 library on site at MGHPCC in Holyoke, and we expect expert examination and service to begin soon.

    The IBM TS4500 tape library which supports the NESE Tape service began alerting to an internal issue at about 6:30 PM EDT on 7/21/2026 and taking exceptions related to changing tapes in tape drives. It is not yet clear when this first had an effect on the service.

Jun 2026

MGHPCC Outage: NESE Downtime
  • Completed
    June 18, 2026 at 6:30 PMUTC
    Completed
    June 18, 2026 at 6:30 PMUTC
    Maintenance has completed successfully
  • In progress
    June 15, 2026 at 1:00 PMUTC
    In progress
    June 15, 2026 at 1:00 PMUTC
    Maintenance is now in progress
  • Planned
    June 15, 2026 at 1:00 PMUTC
    Planned
    June 15, 2026 at 1:00 PMUTC

    MGHPCC will conduct its annual facility power shutdown from Monday, June 15th, through Wednesday, June 17th. During this time, the NESE storage service, including NESE Tape and NESE Disk Globus endpoints, will be unavailable. We will begin with the power-off procedures on Monday, June 15th, in the afternoon and start restoring access to services on Thursday, June 18th, early in the morning. Please refer to the following timeline for details on the expected downtime of the NESE services:

    Please reference the timeline below for details: 

    NESE services will go offline on Monday, June 15th at: : 

    • 3:00 pm: Globus Tape and Disk endpoints; MinIO

    • 3:00 pm: dCache DTNs; xRootD DTNs

    • 4:00 pm: Disk services, Ceph RBD and dCache Storage Cluster

    Access to NESE services will be resumed on Thursday, June 18th at:  

    • 9:00 am: The Disk services, Ceph RBD and dCache Storage Cluster

    • 2:00 pm: dCache DTNs; xRootD DTNs

    • 2:00 pm: Globus Tape and Disk endpoints; MinIO

    Nese Tape should be fully operational, including data retrieval, by 6:00 pm

    Action Items for NESE Services Clients:

    • Unmount and detach RBD storage by 4:00 pm on Monday, June 15th

    • Stop Ceph RGW nodes by 4:00 pm on Monday, June 15th

    Reminders for Monitoring:

Jun 2026 to Aug 2026

Next

For issues not shown here, please contact the NESE team via email help@nese.mghpcc.org