NESE Status - Incident history https://nese.instatus.com NESE Mon, 17 Aug 2026 13:00:00 +0000 NESE Software Upgrade (Downtime) Type: Maintenance Duration: 3 hours Affected Components: NESE Globus Service, NESE Disk Storage, NESE Tape Storage Aug 17, 16:55:00 GMT+0 - Identified - The NESE team will be performing a software upgrade on **Monday, August 17th starting at around 9:00 am (EST)**. The upgrade will not require downtime of uploads or downloads of files already recalled from disk. The transfer of files to and from tape will be suspended for one or more short intervals during preparation and execution of the upgrade. All work is planned to be completed by **12:00 Noon (EST)**, and we expect full service to be available uninterrupted once we are done. Aug 17, 13:00:01 GMT+0 - Identified - Maintenance is now in progress Aug 17, 16:00:00 GMT+0 - Completed - Maintenance has completed successfully Type: Maintenance

Duration: 3 hours

Affected Components: , ,

<p><small>Aug <var data-var='date'> 17</var>, <var data-var='time'>16:55:00</var> GMT+0</small><br><strong>Identified</strong> - The NESE team will be performing a software upgrade on **Monday, August 17th starting at around 9:00 am (EST)**. The upgrade will not require downtime of uploads or downloads of files already recalled from disk. The transfer of files to and from tape will be suspended for one or more short intervals during preparation and execution of the upgrade. All work is planned to be completed by **12:00 Noon (EST)**, and we expect full service to be available uninterrupted once we are done..</p> <p><small>Aug <var data-var='date'> 17</var>, <var data-var='time'>13:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Aug <var data-var='date'> 17</var>, <var data-var='time'>16:00:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> ]]> Mon, 17 Aug 2026 13:00:00 +0000 https://nese.instatus.com/maintenance/cmsowj9320bz40rlonosllmkn https://nese.instatus.com/maintenance/cmsowj9320bz40rlonosllmkn NESE Tape Servicd Down Due to Tape Library Faults Type: Incident Duration: 1 day and 18 minutes Affected Components: NESE Tape Storage Jul 21, 22:30:00 GMT+0 - Investigating - NESE Tape service may have been impaired or unavailable since approximately 6:30 PM EDT on 7/21/2026\. The service has been administratively shut down since 9:45 AM on 7/22/2026\. The specific symptoms are that the tape library is failing to move and load tape cartridges and the library cannot be reached by the software that manages migating files between tape and disk. The vendor, IBM, has been engaged at the highest supported priority, "Severity One." IBM routinely has techicians trained on the TS4500 library on site at MGHPCC in Holyoke, and we expect expert examination and service to begin soon. The IBM TS4500 tape library which supports the NESE Tape service began alerting to an internal issue at about 6:30 PM EDT on 7/21/2026 and taking exceptions related to changing tapes in tape drives. It is not yet clear when this first had an effect on the service. Jul 22, 13:18:36 GMT+0 - Resolved - This incident has been resolved. Jul 22, 14:12:09 GMT+0 - Investigating - Apologies for the false sense of resolution, colleagues. This is my first time running an incident in InStatus and I appear to have clicked unwisely. Jul 22, 22:48:11 GMT+0 - Resolved - Vendor technicians have applied temporary repairs, and NESE Tape service is fully available. The immediate cause of the tape library stopping all tape movements, which was the reason for suspending tape recalls, was a mechanical failure involving a tape drive and the "robot arm" which moves tapes. Essentially, neither one would let go first. The vendor technicians released the drive, tape, and robot arm from this "deadly embrace" and removed the stuck "gripper" from the arm assembly. Technicians will return later to complete the repair, and no downtime is expected to be needed for completion. Since the tape library was returned to service, I have observed successful recalls from tape. We will continue working with the vendor to find root causes and apply fixes to make the NESE Tape service more robust. Type: Incident

Duration: 1 day and 18 minutes

Affected Components:

<p><small>Jul <var data-var='date'> 21</var>, <var data-var='time'>22:30:00</var> GMT+0</small><br><strong>Investigating</strong> - NESE Tape service may have been impaired or unavailable since approximately 6:30 PM EDT on 7/21/2026\. The service has been administratively shut down since 9:45 AM on 7/22/2026\. The specific symptoms are that the tape library is failing to move and load tape cartridges and the library cannot be reached by the software that manages migating files between tape and disk. The vendor, IBM, has been engaged at the highest supported priority, "Severity One." IBM routinely has techicians trained on the TS4500 library on site at MGHPCC in Holyoke, and we expect expert examination and service to begin soon. The IBM TS4500 tape library which supports the NESE Tape service began alerting to an internal issue at about 6:30 PM EDT on 7/21/2026 and taking exceptions related to changing tapes in tape drives. It is not yet clear when this first had an effect on the service..</p> <p><small>Jul <var data-var='date'> 22</var>, <var data-var='time'>13:18:36</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Jul <var data-var='date'> 22</var>, <var data-var='time'>14:12:09</var> GMT+0</small><br><strong>Investigating</strong> - Apologies for the false sense of resolution, colleagues. This is my first time running an incident in InStatus and I appear to have clicked unwisely..</p> <p><small>Jul <var data-var='date'> 22</var>, <var data-var='time'>22:48:11</var> GMT+0</small><br><strong>Resolved</strong> - Vendor technicians have applied temporary repairs, and NESE Tape service is fully available. The immediate cause of the tape library stopping all tape movements, which was the reason for suspending tape recalls, was a mechanical failure involving a tape drive and the "robot arm" which moves tapes. Essentially, neither one would let go first. The vendor technicians released the drive, tape, and robot arm from this "deadly embrace" and removed the stuck "gripper" from the arm assembly. Technicians will return later to complete the repair, and no downtime is expected to be needed for completion. Since the tape library was returned to service, I have observed successful recalls from tape. We will continue working with the vendor to find root causes and apply fixes to make the NESE Tape service more robust..</p> ]]> Tue, 21 Jul 2026 22:30:00 +0000 https://nese.instatus.com/incident/cmrw5eh2t00oi1anrxv0gat79 https://nese.instatus.com/incident/cmrw5eh2t00oi1anrxv0gat79 MGHPCC Outage: NESE Downtime Type: Maintenance Duration: 3 days, 5 hours and 30 minutes Affected Components: NESE Globus Service, NESE Disk Storage, NESE Tape Storage Jun 15, 13:00:01 GMT+0 - Identified - Maintenance is now in progress Jun 18, 18:30:00 GMT+0 - Completed - Maintenance has completed successfully Jun 15, 13:00:00 GMT+0 - Identified - MGHPCC will conduct its annual facility power shutdown from Monday, June 15th, through Wednesday, June 17th. During this time, the NESE storage service, including NESE Tape and NESE Disk Globus endpoints, will be unavailable. We will begin with the power-off procedures on Monday, June 15th, in the afternoon and start restoring access to services on Thursday, June 18th, early in the morning. Please refer to the following timeline for details on the expected downtime of the NESE services: Please reference the timeline below for details: NESE services will go offline on Monday, June 15th at: : * 3:00 pm: Globus Tape and Disk endpoints; MinIO * 3:00 pm: dCache DTNs; xRootD DTNs * 4:00 pm: Disk services, Ceph RBD and dCache Storage Cluster Access to NESE services will be resumed on Thursday, June 18th at: * 9:00 am: The Disk services, Ceph RBD and dCache Storage Cluster * 2:00 pm: dCache DTNs; xRootD DTNs * 2:00 pm: Globus Tape and Disk endpoints; MinIO Nese Tape should be fully operational, including data retrieval, by 6:00 pm Action Items for NESE Services Clients: * Unmount and detach RBD storage by 4:00 pm on Monday, June 15th * Stop Ceph RGW nodes by 4:00 pm on Monday, June 15th Reminders for Monitoring: * Monitor the MGHPCC Slack #nese channel for status updates and announcements * Visit[ https://nese.instatus.com/](https://nese.instatus.com/) for real-time updates on progress * Subscribe at[ https://nese.instatus.com/subscribe/email](https://nese.instatus.com/subscribe/email) for updates and announcements * Tape and Disk Globus transfers should resume after the NESE Globus endpoint comes back online Type: Maintenance

Duration: 3 days, 5 hours and 30 minutes

Affected Components: , ,

<p><small>Jun <var data-var='date'> 15</var>, <var data-var='time'>13:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Jun <var data-var='date'> 18</var>, <var data-var='time'>18:30:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> <p><small>Jun <var data-var='date'> 15</var>, <var data-var='time'>13:00:00</var> GMT+0</small><br><strong>Identified</strong> - MGHPCC will conduct its annual facility power shutdown from Monday, June 15th, through Wednesday, June 17th. During this time, the NESE storage service, including NESE Tape and NESE Disk Globus endpoints, will be unavailable. We will begin with the power-off procedures on Monday, June 15th, in the afternoon and start restoring access to services on Thursday, June 18th, early in the morning. Please refer to the following timeline for details on the expected downtime of the NESE services: Please reference the timeline below for details: NESE services will go offline on Monday, June 15th at: : * 3:00 pm: Globus Tape and Disk endpoints; MinIO * 3:00 pm: dCache DTNs; xRootD DTNs * 4:00 pm: Disk services, Ceph RBD and dCache Storage Cluster Access to NESE services will be resumed on Thursday, June 18th at: * 9:00 am: The Disk services, Ceph RBD and dCache Storage Cluster * 2:00 pm: dCache DTNs; xRootD DTNs * 2:00 pm: Globus Tape and Disk endpoints; MinIO Nese Tape should be fully operational, including data retrieval, by 6:00 pm Action Items for NESE Services Clients: * Unmount and detach RBD storage by 4:00 pm on Monday, June 15th * Stop Ceph RGW nodes by 4:00 pm on Monday, June 15th Reminders for Monitoring: * Monitor the MGHPCC Slack #nese channel for status updates and announcements * Visit[ https://nese.instatus.com/](https://nese.instatus.com/) for real-time updates on progress * Subscribe at[ https://nese.instatus.com/subscribe/email](https://nese.instatus.com/subscribe/email) for updates and announcements * Tape and Disk Globus transfers should resume after the NESE Globus endpoint comes back online.</p> ]]> Mon, 15 Jun 2026 13:00:00 +0000 https://nese.instatus.com/maintenance/cmpi8gae60846pbheozzr24em https://nese.instatus.com/maintenance/cmpi8gae60846pbheozzr24em Emergency Maintenance on NESE Tape Library Type: Maintenance Duration: 4 days, 21 hours and 23 minutes Affected Components: NESE Tape Storage Mar 2, 11:32:58 GMT+0 - Identified - We are still working with IBM technical support at restoring the inventory. The expected downtime is extended until Tuesday March 3rd, 9am. Apologies for the inconvenvenience. Mar 3, 13:59:37 GMT+0 - Identified - We are still working with IBM to resolve the issue with the library inventory. We will further extend the emergency downtime for the tape library to Wednesday, March 4th, 9 am. IBM is sending a hardware replacement part for the library tomorrow, and their support engineer will install that part tomorrow and re-launch the scan of the tapes in the library. According to support, we expect that replacing that component should resolve the issue. If you have questions or concerns, of if you would like more details on the issue, please feel free to connect to NESE office hours session tomorrow starting at 1 pm, where we can provide any additional information we have, and the latest updates. Thank you for your patience. Office hours on Tuesday: 1:00pm -- 2:45pm <https://harvard.zoom.us/j/97215076930?pwd=Z3hua1FpWlVjWDVsenFVMWhpbzc0dz09> Feb 27, 17:00:00 GMT+0 - Identified - NESE Tape service is currently not able to store or recall files to and from tape due to vendor firmware issues in the tape library. The issue is related to the library robotics and cartridge database, and we do NOT expect any data loss from this issue. The issue is apparently due to an issue with the inventory database related to a recent firmware update. This database can be scrubbed and reconstructed by the library, which will scan the bar code labels on all the cartridges to rebuild the inventory. Association of files in Globus to tapes is handled separately from the tape library and is not affected by the firmware update. Similarly, USATLAS and CMS file to tape association was not affected by the firmware update. We are taking additional measures to preserve this information as we complete the work with the tape library inventory. Mar 4, 14:22:59 GMT+0 - Completed - The Maintenance has completed successfully. One hardware components in the library has been replaced. The tape cartridges inventory has been reconciled. The Storage Archive service and Tape Library are up and running. Storage Archive shows NO missing or duplicated cartridges. We will keep closely monitoring the service to confirm everything proceeds as normal. Remaining items: One of the two accessors in the library robotic components was still showing small issues with calibration, so we have decided to leave it disabled for the time being, waiting for further troubleshooting from IBM. That will have only minimal impact on performance of the service. Type: Maintenance

Duration: 4 days, 21 hours and 23 minutes

Affected Components:

<p><small>Mar <var data-var='date'> 2</var>, <var data-var='time'>11:32:58</var> GMT+0</small><br><strong>Identified</strong> - We are still working with IBM technical support at restoring the inventory. The expected downtime is extended until Tuesday March 3rd, 9am. Apologies for the inconvenvenience..</p> <p><small>Mar <var data-var='date'> 3</var>, <var data-var='time'>13:59:37</var> GMT+0</small><br><strong>Identified</strong> - We are still working with IBM to resolve the issue with the library inventory. We will further extend the emergency downtime for the tape library to Wednesday, March 4th, 9 am. IBM is sending a hardware replacement part for the library tomorrow, and their support engineer will install that part tomorrow and re-launch the scan of the tapes in the library. According to support, we expect that replacing that component should resolve the issue. If you have questions or concerns, of if you would like more details on the issue, please feel free to connect to NESE office hours session tomorrow starting at 1 pm, where we can provide any additional information we have, and the latest updates. Thank you for your patience. Office hours on Tuesday: 1:00pm -- 2:45pm <https://harvard.zoom.us/j/97215076930?pwd=Z3hua1FpWlVjWDVsenFVMWhpbzc0dz09>.</p> <p><small>Feb <var data-var='date'> 27</var>, <var data-var='time'>17:00:00</var> GMT+0</small><br><strong>Identified</strong> - NESE Tape service is currently not able to store or recall files to and from tape due to vendor firmware issues in the tape library. The issue is related to the library robotics and cartridge database, and we do NOT expect any data loss from this issue. The issue is apparently due to an issue with the inventory database related to a recent firmware update. This database can be scrubbed and reconstructed by the library, which will scan the bar code labels on all the cartridges to rebuild the inventory. Association of files in Globus to tapes is handled separately from the tape library and is not affected by the firmware update. Similarly, USATLAS and CMS file to tape association was not affected by the firmware update. We are taking additional measures to preserve this information as we complete the work with the tape library inventory..</p> <p><small>Mar <var data-var='date'> 4</var>, <var data-var='time'>14:22:59</var> GMT+0</small><br><strong>Completed</strong> - The Maintenance has completed successfully. One hardware components in the library has been replaced. The tape cartridges inventory has been reconciled. The Storage Archive service and Tape Library are up and running. Storage Archive shows NO missing or duplicated cartridges. We will keep closely monitoring the service to confirm everything proceeds as normal. Remaining items: One of the two accessors in the library robotic components was still showing small issues with calibration, so we have decided to leave it disabled for the time being, waiting for further troubleshooting from IBM. That will have only minimal impact on performance of the service..</p> ]]> Fri, 27 Feb 2026 17:00:00 +0000 https://nese.instatus.com/maintenance/cmm5chhxe02gudl4fdszkoaad https://nese.instatus.com/maintenance/cmm5chhxe02gudl4fdszkoaad SERVICE OUTAGE for Tape Frame Installation Type: Maintenance Duration: 9 hours Affected Components: NESE Tape Storage Feb 19, 13:00:00 GMT+0 - Identified - We are installing four new tape frames, which will bring the tape system raw storage capacity to 253 petabytes. **Service Affected:** NESE Tape Service **Maintenance Window:** 8:00 AM - 5:00 PM (EST) * The tape service will be unavailable. * All upgrade activities are expected to be completed on the same day. NOTES: * Monitor the MGHPCC Slack #nese channel for status updates and announcements * Monitor <https://nese.instatus.com/> for real-time updates on progress Subscribe to <https://nese.instatus.com/subscribe/email> for updates and announcements Feb 19, 13:00:01 GMT+0 - Identified - Maintenance is now in progress Feb 19, 22:00:00 GMT+0 - Completed - Maintenance has completed successfully Type: Maintenance

Duration: 9 hours

Affected Components:

<p><small>Feb <var data-var='date'> 19</var>, <var data-var='time'>13:00:00</var> GMT+0</small><br><strong>Identified</strong> - We are installing four new tape frames, which will bring the tape system raw storage capacity to 253 petabytes. **Service Affected:** NESE Tape Service **Maintenance Window:** 8:00 AM - 5:00 PM (EST) * The tape service will be unavailable. * All upgrade activities are expected to be completed on the same day. NOTES: * Monitor the MGHPCC Slack #nese channel for status updates and announcements * Monitor <https://nese.instatus.com/> for real-time updates on progress Subscribe to <https://nese.instatus.com/subscribe/email> for updates and announcements.</p> <p><small>Feb <var data-var='date'> 19</var>, <var data-var='time'>13:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Feb <var data-var='date'> 19</var>, <var data-var='time'>22:00:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> ]]> Thu, 19 Feb 2026 13:00:00 +0000 https://nese.instatus.com/maintenance/cmkwugmpi00een2jwcmsqie9e https://nese.instatus.com/maintenance/cmkwugmpi00een2jwcmsqie9e Upgrade: Installing Storage Scale 6000 Type: Maintenance Duration: 8 hours Affected Components: NESE Tape Storage Feb 9, 13:00:00 GMT+0 - Identified - In the process of the tape front-end file caching system upgrade, we will be installing a new IBM Storage Scale System 6000\. We will provide an additional update for when the software integration and data transfer from the current IBM Elastic Storage System 5000 will be performed. **Service Affected:** NESE Tape Service **Maintenance Window: No Downtime expected** NOTES: * Monitor the MGHPCC Slack #nese channel for status updates and announcements * Monitor <https://nese.instatus.com/> for real-time updates on progress * Subscribe to <https://nese.instatus.com/subscribe/email> for updates and announcements Please let us know if you have any questions or concerns Feb 9, 13:00:01 GMT+0 - Identified - Maintenance is now in progress Feb 9, 21:00:00 GMT+0 - Completed - Maintenance has completed successfully Type: Maintenance

Duration: 8 hours

Affected Components:

<p><small>Feb <var data-var='date'> 9</var>, <var data-var='time'>13:00:00</var> GMT+0</small><br><strong>Identified</strong> - In the process of the tape front-end file caching system upgrade, we will be installing a new IBM Storage Scale System 6000\. We will provide an additional update for when the software integration and data transfer from the current IBM Elastic Storage System 5000 will be performed. **Service Affected:** NESE Tape Service **Maintenance Window: No Downtime expected** NOTES: * Monitor the MGHPCC Slack #nese channel for status updates and announcements * Monitor <https://nese.instatus.com/> for real-time updates on progress * Subscribe to <https://nese.instatus.com/subscribe/email> for updates and announcements Please let us know if you have any questions or concerns.</p> <p><small>Feb <var data-var='date'> 9</var>, <var data-var='time'>13:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Feb <var data-var='date'> 9</var>, <var data-var='time'>21:00:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> ]]> Mon, 9 Feb 2026 13:00:00 +0000 https://nese.instatus.com/maintenance/cmkwuc5kj00cftpjj07vkn3yi https://nese.instatus.com/maintenance/cmkwuc5kj00cftpjj07vkn3yi NESE Tape System Maintenance Type: Maintenance Duration: 4 days and 8 hours Affected Components: NESE Globus Service, NESE Tape Storage Dec 1, 14:00:01 GMT+0 - Identified - Maintenance is now in progress Dec 1, 14:00:00 GMT+0 - Identified - NESE tape system maintenance and a major software upgrade are scheduled to begin on December 1, 2025\. As a result, the **NESE Tape service will be offline from December 1 to December 5**. Starting December 8 through December 12, the service will be back online with reduced performance. All maintenance activities are planned to conclude on December 12, 2025. **NOTES:** * Monitor the MGHPCC Slack #nese channel for status updates and announcements * Monitor <https://nese.instatus.com/> for real-time updates on progress * Subscribe to <https://nese.instatus.com/subscribe/email> for updates and announcements Please let us know if you have any questions or concerns. Dec 5, 22:00:00 GMT+0 - Completed - Maintenance has completed successfully Type: Maintenance

Duration: 4 days and 8 hours

Affected Components: ,

<p><small>Dec <var data-var='date'> 1</var>, <var data-var='time'>14:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Dec <var data-var='date'> 1</var>, <var data-var='time'>14:00:00</var> GMT+0</small><br><strong>Identified</strong> - NESE tape system maintenance and a major software upgrade are scheduled to begin on December 1, 2025\. As a result, the **NESE Tape service will be offline from December 1 to December 5**. Starting December 8 through December 12, the service will be back online with reduced performance. All maintenance activities are planned to conclude on December 12, 2025. **NOTES:** * Monitor the MGHPCC Slack #nese channel for status updates and announcements * Monitor <https://nese.instatus.com/> for real-time updates on progress * Subscribe to <https://nese.instatus.com/subscribe/email> for updates and announcements Please let us know if you have any questions or concerns..</p> <p><small>Dec <var data-var='date'> 5</var>, <var data-var='time'>22:00:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> ]]> Mon, 1 Dec 2025 14:00:00 +0000 https://nese.instatus.com/maintenance/cmhj6tgf400jw71ra1d9xz0y8 https://nese.instatus.com/maintenance/cmhj6tgf400jw71ra1d9xz0y8