Group Responsible: IT Services

Affected Area: OpenShift Virtual Enviroment

Maintenance Type: transparent

The OpenShift cluster will be undergoing planned upgrade/maintenance tomorrow. This will begin on 5/28 @ 8AM. This should be "transparent" to all. If you see any issues or problem, please let notify us.

Group Responsible: IT Services

Affected Area: Phenix GPFS (GPFS02)

Maintenance Type: scheduled downtime

Due to a hardware issue on the Phenix gpfs Cluster (GPFS02), we will be scheduling down time for the cluster for Thursday 5/21 @10AM ~ 12PM. This will bring all storage for the GPSF02 cluster offline.

Group Responsible: Service & Tools

Affected Area: SCDF Mattermost

Maintenance Type: service interruption

SCDF Mattermost will be down for a short reboot to apply patches at approximately 23:00 EST. Users may experience a brief interruption while the server reboots but should be able to resume normal usage following this update.

Group Responsible: IT Services

Affected Area: OpenShift Virtual Enviroment

Maintenance Type: transparent

The OpenShift cluster will be undergoing planned maintenance today to upgrade cluster version from 4.18.26 to 4.18.34. This was begun on 5/12 with some issues, they have been resolved and the upgrade will now be completed. This should be "transparent" to all. If you see any issues or problem, please let notify us.

Group Responsible: IT Fabric

Affected Area: Openshift cluster - all VM networking and container workloads.

Maintenance Type: transparent

The Openshift cluster will be undergoing planned maintenance to upgrade cluster version from 4.18.26 to 4.18.34. This **should** be transparent according to Redhat. They claim they fixed the issue that caused all VM secondary networks to drop during the master node upgrade portion during last upgrade. If all goes well, early Thursday morning we will upgrade to 4.19.30 but another notification will be sent on Wednesday.

Group Responsible: Service & Tools

Affected Area: The NX service will not be available during this time

Maintenance Type: service interruption

The NX servers will be rebooted to remediate a significant linux vulnerability. All NX sessions will be terminated, please save your work.

Group Responsible: IT Fabric

Affected Area: sphnx and spool condor pools

Maintenance Type: service interruption

sphnx and spool condor pool worker nodes are being drained and rebooted to remediate a critical linux vulnerability. sphnx worker nodes will be fully drained en mass, running jobs will be allowed to run to completion (unless killed by their owner), but no new jobs will be able to run until some nodes fully drain, reboot, and start accepting new jobs again. spool worker nodes will get a rolling partial drain + reboot to try to best balance remediation speed and minimization of disruption. Some jobs will be killed*, but unless they include their own hold expression specifying otherwise, they'll get automatically restarted on another node. * full drain with no jobs killed would take way too long for a rolling remediation. When a node is remediated, jobs running for <2hrs will be killed, as will remaining jobs once a node reaches a 90% drain level (90% of it's cpu's unallocated), but as stated, killed jobs should automatically be restarted. The above plans were quickly developed with the largest stakeholders. We recognize this may be problematic for some spool experiments/users, my apologies. Thank you for your patience and understanding.

Group Responsible: IT Services

Affected Area: SSH logins for SCDF Staff will be interrupted.

Maintenance Type: service interruption

Due to the Linux kernel issue explained in CVE-2026-31431, we will be scheduling a reboot of admingw01&02 and staffgw01&02. This will cause connections to drop during the reboot process. Interruption time should be minimal.

Group Responsible: IT Fabric

Affected Area: SCDF managed interactive/submit nodes

Maintenance Type: service interruption

To remediate a significant linux vulnerability, all SCDF managed interactive submit nodes will be rebooted shortly with a ~10min warning mesg. Submitted condor jobs will continue running and reconnect with the submit node after the reboot. For worker nodes, we're working on a solution to balance prompt response with minimizing disruption of running jobs as much as feasible. Interactive logins and processes will be terminated when the int/sub nodes reboot, once the nodes are back up, you should be able to continue your work. Thank you for your patience and understanding.

Group Responsible: Service & Tools

Affected Area: BNLBox

Maintenance Type: scheduled downtime

BNLBox service will be down for upto 1 hour due to its maintenance.

Group Responsible: IT Services

Affected Area: RT

Maintenance Type: service interruption

The RT VM will be migrated off the RHEV platform. The RT web site will be unavailable during the migration but emails will be queued and delivered once the migration is complete. This will begin at 8:30am on Wednesday April 1st and is expected to take 3-4 hours. An update will be sent when the migration is complete.

Group Responsible: Service & Tools

Affected Area: BNLBox

Maintenance Type: transparent

The scheduled BNLBox upgrade for today has been canceled. A new announcement will be issued once downtime has been rescheduled.

Group Responsible: Service & Tools

Affected Area: BNLBox

Maintenance Type: scheduled downtime

BNLBox will be unavailable on 03/25/2026 starting at 08:15 AM EST for approximately one hour to perform a scheduled upgrade. During this time, users will experience a temporary disruption in service. We apologize for the inconvenience and appreciate your patience.

Group Responsible: IT Services

Affected Area: Atlas GPFS

Maintenance Type: transparent

We will be upgrading the Atlas GPFS servers (atlasgpfs01) on Monday 3/2/2026 beginning @10:00 AM. This should be a transparent upgrade to the RHEL and GPFS software. If there are any issues, please open an RT ticket for these to be addressed.

Group Responsible: IT Services

Affected Area: HPC GPFS & RHEL 8

Maintenance Type: transparent

We will be upgrading the HPC/IC2 GPFS servers (hpcgpfs01) beginning @ 9:30AM. This should be a transparent upgrade to the RHEL and GPFS software. If there are any issues, please open an RT ticket for these to be addressed.