Group Responsible: Storage & Infrastructure

Affected Area: US ATLAS dCache storage service

Maintenance Type: scheduled downtime

Dear all, Please note that the BNL US ATLAS dCache storage element will undergo maintenance for a major software upgrade on September 14 from 9:00 AM to 5:00 PM EDT. The storage will be unavailable to users during this period. Thank you in advance for your understanding.

Group Responsible: Network

Affected Area: Internal/External network at the SCDF/ATLAS TIer 1

Maintenance Type: transparent

Selected network switches interconnecting systems at the SCDF/US ATLAS Tier 1 and connecting facility systems to the WAN will be upgraded to newer software. No systems will lose network connectivity, but selected systems will lose network link redundancy as individual switches in a linked pair of switches are upgraded. The upgrade window is from 7:00AM EDT to 9:00AM EDT with the bulk of the changes occuring from 7:00AM to 8:00AM EDT

Group Responsible: Data Center Operations

Affected Area: SCDF Data Center Operations

Maintenance Type: transparent

This maintenance was completed successfully today at 4 pm EDT. As stated, this maintenance was transparent to the data centre and its services, with no power interruptions. Required maintenance of the electricity supply at the BNL site on Saturday, September 12 will interrupt utility power from approximately 6 am to 4 pm EDT. This maintenance was originally scheduled for 2025 and has been postponed until after the end of the RHIC and LHC run periods. It was then re-scheduled for August 12, but was again postponed due to unresolved reliability issues with the SCDF generators. These issues have been resolved, and the generators have been tested (with transfers of our IT load) successfully several times in the past few weeks. BNL Facilities & Operations (F&O) team has assured the SCDF that maintenance and intervention will be transparent to the data centre and its services, with no power interruptions. Some of the maintenance work will be done outdoors, and it may be postponed again in case of inclement weather. We will confirm the maintenance schedule to all stakeholders by 12:00 pm EDT on Friday, September 11 at the latest. SCDF will have staff on call during the maintenance. Communication will remain available via the corresponding experiments and projects MM channels or via email. John McCarthy (mccarthy@bnl.gov) is the SCDF point of contact (PoC) with F&O for this intervention. Past and upcoming announcements are listed at: https://useraccount.sdcc.bnl.gov/announcements

Group Responsible: Data Center Operations

Affected Area: SCDF Data Center Operations

Maintenance Type: transparent

This maintenance was completed successfully today at 4 pm EDT. As stated, this maintenance was transparent to the data centre and it services, with no power interruption. Required maintenance of the electricity supply at the BNL site on Saturday, September 12 will interrupt utility power from approximately 6 am to 4 pm EDT. This maintenance was originally scheduled for 2025 and has been postponed until after the end of the RHIC and LHC run periods. It was then re-scheduled for August 12, but was again postponed due to unresolved reliability issues with the SCDF generators. These issues have been resolved, and the generators have been tested (with transfers of our IT load) successfully several times in the past few weeks. BNL Facilities & Operations (F&O) team has assured the SCDF that maintenance and intervention will be transparent to the data centre and its services, with no power interruptions. Some of the maintenance work will be done outdoors, and it may be postponed again in case of inclement weather. We will confirm the maintenance schedule to all stakeholders by 12:00 pm EDT on Friday, September 11 at the latest. SCDF will have staff on call during the maintenance. Communication will remain available via the corresponding experiments and projects MM channels or via email. John McCarthy (mccarthy@bnl.gov) is the SCDF point of contact (PoC) with F&O for this intervention.

Group Responsible: Data Center Operations

Affected Area: RHIC Webmail

Maintenance Type: scheduled downtime

Upgrade of RHIC webmail complete. https://webmail.rhic.bnl.gov is available again.

Group Responsible: Storage & Infrastructure

Affected Area: RHIC Webmail

Maintenance Type: scheduled downtime

On September 10th, 10 AM EST, the RHIC webmail server webmail.rhic.bnl.gov will be unavailable. This is expected to take no more than 2 hours.

Group Responsible: Services & Tools

Affected Area: TEST

Maintenance Type: transparent

Test message

Group Responsible: Service & Tools

Affected Area: SSH/CSSH Gateways

Maintenance Type: transparent

On Monday, Aug 31, at 8:30am, 3 gateway nodes will be rebooted to install an important security update. These nodes are: - ssh01.sdcc.bnl.gov - ssh02.sdcc.bnl.gov - cssh01.sdcc.bnl.gov The "ssh.sdcc.bnl.gov" and "cssh.sdcc.bnl.gov" aliases have been updated to temporarily exclude them, and as such, any sessions started after this announcement will not be affected. To avoid service interruption, please ensure your sessions are not using the affected nodes.

Group Responsible: Service & Tools

Affected Area: SSH/CSSH/SFTP Gateways

Maintenance Type: transparent

On Thursday, Aug 27, certain gateway nodes will go offline for upgrades, and any active sessions on these nodes will be killed. The "ssh.sdcc.bnl.gov", "cssh.sdcc.bnl.gov", and "sftp.sdcc.bnl.gov" aliases have been updated to exclude the affected nodes. As such, sessions created on or after Thursday, Aug 20 are not affected. The affected nodes are: - ssh03.sdcc.bnl.gov - ssh04.sdcc.bnl.gov - cssh02.sdcc.bnl.gov - sftp02.sdcc.bnl.gov Any active sessions on these nodes will be killed on the morning of Thu, Aug 27. To avoid service interruption, please ensure any active sessions you have are not using these nodes.

Group Responsible: Service & Tools

Affected Area: SSH/CSSH/SFTP Gateways

Maintenance Type: transparent

On Wednesday, Aug 19, certain gateway nodes will go offline for upgrades, and any active sessions on these nodes will be killed. The "ssh.sdcc.bnl.gov", "cssh.sdcc.bnl.gov", and "sftp.sdcc.bnl.gov" aliases have been updated to exclude the affected nodes. As such, sessions created after this announcement are not affected. The affected nodes are: - ssh01.sdcc.bnl.gov - ssh02.sdcc.bnl.gov - cssh01.sdcc.bnl.gov - sftp01.sdcc.bnl.gov Any active sessions on these nodes will be killed on the morning of Wed, Aug 19. To avoid service interruption, please ensure any active sessions you have are not using these nodes.

Group Responsible: IT Services

Affected Area: SCDF Services

Maintenance Type: scheduled downtime

The SCDF has been informed that the electrical maintenance work scheduled for Saturday, August 15 has been postponed until further notice. An announcement will be posted when the work is re-scheduled.

Group Responsible: Experimental Support

Affected Area: SCDF Services

Maintenance Type: scheduled downtime

Required maintenance of the electricity supply at the BNL site on Saturday, August 15 will require the shutdown of most computing services at the SCDF. The SCDF will be scheduled for downtime from 18:00 EDT on Friday, August 14 until 12:00 EDT on Sunday, August 16. We will start draining batch queues on Friday, August 14 at 12:00 EDT and will restore services on Sunday, August 16 at 12:00 EDT. Some of the maintenance work will be done outdoors, and it may be postponed in case of thunderstorms or rain. We will confirm the maintenance schedule to all stakeholders by 09:00 EDT on Friday, August 14 at the latest. This maintenance was originally scheduled for 2025 and has been postponed until after the end of the RHIC and LHC run periods and scheduled during a long NSLS2 shutdown. Services affected include compute farm, tape storage, disk storage (GPFS, Lustre and dCache) and collaborative tools (Git, InvenioRDM, BNLBox, etc). Email, MatterMost, and the SCDF website will remain available to allow communications between SCDF staff and the user community.

Group Responsible: Service & Tools

Affected Area: BNLBox

Maintenance Type: service interruption

BNLBox service is resumed at this time, thank you for your patience.

Group Responsible: Service & Tools

Affected Area: BNLBox

Maintenance Type: service interruption

BNLBox is temporarily down, at this time we are investigating the issue. An update will be provided once service resumes, we apologize for the interruption.

Group Responsible: IT Services

Affected Area: OpenShift Virtual Environment

Maintenance Type: transparent

The OpenShift cluster will be undergoing planned maintenance tomorrow to upgrade cluster version. This will begin on Tuesday 6/23/26 @ 8AM. This should be "transparent" to all. If you see any issues or problem, please let notify us.