EGO Computing Maintenance and Computing Room UPS Replacement

Short description

Planned maintenance of the EGO computing infrastructure, including staged computing updates, UPS replacement activities, and progressive service restart. The intervention will affect authentication, detector computing, storage, core user services, and network availability over the maintenance window.

Current status

• Overall status: ONGOING 
• Last updated: [30 March 2026, 16:00, CET]

- Afternoon of March 25th: 

• Detector Hardware in safe mode
• All Virgo application stopped
• DAQ shutdown (Collection, Storage, Access) (
•  Cm and VPM processes shutdown
• 17:30 CET: Computing maintenance activities are in progress.

Morning of March 28th:

• Computing infrastructure updates are ongoing as planned and in some areas are in advance
• Shutdown window on Monday, March 30 is confirmed as planned

Afternoon of March 30th:

• Pair of temporary generators has been installed
• As planned, computing core services shutdown started at 12:00
• UPS Electrical shutdown started at 13:30 and finished at 15:45
• Computing Center is being powered by generators
• EGO IT team is restoring core services 

Contacts during the shutdown

For urgent issues during the shutdown windows, please use the alternative contacts below, as the EGO mail service may be unavailable.

Phone contacts

• Franco Carbognani: +39 3669826503
• Computing support: (L. Salconi) +39 3428090890
• Interferometer Operation  (V. Dattilo): +39 3292274503
• EGO Administration: +39 3470749281

Alternative email addresses

• fcarbogn@gmail.com
  
  

During the shutdown windows, updates on service status and recovery progress will be published on this Indico page.

Description

Following the planned computing outage at EGO, this event provides the consolidated schedule and expected service impact for the infrastructure update and UPS replacement activities.

To reduce downtime of user-facing services during the UPS intervention, a pair of generators will be used to provide temporary power coverage during part of the work. As a consequence, Core Services, Network, and Auth (IPA) will only be shut down during two specific daytime windows:

• 30 March, from 12:00 until late evening
• 2 April, from 12:00 until late evening

Outside these windows, core services may be available, but throughout the shutdown period services may still degrade because of dependencies and ongoing upgrades.

Overall timeline

• 26–29 March: progressive computing infrastructure updates
• 30 March – 2 April: UPS replacement activities
• 3 April: restart phase begins
• 7 April: completion of full system recovery, including detector-related services

Main impacts

Core services, network, and authentication

The following services are expected to be unavailable during the scheduled shutdown windows on 30 March afternoon and 2 April afternoon, and may degrade outside those windows during the intervention period:

• websites
• mail
• mailing lists
• logbook
• wifi
• network
• vmd
• etmd
• wikies
• indico
• vdi
• tds
• timestamps
• meeting rooms
• admin software
• purchase sw
• git / mattermost ET
• team speak
• Auth (IPA)

Emails sent to EGO addresses while mail is unavailable will be queued and delivered after restoration.

Detector computing and storage

The following services are affected over the longer maintenance window, from 26 March to 7 April:

• farmn
• ctrl
• olservers
• htcondor
• mass storage (/data)
• stol (daq data)
• rtpc
• dms
• web scientist
• file server (/virgoxxx)

These services should be considered unavailable for most of the maintenance period until progressive recovery is completed.

Recovery

• From 3 April: progressive restart of services begins
• By 7 April: full restoration expected

Important note

The schedule may still evolve depending on work progress and unforeseen technical issues.

Services Availability Matrix

Date                    | Auth (IPA)    | Detector Computing | Storage (/data) | Core Services (mail, web, etc.) | Network
----------------+-------------------+--------------------+---------------------+----------------------------------+---------
Mar 26                | may degrade       | DOWN               | DOWN                | may degrade                         | may degrade
Mar 27                | may degrade       | DOWN               | DOWN                | may degrade                          | may degrade
Mar 28                | may degrade       | DOWN               | DOWN                | may degrade                          | may degrade
Mar 29                | may degrade       | DOWN               | DOWN                | may degrade                          | may degrade
Mar 30 morning  | may degrade      | DOWN               | DOWN                | may degrade                          | may degrade
Mar 30 afternoon| DOWN                 | DOWN               | DOWN                | DOWN                                    | DOWN
Mar 31                | may degrade      | DOWN               | DOWN                | may degrade                         | may degrade
Apr 01                 | may degrade    | DOWN               | DOWN                | may degrade                           | may degrade
Apr 02 morning   | may degrade    | DOWN               | DOWN                | may degrade                          |may degrade
Apr 02 afternoon| DOWN                | DOWN               | DOWN                | DOWN                                    | DOWN
Apr 03                 | restarting          | restarting         | restarting            | restarting                               | restarting
Apr 04                 | recovering        | recovering         | recovering          | recovering                              | recovering
Apr 05                 | recovering        | recovering         | recovering          | recovering                              | recovering
Apr 06                 | recovering        | recovering         | recovering          | recovering                              | recovering
Apr 07                 | OK                     | OK                     | OK                     | OK                                           | OK