A cluster instability incident occurred affecting visibility and management of AU0 within the environment. AU1 remained operational but entered a degraded quorum state due to AU0 temporarily dropping out of corosync cluster membership. AU0 was later confirmed to be operational locally and successfully re-synchronised with the cluster after physical console access and service recovery.

Maintenance ended

Incident Report – Network Connectivity Failure (AU0 & AU1)

Date: 1 May 2026
Systems Affected:

• AU0 (Primary Node)
• AU1 (Secondary Node)

---

Summary

A critical network failure occurred impacting both AU0 and AU1. AU0 experienced a DHCP disconnection and failed to rebind to its statically assigned IP address. Concurrently, AU1 became completely inaccessible over the network.

---

Incident Details

AU0 (Primary Node):

• Lost network connectivity due to DHCP failure.
• Attempted fallback/reassignment to statically configured IP was unsuccessful.
• System remained online but unreachable via network.
• No evidence of successful lease renewal or interface recovery.

AU1 (Secondary Node):

• Fully inaccessible during the incident window.
• No response to ping, SSH, or management interface requests.
• Root cause not yet confirmed (potential network-level or host-level failure).

---

Impact

• Loss of access to both nodes resulted in service disruption.
• Any workloads, containers, or hosted services on AU0 and AU1 were unreachable.
• Potential customer-facing downtime depending on service allocation.

---

Root Cause (Preliminary)

• AU0: DHCP failure leading to loss of IP assignment, with static IP fallback not applying correctly (possible misconfiguration or interface binding issue).
• AU1: Likely related network isolation or upstream failure; requires further investigation.

---

Actions Taken

• Initial diagnostics performed via local network inspection.
• Verified AU0 interface presence but confirmed lack of IP binding.
• Attempted reconnection and manual network intervention (unsuccessful).
• AU1 accessibility tested across multiple vectors (no response).