Proxmox VE 8 Multi-Node Clustered Virtualization & Ceph Hyperconverged Storage Deployment

1. Scope, Architecture & Redundancy Model

This standard operating procedure governs the greenfield assembly, cluster initialization, and hyperconverged Ceph storage deployment for Proxmox Virtual Environment (PVE) 8.x enterprise environments.

To guarantee zero single point of failure and eliminate split-brain risks, Proxmox clustering requires:

1. Odd Node Count: Minimum 3 physical nodes (pve-node01, pve-node02, pve-node03) to maintain quorum ((N/2) + 1).
2. Redundant Corosync Communication: Dedicated physical interfaces for Ring 0 and Ring 1 (latency must stay < 5ms).
3. Ceph Hyperconverged Fabric: Dedicated 10G/25G non-routed network for Ceph public and cluster traffic (Jumbo Frames MTU 9000).

 ┌───────────────────────────────────────────────────────────────┐
 │               Physical Top-of-Rack Switch Fabric              │
 └──────┬───────────────────────┬───────────────────────┬────────┘
        │                       │                       │
 ┌──────┴──────────────┐ ┌──────┴──────────────┐ ┌──────┴──────────────┐
 │   pve-node01        │ │   pve-node02        │ │   pve-node03        │
 │ ├─ Corosync Ring 0  │ │ ├─ Corosync Ring 0  │ │ ├─ Corosync Ring 0  │
 │ ├─ Corosync Ring 1  │ │ ├─ Corosync Ring 1  │ │ ├─ Corosync Ring 1  │
 │ ├─ Ceph Public/Sync │ │ ├─ Ceph Public/Sync │ │ ├─ Ceph Public/Sync │
 │ └─ VM Bridges (Trunk)│ │ └─ VM Bridges (Trunk)│ │ └─ VM Bridges (Trunk)│
 └─────────────────────┘ └─────────────────────┘ └─────────────────────┘

---

2. Pre-Flight Network & Storage Matrix

Network Service	Physical Uplink	VLAN	Subnet CIDR	MTU	Notes
PVE Management / Ring 0	eno1 (1G/10G)	VLAN 10	10.10.10.0/24	1500	Corosync Priority 1
Corosync Redundant Ring 1	eno2 (1G/10G)	VLAN 11	172.16.11.0/24	1500	Corosync Priority 2 (Isolated)
Ceph Public & Sync Traffic	ens1f0np0 (10G/25G)	VLAN 91	10.10.91.0/24	9000	Jumbo frames mandatory
VM Traffic / Guest Trunk	ens1f1np1 (10G/25G)	LACP Trunk	Multi-VLAN	1500	vmbr0 bridged to core switch

---

3. Execution: Greenfield Base Host Preparation

Perform on all candidate nodes (pve-node01, pve-node02, pve-node03) prior to cluster formation.

Step 1: Enterprise Repositories & Debian Updates

Disable unvalidated pve-enterprise repository if subscription keys are not active, and enable the enterprise pve-no-subscription repository for baseline patching:

# 1. Disable default enterprise repo
sed -i -e 's/^/#/' /etc/apt/sources.list.d/pve-enterprise.list

# 2. Add verified no-subscription repo (or enterprise if licensed)
cat <<EOF > /etc/apt/sources.list.d/pve-install-repo.list
deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription
EOF

# 3. Add Ceph repository
cat <<EOF > /etc/apt/sources.list.d/ceph.list
deb http://download.proxmox.com/debian/ceph-reef bookworm no-subscription
EOF

# 4. Refresh package caches and perform distribution upgrade
apt update && apt dist-upgrade -y

# 5. Install essential cluster utilities
apt install -y ifupdown2 chrony iotop iftop htop nvme-cli

Step 2: High-Precision NTP Synchronization

Ensure cluster nodes synchronize against internal Enterprise NTP Domain Controllers (per REF-NET-002):

cat <<EOF > /etc/chrony/chrony.conf
server 10.10.10.10 iburst minpoll 4 maxpoll 6
server 10.10.10.11 iburst minpoll 4 maxpoll 6
driftfile /var/lib/chrony/chrony.drift
rtcsync
makestep 1 3
EOF

systemctl restart chrony
chronyc sources -v

Step 3: Network Configuration (/etc/network/interfaces)

Apply baseline interfaces with dedicated Corosync and Ceph links:

cat <<'EOF' > /etc/network/interfaces
auto lo
iface lo inet loopback

# Management & Corosync Ring 0
auto eno1
iface eno1 inet static
    address 10.10.10.31/24
    gateway 10.10.10.1
    dns-nameservers 10.10.10.10 10.10.10.11
    dns-search corp.internal

# Corosync Ring 1 (Isolated heartbeat)
auto eno2
iface eno2 inet static
    address 172.16.11.31/24
    mtu 1500

# Ceph Storage Fabric (Jumbo Frames)
auto ens1f0np0
iface ens1f0np0 inet static
    address 10.10.91.31/24
    mtu 9000

# VM Trunk Bridge
auto ens1f1np1
iface ens1f1np1 inet manual

auto vmbr0
iface vmbr0 inet manual
    bridge-ports ens1f1np1
    bridge-stp off
    bridge-fd 0
    bridge-vlan-aware yes
    bridge-vids 2-4094
EOF

ifreload -a

---

4. Execution: Proxmox Cluster Creation

Step 1: Initialize the Cluster on Node 1 (pve-node01)

Create the cluster defining both Corosync Ring 0 (management) and Ring 1 (heartbeat network):

pvecm create PVE-PROD-CLUSTER \
    --link0 10.10.10.31,priority=1 \
    --link1 172.16.11.31,priority=2

Verify cluster creation:

pvecm status

Step 2: Join Successive Nodes (pve-node02, pve-node03)

On pve-node02 and pve-node03, execute the join command specifying each node’s local link addresses:

# On pve-node02:
pvecm add 10.10.10.31 \
    --link0 10.10.10.32,priority=1 \
    --link1 172.16.11.32,priority=2

# On pve-node03:
pvecm add 10.10.10.31 \
    --link0 10.10.10.33,priority=1 \
    --link1 172.16.11.33,priority=2

Step 3: Quorum and Corosync Verification

Execute on any node:

# Check Corosync link health and round-trip times
corosync-cfgtool -s

# Confirm quorum membership
pvecm status

Expected output: Quorate: Yes, Nodes: 3, Total votes: 3, Quorum: 2.

---

5. Execution: Ceph Reef Hyperconverged Storage Deployment

Step 1: Ceph Initialization

On pve-node01, initialize the cluster Ceph network targeting the dedicated 10G/25G storage fabric:

pveceph install --version reef
pveceph init --network 10.10.91.0/24

Step 2: Deploy Ceph Monitors (MON) and Managers (MGR)

Maintain an odd number of MONs (one per node) across the 3 nodes:

# Deploy MONs across all 3 nodes
pveceph mon create --node pve-node01
pveceph mon create --node pve-node02
pveceph mon create --node pve-node03

# Deploy Managers
pveceph mgr create --node pve-node01
pveceph mgr create --node pve-node02
pveceph mgr create --node pve-node03

Step 3: Provision Ceph OSDs (Object Storage Daemons)

Identify enterprise NVMe or SAS SSDs on each node (e.g., /dev/nvme0n1, /dev/nvme1n1) and register them:

# Run on each corresponding node for each dedicated SSD
pveceph osd create /dev/nvme0n1
pveceph osd create /dev/nvme1n1

Verify storage health:

ceph status
ceph osd tree

Step 4: Create Clustered Ceph Storage Pools

Provision the primary VM disk storage pool with 3x replication (Min Size 2 to avoid data corruption):

# Create Ceph RBD pool with 3/2 replication rule
pveceph pool create ceph-vm-pool --size 3 --min_size 2 --add_storages 1

Confirm pool parameters:

ceph df
pvesm status

---

6. High Availability (HA) & Hardware Fencing

Step 1: Enable Hardware/Linux Watchdog Fencing

Enable the Linux softdog or hardware IPMI watchdog daemon across all nodes:

# Enable watchdog module
modprobe softdog
echo "softdog" >> /etc/modules

# Verify PVE HA manager uses watchdog
systemctl restart pve-ha-crm pve-ha-lrm
ha-manager status

Step 2: Configure HA Resource Groups

Group critical VMs (Domain Controllers, Firewalls, Database servers) into HA groups:

# Define HA group prioritizing pve-node01 and pve-node02
ha-manager groupadd Tier1-Infra --nodes "pve-node01:1,pve-node02:1,pve-node03:2" --nofailback 0

# Assign VM to HA group
ha-manager add vm:100 --group Tier1-Infra --max_restart 2 --max_relocate 1

---

7. Verification & Live Migration Drills

1. Corosync Latency Validation:

corosync-cmapctl | grep runtime.totem.pg.mrp.srp.members

2. Live VM Migration Drill:

# Perform online live migration of VM 100 from node01 to node02
qm migrate 100 pve-node02 --online --with-local-disks 0

Verify zero packet loss on guest VM during cutover.

3. Ceph Failover Simulation:

Temporarily down one OSD to verify auto-repair and continuous I/O:

ceph osd set noout
ceph health detail
ceph osd unset noout

---

8. Rollback & Emergency Node Removal Procedure

If a node experiences catastrophic Corosync partition or unrecoverable hardware failure:

Step 1: Evacuate HA Workloads

ha-manager relocate vm:100 pve-node01

Step 2: Remove Failed Node from Ceph

# Remove Ceph OSDs belonging to failed node
ceph osd out <osd-id>
ceph osd crush remove osd.<osd-id>
ceph auth del osd.<osd-id>
ceph osd rm <osd-id>

# Remove MON
pveceph mon destroy <node-name>

Step 3: Remove Node from Corosync Cluster

# Execute on a healthy surviving node ONLY:
pvecm delnode <failed-node-name>

# If quorum is lost, temporarily force expected votes to survive:
pvecm expected 1

---

9. Operational Audit Trail & Document Approvals

Activity	Verification Checkpoint	Performed By	Date Validated
Corosync Redundant Ring	Dual links verified via corosync-cfgtool -s	Systems Administration Team	Approved
Ceph Reef Pool (3/2)	ceph status returns HEALTH_OK	Systems Administration Team	Approved
Fencing & HA Relay	Softdog armed and CRM/LRM synchronized	Lead Systems Administrator	Approved
Live Migration Benchmark	Zero guest ping loss during memory sync	Lead Systems Administrator	Approved