Preserve Instances, network interfaces and addresses during upgrades #562

Problem

Upgrading an instance should replace its running process without replacing its network identity. Today the rollout controller deletes and recreates the Instance to make its runtime provider replace the backing Pod. A network reservation belongs to that Instance, so this also runs instance-deletion cleanup even though the workload still wants the same instance. An upgrade should normally keep the Instance and replace only its backing runtime.

During infra#7932, Valkey instances in three regions received new addresses after replacement. Their applications kept connecting to the old addresses and failed to start. The console investigation also found that a replacement could start using a previous instance's network reservation while it was being removed.

Desired outcome

If the workload still wants the same instance and its network settings have not changed, an image update, explicit restart or other runtime upgrade preserves the Instance UID, NetworkInterface UIDs and assigned addresses. For Pod-backed runtimes, replace the underlying Pod while keeping the Instance. Customers should not have to opt into a different retention policy just to keep networking through an upgrade.

Removing an instance for good is different: scale-down, workload deletion and an explicit interface removal must still honor the configured cleanup policy. A network configuration change that cannot preserve an interface needs a clearly defined replacement process.

Plan

1. Keep the Instance through ordinary upgrades. Treat the Instance as the stable desired resource and its Pod or other backing runtime as replaceable. Update the existing Instance to the desired configuration and have the runtime provider stop and replace its runtime. The current delete-and-recreate behavior exists because the unikraft provider builds the Pod at creation and does not reconcile an existing Pod to changed runtime settings. Updating the Instance alone is therefore insufficient; add an explicit runtime replacement path. Review API validation and every provider before relying on in-place Instance updates. Reserve Instance deletion for permanent removal or an explicitly documented change that truly requires a new identity.
2. Preserve the interface and reservation through upgrades. Keep the existing NetworkInterface object and addresses while the old runtime stops and its replacement starts. Coordinate compute and the network-services operator so garbage collection cannot release them between replacements. Do not solve this by forcing every customer to use Retain, which would also change cleanup after permanent deletion.
3. Make runtime replacement and the handoff explicit. Track the desired and running runtime revision separately. Mark the Instance as updating until the new runtime is ready; an old Pod's ready status must not complete the upgrade. Preserve ordered rollout behavior and let an update replace a failed old runtime. End shell sessions tied to the old runtime safely even though the Instance UID is unchanged. Do not let two runtimes use the same interface at once. Verify that the old runtime has released it before the replacement attaches, and verify readiness for the current holder. A same-name reservation or a stale ready condition is not enough. Distinguish a deliberately preserved reservation from an unrelated or terminating predecessor reservation.
4. Handle failure and cancellation. Cover controller restarts, delayed cleanup, a failed replacement, rapid consecutive upgrades and a scale-down or deletion during an upgrade. Preserve the address while the instance remains desired and release it only when the cleanup policy calls for that.
5. Migrate existing workloads safely. Move ownership without deleting live interfaces, changing addresses or restarting workloads. Coordinate changes needed in network-services-operator, Galactic and runtime providers; link their implementation issues here.
6. Prove the behavior in staging. Record interface IDs and addresses, upgrade an instance, and verify they remain the same. Repeat across controller upgrades and failure cases. Then test permanent deletion to prove resources are still cleaned up.

Acceptance criteria

• An image update or explicit restart preserves the Instance UID while replacing its backing Pod or runtime with the requested configuration.
• Readiness and rollout progress reflect the new runtime revision; stale status from the old runtime cannot complete an upgrade.
• Shell sessions cannot silently move to a different runtime when the Instance UID stays the same.
• An image update or explicit restart preserves each unchanged NetworkInterface UID and its assigned addresses while the instance remains desired.
• This works with the current Delete policy; permanent removal still releases resources under that policy.
• Retain continues to preserve resources after permanent removal as documented.
• Unchanged interfaces survive an upgrade even when another interface is added, removed or changed.
• The replacement waits for a safe handoff and cannot start from stale readiness belonging to an unrelated or terminating reservation.
• Controller restarts, failed upgrades, repeated updates and deletion during upgrade neither duplicate address use nor leak interfaces.
• Existing workloads migrate without interface recreation, address changes or an unintended restart.
• Staging tests cover both address preservation during upgrade and cleanup after scale-down or workload deletion.
• Any unavoidable address-changing operation is documented and visible to the customer before it is applied.

Related

• Parent incident: infra#7932.
• Upgrading compute to v0.14.0 recreated running instances #560 prevents dependency upgrades from causing unintended replacements. This issue preserves networking during replacements that are intended.
• Support multiple network interfaces per workload instance #442 covers multiple interfaces; preservation and cleanup should work per named interface.