Skip to content

Kali clones: cloud-init's generator can be killed on first boot, leaving the guest with no user #1127

Closed

Description

@Gerrrt

Found on 2026-10-10 while building garuda (#921), the first real clone of template 910 (tpl-kali-saruman). 902 (tpl-kali, for ifrit, #790) shares the same preseed and package set, so it should inherit this.

What happened

tofu apply made VMID 162 from 910 and booted it. On that first boot cloud-init never ran:

  • cloud-init status reported not started.
  • The hostname stayed tpl-kali-saruman.
  • There was no analyst user and no authorized_keys, so SSH from phoenix was refused (Invalid user analyst).

The datasource was fine. ds-identify.log shows Found single datasource: NoCloud, got label=cidata and returning 0.

The generator, though, was cut off mid-script. /run/cloud-init/cloud-init-generator.log ends at:

checking for datasource

The script's next line logs ds-identify rc=…, which never appears. So cloud-init.target was never linked into multi-user.target: /run/systemd/generator.early/ did not exist, and the target stayed inactive (dead).

In the same boot's journal, a few microseconds apart (systemd 262, Kali cloud-init 26.2-1):

[5.608679] systemd[1]: Detected first boot.
[5.608733] systemd-sslh-generator: Configuration directory '/etc/sslh/' does not exist! No units generated.
[5.608748] (generato[283]: '/usr/lib/systemd/system-generators/systemd-sslh-generator' terminated by signal ABRT.
[5.608829] systemd[1]: Failed to preset all unit: ... is transient or generated
[5.609155] systemd[1]: Failed to preset all unit ...: Read-only file system

sslh is there because kali-linux-headless depends on it, and the preseed installs that through kali-linux-default.

qm reboot 162 fixed it. On the second boot, which is no longer systemd's "first boot", the generator logged ds-identify rc=0 and enabled via …/cloud-init.target. cloud-init then ran: hostname garuda, user analyst, and the key in place.

Intermittent, not every time

Both scripts/packer-smoke.sh 910 runs, one after each build, cloned 910 and passed on the clone's first boot (hostname smoke-910, SSH as smoke). So it is 1 failure in 3 first boots so far, which looks like a race between generators on first boot rather than a fixed misconfiguration. Which of these kills the cloud-init generator is not known yet:

  • the sslh generator's ABRT;
  • first-boot preset-all on a still read-only root;
  • systemd 262's generator handling.

A side effect worth knowing

The first boot without cloud-init still let sshd generate host keys. The second boot's cloud-init replaced them, so a key pinned between the two boots goes stale. Pin a Kali clone's host key only after cloud-init status says done.

Options

Not decided:

  1. Remove the trigger from the template. Purge sslh in packer/kali.pkr.hcl's provisioner, or keep it and give it an /etc/sslh/ so its generator exits cleanly. This only helps if sslh is the cause.
  2. Stop depending on the generator. Statically enable cloud-init.target from the template, e.g. a multi-user.target.wants link. Then a killed generator no longer decides whether cloud-init runs.
  3. Detect it. Make packer-smoke.sh assert cloud-init status is done, not only the hostname. Add a runbook check after a Kali clone's first boot: if it is not done, qm reboot once.
  4. Reproduce first: clone 910 several times and count the failures, to tell a race from a one-off.

garuda itself is fine now, after the reboot. This is about the next clone.

Activity

  1. added
    bugSomething isn't working
    labstacks/lab on alexander (VLAN 30)
    on Oct 10, 2026
  2. Gerrrt commented on Oct 10, 2026

    @Gerrrt
    OwnerAuthor

    Fixed in template 910 on 2026-10-10 (#1157).

    Cause. On a clone's first boot, cloud-init's generator was cut off before it enabled cloud-init.target. This affected 2 of 7 first boots of 910: garuda, and one of four disposable reproduction clones. On garuda's first boot, sslh's generator aborted alongside it.

    Fix (both in packer/kali.pkr.hcl's shared provisioner):

    • cloud-init.target is now linked into multi-user.target, so cloud-init no longer depends on the generator.
    • sslh's generator is masked with a /dev/null link. The sslh package stays installed.

    This is option 2 from the issue, plus option 1's masking. 902 inherits it when ifrit builds it.

    Proved. I rebuilt 910 with the fix. Then 12 of 12 first boots of disposable clones ran cloud-init: their own hostname, status: done, and the generator completing. On the last 4, sslh's generator no longer aborts. An earlier version of my check counted an sslh abort on every boot because it matched its own command in the journal. Those sslh counts are void; the cloud-init counts stand.

    Not added: a cloud-init status assertion in packer-smoke.sh. Its hostname and SSH checks already fail when cloud-init does not run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workinglabstacks/lab on alexander (VLAN 30)

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions