hetzner example broken

[rgarcia]

When attempting to use the Hetzner example here, I get the following error:

Error from server (BadRequest): error when creating ".../clusterTemplateFromDocs.yaml": HCloudMachineTemplate in version "v1beta1" cannot be handled as a HCloudMachineTemplate: strict decoding error: unknown field "spec.imageName", unknown field "spec.sshKeys", unknown field "spec.type"

Looking at the reference docs for HCloudMachineTemplate, I edited it from

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: HCloudMachineTemplate
metadata:
  name: hetzner-test-mt
  namespace: default
spec:
  imageName: ubuntu-22.04
  type: cx21
  sshKeys:
    - name: ssh-key

to

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: HCloudMachineTemplate
metadata:
  name: hetzner-test-mt
  namespace: default
spec:
  template:
    spec:
      type: cx21
      imageName: ubuntu-22.04

and I added the ssh key configuration to the HetznerCluster resource since from those docs it seems like that's where it goes:

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: HetznerCluster
...
spec:
...
  sshKeys:
    hcloud:
    - name: <my ssh key in hcloud>

Now the kubectl apply succeeds but it's still broken:

 % kubectl events --for machinedeployment/hetnzer-test-md
LAST SEEN               TYPE      REASON             OBJECT                              MESSAGE
21m (x14 over 21m)      Warning   ReconcileError     MachineDeployment/hetnzer-test-md   failed to retrieve HCloudMachineTemplate external object "default"/"hetzner-test-mt": HCloudMachineTemplate.infrastructure.cluster.x-k8s.io "hetzner-test-mt" not found
11m (x11 over 11m)      Warning   ReconcileError     MachineDeployment/hetnzer-test-md   failed to retrieve HCloudMachineTemplate external object "default"/"hetzner-test-mt": HCloudMachineTemplate.infrastructure.cluster.x-k8s.io "hetzner-test-mt" not found
6m1s (x12 over 6m11s)   Warning   ReconcileError     MachineDeployment/hetnzer-test-md   failed to retrieve HCloudMachineTemplate external object "default"/"hetzner-test-mt": HCloudMachineTemplate.infrastructure.cluster.x-k8s.io "hetzner-test-mt" not found
5m49s (x4 over 5m49s)   Warning   ReconcileError     MachineDeployment/hetnzer-test-md   failed to retrieve HCloudMachineTemplate external object "default"/"hetzner-test-mt": HCloudMachineTemplate.infrastructure.cluster.x-k8s.io "hetzner-test-mt" not found
5m49s                   Warning   ReconcileError     MachineDeployment/hetnzer-test-md   failed to retrieve K0sWorkerConfigTemplate external object "default"/"hetzner-test-machine-config": K0sWorkerConfigTemplate.bootstrap.cluster.x-k8s.io "hetzner-test-machine-config" not found
5m49s                   Normal    SuccessfulCreate   MachineDeployment/hetnzer-test-md   Created MachineSet default/hetnzer-test-md-qmkv5

Looking at the hcloudmachinetemplate reveals a cryptic error

kubectl get hcloudmachinetemplate hetzner-test-mt 
NAME              IMAGE          PLACEMENT GROUP   TYPE   REASON                     MESSAGE
hetzner-test-mt   ubuntu-22.04                     cx21   HetznerSecretUnreachable   could not find HetznerSecret

I don't see anything in the HCloudMachineTemplate docs about configuring a secret... let's dive into the source I guess? Looks like the hcloudmachinetemplate controller pulls secret info from the parent cluster: https://github.com/syself/cluster-api-provider-hetzner/blob/3498a49c7540c1f27b0f66f1bb76625120f74cc0/controllers/hcloudmachinetemplate_controller.go#L120-L124

And my parent cluster has a secret ref that appears to be correct:

kubectl describe hetznercluster hetzner-test 
Name:         hetzner-test
Namespace:    default
Labels:       cluster.x-k8s.io/cluster-name=hetzner-test
Annotations:  cluster.x-k8s.io/managed-by: k0smotron
API Version:  infrastructure.cluster.x-k8s.io/v1beta1
Kind:         HetznerCluster
Metadata:
  Creation Timestamp:  2024-05-02T16:52:26Z
  Generation:          1
  Owner References:
    API Version:           cluster.x-k8s.io/v1beta1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Cluster
    Name:                  hetzner-test
    UID:                   72d92c18-b6a3-4517-ac36-79094feb1a43
  Resource Version:        22148840
  UID:                     e5b6a7d8-6ecc-412e-b0ad-26a4b6c7af55
Spec:
  Control Plane Endpoint:
    Host:  1.2.3.4
    Port:  6443
  Control Plane Load Balancer:
    Algorithm:  round_robin
    Enabled:    false
    Port:       6443
    Type:       lb11
  Control Plane Regions:
    fsn1
  Hcloud Network:
    Cidr Block:         10.0.0.0/16
    Enabled:            false
    Network Zone:       eu-central
    Subnet Cidr Block:  10.0.0.0/24
  Hetzner Secret Ref:
    Key:
      Hcloud Token:            hcloud
      Hetzner Robot Password:  
      Hetzner Robot User:      
    Name:                      hetzner-test
  Ssh Keys:
    Hcloud:
      Name:  metal-4VfxTHuaC4gL6C9jRwZNVm
    Robot Rescue Secret Ref:
      Key:
        Name:         
        Private Key:  
        Public Key:   
      Name:           
Events:               <none>

kubectl describe secret hetzner-test   
kubectl describe secret hetzner-test         
Name:         hetzner-test
Namespace:    default
Labels:       caph.environment=owned
Annotations:  <none>

Type:  Opaque

Data
====
hcloud:  65 bytes

So I'm officially stumped. Will also ask the caph folks if they have any insight.

Thanks for any tips you can provide.