[Summary, full note in Shaarli]
Karpenter: make nodes schedulable only once required DaemonSets are Ready
Approach: a startup taint plus the Node Readiness Controller (https://github.com/kubernetes-sigs/node-readiness-controller). No fixed sleep; keep NodeReady separate from workload initialization.
- NodePool startup taint (only on the relevant NodePools; existing nodes may not get it):
spec.template.spec.startupTaints: [{key: readiness.example.com/required-daemonsets, value: pending, effect: NoSchedule}]
- Install the controller (check the current release and docs):
VERSION=v0.5.0 ; kubectl apply -f .../releases/download/${VERSION}/crds.yaml ; kubectl wait --for=condition=Established crd/nodereadinessrules.readiness.node.x-k8s.io ; kubectl apply -f .../install.yaml
- NodeReadinessRule (readiness.node.x-k8s.io/v1alpha1) required-daemonsets-ready: nodeSelector readiness.example.com/enabled="true"; condition readiness.example.com/RequiredDaemonSetsReady requiredStatus "True"; same taint; enforcementMode: continuous (puts the taint back if the condition turns False).
- Allowlist of required DaemonSets in a ConfigMap (e.g. kube-proxy, aws-node or your CNI, ebs-csi-node, node-exporter).
- A reporter DaemonSet (code to write) that, for its node (downward API), sets the condition True only when every required DaemonSet has a Ready pod it owns there, False if one is missing, unready or terminating; unknown = not ready. RBAC: get/list/watch pods, nodes, daemonsets; patch/update nodes/status.
- Every required DaemonSet and the reporter must tolerate the taint; the controller must run on an already healthy node.
- Test on a non-prod NodePool (kubectl get nodepools / nodes / nodereadinessrules): new node tainted, condition True, taint removed; stop a required pod, taint back.
Limits: NoSchedule doesn't evict running workloads (needs a separate policy). Docs: https://node-readiness-controller.sigs.k8s.io/user-guide/getting-started.html , https://karpenter.sh/docs/concepts/nodepools/