Skip to content

Troubleshooting

Start with the controller status, recent activity, and operator logs.

Terminal window
varroactl describe controller <namespace>/<name>
varroactl activity --controller <name>
varroactl logs <namespace>/<name> --follow
kubectl get events -n <namespace> --sort-by=.lastTimestamp

  • Pending or Provisioning: check namespace eligibility, image pull, storage, and ingress events. Correct the missing dependency, then reconcile.

  • Jenkins does not connect: check Pod readiness, gateway reachability, TLS, and network policy.

  • Running with the pill Mite unreachable: the mite stream is lost, and the pod stays in place. Check the gateway pods, the pod egress to the gateway on TCP 9090, and the mite log.

  • Bundle change is absent: inspect composed bundle status and any rollout or reconciliation hold. Correct the bundle, then approve or reconcile.

  • 403 or 401: run varroactl whoami, then review Varroa RBAC.

  • Lifecycle action waits: inspect pending status for a build, power state, hibernation, or approval. See Reconciliation.

  • Controller condition ReconcileBlocked is True with reason DesiredStatePushFailed, and its message contains above the NATS max payload. The message reads desired-state command is N bytes, above the NATS max payload: nats: maximum payload exceeded, where N is the command size in bytes. The merged desired state is larger than the 1 MiB NATS message limit. A cell carries the content of every tenant bundle in one command. Reduce the bundle content. Or move some tenants to another cell.

  • Plugins are unavailable: check the version profile and Update Center readiness.

  • UpdateCenterFallback is True with DownloadURLNotFound or PluginArchiveFallback: the download URL does not serve the pins. Read the message: kubectl get controller lts-2-570 -n east -o jsonpath='{.status.conditions[?(@.type=="UpdateCenterFallback")].message}'. Set pluginUpdateCenterDownloadURL to <mirror>/download/plugins, then roll the controller.

  • Operator, gateway, or BFF logs nats disconnected followed by repeated nats async error with authorization violation, and the operator pod goes NotReady: the NATS password rotated. It clears in about a minute: each reconnect re-reads the mounted Secret after kubelet syncs it. Confirm the operator is Ready and varroa.bus.connected is 1.

  • Those same authorization errors continue well past a minute, and the operator stays NotReady: the component cannot see the rotated password. Its pod spec supplies the bus password as an environment variable instead of -bus-pass-file, which freezes the value at pod start. Upgrade the release, then run kubectl rollout restart deployment/<release>-varroa-operator deployment/<release>-varroa-gateway deployment/<release>-varroa-bff. A restart during rotation is safe: the component waits NotReady for the synced Secret.

  • A component logs nats permission violation and varroa.bus.permission_violations increases: NATS denied that component the subject the line names. A denied publish is lost; upgrade the core release to match the chart grants. The gateway, operator, BFF, and recorder retry a denied subscription. The gateway marks MiteStreamDegraded until NATS accepts it.

  • nats stream ls fails with a permission violation for every service user: the ACL denies stream discovery. Inspect JetStream through port 8222 of each pod instead: kubectl -n <operator-namespace> port-forward pod/<release>-nats-0 8222:8222, then curl -s 'http://localhost:8222/jsz?streams=true&consumers=true'. /varz gives server status and /connz the connections. Each pod reports only its own streams.

  • Controller sits in Provisioning and the pill shows Boot failed: read kubectl logs <pod> -c jenkins -n <namespace>. A JCasC boot error such as Cloud must have a unique non-empty name exits with code 5. Fix the bundle. The operator restamps the StatefulSet with the new content. A crash-looping pod cannot roll on its own under OrderedReady. The operator deletes the pod when its revision lags the StatefulSet. The StatefulSet recreates the pod and the controller reaches Connected in about two minutes. This applies only to the default RollingUpdate strategy with no partition. An OnDelete or partitioned overlay keeps the manual pod delete. If the pod still runs the old revision after ten minutes, run kubectl delete pod <pod> -n <namespace>. Then report it as a bug.

  • Builds do not start, and the Jenkins queue shows items blocked with the cause “awaiting Varroa claim”: confirm VARROA_CLAIM_GATE on the Jenkins container matches whether recordStore.enabled is true. A mismatch after a chart change needs a pod roll to pick up the new value. Otherwise check the Jenkins log for the claim handler’s lines. An INFO line starting VarroaClaimQueue: duplicate claim for run means a different run already claimed the same intent key, or the ledger recorded the start of the same run. A WARNING line starting VarroaClaimQueue: claim refused for job names the job, the run, the veto reason, and the message. A line starting VarroaClaimQueue: NUMBER_CONFLICT means the job’s counter already passed the grant; the plugin keeps the item and claims again with number_conflict. A BuildNumberGate line names a run that started under another number than its grant, or a grant that did not start in 60 seconds. A line starting VarroaClaimQueue: giving up on job means one hour of retries failed. The gateway logs claim answered unavailable for each failed claim, and the recorder later marks the run CLAIM_EXPIRED. Check the mite’s claim counters to see which outcome dominates: varroa.mite.claims.granted, varroa.mite.claims.duplicate, varroa.mite.claims.vetoed, and varroa.mite.claims.failed. See Build claims for the full decision table and the symptoms of a recorder outage.

  • Pill shows Blocked with plugin X requested at A conflicts with profile lock B: every plugin pinned in a bundle must equal the JenkinsVersionProfile lock for the controller’s Jenkins version. Read the lock with kubectl get configmap jenkins-version-<v>-pluginset-content -n <operator-namespace> -o jsonpath='{.data.plugins\.yaml}', then pin that plugin to B where the pin lives. The message names the source: spec.pluginSpec means the controller’s own spec, and the bundle means the catalog or bundle entry. A bundle composed from catalog items does not recompose on a catalog change by itself. After the catalog re-syncs, delete the bundle’s <name>-content ConfigMap so the operator re-materializes it. The controller reconciles through on the next tick.

  • A run page shows Not started and gives no reason: the record store holds no row for that run. This state means the trigger produced no result. The usual cause is a gateway restart, which loses the pending command a result must correlate with. A cell pod roll does the same to a command the old pod never ran. The stream redelivers that command to the replacement mite, and the next result records the outcome. The redelivered trigger carries the same run identifier and the same intent key, so it builds once. A run whose result is TRIGGER_FAILED names its reason on the page. A build the cell accepted and then lost is not Not started either: the page shows Queued at once, and the recorder marks it interrupted with the reason cell-rolled once the cell is confirmed gone. See Build records.

  • A pipelines page lists no declared job, or a job page shows no definition, no parameters, and no template row: the BFF cannot read the bundle’s content ConfigMap. The page still shows build history, because that comes from the build record store. Read the bundle’s status.contentRef and its namespace, then test the read: kubectl auth can-i get configmaps/<contentRef> -n <bundle-namespace> --as=system:serviceaccount:<release-namespace>:<release>-varroa-bff. An answer of no means the grant is absent. Confirm the operator holds the subject name: kubectl get deployment <release>-varroa-operator -n <release-namespace> -o jsonpath='{.spec.template.spec.containers[0].env}' must show VARROA_BFF_SERVICE_ACCOUNT. A hive install sets no such variable and serves no dashboard, so this entry does not apply there. Then look for the grant objects: kubectl get role,rolebinding <bundle>-content-reader -n <bundle-namespace>. The operator checks the pair on every reconcile. It writes only when a field it owns has drifted, so a steady grant causes no write. A deleted pair returns within one tick. Check the bundle’s status.warnings for an entry starting content read grant. That entry names the API error, which is usually an operator permission that scoped mode does not cover in that namespace.

  • A tenant agent pod logs Failed to connect in a loop, and no build uses it: the cell instance that launched it is gone. A replacement instance holds no node for the pod. A stale pod that is not Running goes within about four minutes. A stale Running pod stays for about 41 minutes, because a resume can still adopt it. Read the refresh time with kubectl get pod <pod> -n <agent-namespace> -o jsonpath='{.metadata.annotations.kubernetes\.jenkins\.io/last-refresh}', which is in epoch milliseconds. The operator logs deleted a tenant agent pod with the pod and the reason. A pod without that annotation, or one that runs as another ServiceAccount, is never deleted. See Agent pod lifetime.

  • A step fails with varroaArtifact: NOT_FOUND: no artifact at path: that path was not recorded. handle.artifacts lists the recorded paths. See Step failures.

For a support case, provide the controller YAML with secrets removed, varroactl describe output, relevant Kubernetes events, and the time range of the failed action. Do not share API keys, passwords, or private bundle credentials.