Blog › ICP guides
Kubernetes engineer on retainer: Pod scheduling, workload controllers, networking, RBAC security, and Helm chart governance on monthly retainer
August 12, 2026 · ~20 min read
A growth-stage SaaS company had run three production incidents in two months from the same Kubernetes cluster. The first: during a traffic spike, a memory-constrained node evicted pods mid-request. The root cause was straightforward after the fact — no PodDisruptionBudget was configured on the affected Deployment, and no ResourceQuota was applied to the namespace, so the scheduler had no reliable guidance on which pods were safe to evict and which were serving live traffic. The second: a DaemonSet log collector consumed 2 CPU cores unexpectedly during a log burst from a high-volume service, throttling application pods on the same node to the point of timeout cascades. The root cause was equally straightforward: the DaemonSet had resources.requests.cpu: "0" and no limits section at all, so the CPU scheduler gave it uncapped access during bursts with no cost accounting. The third: a junior engineer pushed a networking change that opened traffic between the payments namespace and the analytics namespace, which should have been isolated by policy. The root cause: no NetworkPolicy existed in either namespace, so the cluster’s default allow-all posture permitted the traffic without any warning.
A fractional Kubernetes platform engineer on monthly retainer designed the remediation across three sessions. For the eviction problem: PodDisruptionBudget with minAvailable: 2 on each production Deployment, plus a ResourceQuota per namespace enforcing requests.cpu, limits.memory, and count/pods ceilings. For the DaemonSet overconsumption: a LimitRange in every namespace setting default CPU requests and limits, making it structurally impossible to schedule a container with zero CPU request. For the namespace isolation failure: ingress and egress NetworkPolicy rules establishing a default-deny posture in every namespace, followed by explicit allow rules for the services that required cross-namespace communication.
None of the remediation produced a visible feature. The PodDisruptionBudget, ResourceQuota, LimitRange, and NetworkPolicy YAML files were each under 30 lines. The platform engineer’s work was in the hours of cluster audit, namespace budget design, eviction behavior testing, and NetworkPolicy rule verification that preceded those 30-line files — invisible in the YAML, visible only in the absence of the three incident classes that had caused the previous two months of on-call escalations.
Kubernetes engineers, platform engineers, and Kubernetes architects on monthly retainer — fractional Kubernetes platform engineers, Kubernetes consultant retainers, and Kubernetes advisory engagements — do their highest-value work in the Pod scheduling design, ResourceQuota and LimitRange governance, NetworkPolicy isolation, HPA autoscaling configuration, and Helm chart architecture that produces the stable, secure, and well-governed cluster the engineering director reports on to the CTO. This guide covers the Kubernetes platform layer in depth: scheduling, workload controllers, networking, security, observability, and Helm — and how to structure a Kubernetes engineer retainer that makes the hours behind each platform function visible.
Pod scheduling and resource management
Pod scheduling is the process by which the Kubernetes scheduler assigns a Pod to a node. The scheduler filters nodes by hard constraints (the Pod cannot run here) and then scores remaining nodes by soft preferences (the Pod prefers to run here). A platform engineer on retainer spends meaningful hours designing the constraint and preference rules that determine where workloads land — work that is entirely invisible in the absence of the scheduling failures those rules prevent.
NodeSelector and NodeAffinity
nodeSelector is the simplest scheduling constraint: a map of label key-value pairs that the target node must match. It is a hard requirement — no matching node means the Pod remains Pending. NodeAffinity provides the same capability with richer expression and the ability to specify soft preferences in addition to hard requirements:
apiVersion: apps/v1
kind: Deployment
metadata:
name: api-server
spec:
template:
spec:
# Simple hard requirement: node must have this label
# nodeSelector:
# topology.kubernetes.io/zone: us-east-1a
affinity:
nodeAffinity:
# Hard requirement: Pod will NOT schedule if no node matches
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/arch
operator: In
values:
- amd64
- arm64
- key: node-role.kubernetes.io/spot
operator: DoesNotExist # Avoid spot instances for API servers
# Soft preference: prefer nodes in us-east-1a, fall back to us-east-1b
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 80
preference:
matchExpressions:
- key: topology.kubernetes.io/zone
operator: In
values:
- us-east-1a
- weight: 20
preference:
matchExpressions:
- key: topology.kubernetes.io/zone
operator: In
values:
- us-east-1b
The operator field in matchExpressions supports In (label value is in the list), NotIn (label value is not in the list), Exists (label key exists, any value), and DoesNotExist (label key is absent). The Gt and Lt operators work with numeric string values. The IgnoredDuringExecution suffix on both affinity types means that if a running pod’s node later loses the matching label, the pod continues running — eviction does not occur. A hypothetical RequiredDuringExecution form would evict pods when nodes lose labels, but this is not yet implemented in Kubernetes as of 2026.
PodAffinity and PodAntiAffinity
While NodeAffinity constrains scheduling based on node labels, PodAffinity and PodAntiAffinity constrain scheduling based on the labels of other pods already running in the cluster. This enables co-location (place this pod near pods with label app: cache) and spreading (do not place two pods with label app: api on the same node):
affinity:
podAntiAffinity:
# Hard requirement: no two api-server pods on the same node
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app: api-server
topologyKey: kubernetes.io/hostname # "same node" = same hostname
# Soft preference: spread api-server pods across availability zones
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchLabels:
app: api-server
topologyKey: topology.kubernetes.io/zone # "same zone" = same AZ label
podAffinity:
# Soft preference: co-locate api-server near its cache sidecar service
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 50
podAffinityTerm:
labelSelector:
matchLabels:
app: redis-cache
topologyKey: kubernetes.io/hostname
The topologyKey field defines what “co-located” or “spread apart” means. With topologyKey: kubernetes.io/hostname, two pods are considered co-located if they are on the same node. With topologyKey: topology.kubernetes.io/zone, two pods are co-located if they are in the same availability zone. The anti-affinity rule with topologyKey: kubernetes.io/hostname and a hard requirement is the standard pattern for ensuring that a Deployment’s replicas land on distinct nodes, preventing a single node failure from taking down all replicas simultaneously.
Taints and tolerations
Taints allow a node to repel pods; tolerations allow a pod to be scheduled on tainted nodes. The combination enables dedicated node pools for specific workloads (GPU nodes for ML workloads, high-memory nodes for analytics, spot nodes for batch jobs) without requiring every pod to affirmatively request those nodes:
# Taint a node so only GPU workloads are scheduled on it:
kubectl taint nodes gpu-node-1 nvidia.com/gpu=present:NoSchedule
# Pods without a matching toleration will not be scheduled on gpu-node-1.
# Pods with this toleration CAN be scheduled there (but still need NodeAffinity
# to be REQUIRED to land there — toleration is a gate, not a magnet):
spec:
tolerations:
# Exact match: key, operator Equal, value, effect must all match
- key: "nvidia.com/gpu"
operator: Equal
value: "present"
effect: NoSchedule
# Wildcard: tolerate ALL taints with this effect
- operator: Exists
effect: NoSchedule
# NoExecute with tolerationSeconds: pod is evicted after 300s if taint is applied
- key: "node.kubernetes.io/not-ready"
operator: Exists
effect: NoExecute
tolerationSeconds: 300 # Wait 300s before evicting — useful for DaemonSets
- key: "node.kubernetes.io/unreachable"
operator: Exists
effect: NoExecute
tolerationSeconds: 300
The three taint effects differ in behavior: NoSchedule prevents new pods from scheduling (existing pods are unaffected); PreferNoSchedule is a soft version that discourages but does not prevent scheduling; NoExecute is the strongest — it both prevents new scheduling and evicts existing pods that do not tolerate the taint (with tolerationSeconds providing a grace period before eviction). DaemonSets for logging and monitoring routinely tolerate node.kubernetes.io/not-ready:NoExecute and node.kubernetes.io/unreachable:NoExecute with generous tolerationSeconds to ensure they remain scheduled on nodes during transient network issues or brief health-check failures.
Resource requests, limits, and LimitRange
Resource requests and limits are the most operationally consequential fields in a Pod spec. The platform engineer’s recurring retainer work is auditing clusters where application engineers have set these incorrectly — or not at all:
resources:
requests:
cpu: "250m" # Scheduler guarantee: this pod needs 0.25 CPU cores
memory: "256Mi" # Scheduler guarantee: this pod needs 256 MiB memory
limits:
cpu: "1000m" # CFS bandwidth quota: max 1 CPU core (throttled, not killed)
memory: "512Mi" # OOMKill threshold: container killed immediately if exceeded
---
# LimitRange: namespace-level enforcement for all Pods and Containers
apiVersion: v1
kind: LimitRange
metadata:
name: default-limits
namespace: production
spec:
limits:
- type: Container
default:
cpu: "500m"
memory: "512Mi"
defaultRequest:
cpu: "100m"
memory: "128Mi"
min:
cpu: "50m"
memory: "64Mi"
max:
cpu: "4"
memory: "8Gi"
- type: Pod
max:
cpu: "8"
memory: "16Gi"
CPU and memory behave fundamentally differently when a limit is exceeded. CPU throttling: when a container exceeds its CPU limit, the Linux kernel’s CFS bandwidth quota mechanism throttles it — it does not receive CPU cycles for the remainder of the quota period (typically 100ms). The container continues running, but slower. This is why CPU-heavy containers with low limits exhibit high request latency: they are running but waiting for quota replenishment. Memory eviction: when a container exceeds its memory limit, the kernel sends SIGKILL immediately — there is no grace period, no SIGTERM, no opportunity for graceful shutdown. The pod restarts with an OOMKilled status. The LimitRange object prevents the zero-request DaemonSet scenario described in the opening: any container submitted to the namespace without explicit requests receives the defaultRequest values, and any container submitted with requests below the min values is rejected at admission.
ResourceQuota and PriorityClass
apiVersion: v1
kind: ResourceQuota
metadata:
name: production-quota
namespace: production
spec:
hard:
requests.cpu: "20"
requests.memory: "40Gi"
limits.cpu: "60"
limits.memory: "120Gi"
count/pods: "150"
count/services: "30"
count/services.loadbalancers: "5"
count/persistentvolumeclaims: "20"
---
# PriorityClass for guaranteed scheduling of critical services
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: production-critical
value: 1000000
preemptionPolicy: PreemptLowerPriority
globalDefault: false
description: "Production-critical services that preempt lower-priority workloads during resource contention"
---
# Built-in system priority classes (do not modify these values):
# system-cluster-critical: 2000000000 (kube-dns, metrics-server)
# system-node-critical: 2000001000 (kubelet, node-local-dns)
ResourceQuota is enforced at the time a resource is created or updated — not at scheduling time. A pod that exceeds the namespace quota is rejected at admission with a 403 Forbidden; it never enters the Pending state. This makes ResourceQuota the correct mechanism for preventing a runaway deployment from consuming an entire cluster’s resources. PriorityClass with PreemptLowerPriority enables the scheduler to evict lower-priority pods to make room for higher-priority pods when the cluster is under resource pressure. The value field is an integer; higher values are higher priority. Pods without a priorityClassName receive priority 0.
Workload controllers
The workload controllers — Deployment, StatefulSet, DaemonSet, CronJob, Job, and the autoscalers — are the layer above raw Pod scheduling where application teams interact with Kubernetes most directly. The platform engineer’s retainer work in this layer is configuring the controllers for the operational properties that application teams cannot or do not configure themselves: zero-downtime rolling updates, ordered stateful pod initialization, DaemonSet resource governance, and autoscaler behavior tuning.
Deployment: rolling updates and PodDisruptionBudget
apiVersion: apps/v1
kind: Deployment
metadata:
name: api-server
namespace: production
spec:
replicas: 6
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 2 # Spin up 2 extra pods before terminating old ones
maxUnavailable: 0 # Never reduce below 6 available pods during rollout
minReadySeconds: 30 # Pod must be ready for 30s before counted as Available
progressDeadlineSeconds: 600 # Rollout must complete within 10 minutes
selector:
matchLabels:
app: api-server
template:
spec:
terminationGracePeriodSeconds: 60
containers:
- name: api-server
image: registry.example.com/api-server:v2.1.0
readinessProbe:
httpGet:
path: /healthz/ready
port: 8080
initialDelaySeconds: 10
periodSeconds: 5
failureThreshold: 3
---
# PodDisruptionBudget: guarantees minimum availability during voluntary disruptions
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: api-server-pdb
namespace: production
spec:
minAvailable: 4 # At least 4 pods must be available during node drains
selector:
matchLabels:
app: api-server
The maxSurge: 2, maxUnavailable: 0 combination is the zero-downtime rolling update pattern: the deployment controller creates 2 new pods (surge) before terminating any old ones, ensuring the total available pod count never drops below the desired replica count during the rollout. minReadySeconds: 30 prevents a pod that passes its readiness probe once but then fails immediately afterward from being counted as successfully updated — it must remain ready for 30 continuous seconds before the rollout controller considers it healthy and proceeds to update the next batch. The PodDisruptionBudget with minAvailable: 4 constrains voluntary disruptions (node drains for maintenance, cluster upgrades): kubectl drain will not proceed if draining a node would leave fewer than 4 api-server pods available.
StatefulSet: ordered initialization and stable storage
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: postgres
namespace: database
spec:
serviceName: postgres-headless # Required: links pods to a headless Service
replicas: 3
podManagementPolicy: OrderedReady # Default: pod-N waits for pod-(N-1) to be Running+Ready
# podManagementPolicy: Parallel # Use when pods are independent (e.g., worker replicas)
updateStrategy:
type: RollingUpdate
rollingUpdate:
partition: 0 # Set partition: 2 to update only pods with ordinal >= 2 (canary rollout)
selector:
matchLabels:
app: postgres
template:
metadata:
labels:
app: postgres
spec:
containers:
- name: postgres
image: postgres:16
env:
- name: PGDATA
value: /var/lib/postgresql/data/pgdata
volumeMounts:
- name: postgres-data
mountPath: /var/lib/postgresql/data
volumeClaimTemplates:
- metadata:
name: postgres-data
spec:
accessModes: ["ReadWriteOnce"]
storageClassName: gp3-encrypted
resources:
requests:
storage: 100Gi
---
# Headless Service: enables stable DNS for StatefulSet pods
apiVersion: v1
kind: Service
metadata:
name: postgres-headless
namespace: database
spec:
clusterIP: None # Headless: no VIP, DNS returns pod IPs directly
selector:
app: postgres
ports:
- port: 5432
targetPort: 5432
StatefulSet pods have stable, predictable identities: the pod named postgres-0 is always the first replica, postgres-1 the second, and postgres-2 the third. With a headless service, each pod gets a stable DNS name: postgres-0.postgres-headless.database.svc.cluster.local, postgres-1.postgres-headless.database.svc.cluster.local, and so on. Clients (such as a Patroni-managed PostgreSQL cluster primary election agent) can address individual pods by name, which is not possible with a standard Deployment where pods have random names and share a ClusterIP service VIP. The volumeClaimTemplates creates a separate PVC per replica; these PVCs are NOT deleted when the StatefulSet is scaled down, preserving data for recovery when the pod is rescheduled.
DaemonSet: node coverage and resource governance
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: fluent-bit
namespace: logging
spec:
updateStrategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1 # Update one node at a time
selector:
matchLabels:
app: fluent-bit
template:
spec:
tolerations:
# Tolerate not-ready and unreachable so logs are collected even on degraded nodes
- key: "node.kubernetes.io/not-ready"
operator: Exists
effect: NoExecute
tolerationSeconds: 300
- key: "node.kubernetes.io/unreachable"
operator: Exists
effect: NoExecute
tolerationSeconds: 300
# Also tolerate control-plane nodes if logging there is required
- key: "node-role.kubernetes.io/control-plane"
operator: Exists
effect: NoSchedule
containers:
- name: fluent-bit
image: fluent/fluent-bit:3.1
resources:
requests:
cpu: "100m"
memory: "128Mi"
limits:
cpu: "500m" # Capped: prevents log burst from consuming 2+ CPU cores
memory: "256Mi"
volumeMounts:
- name: varlog
mountPath: /var/log
readOnly: true
volumes:
- name: varlog
hostPath:
path: /var/log
The DaemonSet resource governance incident in the opening (2 CPU cores consumed during a log burst) is prevented by the explicit limits.cpu: "500m": even during a log burst that would otherwise demand 2 full cores, the CFS bandwidth quota limits the fluent-bit container to 0.5 cores per 100ms quota period. The application pods on the same node retain their CPU access. The tolerationSeconds: 300 on not-ready and unreachable allows the log collector to stay running for 5 minutes before eviction when a node enters a degraded state, preventing log gaps during transient network partitions or health-check failures.
CronJob and Job completion patterns
apiVersion: batch/v1
kind: CronJob
metadata:
name: nightly-report
namespace: production
spec:
schedule: "0 2 * * *" # 02:00 UTC daily
concurrencyPolicy: Forbid # Skip if previous run is still running
startingDeadlineSeconds: 3600 # Do not start if more than 1 hour overdue
successfulJobsHistoryLimit: 3
failedJobsHistoryLimit: 5
suspend: false # Set true to pause without deleting
jobTemplate:
spec:
backoffLimit: 2 # Retry failed pods up to 2 times
activeDeadlineSeconds: 7200 # Kill job if it runs longer than 2 hours
template:
spec:
restartPolicy: OnFailure
containers:
- name: reporter
image: registry.example.com/reporter:v1.4
---
# Indexed Job: work-queue pattern where each pod processes a specific shard
apiVersion: batch/v1
kind: Job
metadata:
name: data-migration
spec:
completions: 10 # 10 total completions needed
parallelism: 3 # Run at most 3 pods simultaneously
completionMode: Indexed # Each pod gets a unique index (JOB_COMPLETION_INDEX env var)
backoffLimit: 4
template:
spec:
restartPolicy: Never
containers:
- name: migrator
image: registry.example.com/migrator:v2
env:
- name: JOB_COMPLETION_INDEX
valueFrom:
fieldRef:
fieldPath: metadata.annotations['batch.kubernetes.io/job-completion-index']
concurrencyPolicy: Forbid is the safe default for CronJobs that perform database operations or generate reports: if the previous run is still executing when the next scheduled trigger fires, the new run is skipped entirely rather than running concurrently and producing duplicate output or database contention. concurrencyPolicy: Replace terminates the previous run and starts a new one — useful for jobs where freshness matters more than completion of the previous run. startingDeadlineSeconds: 3600 prevents a large backlog from forming if the CronJob controller misses multiple scheduled triggers (for example, during a control plane outage): the job is only started if the scheduled time was less than one hour ago.
HorizontalPodAutoscaler v2 and VerticalPodAutoscaler
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: api-server-hpa
namespace: production
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: api-server
minReplicas: 3
maxReplicas: 20
metrics:
# Scale on CPU utilization relative to requests
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70 # Scale out when average CPU > 70% of request
# Scale on memory utilization
- type: Resource
resource:
name: memory
target:
type: Utilization
averageUtilization: 80
# Scale on custom Prometheus metric via prometheus-adapter
- type: External
external:
metric:
name: http_requests_per_second
selector:
matchLabels:
service: api-server
target:
type: AverageValue
averageValue: "500" # Scale out when >500 req/s per pod
behavior:
scaleUp:
stabilizationWindowSeconds: 60 # Wait 60s before scaling up again
policies:
- type: Pods
value: 4
periodSeconds: 60 # Add at most 4 pods per 60 seconds
scaleDown:
stabilizationWindowSeconds: 300 # Wait 5 minutes before scaling down
policies:
- type: Percent
value: 10
periodSeconds: 60 # Remove at most 10% of pods per 60 seconds
---
# VerticalPodAutoscaler: recommendation-only mode (does not restart pods)
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
name: api-server-vpa
namespace: production
spec:
targetRef:
apiVersion: apps/v1
kind: Deployment
name: api-server
updatePolicy:
updateMode: "Off" # "Off" = recommendations only, no automatic restarts
# "Auto" = automatically adjusts requests and restarts pods
resourcePolicy:
containerPolicies:
- containerName: api-server
minAllowed:
cpu: 100m
memory: 128Mi
maxAllowed:
cpu: 4
memory: 4Gi
The behavior.scaleDown.stabilizationWindowSeconds: 300 is one of the most important HPA tuning parameters. Without it, the HPA evaluates the current metric every 15 seconds and can scale down aggressively after a traffic spike, removing pods just before a second spike arrives and causing a scale-thrash cycle. The 5-minute stabilization window requires that the metric remain below the scale-down threshold for 5 consecutive minutes before pods are removed, absorbing the tail of a traffic spike. The VPA in updateMode: "Off" is the platform engineer’s observability tool during the initial retainer engagement: it watches historical resource usage and generates recommendations visible in kubectl describe vpa api-server-vpa without actually restarting pods. After reviewing the recommendations against observed behavior, the platform engineer updates the Deployment’s requests to align with actual usage.
Kubernetes networking
Kubernetes networking is the layer where the most subtle production failures and the most consequential security failures occur. The platform engineer on retainer designs the Service types, Ingress configuration, and NetworkPolicy rules that determine how traffic flows into, within, and out of the cluster — and which traffic is prohibited.
Service types
# ClusterIP: stable VIP, kube-proxy iptables/IPVS rules route to pod IPs
apiVersion: v1
kind: Service
metadata:
name: api-server
namespace: production
spec:
type: ClusterIP
selector:
app: api-server
ports:
- port: 80
targetPort: 8080
protocol: TCP
---
# LoadBalancer: provisions a cloud load balancer (NLB/ALB via AWS LBC, GCP LB, etc.)
apiVersion: v1
kind: Service
metadata:
name: api-server-lb
namespace: production
annotations:
service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
service.beta.kubernetes.io/aws-load-balancer-internal: "true"
spec:
type: LoadBalancer
# spec.loadBalancerIP: "10.0.0.100" # Request specific IP (cloud-provider-dependent)
externalTrafficPolicy: Local # Preserve client source IP; only route to local pods
selector:
app: api-server
ports:
- port: 443
targetPort: 8443
---
# Headless Service: clusterIP: None, DNS returns individual pod IPs (StatefulSet use)
apiVersion: v1
kind: Service
metadata:
name: postgres-headless
namespace: database
spec:
clusterIP: None
selector:
app: postgres
ports:
- port: 5432
---
# ExternalName: CNAME alias to external service (no selector, no endpoint tracking)
apiVersion: v1
kind: Service
metadata:
name: legacy-api
namespace: production
spec:
type: ExternalName
externalName: legacy.internal.example.com
externalTrafficPolicy: Local on a LoadBalancer or NodePort service instructs kube-proxy to only forward traffic to pods on the local node rather than load-balancing across all cluster nodes. This preserves the original client IP in the X-Forwarded-For header and eliminates the extra network hop for cross-node forwarding — but it means nodes with no local pods will receive traffic that is then dropped, making load distribution across pods uneven unless an external load balancer is configured to perform health checks against the NodePort and route only to nodes with healthy local pods.
Ingress with nginx and cert-manager TLS
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: api-ingress
namespace: production
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /$2
nginx.ingress.kubernetes.io/proxy-body-size: "50m"
nginx.ingress.kubernetes.io/proxy-read-timeout: "60"
nginx.ingress.kubernetes.io/ssl-redirect: "true"
cert-manager.io/cluster-issuer: "letsencrypt-prod"
spec:
ingressClassName: nginx
tls:
- hosts:
- api.example.com
secretName: api-example-com-tls # cert-manager populates this Secret
rules:
- host: api.example.com
http:
paths:
- path: /v2(/|$)(.*)
pathType: Prefix
backend:
service:
name: api-server-v2
port:
number: 80
- path: /v1(/|$)(.*)
pathType: Prefix
backend:
service:
name: api-server-v1
port:
number: 80
- path: /
pathType: Prefix
backend:
service:
name: api-server-v2
port:
number: 80
---
# cert-manager Certificate resource (created automatically by the Ingress annotation,
# or explicitly for more control):
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: api-example-com
namespace: production
spec:
secretName: api-example-com-tls
issuerRef:
name: letsencrypt-prod
kind: ClusterIssuer
dnsNames:
- api.example.com
- www.api.example.com
duration: 2160h # 90 days
renewBefore: 360h # Renew 15 days before expiryNetworkPolicy: namespace isolation and default-deny
NetworkPolicy is the Kubernetes mechanism for enforcing network segmentation at the pod level. It is the platform layer that would have prevented the namespace isolation failure in the opening scenario. NetworkPolicy is only enforced if the cluster’s CNI plugin supports it (Calico, Cilium, Weave, and Amazon VPC CNI with Network Policy add-on all support it; Flannel without a network policy plugin does not):
# Pattern 1: Default deny-all (apply first in every namespace)
# Empty podSelector matches ALL pods in namespace
# Empty policyTypes with no ingress/egress rules = deny all traffic
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-all
namespace: payments
spec:
podSelector: {} # Applies to ALL pods in namespace
policyTypes:
- Ingress
- Egress
---
# Pattern 2: Allow traffic within the same namespace
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-same-namespace
namespace: payments
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
ingress:
- from:
- podSelector: {} # Any pod in this namespace
egress:
- to:
- podSelector: {} # Any pod in this namespace
---
# Pattern 3: Allow ingress from the nginx ingress controller namespace
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-from-ingress-controller
namespace: payments
spec:
podSelector:
matchLabels:
app: payments-api
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: ingress-nginx
podSelector:
matchLabels:
app.kubernetes.io/name: ingress-nginx
ports:
- protocol: TCP
port: 8080
---
# Pattern 4: Allow specific cross-namespace traffic (payments -> external DNS only)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-dns-egress
namespace: payments
spec:
podSelector: {}
policyTypes:
- Egress
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
The default-deny-all pattern applied first to the payments namespace means that the junior engineer’s networking change in the opening scenario — deploying a service that accepted connections from the analytics namespace — would have been silently blocked by the existing NetworkPolicy before any traffic could flow. The namespaceSelector and podSelector used together in the same from entry (Pattern 3) create an AND condition: the source pod must be in the matching namespace AND have the matching pod labels. Placed in separate array elements under from, they would create an OR condition: the source can be any pod in the matching namespace OR any pod with the matching labels in any namespace — a common and consequential distinction that the platform engineer validates during NetworkPolicy authoring.
CoreDNS and service discovery
# Service DNS: <service>.<namespace>.svc.cluster.local
# Pod DNS: <pod-ip-dashes>.<namespace>.pod.cluster.local
# ndots:5 means any name with fewer than 5 dots is tried as relative first,
# so "api-server" resolves as:
# api-server.production.svc.cluster.local (found — returns ClusterIP)
# while "api.example.com" (2 dots) is tried as:
# api.example.com.production.svc.cluster.local (fails)
# api.example.com.svc.cluster.local (fails)
# api.example.com.cluster.local (fails)
# api.example.com. (succeeds — external DNS)
# This causes up to 3 wasted DNS queries per external hostname lookup.
# Fix: use FQDNs with trailing dots in configs, or set ndots:2 for workloads
# that don't need short-name in-cluster resolution.
---
# CoreDNS ConfigMap: add a stub zone for an on-premises service
apiVersion: v1
kind: ConfigMap
metadata:
name: coredns
namespace: kube-system
data:
Corefile: |
.:53 {
errors
health {
lameduck 5s
}
ready
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
fallthrough in-addr.arpa ip6.arpa
ttl 30
}
# Stub zone: forward internal.example.com queries to on-prem DNS
internal.example.com:53 {
forward . 10.0.0.2 10.0.0.3
}
prometheus :9153
forward . /etc/resolv.conf {
max_concurrent 1000
}
cache 30
loop
reload
loadbalance
}Kubernetes security
Kubernetes security at the platform layer is principally about RBAC design, service account token governance, and Pod security admission controls that constrain what workloads can do at the kernel and network level. The platform engineer designs these controls so that a compromised application container cannot escalate privilege, read secrets from other namespaces, or access the Kubernetes API with more permissions than it needs.
RBAC: Roles, ClusterRoles, and least-privilege design
# Role: namespace-scoped. Only grants permissions within the payments namespace.
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: payments-app-reader
namespace: payments
rules:
- apiGroups: [""]
resources: ["configmaps", "secrets"]
verbs: ["get", "list"]
- apiGroups: ["apps"]
resources: ["deployments"]
verbs: ["get", "list", "watch"]
---
# RoleBinding: grants the Role to a ServiceAccount in the same namespace
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: payments-app-reader-binding
namespace: payments
subjects:
- kind: ServiceAccount
name: payments-app
namespace: payments
roleRef:
kind: Role
name: payments-app-reader
apiGroup: rbac.authorization.k8s.io
---
# ClusterRole: cluster-scoped permissions (or granted namespace-scoped via RoleBinding)
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: node-reader
rules:
- apiGroups: [""]
resources: ["nodes"]
verbs: ["get", "list", "watch"]
---
# ClusterRoleBinding: grants a ClusterRole cluster-wide
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: node-reader-binding
subjects:
- kind: Group
name: platform-engineers
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: ClusterRole
name: node-reader
apiGroup: rbac.authorization.k8s.io
The platform engineer’s RBAC design work follows three principles. First, use Role and RoleBinding in preference to ClusterRole and ClusterRoleBinding whenever the permission only needs to apply within a specific namespace — a ClusterRole granted to an application service account gives that account permission to read secrets across every namespace in the cluster. Second, design service account roles by enumerating only the resources and verbs the application actually needs (an application that reads ConfigMaps and Secrets should not have update or create permissions). Third, audit with kubectl auth can-i --list --as=system:serviceaccount:payments:payments-app to verify what permissions each service account actually holds after configuration.
ServiceAccount token projection and automount controls
# ServiceAccount: disable auto-mounting for accounts used by web-facing pods
apiVersion: v1
kind: ServiceAccount
metadata:
name: api-server
namespace: production
automountServiceAccountToken: false # Default mount is disabled at SA level
---
# Pod spec: explicitly mount a projected token only for pods that need API access
spec:
serviceAccountName: api-server
automountServiceAccountToken: false # Pod-level override (takes precedence over SA)
volumes:
- name: kube-api-token
projected:
sources:
- serviceAccountToken:
path: token
expirationSeconds: 3600 # Short-lived token; automatically rotated
audience: "https://kubernetes.default.svc"
containers:
- name: api-server
volumeMounts:
- name: kube-api-token
mountPath: /var/run/secrets/kubernetes.io/serviceaccount
readOnly: truePodSecurityAdmission and securityContext
# Apply PodSecurityAdmission labels to a namespace:
# "enforce" rejects pods that violate the policy
# "warn" allows pods but emits a warning
# "audit" allows pods but logs a violation
apiVersion: v1
kind: Namespace
metadata:
name: production
labels:
pod-security.kubernetes.io/enforce: restricted
pod-security.kubernetes.io/enforce-version: latest
pod-security.kubernetes.io/warn: restricted
pod-security.kubernetes.io/warn-version: latest
---
# Pod spec that satisfies the "restricted" PodSecurity profile:
spec:
securityContext:
runAsNonRoot: true
runAsUser: 1000
runAsGroup: 3000
fsGroup: 2000
seccompProfile:
type: RuntimeDefault # Required by restricted profile
containers:
- name: api-server
image: registry.example.com/api-server:v2.1.0
securityContext:
allowPrivilegeEscalation: false # Required by restricted profile
readOnlyRootFilesystem: true # Required by restricted profile
runAsNonRoot: true
capabilities:
drop:
- "ALL" # Drop all Linux capabilities
add:
- "NET_BIND_SERVICE" # Re-add only if port < 1024 binding needed
volumeMounts:
- name: tmp
mountPath: /tmp # Writable tmp volume for readOnlyRootFilesystem
- name: cache
mountPath: /app/cache
volumes:
- name: tmp
emptyDir: {}
- name: cache
emptyDir: {}
The restricted PodSecurityAdmission profile (replacing the deprecated PodSecurityPolicy) enforces the security baseline that prevents the most common container privilege escalation paths: runAsNonRoot: true prevents containers from running as root even if the image entrypoint is root by default; readOnlyRootFilesystem: true prevents the container from writing to its own filesystem (attackers cannot drop persistent tooling); allowPrivilegeEscalation: false prevents setuid binaries from escalating to root; capabilities.drop: ["ALL"] removes all Linux capabilities from the container (including NET_ADMIN, SYS_PTRACE, and SYS_MODULE that would allow network configuration, process inspection, and kernel module loading). The platform engineer validates that each workload can run under the restricted profile and provides emptyDir volumes for the paths that the application legitimately needs to write to (temporary files, cache directories) when readOnlyRootFilesystem is enabled.
Service mesh mTLS with Istio
# PeerAuthentication: require mTLS for all pods in the namespace
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
name: default
namespace: production
spec:
mtls:
mode: STRICT # All pod-to-pod traffic must use mTLS; plaintext rejected
---
# AuthorizationPolicy: restrict which services can call the payments API
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: payments-api-authz
namespace: payments
spec:
selector:
matchLabels:
app: payments-api
action: ALLOW
rules:
- from:
- source:
principals:
- "cluster.local/ns/production/sa/order-service"
- "cluster.local/ns/production/sa/checkout-service"
to:
- operation:
methods: ["POST"]
paths: ["/v1/charge", "/v1/refund"]Kubernetes observability
Kubernetes observability at the platform layer means configuring the components that make cluster resource usage, workload health, and scheduling behavior visible: Prometheus scraping via ServiceMonitors, metrics-server for HPA and kubectl top, and structured pod probes that distinguish slow-starting containers from unhealthy ones.
Prometheus Operator ServiceMonitor and PrometheusRule
# ServiceMonitor: tells Prometheus Operator which services to scrape
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: api-server-monitor
namespace: production
labels:
release: prometheus # Must match Prometheus CR's serviceMonitorSelector
spec:
selector:
matchLabels:
app: api-server
monitoring: enabled
endpoints:
- port: metrics # Named port on the Service
path: /metrics
interval: 30s
scrapeTimeout: 10s
scheme: http
---
# PrometheusRule: alerting rules for the api-server workload
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: api-server-alerts
namespace: production
labels:
release: prometheus
spec:
groups:
- name: api-server.rules
interval: 30s
rules:
- alert: ApiServerHighErrorRate
expr: |
rate(http_requests_total{job="api-server", status=~"5.."}[5m])
/ rate(http_requests_total{job="api-server"}[5m]) > 0.05
for: 2m
labels:
severity: critical
annotations:
summary: "API server error rate above 5% for 2 minutes"
description: "Error rate is {{ $value | humanizePercentage }}"
- alert: ApiServerHPANearMaxReplicas
expr: |
kube_horizontalpodautoscaler_status_current_replicas{
horizontalpodautoscaler="api-server-hpa"
}
/
kube_horizontalpodautoscaler_spec_max_replicas{
horizontalpodautoscaler="api-server-hpa"
} > 0.9
for: 5m
labels:
severity: warning
annotations:
summary: "HPA is at >90% of max replicas — scale ceiling approaching"Pod probes: readiness, liveness, and startup
containers:
- name: api-server
image: registry.example.com/api-server:v2.1.0
ports:
- containerPort: 8080
name: http
# startupProbe: allows slow-starting containers time to initialize
# Before startupProbe succeeds, liveness and readiness probes are suspended
startupProbe:
httpGet:
path: /healthz/startup
port: 8080
failureThreshold: 30 # 30 failures × 10s period = 5 minutes max startup time
periodSeconds: 10
successThreshold: 1
# readinessProbe: controls whether pod receives traffic from Services
# Failing readiness removes pod from Service endpoints (no traffic)
# Passing readiness adds pod back to Service endpoints
readinessProbe:
httpGet:
path: /healthz/ready
port: 8080
initialDelaySeconds: 5
periodSeconds: 5
failureThreshold: 3 # Remove from rotation after 3 failures (15s)
successThreshold: 1 # Restore to rotation after 1 success
# livenessProbe: controls whether pod is restarted
# Failing liveness causes kubelet to kill and restart the container
livenessProbe:
httpGet:
path: /healthz/live
port: 8080
initialDelaySeconds: 15
periodSeconds: 10
failureThreshold: 5 # Restart after 5 consecutive failures (50s)
successThreshold: 1
# exec probe for a gRPC service without an HTTP health endpoint:
# livenessProbe:
# exec:
# command: ["/bin/grpc_health_probe", "-addr=:9090"]
# initialDelaySeconds: 10
# periodSeconds: 10
The three probe types serve distinct functions and are commonly misconfigured by application teams. startupProbe is the platform engineer’s solution to a recurring problem: the failureThreshold on livenessProbe is set conservatively (5 failures × 10 seconds = 50 seconds before restart) to avoid restarting healthy pods under load, but a slow-starting container (a JVM warming up, a Python application initializing a large model) takes 3 minutes to become ready. Without startupProbe, the liveness probe fires before the application is ready and restarts it repeatedly, preventing it from ever completing initialization. With startupProbe configured with a high failureThreshold, the liveness and readiness probes are suspended until startup succeeds, then take over normal health checking. The readinessProbe and livenessProbe must target different health check semantics: readiness reflects whether the pod is ready to serve traffic (database connection established, warmup complete, feature flags loaded); liveness reflects whether the pod process is alive and not deadlocked (if liveness fails, a restart is the correct remediation).
Kubernetes Events and resource metrics
# Inspect recent cluster events sorted by timestamp:
kubectl get events -n production --sort-by=lastTimestamp
# Common event reasons to monitor:
# BackOff — container is crash-looping; check logs
# OOMKilled — container exceeded memory limit; increase limit or fix leak
# Evicted — pod evicted by kubelet; check node memory pressure
# FailedScheduling — no node satisfies constraints; check affinity, taints, quota
# Pulling — image pull in progress (long pulls indicate registry issues)
# Failed — image pull failed; check image name and pull secret
# Resource usage from metrics-server:
kubectl top pods -n production --containers
kubectl top nodes
# metrics-server resource requirements (small clusters):
# resources:
# requests:
# cpu: 100m
# memory: 200Mi
# limits:
# cpu: 500m
# memory: 500MiHelm chart authoring
Helm chart authoring is the platform layer that governs how applications are packaged, configured, and deployed across environments. The platform engineer on retainer spends meaningful hours designing chart structures that make per-environment parameterization explicit, mandatory fields validated at render time, and multi-environment deployments repeatable.
values.yaml: required fields, defaults, and toYaml
# values.yaml: default values with documentation
replicaCount: 3
image:
repository: registry.example.com/api-server
tag: "" # Must be provided at deploy time; empty default forces explicit set
pullPolicy: IfNotPresent
service:
type: ClusterIP
port: 80
targetPort: 8080
resources:
requests:
cpu: "250m"
memory: "256Mi"
limits:
cpu: "1000m"
memory: "512Mi"
nodeSelector: {}
tolerations: []
affinity: {}
podDisruptionBudget:
enabled: true
minAvailable: 2
hpa:
enabled: false
minReplicas: 3
maxReplicas: 20
targetCPUUtilizationPercentage: 70
---
# templates/deployment.yaml: use required and default functions
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "api-server.fullname" . }}
labels:
{{- include "api-server.labels" . | nindent 4 }}
spec:
replicas: {{ .Values.replicaCount }}
template:
spec:
containers:
- name: {{ .Chart.Name }}
image: "{{ .Values.image.repository }}:{{ required "image.tag is required" .Values.image.tag }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
resources:
{{- toYaml .Values.resources | nindent 12 }}
{{- with .Values.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.tolerations }}
tolerations:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.affinity }}
affinity:
{{- toYaml . | nindent 8 }}
{{- end }}_helpers.tpl: consistent naming across all resources
{{/*
templates/_helpers.tpl
*/}}
{{/* Expand the name of the chart */}}
{{- define "api-server.name" -}}
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }}
{{- end }}
{{/* Create a default fully qualified app name, capped at 63 characters */}}
{{- define "api-server.fullname" -}}
{{- if .Values.fullnameOverride }}
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- $name := default .Chart.Name .Values.nameOverride }}
{{- if contains $name .Release.Name }}
{{- .Release.Name | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }}
{{- end }}
{{- end }}
{{- end }}
{{/* Common labels applied to all resources */}}
{{- define "api-server.labels" -}}
helm.sh/chart: {{ include "api-server.chart" . }}
{{ include "api-server.selectorLabels" . }}
{{- if .Chart.AppVersion }}
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
{{- end }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
{{- end }}
{{/* Selector labels (must be stable across upgrades) */}}
{{- define "api-server.selectorLabels" -}}
app.kubernetes.io/name: {{ include "api-server.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
{{- end }}
{{/* Create chart name and version label */}}
{{- define "api-server.chart" -}}
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }}
{{- end }}
{{/* ServiceAccount name */}}
{{- define "api-server.serviceAccountName" -}}
{{- if .Values.serviceAccount.create }}
{{- default (include "api-server.fullname" .) .Values.serviceAccount.name }}
{{- else }}
{{- default "default" .Values.serviceAccount.name }}
{{- end }}
{{- end }}Chart dependencies, conditional resources, and chart testing
# Chart.yaml: declare chart dependencies
apiVersion: v2
name: api-server
description: API server with optional Redis cache and PostgreSQL
type: application
version: 1.4.0
appVersion: "2.1.0"
dependencies:
- name: redis
version: "19.x.x"
repository: "https://charts.bitnami.com/bitnami"
condition: redis.enabled # Only installed if redis.enabled: true in values
- name: postgresql
version: "14.x.x"
repository: "https://charts.bitnami.com/bitnami"
condition: postgresql.enabled
---
# Update and vendor chart dependencies:
# helm dependency update ./api-server
# Creates charts/ directory with .tgz archives of dependencies
---
# templates/hpa.yaml: conditional resource (only rendered if hpa.enabled)
{{- if .Values.hpa.enabled }}
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: {{ include "api-server.fullname" . }}
labels:
{{- include "api-server.labels" . | nindent 4 }}
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: {{ include "api-server.fullname" . }}
minReplicas: {{ .Values.hpa.minReplicas }}
maxReplicas: {{ .Values.hpa.maxReplicas }}
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: {{ .Values.hpa.targetCPUUtilizationPercentage }}
{{- end }}
---
# Chart testing with chart-testing (ct):
# ct lint --charts charts/api-server # Validates Chart.yaml, values, and rendered templates
# ct install --charts charts/api-server # Installs into temp namespace, runs tests, then deletes
---
# Deploy with atomic rollback on failure:
helm upgrade --install api-server ./api-server \
--namespace production \
--create-namespace \
--values ./values/production.yaml \
--set image.tag=${CI_COMMIT_SHA} \
--atomic \
--timeout 5m \
--history-max 10
# Rollback to previous release:
helm rollback api-server 0 --namespace production # 0 = previous release
# Inspect release history:
helm history api-server --namespace production
The --atomic flag on helm upgrade --install is the platform engineer’s deployment safety net: if any resource in the new release fails to reach a ready state within the timeout window (5 minutes in this example), Helm automatically rolls back to the previous release and exits non-zero. This prevents a partial upgrade from leaving the cluster in an inconsistent state where some pods are running the new version and others the old. The --history-max 10 flag limits the number of release history entries Kubernetes stores as Secrets, preventing unbounded Secret accumulation in clusters with frequent deployments.
HourTab for Kubernetes platform engineer retainers
Kubernetes platform engineer retainer work produces visible infrastructure outcomes: zero pod evictions during traffic spikes after PodDisruptionBudget and ResourceQuota governance is in place; namespace isolation enforced after NetworkPolicy default-deny posture is established; HPA scaling that stabilizes correctly after scaleDown.stabilizationWindowSeconds tuning prevents oscillation; and Helm chart deployments that roll back atomically when a new version fails readiness checks. Each outcome is visible. The hours that produced it are not.
The ResourceQuota and LimitRange design that preceded the eviction prevention: auditing every Deployment and DaemonSet in the namespace to identify zero-request containers and unset limits, calculating namespace budget ceilings that allow peak-traffic burst without starving adjacent workloads, writing and testing the quota and limit range configuration, and verifying enforcement behavior by submitting quota-exceeding test pods at the admission webhook. This work produced a 30-line ResourceQuota YAML and a 25-line LimitRange YAML. The hours were 10 to 16, depending on namespace complexity and the number of workloads requiring remediation.
The NetworkPolicy audit that preceded the namespace isolation enforcement: inventorying every existing inter-namespace traffic flow that was legitimate and needed an explicit allow rule, writing the default-deny-all policy, writing the allow-same-namespace and allow-from-ingress-controller policies, testing each allow rule with ephemeral curl debug pods to verify traffic flows as intended, and verifying that previously unauthorized traffic paths (like analytics to payments) are now silently dropped. This produced four to eight NetworkPolicy manifests. The hours were 12 to 20, depending on the number of legitimate cross-namespace traffic flows and the complexity of the ingress controller configuration.
The HPA behavior tuning that preceded the stable scale-down: monitoring the HPA’s scaling events over several traffic cycles to observe oscillation patterns, calibrating stabilizationWindowSeconds and scaleDown.policies to match the traffic pattern’s post-spike tail, configuring external Prometheus metrics via prometheus-adapter for queue-depth-driven autoscaling, and load-testing the tuned configuration under synthetic traffic to verify stable behavior. This produced changes to an existing HPA manifest. The hours were 6 to 14, depending on the number of metrics and the stability of the traffic pattern.
The Helm chart refactoring that preceded the multi-environment deployment standardization: converting per-environment ad hoc kubectl apply YAML directories into a parameterized Helm chart with values schema, _helpers.tpl template libraries for consistent naming, conditional resources controlled by values flags, dependency declaration for Redis and PostgreSQL subcharts, and ct lint and ct install chart testing in CI. This produced the chart directory. The hours were 12 to 25, depending on the number of resources being parameterized and the number of environments requiring distinct values files.
Monthly retainer amounts for Kubernetes platform engineer advisory and architecture consulting typically range from $6,000 to $12,000 per month for platform advisory and architecture review, increasing to $13,000 to $30,000 per month for full-time fractional platform engineering covering hands-on cluster configuration, workload migration, and multi-cluster governance.
HourTab gives Kubernetes platform engineers and Kubernetes architects a retainer dashboard their engineering directors can bookmark without creating an account: the month’s committed hours, the hours consumed, and the work log entries that connect each block to the specific platform function performed. When the VP Engineering can see that 14 of the month’s 40 retainer hours went to NetworkPolicy authoring and testing across three namespaces and 10 went to HPA behavior tuning and load testing for the two highest-traffic Deployments, the retainer renewal conversation is grounded in the actual distribution of Kubernetes platform advisory work rather than an abstract sense of whether the cluster stability investment produced value.
The retainer model fits Kubernetes platform engineering because Kubernetes clusters are living systems: every new workload deployed by an application team may arrive without resource requests or limits and require LimitRange remediation; every new namespace requires a NetworkPolicy default-deny posture and a ResourceQuota budget; every Kubernetes version upgrade introduces deprecated API versions that Helm charts must be updated to use; and every new cloud provider feature (NLB annotations, storage class configurations, node pool taint propagation) requires platform-layer evaluation before application teams adopt it. A monthly hour commitment provides the Kubernetes platform engineer’s sustained availability across the full cluster maintenance and evolution calendar.
For Kubernetes consultants documenting retainer work, sharing a live hours dashboard replaces the weekly status email: the client sees the current month’s hour consumption and the work log entries that narrate what each block of Kubernetes platform advisory hours accomplished.
Frequently asked questions
What does a Kubernetes engineer on retainer typically do?
A Kubernetes engineer or platform engineer on monthly retainer provides ongoing Kubernetes platform advisory and hands-on configuration across five principal service areas. Pod scheduling and resource governance: designing NodeAffinity and PodAntiAffinity rules that spread workloads across failure domains, configuring LimitRange and ResourceQuota per namespace to enforce resource boundaries and prevent CPU starvation and memory OOMKills, setting PriorityClass for critical workloads, and implementing PodDisruptionBudgets that guarantee minimum availability during voluntary disruptions. Workload controller design: configuring Deployment rolling update strategies for zero-downtime deploys, designing StatefulSet pod management for ordered initialization of clustered databases, tuning HorizontalPodAutoscaler behavior including scaleDown stabilization windows and custom Prometheus metrics, and auditing DaemonSet resource limits to prevent log collectors from consuming unbounded CPU during log bursts. Networking and NetworkPolicy: writing ingress and egress NetworkPolicy rules that enforce namespace isolation, configuring Ingress controllers with TLS termination via cert-manager, and tuning CoreDNS for custom stub zones. RBAC and security: designing Role and ClusterRole hierarchies on the principle of least privilege, disabling automountServiceAccountToken on Pods that do not require API server access, configuring PodSecurityAdmission namespace labels for the restricted profile, and auditing securityContext settings for runAsNonRoot, readOnlyRootFilesystem, and capability dropping. Helm chart authoring and governance: designing chart values schemas with required field validation, maintaining _helpers.tpl template libraries, configuring chart dependencies and condition flags, and establishing helm upgrade --atomic --timeout deployment governance across environments.
What Kubernetes work is most commonly underlogged in a retainer?
The most systematically underlogged categories are: ResourceQuota and LimitRange design (defining per-namespace CPU and memory budgets, enforcing minimum request values to prevent containers with zero requests from monopolizing node resources, calibrating quota ceilings that allow legitimate burst — typically 8 to 16 hours per cluster namespace audit invisible in the YAML files); NetworkPolicy authoring (designing ingress and egress rules for namespace isolation, writing the default-deny-all and allow-same-namespace policies, testing traffic flow with ephemeral debug pods — typically 10 to 20 hours per cluster security posture review invisible in the absence of unauthorized cross-namespace traffic); HPA behavior tuning (calibrating scaleDown.stabilizationWindowSeconds to prevent oscillation, configuring external Prometheus metrics for queue-depth-driven autoscaling, testing scale behavior under synthetic load — typically 6 to 14 hours per workload invisible in the stable scaling curve); and Helm chart refactoring (converting per-environment ad hoc YAML overlays into a parameterized chart with values schema validation, _helpers.tpl template libraries, and dependency condition flags — typically 12 to 25 hours per chart invisible in the values.yaml diff). Detailed work log entries that capture the specific cluster configurations applied and the incidents they prevent connect the invisible Kubernetes platform investment to its concrete outcomes.
What should a Kubernetes engineer retainer agreement include?
Kubernetes engineer retainer agreements should specify: scope boundary between cluster platform advisory, security audit, workload optimization, and feature-specific engineering (RBAC design and NetworkPolicy authoring produce no deployable application artifact — define these explicitly as in-scope platform functions with their own hour allocation); cluster access level required (read-only kubeconfig for audit and review; write access to specific namespaces for configuration changes; cluster-admin for LimitRange, ResourceQuota, and PodSecurityAdmission configuration); namespace governance scope (which namespaces the platform engineer owns the ResourceQuota and LimitRange configuration for, versus namespaces owned by application teams); Helm chart ownership (whether the retainer covers chart authoring, chart review only, or values governance for application teams authoring their own charts); security posture scope (whether PodSecurityAdmission enforcement level changes, NetworkPolicy authoring, and RBAC audit are included, and whether Istio mTLS or service mesh configuration is in scope); and a shared work log documenting each scheduling configuration session, ResourceQuota and LimitRange design sprint, NetworkPolicy authoring engagement, HPA tuning session, and Helm chart refactoring engagement. Monthly retainer amounts for Kubernetes platform engineer advisory typically range from $6,000 to $12,000 per month for platform advisory and architecture review, increasing to $13,000 to $30,000 per month for full-time fractional platform engineering.
What are typical retainer rates for Kubernetes engineers and platform engineers?
Kubernetes engineers command rates that reflect the platform’s operational depth, the scheduling and networking expertise required for production-grade cluster configuration, and the RBAC and security design skill that high-value retainer work demands. Entry-level Kubernetes engineers with 1 to 3 years of experience, CKA certification, and basic cluster operations skill typically bill $85 to $145 per hour, with monthly retainers running 10 to 20 hours for cluster health review and advisory work. Mid-level Kubernetes platform engineers with 3 to 8 years of experience, expertise in RBAC design, HPA configuration, Helm chart authoring, and NetworkPolicy design, typically bill $140 to $250 per hour, with monthly retainers running 15 to 35 hours. Senior Kubernetes architects and platform engineers with 8 to 14 years of experience, expertise in multi-cluster federation, service mesh (Istio), advanced scheduling, and enterprise security posture, typically bill $200 to $390 per hour, with monthly retainers running 20 to 50 hours. Kubernetes consulting firms and platform engineering consultancies typically bill $170 to $310 per hour. Monthly retainer amounts range from $6,000 to $12,000 per month for platform advisory and architecture review retainers, increasing to $13,000 to $30,000 per month for full-time fractional platform engineering engagements.
How should Kubernetes engineer retainer hours be logged?
Kubernetes retainer work log entries should capture the platform advisory category (Pod scheduling design, ResourceQuota and LimitRange governance, NetworkPolicy authoring, HPA tuning, Helm chart refactoring, RBAC audit, observability configuration), the specific namespace or workload, the task performed, and the finding or deliverable. Example: “ResourceQuota and LimitRange Governance — production namespace. Task: design and apply namespace-level resource governance to prevent memory OOMKill and CPU starvation. Work: audited all Deployments and DaemonSets in production namespace — found 6 containers with cpu requests: 0, 3 containers with no memory limits, and 2 DaemonSets with no resource section — 2 hours; designed LimitRange with default CPU request 100m, default CPU limit 500m, default memory request 128Mi, default memory limit 512Mi, min CPU 50m, min memory 64Mi — applied and verified with kubectl describe limitrange — 3 hours; designed ResourceQuota with requests.cpu: 8, limits.cpu: 20, requests.memory: 16Gi, limits.memory: 48Gi, count/pods: 100 — verified enforcement by submitting a quota-exceeding pod at admission webhook — 2 hours; applied PodDisruptionBudget minAvailable: 2 to each stateful Deployment; verified behavior during kubectl drain simulation — 3 hours. Total: 10 hours. Production incidents prevented at platform layer: CPU starvation from zero-request DaemonSet, memory OOMKill from unlimited container, unprotected voluntary disruption during node maintenance. Application code changes: zero.” Entries that document the specific cluster configurations applied and the incidents they prevent connect the 10 hours of platform governance work to the production stability it produced, making the Kubernetes platform retainer investment legible to the engineering director reviewing the work log.