Title: Kubernetes Orchestration — LearnCoachAssist

Q: What is a Pod in Kubernetes?
A: A Pod is the smallest deployable unit in Kubernetes. It represents a single instance of a running process and can contain one or more containers that share:
- The same network namespace (IP address and ports)
- Storage volumes
- A common lifecycle

Q: Why would you run multiple containers in a single Pod?
A: Multiple containers in a Pod share the same network and storage, making them ideal for tightly coupled processes. Common patterns include:
- Sidecar – adds functionality (e.g., logging agent)
- Ambassador – proxies network traffic
- Adapter – transforms output for external consumers

Q: What is an init container?
A: An init container runs before the main application containers start. It runs to completion and must succeed before the next init container (or main container) starts. Use cases include:
- Pre-populating shared volumes with data
- Waiting for a dependent service to become available
- Running database migrations

Q: What is the sidecar pattern in Kubernetes?
A: The sidecar pattern places a helper container alongside the main application container within the same Pod. The sidecar extends or enhances the main container's functionality. Examples:
- Log shipping (e.g., Fluentd sidecar)
- Service mesh proxy (e.g., Envoy in Istio)
- Configuration reloading

Q: What is a Kubernetes Service?
A: A Service is an abstraction that defines a logical set of Pods and a policy to access them. It provides:
- A stable IP address and DNS name
- Load balancing across matching Pods
- Decoupling of consumers from individual Pod IPs
Services use label selectors to target Pods.

Q: What is a ClusterIP Service?
A: ClusterIP is the default Service type. It exposes the Service on an internal IP accessible only within the cluster. Use it for:
- Internal microservice-to-microservice communication
- Backend services not exposed externally
Example: spec.type: ClusterIP

Q: What is a NodePort Service?
A: A NodePort Service exposes the Service on a static port (range 30000–32767) on every node's IP. External traffic can reach the Service via <NodeIP>:<NodePort>. It automatically creates a ClusterIP Service that the NodePort routes to.

Q: What is a LoadBalancer Service?
A: A LoadBalancer Service provisions an external load balancer from the cloud provider (AWS ELB, GCP LB, etc.). It automatically creates NodePort and ClusterIP Services. Traffic flows: External LB → NodePort → ClusterIP → Pod. Best for production workloads needing external access.

Q: What is an ExternalName Service?
A: An ExternalName Service maps a Service to an external DNS name using a CNAME record. It does not use selectors or define endpoints. Example: spec.externalName: my.database.example.com. Useful for referencing external services with a cluster-internal DNS alias.

Q: What is a Deployment in Kubernetes?
A: A Deployment manages a set of identical Pods via a ReplicaSet. It provides:
- Declarative updates – describe the desired state
- Rolling updates – zero-downtime deployments
- Rollback – revert to previous revisions
- Scaling – adjust replica count

Q: How do rolling updates work in a Deployment?
A: During a rolling update, Kubernetes incrementally replaces old Pods with new ones. Key parameters:
- maxSurge – max extra Pods above desired count (default 25%)
- maxUnavailable – max Pods that can be unavailable (default 25%)
This ensures the application stays available throughout the update.

Q: How do you rollback a Deployment?
A: Use kubectl rollout undo deployment/<name> to revert to the previous revision. To rollback to a specific revision: kubectl rollout undo deployment/<name> --to-revision=2. View revision history with kubectl rollout history deployment/<name>.

Q: What is a ReplicaSet?
A: A ReplicaSet ensures a specified number of Pod replicas are running at all times. If a Pod fails, the ReplicaSet creates a replacement. Key fields:
- spec.replicas – desired Pod count
- spec.selector – label selector to identify Pods
- spec.template – Pod template
Note: Deployments manage ReplicaSets; use Deployments instead of bare ReplicaSets.

Q: What is a ConfigMap?
A: A ConfigMap stores non-confidential configuration data as key-value pairs. Pods consume ConfigMaps as:
- Environment variables
- Command-line arguments
- Mounted files in a volume
Create with: kubectl create configmap my-config --from-literal=key=value

Q: What is a Secret in Kubernetes?
A: A Secret stores sensitive data such as passwords, tokens, or TLS certificates. Data is base64-encoded (not encrypted by default). Types include:
- Opaque – arbitrary key-value data
- kubernetes.io/tls – TLS cert and key
- kubernetes.io/dockerconfigjson – registry credentials
Enable encryption at rest for production clusters.

Q: What are Namespaces in Kubernetes?
A: Namespaces provide logical isolation within a cluster. They partition resources and allow:
- Separate environments (dev, staging, prod)
- Resource quotas per namespace
- RBAC scoping
Default namespaces: default, kube-system, kube-public, kube-node-lease.

Q: What is an Ingress resource?
A: An Ingress manages external HTTP/HTTPS access to Services within the cluster. It provides:
- Host-based routing (e.g., app.example.com)
- Path-based routing (e.g., /api → service-a)
- TLS termination
Requires an Ingress Controller (e.g., NGINX, Traefik) to function.

Q: What is an Ingress Controller?
A: An Ingress Controller is the component that implements the Ingress resource rules. It watches for Ingress objects and configures the underlying load balancer. Popular controllers:
- NGINX Ingress Controller
- Traefik
- HAProxy
- AWS ALB Ingress Controller
Without a controller, Ingress resources have no effect.

Q: How do you configure TLS on an Ingress?
A: Add a tls section to the Ingress spec:
spec:
  tls:
  - hosts:
    - app.example.com
    secretName: tls-secret
The Secret must contain tls.crt and tls.key. Tools like cert-manager can automate certificate issuance from Let's Encrypt.

Q: What is Helm in Kubernetes?
A: Helm is the package manager for Kubernetes. Key concepts:
- Chart – a packaged set of Kubernetes manifests
- Values – configurable parameters for a chart
- Release – an installed instance of a chart
- Repository – a collection of charts
Install with: helm install my-release my-chart

Q: What is a Helm chart composed of?
A: A Helm chart directory typically contains:
- Chart.yaml – metadata (name, version, description)
- values.yaml – default configuration values
- templates/ – Kubernetes manifest templates using Go templating
- charts/ – dependency sub-charts
- README.md – documentation

Q: What is the Horizontal Pod Autoscaler (HPA)?
A: The HPA automatically scales the number of Pod replicas based on observed metrics. It adjusts replicas to meet target utilization:
- CPU utilization
- Memory utilization
- Custom/external metrics
Create with: kubectl autoscale deployment my-app --cpu-percent=50 --min=2 --max=10

Q: What is the Vertical Pod Autoscaler (VPA)?
A: The VPA automatically adjusts CPU and memory requests/limits for containers based on actual usage. Modes:
- Off – only recommends, no changes
- Auto – applies recommendations (may restart Pods)
- Initial – sets resources only at Pod creation
Note: Do not use VPA and HPA on the same CPU/memory metrics simultaneously.

Q: What is a StatefulSet?
A: A StatefulSet manages stateful applications with guarantees for:
- Stable network identity – each Pod gets a persistent hostname (e.g., web-0, web-1)
- Ordered deployment and scaling
- Stable persistent storage via PersistentVolumeClaims
Use for databases (MySQL, PostgreSQL), distributed systems (Kafka, ZooKeeper).

Q: What is a DaemonSet?
A: A DaemonSet ensures that a copy of a Pod runs on every node (or a subset via node selectors). When nodes are added, the DaemonSet Pod is automatically scheduled. Common use cases:
- Log collectors (Fluentd, Filebeat)
- Monitoring agents (Prometheus node-exporter)
- Network plugins (Calico, Cilium)

Q: What is a Job in Kubernetes?
A: A Job creates one or more Pods and ensures they run to successful completion. Key fields:
- spec.completions – number of successful completions needed
- spec.parallelism – how many Pods run concurrently
- spec.backoffLimit – retries before marking as failed
Use for batch processing, data migrations, one-off tasks.

Q: What is a CronJob in Kubernetes?
A: A CronJob creates Jobs on a recurring schedule using cron syntax. Example: spec.schedule: "0 2 * * *" runs daily at 2 AM. Important fields:
- concurrencyPolicy – Allow, Forbid, or Replace
- successfulJobsHistoryLimit
- failedJobsHistoryLimit

Q: How do you list resources with kubectl?
A: Use kubectl get <resource>. Examples:
- kubectl get pods – list Pods in current namespace
- kubectl get pods -A – list Pods in all namespaces
- kubectl get svc,deploy – list Services and Deployments
- kubectl get pods -o wide – show extra info (node, IP)
- kubectl get pods -o yaml – full YAML output

Q: How do you inspect a resource with kubectl describe?
A: kubectl describe <resource> <name> shows detailed information including:
- Metadata and labels
- Status and conditions
- Events – useful for debugging scheduling/pull errors
Example: kubectl describe pod my-pod reveals image pull errors, OOMKilled events, and probe failures.

Q: How do you apply and delete resources with kubectl?
A: Apply: kubectl apply -f manifest.yaml creates or updates resources declaratively. Supports directories: kubectl apply -f ./k8s/.
Delete: kubectl delete -f manifest.yaml or kubectl delete pod my-pod. Use --grace-period=0 --force for immediate deletion (use with caution).

Q: How do you view container logs with kubectl?
A: Use kubectl logs <pod-name>. Useful flags:
- -f – stream logs in real time
- --tail=100 – show last 100 lines
- -c <container> – specify container in multi-container Pod
- --previous – logs from a previous (crashed) instance
- -l app=nginx – logs from all Pods matching a label

Q: How do you execute commands inside a running container?
A: Use kubectl exec:
- kubectl exec -it my-pod -- /bin/sh – interactive shell
- kubectl exec my-pod -- cat /etc/config – run a single command
- kubectl exec -it my-pod -c sidecar -- /bin/bash – target a specific container
The -it flags enable interactive TTY mode.

Q: What does kubectl port-forward do?
A: kubectl port-forward forwards local ports to a Pod or Service for debugging:
- kubectl port-forward pod/my-pod 8080:80 – forward local 8080 to Pod port 80
- kubectl port-forward svc/my-svc 8080:80 – forward to a Service
Traffic goes directly without needing Ingress or LoadBalancer. Useful for local development and debugging.

Q: How do you scale a Deployment with kubectl?
A: Use kubectl scale deployment/<name> --replicas=<count>. Examples:
- kubectl scale deployment/nginx --replicas=5
- kubectl scale --replicas=0 deployment/nginx – scale to zero
For automatic scaling, use the HPA: kubectl autoscale deployment/nginx --min=2 --max=10 --cpu-percent=80

Q: How do you manage rollouts with kubectl?
A: Key kubectl rollout subcommands:
- kubectl rollout status deployment/app – watch rollout progress
- kubectl rollout history deployment/app – view revision history
- kubectl rollout undo deployment/app – rollback to previous
- kubectl rollout pause deployment/app – pause a rollout
- kubectl rollout resume deployment/app – resume a paused rollout

Q: What is RBAC in Kubernetes?
A: Role-Based Access Control (RBAC) regulates access to Kubernetes resources. It uses four objects:
- Role – permissions within a namespace
- ClusterRole – cluster-wide permissions
- RoleBinding – binds a Role to users/groups/service accounts
- ClusterRoleBinding – binds a ClusterRole cluster-wide

Q: What is the difference between a Role and a ClusterRole?
A: A Role grants permissions within a specific namespace (e.g., read Pods in "dev"). A ClusterRole grants permissions cluster-wide or on non-namespaced resources (nodes, PVs). ClusterRoles can also be bound to a namespace via a RoleBinding to reuse permission sets across namespaces.

Q: What is a RoleBinding in Kubernetes?
A: A RoleBinding associates a Role (or ClusterRole) with subjects within a namespace. Subjects can be:
- User – an authenticated user
- Group – a set of users
- ServiceAccount – an identity for Pods
Example: Bind role "pod-reader" to user "jane" in namespace "dev".

Q: What is a PersistentVolume (PV)?
A: A PersistentVolume is a cluster-level storage resource provisioned by an admin or dynamically via a StorageClass. Key attributes:
- capacity – storage size
- accessModes – ReadWriteOnce, ReadOnlyMany, ReadWriteMany
- persistentVolumeReclaimPolicy – Retain, Delete, or Recycle
PVs exist independently of any Pod.

Q: What is a PersistentVolumeClaim (PVC)?
A: A PVC is a user's request for storage. It specifies size, access mode, and optionally a StorageClass. Kubernetes binds a matching PV to the PVC. Pods reference PVCs in their volume spec:
volumes:
  - name: data
    persistentVolumeClaim:
      claimName: my-pvc

Q: What is a StorageClass?
A: A StorageClass defines how storage is dynamically provisioned. It specifies:
- provisioner – e.g., kubernetes.io/aws-ebs, pd.csi.storage.gke.io
- parameters – provisioner-specific settings (disk type, IOPS)
- reclaimPolicy – Delete or Retain
- volumeBindingMode – Immediate or WaitForFirstConsumer

Q: What are resource requests and limits?
A: Requests define the minimum resources a container needs for scheduling. Limits define the maximum resources a container can use.
resources:
  requests:
    cpu: "250m"
    memory: "128Mi"
  limits:
    cpu: "500m"
    memory: "256Mi"
Exceeding memory limits causes OOMKill; exceeding CPU limits causes throttling.

Q: What is a liveness probe?
A: A liveness probe checks if a container is still running. If it fails, Kubernetes restarts the container. Probe types:
- httpGet – HTTP request to an endpoint
- tcpSocket – TCP connection check
- exec – run a command inside the container
Configure with initialDelaySeconds, periodSeconds, and failureThreshold.

Q: What is a readiness probe?
A: A readiness probe checks if a container is ready to accept traffic. If it fails, the Pod is removed from Service endpoints (no traffic routed to it) but is not restarted. Use readiness probes for:
- Waiting for cache warm-up
- Loading large datasets
- Dependency health checks

Q: What are taints and tolerations?
A: Taints are applied to nodes to repel Pods unless those Pods have matching tolerations. Taint effects:
- NoSchedule – Pods won't be scheduled
- PreferNoSchedule – soft preference to avoid
- NoExecute – evicts existing Pods without tolerations
Example: kubectl taint nodes node1 gpu=true:NoSchedule

Q: What is node affinity?
A: Node affinity constrains which nodes a Pod can be scheduled on based on node labels. Two types:
- requiredDuringSchedulingIgnoredDuringExecution – hard requirement (must match)
- preferredDuringSchedulingIgnoredDuringExecution – soft preference (try to match)
Use it to place Pods on specific hardware, availability zones, or node pools.

Q: What are labels and selectors in Kubernetes?
A: Labels are key-value pairs attached to objects (e.g., app: nginx, env: prod). Selectors filter objects by labels. Two types:
- Equality-based: app=nginx
- Set-based: env in (prod, staging)
Services, Deployments, and ReplicaSets all use selectors to target Pods.

Q: How do you create a Namespace?
A: Create with kubectl create namespace <name> or declaratively:
apiVersion: v1
kind: Namespace
metadata:
  name: my-namespace
Set your default namespace with: kubectl config set-context --current --namespace=my-namespace. Use -n <namespace> flag to target a specific namespace.

Q: What is a Kubernetes startup probe?
A: A startup probe checks if a container application has started successfully. While the startup probe is active, liveness and readiness probes are disabled. This is ideal for slow-starting containers that need extra time to initialize. Once the startup probe succeeds, liveness and readiness probes take over. Configure with failureThreshold × periodSeconds to set the maximum startup time.

Q: What is a Helm release and repository?
A: A release is a running instance of a Helm chart with a specific configuration. You can install multiple releases of the same chart with different names/values.
A repository is a hosted collection of charts. Commands:
- helm repo add bitnami https://charts.bitnami.com/bitnami
- helm repo update
- helm search repo nginx

Q: How does the Kubernetes control plane differ from the worker nodes?
A: The control plane runs components that make global decisions: kube-apiserver, etcd, kube-scheduler, and kube-controller-manager. Worker nodes run the workloads via kubelet, kube-proxy, and the container runtime (e.g., containerd). The control plane exposes the API; nodes actually schedule and run Pods.

Q: What is the role of kube-apiserver?
A: The kube-apiserver is the front-end of the Kubernetes control plane and the only component that talks directly to etcd. It validates and processes REST requests, authenticates/authorizes callers, and serves as the central hub for all cluster component communication.

Q: What is etcd in a Kubernetes cluster?
A: etcd is a distributed, consistent key-value store that serves as the source of truth for all cluster state — node information, Pod specs, Secrets, ConfigMaps, etc. It uses the Raft consensus algorithm. Losing etcd data effectively destroys the cluster, so always back it up.

Q: What does kube-scheduler do?
A: The kube-scheduler watches for newly created Pods with no assigned node and selects an optimal node based on:
- Resource requests vs. node capacity
- Affinity/anti-affinity rules
- Taints and tolerations
- Topology spread constraints
- Node selectors

Q: What does kube-controller-manager do?
A: The kube-controller-manager runs core controller loops that reconcile the cluster's actual state toward the desired state. Examples: Node controller, Job controller, Endpoint controller, ServiceAccount & Token controllers. Each is a separate process compiled into a single binary.

Q: What is the kubelet?
A: The kubelet is the agent that runs on each worker node. It registers the node with the apiserver, watches for Pod specs assigned to it, ensures containers are healthy via probes, and reports Pod/node status back to the control plane.

Q: What is kube-proxy?
A: kube-proxy is a network component on each node that maintains iptables/IPVS rules (or eBPF programs) so traffic to Service cluster IPs gets load-balanced to the correct backend Pods. Without it, Service IPs would not resolve inside the cluster.

Q: What is a container runtime?
A: The container runtime is the software responsible for running containers. Kubernetes supports any runtime implementing the CRI (Container Runtime Interface), such as containerd, CRI-O, or Mirantis Container Runtime. Docker is no longer supported directly since v1.24 — clusters now use containerd/CRI-O under the hood.

Q: What is the Container Runtime Interface (CRI)?
A: The CRI is a plugin API that lets kubelet communicate with any compatible container runtime. It defines gRPC services for pulling images, creating and managing containers, and inspecting their state, decoupling Kubernetes from any single runtime.

Q: What is the Container Network Interface (CNI)?
A: The CNI (Container Network Interface) is a specification that lets Kubernetes delegate Pod networking setup to third-party plugins such as Calico, Cilium, Flannel, or Weave Net. When a Pod starts, the kubelet calls the CNI plugin to assign an IP and set up routes.

Q: What is a Kubernetes Node?
A: A Node is a worker machine (VM or bare-metal) that runs Pods. Each node has:
- A kubelet
- A container runtime
- A kube-proxy
- Capacity attributes (CPU, memory, GPUs, labels)
Nodes are registered via the apiserver and managed by the Node controller.

Q: What is a kubelet's Pod lifecycle?
A: The kubelet's lifecycle per Pod:
- Receives Pod spec from apiserver
- Runs any init containers in order
- Pulls images via the CRI
- Sets up networking via CNI
- Mounts volumes
- Starts containers and runs probes
- Reports status back

Q: What is the Kubernetes API server endpoint?
A: The apiserver is reachable at https://<apiserver>:6443 by default. Tools and components authenticate via client certificates, bearer tokens, or OIDC. The endpoint serves the OpenAPI spec and is the only place from which cluster state can be mutated.

Q: How does kubectl authenticate to a cluster?
A: kubectl reads the ~/.kube/config file, which stores:
- Clusters (apiserver URL, CA cert)
- Users (credentials: cert/token/oidc)
- Contexts (cluster + user + namespace tuples)
You switch contexts with kubectl config use-context <name>.

Q: What is kubectl context?
A: A context is a triple (cluster, user, namespace) saved in the kubeconfig. It lets you target different clusters/users without re-typing flags. Common commands: kubectl config get-contexts, kubectl config use-context prod.

Q: What does kubectl drain do?
A: kubectl drain <node> safely evicts all Pods from a node before maintenance (kernel upgrade, decommissioning). It respects PodDisruptionBudgets and respects taints. Combine with --ignore-daemonsets for DaemonSet Pods and --force for bare Pods without controllers.

Q: What does kubectl cordon do?
A: kubectl cordon <node> marks a node as unschedulable so the scheduler stops placing new Pods on it, without evicting existing workloads. Use it before draining to gracefully migrate workloads. kubectl uncordon reverses it.

Q: What is kubectl taint?
A: kubectl taint applies/removes taints on nodes. Examples:
kubectl taint nodes node1 dedicated=db:NoSchedule
kubectl taint nodes node1 dedicated- (removes taint). Without matching tolerations, Pods will not be scheduled.

Q: What is a PodDisruptionBudget (PDB)?
A: A PDB limits the number of Pods of an application that can be voluntarily disrupted at once (e.g., during a drain or autoscaling event). Specify minAvailable or maxUnavailable. Cluster autoscalers and node drains respect PDBs to protect availability.

Q: What is the difference between PodDisruptionBudget and Deployment replicas?
A: Replicas set the desired steady-state count. PDB caps how many can be simultaneously unavailable during voluntary disruptions. You can have 3 replicas but a PDB of minAvailable: 2, ensuring that during a drain at most 1 Pod is gone at any time.

Q: What is a Kubernetes Manifest?
A: A manifest is a YAML or JSON file that declaratively describes the desired state of a Kubernetes object. Each manifest has four required top-level keys: apiVersion, kind, metadata, and spec. Apply with kubectl apply -f file.yaml.

Q: What is the structure of a Kubernetes YAML manifest?
A: Every manifest has:
apiVersion — the API group/version (e.g., apps/v1)
kind — the object type (Pod, Deployment, etc.)
metadata — name, namespace, labels, annotations
spec — the desired state
Optional: status (read-only, set by the cluster).

Q: What is the difference between imperative and declarative Kubernetes commands?
A: Imperative: direct commands like kubectl run nginx --image=nginx or kubectl create deployment — fast, but doesn't track history.
Declarative: write a manifest and apply with kubectl apply -f. Recommended for production — versionable, repeatable, supports diffing via kubectl diff -f.

Q: What does kubectl diff do?
A: kubectl diff -f manifest.yaml shows the differences between the live cluster state and the manifests on disk before applying. It prevents accidental drift and is a best-practice preflight step in CI/CD pipelines.

Q: How do you label and annotate Kubernetes resources?
A: Labels are identifying key-value pairs used by selectors. Annotations are non-identifying metadata used by tools and libraries.
kubectl label pod my-pod env=prod
kubectl annotate pod my-pod owner="team-a"
Use --overwrite to update existing.

Q: What is kubectl explain?
A: kubectl explain <resource>.<field> returns inline API documentation. Examples:
kubectl explain pod.spec.containers
kubectl explain deployment.spec.strategy
It's like a built-in man page — handy when you don't have web access.

Q: What are Kubernetes annotations?
A: Annotations are non-identifying metadata attached to objects, used by external tools (service meshes, ingress controllers, monitoring) to store configuration or context. They cannot be used in label selectors and have no semantic meaning to Kubernetes core.

Q: What are finalizers in Kubernetes?
A: Finalizers are metadata strings on objects that block deletion until cleared. They let controllers run cleanup (e.g., releasing cloud resources) before the object is fully removed. Common examples: kubernetes.io/pv-protection on PVs.

Q: What are ownerReferences?
A: ownerReferences link a child object (e.g., a Pod) to its parent (e.g., a ReplicaSet). When the parent is deleted, the garbage collector cascades and deletes children automatically. This is how Deployments, ReplicaSets, and Pods stay linked.

Q: What is the Kubernetes garbage collector?
A: The garbage collector runs in the controller-manager and removes objects whose owners no longer exist (cascade deletion). It supports both background and foreground cascading — foreground waits for dependents to be cleaned up first.

Q: What is kubectl edit?
A: kubectl edit <resource>/<name> opens the live object's YAML in $EDITOR, lets you modify it, and applies the changes back to the cluster. Useful for quick tweaks; for production prefer version-controlled manifests with apply.

Q: What is kubectl patch?
A: kubectl patch applies a JSON/RFC6902 patch to a live object. Types:
--type=strategic — strategic merge patch (default)
--type=merge — JSON merge patch
--type=json — JSON patch
Example: kubectl patch deployment app -p '{"spec":{"replicas":5}}'

Q: What is kubectl replace?
A: kubectl replace -f new.yaml replaces a live object with a new manifest. It fails if the resource doesn't exist (unlike apply) and does not merge — every field must be re-declared. Use it when you want full replacement, not partial updates.

Q: What does kubectl wait do?
A: kubectl wait blocks until a condition is met on a resource. Examples:
kubectl wait --for=condition=Ready pod/my-pod --timeout=120s
kubectl wait --for=jsonpath='{.status.readyReplicas}'=3 deployment/app
Often used in CI/CD pipelines.

Q: What is kubectl top?
A: kubectl top node and kubectl top pod show live CPU/memory usage, sourced from the Metrics Server. Without the Metrics Server installed, the command fails. Useful for quick resource checks and validating HPA targets.

Q: What is the Metrics Server?
A: The Metrics Server is a cluster-wide aggregator of resource usage data (CPU/memory) scraped from kubelets. It exposes them via the Kubernetes API for the HPA, VPA, and kubectl top. It's not a long-term monitoring solution — that's Prometheus's role.

Q: What are Custom Resources?
A: A Custom Resource (CR) extends the Kubernetes API with a new object type you define yourself. Custom Resources alone only store data; combined with a Custom Controller, they become a CustomResourceDefinition (CRD) backed, fully declarative control loop.

Q: What is a CustomResourceDefinition (CRD)?
A: A CRD is a manifest that registers a new resource kind with the apiserver. Once applied, you can kubectl get and kubectl apply objects of that kind. Examples: Certificate from cert-manager, VirtualService from Istio.

Q: What is the Operator pattern?
A: The Operator pattern combines a CRD with a custom controller that encodes operational knowledge (backups, upgrades, scaling, healing) for a specific application. Operators automate Day-2 operations. Examples: etcd-operator, postgres-operator, cert-manager.

Q: What is a Kubernetes controller?
A: A controller is a control loop that watches the state of the cluster via the apiserver and converges actual state toward desired state. Examples: Deployment controller, ReplicaSet controller, Job controller. The Operator pattern is custom controllers + CRDs.

Q: What is the Operator Framework?
A: The Operator Framework is a set of tools for building and managing Operators:
- Operator SDK — scaffolding tool
- Operator Lifecycle Manager (OLM) — install/upgrade Operators
- OperatorHub.io — public registry

Q: What is a headless service?
A: A headless Service has clusterIP: None, returning the Pod IPs directly via DNS instead of a single virtual IP. Useful with StatefulSets to give each Pod a stable DNS name like pod-0.svc.namespace.svc.cluster.local for direct peer access.

Q: What is a Service Mesh?
A: A service mesh is a dedicated infrastructure layer for service-to-service communication, typically implemented by sidecar proxies (Envoy/Linkerd). It provides mTLS, retries, traffic splitting, and observability without changing app code. Examples: Istio, Linkerd, Consul Connect.

Q: What is an Istio VirtualService?
A: A VirtualService defines routing rules for traffic to a service — weights for canary releases, header-based routing, retries, timeouts. It decouples routing intent from the Service definition. Common in canary and A/B test deployments.

Q: What is an Istio DestinationRule?
A: A DestinationRule configures traffic policies after routing decisions: connection pool sizes, outlier detection (retries on failures), TLS mode, and load-balancer strategies. Combined with VirtualService, it gives fine-grained traffic control.

Q: What is mTLS in a service mesh?
A: Mutual TLS means both client and server present certificates to each other. Service meshes like Istio and Linkerd enforce it transparently via sidecars, ensuring all inter-Pod traffic is authenticated and encrypted — even without changes to app code.

Q: What are Network Policies?
A: Network Policies are namespace-scoped rules that control Pod-to-Pod (and Pod-to-external) traffic at the IP/port level. They're enforced by a CNI plugin (Calico, Cilium). Without any Network Policy, all Pods can talk to all Pods (default allow).

Q: What is a default-deny Network Policy?
A: A default-deny policy selects all Pods in a namespace and denies all ingress and egress traffic. It's the recommended baseline; you then layer allow rules for specific traffic. Without this, anything not explicitly blocked is allowed.

Q: How do you allow DNS in a default-deny Network Policy?
A: You must add an explicit egress rule to allow UDP port 53 to the kube-system namespace (or wherever CoreDNS runs):
egress:
  - to: [{namespaceSelector: {matchLabels: {kubernetes.io/metadata.name: kube-system}}} ]
    ports: [{protocol: UDP, port: 53}]

Q: What is Kubernetes PodSecurity Admission?
A: PodSecurity Admission is a built-in admission controller that enforces the Pod Security Standards at the namespace level via labels: pod-security.kubernetes.io/enforce: restricted (or baseline / privileged). It replaced PodSecurityPolicy, which was removed in v1.25.

Q: What are Pod Security Standards?
A: Pod Security Standards define three levels:
- Privileged — unrestricted
- Baseline — blocks known dangerous escalations
- Restricted — hardened best practices (no root, no host network, etc.)
Apply via namespace labels with PodSecurity Admission.

Q: What is a SecurityContext?
A: A SecurityContext defines privilege and access-control settings for a Pod or container. Fields include runAsNonRoot, runAsUser, readOnlyRootFilesystem, allowPrivilegeEscalation, and capabilities.drop. It's the primary tool for hardening Pods.

Q: What is the difference between runAsUser and runAsNonRoot?
A: runAsUser sets a specific numeric UID for the container process. runAsNonRoot: true only requires that the UID be non-zero — the actual UID is decided by the image. Use runAsNonRoot as a baseline; combine with runAsUser for strictness.

Q: What is readOnlyRootFilesystem?
A: readOnlyRootFilesystem: true mounts the container's root filesystem read-only. Combined with writable emptyDir volumes for writable paths (e.g., /tmp), it limits the blast radius of compromised processes and is part of the restricted Pod Security Standard.

Q: What is a ServiceAccount?
A: A ServiceAccount provides an identity for processes running in a Pod. It's mounted at /var/run/secrets/kubernetes.io/serviceaccount/ by default. Pods use it to authenticate to the apiserver (e.g., for the controller manager or Prometheus operator).

Q: What is the default ServiceAccount?
A: Every namespace has a default ServiceAccount. Pods that don't specify one get it. For tighter security, create a dedicated ServiceAccount per workload and disable automounting via automountServiceAccountToken: false.

Q: How do you disable automounting the ServiceAccount token?
A: Set automountServiceAccountToken: false on either the Pod spec or the ServiceAccount. This prevents the projected SA token from being mounted at /var/run/secrets/..., which is the recommended default unless the Pod actually calls the apiserver.

Q: What is RBAC verbs?
A: In RBAC, verbs are the actions allowed on a resource: get, list, watch, create, update, patch, delete, and deletecollection. The special verb * grants all permissions (avoid in production).

Q: What is kubectl auth can-i?
A: kubectl auth can-i create pods checks if the current user can perform an action. Add --as <user> to impersonate: kubectl auth can-i list secrets --as=system:serviceaccount:dev:app-sa. Useful for verifying RBAC roles work as intended.

Q: What is kubeconfig user impersonation?
A: kubectl --as <user> --as-group <group> ... lets admins act on behalf of a user/group to debug permissions without sharing credentials. Combined with kubectl auth can-i --as, this is the standard way to audit RBAC.

Q: What is a Kubernetes API group?
A: API groups organize resources into collections. Core resources (Pod, Service) are in the core group (empty string). CRDs and other resources live in named groups: apps/v1, batch/v1, networking.k8s.io/v1. apiVersion encodes this.

Q: What is an API version lifecycle in Kubernetes?
A: Kubernetes API versions progress through stages:
- Alpha — disabled by default, may break, no guarantee
- Beta — enabled by default, stable features, no breaking changes
- GA / Stable — versioned like v1; safe for production
Deprecated APIs are removed after the deprecation policy window.

Q: What was the deprecated PodSecurityPolicy?
A: PodSecurityPolicy (PSP) was a predecessor to PodSecurity Admission, removed in v1.25. It allowed cluster-wide policy via RBAC. PSPs had a confusing precedence model and were replaced by the simpler, namespace-scoped PodSecurity Admission.

Q: How do you set namespace resource quotas?
A: A ResourceQuota object limits total resource consumption in a namespace:
apiVersion: v1
kind: ResourceQuota
metadata: {name: compute-quota}
spec:
  hard:
    requests.cpu: "10"
    requests.memory: 20Gi
    pods: "50"

Q: What is a LimitRange?
A: A LimitRange sets default and min/max constraints per-container or per-Pod within a namespace. It pairs with ResourceQuota to enforce both defaults (so every Pod gets sensible limits) and ceilings (so no single workload hogs the namespace).

Q: What is the Quality of Service (QoS) class of a Pod?
A: Kubernetes assigns each Pod a QoS class based on its resource settings:
- Guaranteed — every container has requests == limits (both CPU and memory)
- Burstable — at least one request/limit is set but not guaranteed
- BestEffort — no requests or limits anywhere
QoS influences eviction priority under node pressure.

Q: How does QoS affect Pod eviction?
A: Under node resource pressure, the kubelet evicts Pods in this order: BestEffort → Burstable → Guaranteed. Guaranteed Pods are only killed if the node is critically short. This is why setting requests/limits matters for production reliability.

Q: What is a PriorityClass?
A: A PriorityClass assigns a priority value to Pods so the scheduler can preempt lower-priority Pods when the cluster is full. Higher-priority Pods evict lower-priority ones. System critical Pods use classes like system-node-critical and system-cluster-critical.

Q: What is Pod preemption?
A: When the scheduler can't fit a high-priority Pod, it may preempt (evict) lower-priority Pods to make room. Preempted Pods get a graceful shutdown (respecting terminationGracePeriodSeconds), and the scheduler may reschedule them elsewhere.

Q: What is topology spread constraints?
A: Topology spread constraints distribute Pods across failure domains (zones, regions, nodes). Example:
topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
This keeps replicas balanced across zones.

Q: What is pod anti-affinity used for?
A: Pod anti-affinity prevents two Pods of the same kind from landing on the same node/zone, useful for high availability. Example: a Deployment with 3 replicas and requiredDuringSchedulingIgnoredDuringExecution anti-affinity on topologyKey: kubernetes.io/hostname ensures each replica is on a different node.

Q: What is the difference between soft and hard affinity rules?
A: Hard (requiredDuringScheduling...) makes the constraint mandatory — Pods won't schedule unless it can be met. Soft (preferredDuringScheduling...) is a preference; the scheduler tries to honor it but will still schedule the Pod if not possible. Soft rules have a weight field.

Q: What is an emptyDir volume?
A: An emptyDir volume is created when a Pod is assigned to a node and exists for the Pod's lifetime. Initially empty, it's shared between containers in the same Pod. Useful for scratch space, caches, and checkpoint data. Lost on Pod restart.

Q: What is a hostPath volume?
A: A hostPath volume mounts a file or directory from the host node's filesystem into a Pod. It's powerful but dangerous — content varies per node and creates tight coupling. Restricted Pod Security Standard disallows most hostPath usage. Use CSI volumes instead.

Q: What is a CSI volume?
A: A CSI (Container Storage Interface) volume is a persistent volume backed by an external driver (e.g., AWS EBS, GCP PD, Azure Disk). The driver runs as a DaemonSet or Deployment. CSI is the standard since in-tree plugins were removed in v1.26.

Q: What is VolumeSnapshot?
A: A VolumeSnapshot is a point-in-time copy of a PVC, similar to a database snapshot. You define a VolumeSnapshotClass (which CSI driver to use) and a VolumeSnapshot resource. Useful for backups and cloning databases on Kubernetes.

Q: What is dynamic volume provisioning?
A: Dynamic provisioning creates PVs on demand when a PVC references a StorageClass. The StorageClass's provisioner field points to a driver (e.g., ebs.csi.aws.com) that creates the underlying storage. No admin pre-provisioning needed.

Q: What is volumeBindingMode: WaitForFirstConsumer?
A: WaitForFirstConsumer delays PV binding until a Pod is scheduled, so the volume is created in the same zone as the Pod. Without it (Immediate), PVs may be provisioned in zones the Pod cannot reach, leaving PVCs Pending.

Q: What is the access mode ReadWriteMany?
A: ReadWriteMany (RWX) means the volume can be mounted read-write by many nodes simultaneously. Only certain CSI drivers support it (e.g., NFS, CephFS, AWS EFS). It's required for shared logs or shared config across multiple Pods on different nodes.

Q: What are the differences between access modes?
A: - ReadWriteOnce (RWO) — single-node read-write
- ReadOnlyMany (ROX) — many nodes read-only
- ReadWriteMany (RWX) — many nodes read-write
- ReadWriteOncePod (RWOP) — single Pod read-write (CSI 1.1+)

Q: What is kubectl cp?
A: kubectl cp <pod>:<remote_path> <local_path> copies files between a container and your local machine. Reverse direction is supported. Internally it uses tar streams through kubectl exec. Useful for retrieving logs or uploading configs.

Q: What is kubectl debug?
A: kubectl debug launches an ephemeral container in an existing Pod for troubleshooting — ideal for distroless or minimal images where you can't shell in. Examples:
kubectl debug my-pod -it --image=busybox --target=app

Q: What are ephemeral containers?
A: Ephemeral containers are special containers added to a running Pod via kubectl debug for diagnostics. They don't have resources/ports/liveness probes and run until removed. They're the only way to attach a debugger to a distroless image Pod.

Q: What is kubectl events?
A: kubectl get events lists cluster events sorted by time. Use flags like --field-selector type=Warning or --sort-by=.lastTimestamp. kubectl events --for pod/my-pod shows events for a specific object. Events expire after ~1 hour by default.

Q: What is kubectl get -o jsonpath?
A: -o jsonpath='{.items[*].metadata.name}' extracts specific fields from resources via JSONPath expressions. Useful in scripts: kubectl get pods -o jsonpath='{.items[*].status.podIP}'. Combine with --template for reusable formats.

Q: What is kubectl get --show-labels?
A: --show-labels adds a final LABELS column to kubectl get output, exposing all labels at once. Combine with -L key to show only specific labels as their own columns. Useful for quickly auditing label coverage.

Q: What is the Kubernetes downward API?
A: The downward API exposes Pod metadata (name, namespace, labels, resource limits) to running containers as environment variables or files in a downwardAPI volume. It lets containers react to their own context without external lookups. Example: fieldRef: {fieldPath: metadata.name}.

Q: What is the Kubernetes API?
A: The Kubernetes API is the central REST interface through which all cluster interactions occur. Every component — kubelet, scheduler, controllers, kubectl — talks to it. It is versioned, discoverable (via OpenAPI), and serves resources like Pod, Service, Deployment, etc.

Q: What is OpenAPI in Kubernetes?
A: Kubernetes serves an OpenAPI spec at /openapi/v2 and /openapi/v3. Tools like kubectl, client-go, and Swagger use it to generate clients and validators. You can browse it directly: kubectl get --raw /openapi/v2 | jq.

Q: What is an admission controller?
A: Admission controllers are plugins that intercept requests to the apiserver after authentication/authorization but before persistence. They can mutate or validate objects. Examples: MutatingAdmissionWebhook, ValidatingAdmissionWebhook, PodSecurity, ResourceQuota.

Q: What is a Mutating Admission Webhook?
A: A Mutating Admission Webhook is an HTTPS endpoint the apiserver calls with incoming requests, allowing the webhook to modify the object before it's stored. Common uses: injecting sidecars, defaulting fields, secret rotation. Order matters — multiple mutations chain.

Q: What is a Validating Admission Webhook?
A: A Validating Admission Webhook is called after mutating webhooks to accept or reject an object. It cannot modify the request. Common uses: enforce naming conventions, block disallowed image registries, require specific labels.

Q: What is a service mesh sidecar injection?
A: Sidecar injection automatically appends a proxy container (Envoy in Istio, linkerd-proxy in Linkerd) to each Pod via a Mutating Admission Webhook. The proxy intercepts and manages all inbound/outbound traffic for the Pod without app changes.

Q: What is cert-manager?
A: cert-manager is a Kubernetes Operator that automates TLS certificate issuance, renewal, and storage. It integrates with Let's Encrypt (ACME), HashiCorp Vault, Venafi, and others. You declare a Certificate resource; cert-manager creates a Secret with the actual cert.

Q: What is an ACME challenge?
A: ACME (Automatic Certificate Management Environment) is the protocol Let's Encrypt uses. The two main challenge types:
- HTTP-01 — serves a token over HTTP on port 80
- DNS-01 — adds a TXT record to DNS
DNS-01 supports wildcard certs.

Q: What is a Kubernetes secret backend?
A: Secrets are stored in etcd by default. To encrypt them at rest, configure the kube-apiserver with --encryption-provider-config pointing to a config that defines providers like aescbc, secretbox, or aws-kms. Without this, Secrets are only base64-encoded.

Q: What is external Secrets management?
A: External secret managers like HashiCorp Vault, AWS Secrets Manager, or Azure Key Vault hold sensitive values. The External Secrets Operator (or AWS Secrets Store CSI Driver) syncs them into Kubernetes Secrets so apps can mount them as usual.

Q: What is the kubelet configuration?
A: The kubelet is configured via /var/lib/kubelet/config.yaml (or via flags). It controls cluster DNS, image pull policy, cgroup driver, authentication mode, and event recording. After editing, restart the kubelet or send SIGHUP.

Q: What is the cgroup driver in Kubernetes?
A: The cgroup driver manages cgroups (resource control groups) on the node. cgroupfs and systemd are the two options. It must match the container runtime's cgroup driver — mismatches cause kubelet errors. Modern Linux distros default to systemd.

Q: What is a Kubernetes pause container?
A: The pause container (image k8s.gcr.io/pause:3.x) is a tiny, do-nothing container that holds the network namespace for a Pod. Every Pod starts with one; it's the parent of all other containers in the Pod, providing the shared IP and volumes.

Q: What is container imagePullPolicy?
A: imagePullPolicy controls when the kubelet pulls an image:
- Always — every start
- IfNotPresent — only if missing (default for tagged images)
- Never — must be present locally
Use :latest with care; pin tags for reproducibility.

Q: What is imagePullSecrets?
A: imagePullSecrets is a list of Secret references in a Pod spec that authenticate to a private container registry. Format:
imagePullSecrets:
  - name: regcred
Create regcred via kubectl create secret docker-registry regcred --docker-server=... --docker-username=... --docker-password=....

Q: What is the Pod lifecycle phases?
A: A Pod transitions through these phases:
- Pending — accepted, not yet running (image pull, scheduling)
- Running — at least one container is running
- Succeeded — all containers exited 0 and won't restart
- Failed — at least one container exited non-zero
- Unknown — apiserver can't reach the kubelet

Q: What is Pod restartPolicy?
A: restartPolicy controls when containers are restarted:
- Always — every termination (default for Deployments, StatefulSets)
- OnFailure — only on non-zero exit (Jobs)
- Never — once exited, stay exited (Jobs, init containers)

Q: What are Pod conditions?
A: Pod conditions are status fields reflecting lifecycle states:
- PodScheduled — bound to a node
- Initialized — init containers completed
- ContainersReady — all containers ready
- Ready — able to serve traffic
Each has status (True/False/Unknown) and reason.

Q: What are container probes' delay parameters?
A: Each probe type supports:
- initialDelaySeconds — wait before first check
- periodSeconds — interval between checks
- timeoutSeconds — per-probe timeout
- successThreshold — successes to mark healthy
- failureThreshold — failures to mark unhealthy
Startup probes can replace initialDelaySeconds with a long failureThreshold.

Q: What is the difference between liveness and readiness failure?
A: Liveness failure restarts the container.
Readiness failure removes the Pod from Service endpoints but does not restart it. Use liveness for "process is deadlocked" and readiness for "not yet ready to serve traffic".

Q: What is a graceful pod shutdown?
A: Graceful shutdown means the kubelet sends SIGTERM to containers, waits terminationGracePeriodSeconds (default 30s), then sends SIGKILL. Apps should trap SIGTERM to flush state, drain connections, and exit cleanly. preStop hooks can add cleanup logic.

Q: What is a preStop hook?
A: A preStop hook is a command or HTTP request executed inside a container just before it's terminated. It's useful when an app doesn't handle SIGTERM cleanly. Example:
lifecycle:
  preStop:
    exec:
      command: ["sh", "-c", "sleep 10"]

Q: What is kubectl proxy?
A: kubectl proxy starts a local HTTP proxy to the apiserver using your kubeconfig credentials. Useful for browsing the API at http://localhost:8001/api/... or for connecting kubectl-aware tools without exposing the apiserver externally.

Q: What is kubectl api-resources?
A: kubectl api-resources lists every resource kind, its API group/version, and whether it's namespaced. Examples:
kubectl api-resources --namespaced=false — cluster-scoped resources only
kubectl api-resources -o name — names only, for scripting

Q: What is kubectl api-versions?
A: kubectl api-versions lists every API group/version the cluster supports (e.g., apps/v1, networking.k8s.io/v1). Useful when checking whether a feature is available on the cluster.

Q: What is kube-state-metrics?
A: kube-state-metrics is a service that generates Prometheus metrics from the Kubernetes API (number of Pods, Deployment status, Node conditions, etc.). Unlike the Metrics Server, it provides raw object state, not resource usage. Deployed via a separate Service/Deployment.

Q: What is the Kubernetes API server audit log?
A: Audit logs record every request to the apiserver: user, verb, resource, response code, timestamp. Enable by configuring --audit-policy-file and --audit-log-path (or a webhook backend). Critical for security incident analysis.

Q: What is a Kubernetes admission policy engine?
A: OPA Gatekeeper and Kyverno are policy engines that enforce custom rules across the cluster via admission webhooks. They answer questions like "must all images come from a trusted registry?" or "must all Deployments have a specific label?". Stored as code (Rego for OPA, YAML for Kyverno).

Q: What is Kyverno?
A: Kyverno is a Kubernetes-native policy engine using YAML for rules. It supports validate, mutate, generate, and verifyImage operations. Example: auto-inject a default team label, deny images from latest, or generate a default NetworkPolicy for new namespaces.

Q: What is the difference between HPA and Cluster Autoscaler?
A: HPA scales the number of Pods in a Deployment/ReplicaSet based on metrics. Cluster Autoscaler adds or removes worker nodes when Pods can't be scheduled (Pending) or when nodes are underutilized. They work together: HPA handles demand, Cluster Autoscaler handles capacity.

Q: What is a Pod's effective request vs. limit?
A: Requests drive scheduling decisions — the sum of all Pods' requests must fit on a node. Limits drive runtime enforcement — exceeding CPU limit throttles; exceeding memory limit triggers OOMKill. Set requests for proper bin-packing; set limits to prevent runaway resource use.

Q: What is the CPU unit in Kubernetes?
A: CPU resources are measured in millicores or m. 1 = 1 vCPU/core, 500m = half a core. 100m = 0.1 CPU. Decimals (0.5) and fractions (1.5) are also accepted.