Kubernetes Extended Resource Management

Kubernetes Extended Resource Management is the mechanism by which Kubernetes exposes and schedules against hardware and software resources beyond the built-in CPU, memory, and ephemeral storage types — GPUs, FPGAs, high-performance NICs, or any other quantifiable, node-specific capability — using a namespaced resource-name convention and the device plugin framework to advertise availability to the scheduler. Extended resources are requested and limited using the same resources syntax as ordinary resources, but they differ in fundamental ways: they are typically integer-only (no fractional or millicore-style subdivision), they are advertised by device plugins rather than being universally present on every node, and they generally do not support the request-lower-than-limit overcommitment model that CPU and memory allow.

Because extended resources represent genuinely scarce, often expensive hardware, correctly managing their advertisement, allocation, and scheduling constraints is central to any cluster running specialized workloads like machine learning training or hardware-accelerated processing.

Naming Convention

Domain-Qualified Resource Names

Extended resources are named using a domain prefix to avoid collisions with built-in resources and with other vendors' extended resources — nvidia.com/gpu, amd.com/gpu, example.com/fpga — following the same fully-qualified naming pattern used elsewhere in Kubernetes for custom API extensions.

resources:
  limits:
    nvidia.com/gpu: 1

Device Plugins

Advertising Resource Availability

A device plugin is a small process, typically running as a DaemonSet, that runs on each node possessing the relevant hardware and communicates with the kubelet over a local gRPC socket, reporting how many units of the extended resource that node has available — the kubelet then advertises this count as part of the node's status, making it visible to the scheduler.

kubectl describe node node-gpu-01 | grep -A 3 "Capacity"

Health Monitoring and Allocation

Beyond simple advertisement, device plugins also monitor device health, un-advertising a device if it becomes unhealthy, and handle the actual allocation step during Pod startup — informing the kubelet exactly which physical device the container should be given access to, since extended resources are not directly interchangeable the way CPU cores implicitly are.

Requesting Extended Resources

Integer-Only, No Fractional Requests

Extended resources generally must be requested and limited as whole integers — a Pod cannot request 0.5 of a GPU under the standard extended resource model, reflecting that most such hardware cannot be meaningfully subdivided at the Kubernetes scheduling level (though some newer GPU sharing mechanisms, layered on top of the base extended resource model, do enable finer-grained sharing through vendor-specific solutions).

Requests Must Equal Limits

Unlike CPU and memory, extended resources do not support setting a request lower than the limit — a Pod either gets the full number of units it specifies, or it is not scheduled at all, since there is no meaningful "burstable" model for most extended hardware resources.

resources:
  requests:
    nvidia.com/gpu: 1
  limits:
    nvidia.com/gpu: 1

Scheduling Behavior

Hard Filtering, Not Scoring

A node lacking a requested extended resource in sufficient quantity is filtered out of the candidate set entirely during scheduling, exactly like an ordinary insufficient-CPU or insufficient-memory rejection — there is no soft, preference-based equivalent for extended resources in the base model.

kubectl describe pod codartium-gpu-app | grep -A 3 "FailedScheduling"

A FailedScheduling message citing Insufficient nvidia.com/gpu indicates no node currently has an available unit of that specific extended resource, distinct from an ordinary CPU or memory shortage.

Combining Extended Resources with Node Placement

Ensuring Correct Node Targeting

Because extended resources are only present on nodes with the corresponding hardware, Pods requesting them are typically also configured with a matching nodeSelector or nodeAffinity and any required tolerations for taints protecting that specialized hardware pool — the extended resource request alone guarantees hardware availability, but combining it with explicit placement rules improves scheduling efficiency by steering the scheduler toward the right node class more directly.

apiVersion: v1
kind: Pod
metadata:
  name: codartium-gpu-workload
spec:
  nodeSelector:
    gpu: "true"
  tolerations:
    - key: "nvidia.com/gpu"
      operator: "Exists"
      effect: "NoSchedule"
  containers:
    - name: training
      image: codartium/gpu-training:latest
      resources:
        requests:
          nvidia.com/gpu: 2
        limits:
          nvidia.com/gpu: 2

Example

apiVersion: v1
kind: Pod
metadata:
  name: codartium-extended-resource-example
spec:
  containers:
    - name: gpu-app
      image: codartium/gpu-app:latest
      resources:
        requests:
          nvidia.com/gpu: 1
          cpu: "1"
          memory: "2Gi"
        limits:
          nvidia.com/gpu: 1
          cpu: "2"
          memory: "4Gi"