KubeRay in production: best practices from the field

Part 2 of our KubeRay series, from the field. Part 1: Running Ray at scale on Kubernetes with KubeRay covers what KubeRay is, its custom resources, and how to deploy your first RayJob.

Part 1 ended on a good moment: a RayJob reaching SUCCEEDED. This post picks up from there, with what changes when that one job becomes hundreds, spread across teams and clusters. KubeRay handles that growth well; a public Ray Summit 2024 talk showed Ray Train scaled to 10,000 Kubernetes nodes.

If you've run Ray on Kubernetes for a while, you may know the moment. Somewhere past a handful of nodes and a second team, a failed GPU job stops being a quick kubectl logs. The cluster that held the answer may already be gone, and the GPUs reserved for the rerun keep billing while someone works out whether it was the code, the data or the hardware.

As a field engineer, I spend a lot of time with two kinds of teams: enterprises fitting Ray into clusters built for something else, and digital natives pushing Ray further than it has been pushed before. They tend to meet the same handful of scenarios, and they've taught me which practices hold up. That's what this post shares.

LinkFrom the quickstart to production

Part 1's example trains Fashion MNIST on CPUs. A production RayJob uses the same resource, with four changes: it requests GPUs, queues behind team quotas through Kueue, keeps work off the head node, and scales workers from zero.

1apiVersion: ray.io/v1
2kind: RayJob
3metadata:
4  name: nightly-train
5  labels:
6    kueue.x-k8s.io/queue-name: team-ml   # same quotas as every other GPU workload
7spec:
8  entrypoint: python train.py
9  shutdownAfterJobFinishes: true         # GPUs go back to the pool when done
10  ttlSecondsAfterFinished: 300
11  rayClusterSpec:
12    enableInTreeAutoscaling: true
13    headGroupSpec:
14      rayStartParams: { num-cpus: "0" }   # keep work off the head node
15      template: { ... }
16    workerGroupSpecs:
17      - groupName: gpu-workers
18        minReplicas: 0
19        maxReplicas: 16
20        template: { ... }                # nvidia.com/gpu requests go here

At this scale, the features from Part 1 do more of the work: GPU nodes shared across jobs, workers that scale back to zero, and quotas that keep teams fair. For one team on one cluster, this spec is often all you need.

Note the shutdownAfterJobFinishes: true line, though. It is good for cost, and it is also where the first production scenario starts.

LinkFive production scenarios to plan for

Once Ray becomes shared infrastructure, with more teams, more clusters and GPUs you can't afford to waste, the same pattern shows up in the field: defaults that work well at small scale need returning under production parallelism and contention. These are not KubeRay issues. Each sits on the boundary between Ray, Kubernetes and the organization running both.

1. Debugging data is tied to the cluster's lifetime. A RayJob that cleans up after itself also deletes its Ray dashboard. The hardest case is a driver that exits in its first few minutes: GPU workers come up, receive no tasks, and the cluster is torn down before anyone captures the driver log. That leaves little to investigate afterward.

2. Hardware faults surface as application errors. A PCIe or NVLink fault, or a hung NFS mount, can interrupt a training run while the pod still reports Ready. Ray continues to schedule work there, so the failure presents as an application error. Some teams have built a modified operator and a custom resource to detect failing hardware.

3. Default settings need tuning at scale. With 20 clusters competing for one GPU pool, an idle GPU worker can be reaped while its placement group is still forming, and training can't begin. A maxReplicas set with no headroom can stall the autoscaler when Ray Train retries an out-of-memory task. Each retry can also leave a placement group behind. Teams in the field report weeks of platform engineering before reliability stabilizes.

4. Head node disruptions affect the whole cluster. Node drains for node-image updates and security patches can take Ray nodes down, and memory pressure can evict the head pod, as Klaviyo's engineering team describes from a year of running Ray Serve on KubeRay. Without GCS fault tolerance, the whole cluster fails. With it, workers keep running only if the head recovers within about ten minutes, KubeRay's default worker reconnect timeout.

5. Ray shares the cluster with existing platform services. Service meshes, shared namespaces, quotas and audit requirements were there first. One enterprise team serving thousands of models on Ray Serve found its Envoy sidecars creating a listener for every Ray port, which strained the mesh control plane and affected neighboring services.

LinkSix best practices for production KubeRay

None of these needs new tooling. Each practice maps to one of the scenarios above and uses what the Kubernetes ecosystem already gives you.

Link1. Persist logs and events before the cluster goes away

Plan for the Ray dashboard of an ephemeral cluster to go away. Turn on the History Server from Part 1 so Ray events and logs land in object storage, and test it against short-lived runs: a driver that crashes in its first minutes is the case teams most often miss. If your team debugs in an existing log stack, also ship logs there with a sidecar such as Fluent Bit. For a job you are actively debugging, raise ttlSecondsAfterFinished so the cluster outlives the failure long enough to inspect.

Some context on why this matters. Seeing logs and dashboards for a finished RayJob is one of the oldest asks in the KubeRay community; users were raising it as early as 2023. It is hard because ephemeral clusters are the whole point of RayJob, so the evidence has to leave the cluster before the cluster does. The History Server is the community's answer, built with contributors from several large platform teams. It was introduced as alpha in KubeRay v1.6 and promoted to beta in v1.7. It writes Ray events and logs to object storage and lets you inspect them after the cluster is deleted.

Link2. Treat GPU health as a scheduling signal

Run NVIDIA's DCGM exporter and alert on GPU Xid errors, not only on utilization. Use node health checks to taint or cordon a failing node so new Ray pods stop landing there. Then use the label-aware scheduling from Part 1 to keep new Ray work on healthy nodes. KubeRay maintainers are also discussing forwarding Kubernetes Pod events, including device-related failures, to RayCluster resources for remediation (issue #5304).

Link3. Tune for contention, not for a quiet cluster

Test your configuration at production parallelism, not on one cluster. Five settings come up repeatedly in the field:

• Gang scheduling: use one of the gang schedulers from Part 1 (Volcano, YuniKorn, scheduler-plugins or NVIDIA KAI Scheduler) so a job's workers start together or not at all. This stops early workers from sitting idle while a placement group forms.

• Idle timeout: if you don't gang-schedule, raise the idle timeout so early GPU workers survive while the rest of a placement group forms.

• maxReplicas headroom: leave room for the replacement worker Ray Train requests after an out-of-memory kill. Size it for your needed workers plus at least two.

• Bounded retries: cap out-of-memory retries with RAY_task_oom_retries so repeated retries can't run for hours.

• Mixed CPU and GPU pools: watch whether the autoscaler adds CPU nodes while GPUs are the bottleneck, and whether idle CPU nodes scale down during GPU-bound stages.

1autoscalerOptions:
2  idleTimeoutSeconds: 600   # default is 60
3workerGroupSpecs:
4  - groupName: gpu-workers
5    maxReplicas: 10          # 8 needed + headroom for retries

Watch for placement groups left behind after retries, too. They hold resources the autoscaler can't reclaim.

Link4. Design for losing the head node

• Set num-cpus: "0" on the head so CPU tasks and actors aren't scheduled there.

• When you enable GCS fault tolerance, give each critical workload its own external Redis so a Redis issue stays contained to one workload.

• Bake dependencies into the image instead of installing them at runtime, which can fill ephemeral storage and get pods evicted.

• For Ray Serve, set max_replicas_per_node so a single node failure affects only part of a deployment.

Link5. Fit Ray into the cluster you already have

• Give Ray its own namespace, with the RBAC and NetworkPolicy controls from Part 1, so Ray workloads stay isolated from neighboring services.

• If you run a service mesh, keep it at the edge. Exclude Ray's internal ports from the sidecar, and encrypt Pod-to-Pod traffic with Ray's own TLS or KubeRay's mTLS support (alpha in v1.7) instead. A sidecar listener for every Ray port can overload the mesh.

• Label every RayJob and RayService with its team and cost center, so the Kueue quotas from Part 1 also give you attribution and chargeback.

Link6. Alert on the failures that matter, and automate the first response

Alert on RayJob Failed status, head pod restarts and GPU Xid errors. Start from what Part 1 covered: KubeRay's Kubernetes-native status fields and Events already report RayJob and RayCluster state, and kubectl ray gives on-call engineers a quick view. Where the response is always the same, such as cordoning a node with a failing GPU, automate it so the on-call engineer starts from an answer instead of a dashboard.

LinkWhen the glue starts to cost you

In Part 1, KubeRay replaced the glue between Ray and Kubernetes. At fleet scale, a new layer of glue appears. Every practice above is doable. Each one is also glue: something a person on your team builds, maintains and supports. With one team on one cluster, that's manageable. It changes with the second team and the second cluster, when a failed job, a three-hour queue wait or an auditor's question each take real time to answer.

A quick self-check. Do you:

• Maintain a pipeline just to capture logs before ephemeral RayJobs terminate?

• Spend hours working out whether a failure was the GPU, Kubernetes or Ray?

• Hand-tune autoscaler defaults to keep production jobs from stalling?

• Run redundant clusters or a custom health checker to keep Ray Serve up?

• Map workloads to teams by hand for audit or chargeback?

• Operate Ray across several clusters, clouds or regions, beyond what MultiKueue covers?

If two or more describe your week, a platform layer is likely worth a look.

LinkWhat comes next: KubeRay Connect

We built Anyscale KubeRay Connect, now in private preview, so teams don't have to keep building that layer themselves. It sits on top of the KubeRay you already run. Your operator keeps reconciling your CRs, you keep submitting with kubectl apply, and your Kueue queues and GitOps stay as they are. Nothing is rewritten.

Whether you're just starting with KubeRay or already running it in production, keep your existing Kubernetes and KubeRay stack and extend your platform with full-stack observability, multi-cloud orchestration, security, governance, and enterprise support, so your developers stay productive and ship faster.

KubeRay Connect installs with a Helm chart as a thin layer beside your workloads. A small Anyscale Connector patches your KubeRay specs at admission and injects lightweight telemetry sidecars. A collector ships metrics and Kubernetes events to the Anyscale platform.

Here is how Connect maps to the five scenarios:

Connect works alongside your cluster design: mesh configuration, Redis topology and image size stay your choice, and the practices above still apply. Mapping identity to people and placing work across a fleet stay with you today, and preview users are shaping what comes next.

The security model is built for review. Every connection starts inside your cluster, so the control plane needs no ingress. Authentication uses projected ServiceAccount OIDC tokens instead of API keys. Admission is fail-closed by default, so a workload is never silently admitted without telemetry.

LinkGet started

KubeRay is a strong foundation, and the six practices above will take you a long way. When you'd rather spend your team's time on models than on glue, a maintained platform layer gives that time back.

• Running KubeRay in production? Apply for the KubeRay Connect private preview.

• At KubeCon + CloudNativeCon North America 2026 (November 9–12, Salt Lake City)? Register and come say hello. We'd love to hear about your clusters and what your team is working through.

• New to KubeRay? Start with Part 1: Running Ray at scale on Kubernetes with KubeRay.

• Join the community. Share what you've learned in #kuberay on the Ray Slack, or at the bi-weekly KubeRay community meeting.