News

User Story

Engineering

Culture

Stephanie Wang

Scott Lee

Cheng Su

Hao Chen

Eric Liang

Eric Liang headshot

Data loading & preprocessing can easily become the performance bottleneck in ML pipelines. With Ray Data, data loading can be fast, flexible and scalable.

featured image ml training with ray data

Fast, Flexible and Scalable Data Loading for ML Training

Fast, flexible, and scalable data loading for ML training with Ray Data

Cade Daniel

Chen Shen

Richard Liaw

Richard Liaw headshot

In this blog, we discuss continuous batching, a critical systems-level optimization that improves both throughput and latency under load for LLMs.

Cover image for 23X batch blog

Achieve 23x LLM Inference Throughput & Reduce p50 Latency

How continuous batching enables 23x throughput in LLM inference while reducing p50 latency

This blog post delves into how Ray Data streaming works and how to use it for your own ML pipelines distributed across both CPU and GPU devices.

main_streaming_image

How to Stream Distributed Execution Across CPUs & GPUs

Streaming distributed execution across CPUs and GPUs

Amog Kamsetty

Jules S. Damji

We conduct a comparison of three different solutions for offline batch inference: AWS SageMaker Batch Transform, Apache Spark, and Ray Data.

Figure 1. Throughput difference among three different solutions for 10GB batch inference


Figure 1

Offline Batch Inference | Ray, Apache Spark & SageMaker

Offline Batch Inference: Comparing Ray, Apache Spark, and SageMaker

Antoni Baum

Jun Gong

Kai Fricke

Learn the infrastructure challenges for supporting workloads production deployments around foundation models, and how Ray tackles these challenges.

layered-stack

How Ray Solves Generative AI & LLM Infrastructure Challenges

How Ray solves common production challenges for generative AI infrastructure

Robert Nishihara

More companies are needing to train & deploy many small ML models, often hundreds or thousands. Learn how Ray and Anyscale make that possible.

Many_model_image

Training 1 Million ML Models in Record Time | Anyscale

Training One Million Machine Learning Models in Record Time with Ray

Zhe Zhang

This blog covers three methods of batch inference (distributed model evaluation) in Ray: from low-level using Ray Actors, to high-level using Ray Data.

We'll see how low-level Ray APIs allow you to control exactly "How" Ray is executing computations, while Ray's libraries enable you to specify just the "what" --- for a more out-of-the-box scalable experience.

3 ways of batch inference in Ray.

Model Batch Inference in Ray: Actors, ActorPool & Datasets

Model Batch Inference in Ray: Actors, ActorPool, and Datasets

Clark Zinzow

Alex Wu

Jiajun Yao

blog-ray-datasets-thumb

Ray Datasets for large-scale machine learning ingest and scoring

Waleed Kadous

3rdGenTasks andActors

Data Ingest in a Third-Generation ML Architecture | Anyscale

Deep Dive: Data Ingest in a Third Generation ML Architecture

mlplatformCropped

Why 3rd Generation ML Platforms are More Performant

Why Third Generation ML Platforms are More Performant

Ray Distributed Library Patterns (Figure 2)

Ray Distributed Library Patterns

Deploy DeepSeek R1 on Anyscale for full control, seamless scalability, and optimized performance across any cloud or accelerator — without vendor lock-in.

Anyscale + DeepSeek

Deploying DeepSeek R1 on Anyscale

At Ray Summit 2024, we brought together AI experts and practitioners to dive into the future of distributed AI and scalable machine learning.

image3

Ray Summit 2024: Breaking Through the AI Complexity Wall

Explore the new and improved Anyscale Workspaces. Get advanced observability features so you spend less time on busywork and more time building AI apps.

Anyscale Share Image

Explore Anyscale’s New UX

Anyscale’s New User Experience: A Comprehensive Overview

With Anyscale on AWS Marketplace, you can build, scale, and optimize AI/ML and Generative AI—all within the trusted AWS cloud environment.

Group 491 (1)

Anyscale Available on AWS Marketplace

Anyscale Now Available on AWS Marketplace and Achieves Generative AI Competency

This section is used to order the "Types" and "Tags" that show up for filters on the Blog Index

Types

Products / Libraries

Blog

Anyscale is the leading AI application platform. With Anyscale, developers can build, run and scale AI applications instantly.

Anyscale

Featured Posts and News

Deploying DeepSeek R1 on Anyscale

Ray Summit 2024: Breaking Through the AI Complexity Wall

Anyscale’s New User Experience: A Comprehensive Overview

Fast, flexible, and scalable data loading for ML training with Ray Data

Fast, flexible, and scalable data loading for ML training with Ray Data

How continuous batching enables 23x throughput in LLM inference while reducing p50 latency

Streaming distributed execution across CPUs and GPUs

Offline Batch Inference: Comparing Ray, Apache Spark, and SageMaker

How Ray solves common production challenges for generative AI infrastructure

Training One Million Machine Learning Models in Record Time with Ray

Model Batch Inference in Ray: Actors, ActorPool, and Datasets

Ray Datasets for large-scale machine learning ingest and scoring

Deep Dive: Data Ingest in a Third Generation ML Architecture

Why Third Generation ML Platforms are More Performant

Ray Distributed Library Patterns