New Ray Serve tutorial #3709

crypdick · 2025-12-18T23:24:23Z

Adds a new tutorial serving Pytorch models on Ray Serve.

This should be merged after #3694 since otherwise the new Ray version will break the existing Tune tutorial.

I will remove the [WIP] label after the Ray Serve team finishes their review.

pytorch-bot · 2025-12-18T23:24:27Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/tutorials/3709

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Merge Blocking SEVs

There is 1 active merge blocking SEVs. Please view them below:

(merge blocking) CI is down due to a github accident

If you must merge, use @pytorchbot merge -f.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

.gitignore

beginner_source/serving_tutorial.py

… tolerance section.

## Description Automatically exclude common directories (.git, .venv, venv, __pycache__) when uploading working_dir in runtime environment packages. At a minimum we need to exclude `.git/` because unlike the others, nobody includes .git/ in `.gitignore`. This causes Ray to throw a `ray.exceptions.RuntimeEnvSetupError` if your `.git` dir is larger than 512 MiB. I also updated the documentation in handling-dependencies.rst and improved the error message if the env exceeds the GCS_STORAGE_MAX_SIZE limit. ## Related issues N/A ## Additional information This PR pytorch/tutorials#3709 was failing to run because the PyTorch tutorials .git/ folder is huge. --------- Signed-off-by: Ricardo Decal <public@ricardodecal.com> Signed-off-by: Ricardo Decal <crypdick@users.noreply.github.com> Signed-off-by: Ricardo Decal <rdecal@anyscale.com> Co-authored-by: Ricardo Decal <public@ricardodecal.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>

## Description Automatically exclude common directories (.git, .venv, venv, __pycache__) when uploading working_dir in runtime environment packages. At a minimum we need to exclude `.git/` because unlike the others, nobody includes .git/ in `.gitignore`. This causes Ray to throw a `ray.exceptions.RuntimeEnvSetupError` if your `.git` dir is larger than 512 MiB. I also updated the documentation in handling-dependencies.rst and improved the error message if the env exceeds the GCS_STORAGE_MAX_SIZE limit. ## Related issues N/A ## Additional information This PR pytorch/tutorials#3709 was failing to run because the PyTorch tutorials .git/ folder is huge. --------- Signed-off-by: Ricardo Decal <public@ricardodecal.com> Signed-off-by: Ricardo Decal <crypdick@users.noreply.github.com> Signed-off-by: Ricardo Decal <rdecal@anyscale.com> Co-authored-by: Ricardo Decal <public@ricardodecal.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com> Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

## Description Automatically exclude common directories (.git, .venv, venv, __pycache__) when uploading working_dir in runtime environment packages. At a minimum we need to exclude `.git/` because unlike the others, nobody includes .git/ in `.gitignore`. This causes Ray to throw a `ray.exceptions.RuntimeEnvSetupError` if your `.git` dir is larger than 512 MiB. I also updated the documentation in handling-dependencies.rst and improved the error message if the env exceeds the GCS_STORAGE_MAX_SIZE limit. ## Related issues N/A ## Additional information This PR pytorch/tutorials#3709 was failing to run because the PyTorch tutorials .git/ folder is huge. --------- Signed-off-by: Ricardo Decal <public@ricardodecal.com> Signed-off-by: Ricardo Decal <crypdick@users.noreply.github.com> Signed-off-by: Ricardo Decal <rdecal@anyscale.com> Co-authored-by: Ricardo Decal <public@ricardodecal.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com> Signed-off-by: jasonwrwang <jasonwrwang@tencent.com>

AlannaBurke · 2026-01-14T01:54:40Z

This looks sane to me, but let's make sure everything is passing.

… env

crypdick · 2026-01-26T22:22:35Z

@AlannaBurke @svekars I've moved this PR out of draft. CI is blocked by approval, could a maintainer please trigger the tests? Thanks!

svekars · 2026-02-02T20:13:31Z

It looks like there is an error: https://github.com/pytorch/tutorials/actions/runs/21602438372/job/62251972530?pr=3709#step:9:7717

crypdick · 2026-02-02T22:07:07Z

@svekars the issue was that .jenkins/build.sh had a step that installs spacy-3.44, which downgraded pydantic to 1.x.x: pydantic: https://github.com/pytorch/tutorials/actions/runs/21602438372/job/62251972530?pr=3709#step:9:621. I synchronized the spacy versions to 3.7.1, which should preserve the pydantic 2.x from the base docker image.

Please trigger the CI again to confirm that everything is fixed.

crypdick · 2026-02-03T01:13:40Z

Thanks for triggering @svekars ! It looks like a few tests were cancelled due to internal server errors.

## Description Automatically exclude common directories (.git, .venv, venv, __pycache__) when uploading working_dir in runtime environment packages. At a minimum we need to exclude `.git/` because unlike the others, nobody includes .git/ in `.gitignore`. This causes Ray to throw a `ray.exceptions.RuntimeEnvSetupError` if your `.git` dir is larger than 512 MiB. I also updated the documentation in handling-dependencies.rst and improved the error message if the env exceeds the GCS_STORAGE_MAX_SIZE limit. ## Related issues N/A ## Additional information This PR pytorch/tutorials#3709 was failing to run because the PyTorch tutorials .git/ folder is huge. --------- Signed-off-by: Ricardo Decal <public@ricardodecal.com> Signed-off-by: Ricardo Decal <crypdick@users.noreply.github.com> Signed-off-by: Ricardo Decal <rdecal@anyscale.com> Co-authored-by: Ricardo Decal <public@ricardodecal.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>

crypdick · 2026-02-09T19:28:13Z

@svekars thanks for triggering the CI. The linter failed due to a trailing space; hopefully everything is ready to merge now :)

Ricardo Decal and others added 19 commits December 9, 2025 11:47

initial draft

ac21240

progress

b45b2cc

formatting

0c1c18f

bump Ray version

7f2e70a

Create paired tutorial docs

2b7b560

ignore cursor configs

9a64caa

add serve to docker reqs

608aa19

Add completed ray serve tutorial

9257827

add tutorial to index

035d4f1

unstage ipynb and md

1f3e44e

Merge branch 'pytorch:main' into ray-serve-tutorial

6afcd3b

tmp checkpt

83e68e7

sync

fa5d2e0

ignore more data files; ignore intermediates

7002920

ignore more files

3d7a1f0

edit gitignore for symlink

06378e2

finalize serving tutorial

69dfe1c

rm tutorial intermediates

0ac13d6

minor fixes

aa881f3

meta-cla bot added the cla signed label Dec 18, 2025

crypdick commented Dec 18, 2025

View reviewed changes

.gitignore Show resolved Hide resolved

crypdick changed the title ~~[WIP] Ray serve tutorial~~ [WIP] New Ray Serve tutorial Dec 18, 2025

crypdick mentioned this pull request Dec 19, 2025

[Core] Add default excludes for working_dir uploads ray-project/ray#59566

Merged

marwan116 reviewed Jan 5, 2026

View reviewed changes

beginner_source/serving_tutorial.py Outdated Show resolved Hide resolved

Ricardo Decal and others added 4 commits January 7, 2026 13:45

clarify expanation of max_ongoing_requests and max_queued_requests

091d9ca

make linter happy

caa63f9

Merge branch 'main' into ray-serve-tutorial

ae04c95

add grid cards for 'what you will learn' and prerequisites. Add fault…

4f38ddf

… tolerance section.

Merge branch 'main' into ray-serve-tutorial

970805c

abrarsheikh approved these changes Jan 13, 2026

View reviewed changes

crypdick and others added 5 commits January 26, 2026 13:49

Merge branch 'main' into ray-serve-tutorial

9e68127

Ignore localhost links Serving tutorial

1e3f1dc

fix links

239bcd0

add ray serve logo

64a0924

bump pydantic requirement to try to override cached pydantic v1 in CI…

935f12d

… env

crypdick changed the title ~~[WIP] New Ray Serve tutorial~~ New Ray Serve tutorial Jan 26, 2026

crypdick marked this pull request as ready for review January 26, 2026 22:19

crypdick and others added 3 commits January 30, 2026 13:52

Merge branch 'main' into ray-serve-tutorial

b3497bc

Merge branch 'main' into ray-serve-tutorial

8029fbc

Merge branch 'main' into ray-serve-tutorial

fad5559

AlannaBurke self-requested a review February 2, 2026 20:09

bump spacy version so that it doesn't downgrade pydantic

48ac207

crypdick and others added 2 commits February 5, 2026 11:24

Merge branch 'main' into ray-serve-tutorial

3bff5ca

make linter happy

931f24d

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New Ray Serve tutorial #3709

New Ray Serve tutorial #3709

crypdick commented Dec 18, 2025

Uh oh!

pytorch-bot bot commented Dec 18, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

AlannaBurke commented Jan 14, 2026

Uh oh!

crypdick commented Jan 26, 2026

Uh oh!

svekars commented Feb 2, 2026

Uh oh!

crypdick commented Feb 2, 2026 •

edited

Loading

Uh oh!

crypdick commented Feb 3, 2026

Uh oh!

crypdick commented Feb 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

New Ray Serve tutorial #3709

Are you sure you want to change the base?

New Ray Serve tutorial #3709

Conversation

crypdick commented Dec 18, 2025

Uh oh!

pytorch-bot bot commented Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/tutorials/3709

❗ 1 Merge Blocking SEVs

Uh oh!

Uh oh!

Uh oh!

AlannaBurke commented Jan 14, 2026

Uh oh!

crypdick commented Jan 26, 2026

Uh oh!

svekars commented Feb 2, 2026

Uh oh!

crypdick commented Feb 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

crypdick commented Feb 3, 2026

Uh oh!

crypdick commented Feb 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

pytorch-bot bot commented Dec 18, 2025 •

edited

Loading

crypdick commented Feb 2, 2026 •

edited

Loading