Skip to content

Add benchmark tests#9654

Open
akx wants to merge 4 commits into
python-pillow:mainfrom
akx:benchmarks
Open

Add benchmark tests#9654
akx wants to merge 4 commits into
python-pillow:mainfrom
akx:benchmarks

Conversation

@akx

@akx akx commented Jun 3, 2026

Copy link
Copy Markdown
Contributor

Related to #9649 and python-pillow/pillow-perf#161.

This PR ports most Pillow-related tests from pillow-perf, modernized (no more compatibility with ancient versions of Pillow), converted to pytest-benchmark style, and also jams in the blend benchmark I used to measure #9649's impact.

The file lives under Tests/, as it is a pytest file (and some benchmarks use test fixture images), but is not named test_*, so it will not be automatically collected as a test (and nothing untoward should happen if you don't have pytest-benchmark installed).

The second commit in this PR wires benchmarks into CI after successful tests.

On my Macbook, running the whole benchmark suite takes about 200 seconds right now -- in CI, this could be adjusted with e.g. --benchmark-max-time at the possible expense of measurement fidelity. Also, some of the benchmarks are currently somewhat exhaustive (combinatorial parametrizes). EDIT: Ugh, about 5 minutes (so longer than the test suite) in CI 😅

@akx akx force-pushed the benchmarks branch 4 times, most recently from 6eb80ce to 096c161 Compare June 3, 2026 15:52
Comment thread .ci/benchmark.sh
@radarhere

Copy link
Copy Markdown
Member

You are only running this in our Test jobs, not others. I'm guessing you did that because you didn't want to slow down all of our CI jobs?

Comment thread Tests/benchmarks.py Outdated
Comment thread Tests/benchmarks.py Outdated
@akx

akx commented Jun 9, 2026

Copy link
Copy Markdown
Contributor Author

You are only running this in our Test jobs, not others. I'm guessing you did that because you didn't want to slow down all of our CI jobs?

Yep, exactly. Since building Pillow from source doesn't actually seem to take that long, maybe it'd be worth it to split it to a separate build + benchmark only job that's run on a single "known" (given these are GHA cloud runners, we may really never know, and neighbors may be noisy) platform?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants