Feat/new model manager#2251

Draft

grzegorz-roboflow wants to merge 92 commits intomainfrom

feat/new-model-manager

Collaborator

grzegorz-roboflow commented Apr 21, 2026

What does this PR do?

Adding inference_model_manager

Type of Change

New feature (non-breaking change that adds functionality)

Testing

Initial suit of tests added

Checklist

My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code where necessary, particularly in hard-to-understand areas
My changes generate no new warnings or errors
I have updated the documentation accordingly (if applicable)

Additional Context

N/A

github-advanced-security AI found potential problems

View reviewed changes

inference_server/inference_server/app.py Fixed

github-advanced-security AI found potential problems

View reviewed changes

inference_server/inference_server/server.py Fixed

inference_server/inference_server/server.py Fixed

inference_server/inference_server/server.py Fixed

inference_server/inference_server/server.py Fixed

inference_server/inference_server/server.py Fixed

github-advanced-security AI found potential problems

View reviewed changes

inference_server/inference_server/server.py Dismissed

inference_server/inference_server/server.py Dismissed

inference_server/inference_server/server.py Fixed

inference_server/inference_server/server.py Fixed

inference_server/inference_server/server.py Dismissed

inference_server/inference_server/app.py Fixed

github-advanced-security AI found potential problems

View reviewed changes

inference_server/inference_server/server.py Dismissed

inference_server/inference_server/server.py Dismissed

grzegorz-roboflow force-pushed the feat/new-model-manager branch from b0de01d to c27b669 Compare

April 22, 2026 22:05

github-advanced-security AI found potential problems

View reviewed changes

inference_server/inference_server/auth.py Dismissed

PawelPeczek-Roboflow reviewed

View reviewed changes

inference_models/inference_models/models/base/classification.py Outdated

PawelPeczek-Roboflow reviewed

View reviewed changes

inference_models/inference_models/models/auto_loaders/core.py Outdated

PawelPeczek-Roboflow reviewed

View reviewed changes

inference_models/inference_models/models/auto_loaders/core.py Outdated

PawelPeczek-Roboflow reviewed

View reviewed changes

inference_models/inference_models/models/base/documents_parsing.py Outdated

PawelPeczek-Roboflow reviewed

View reviewed changes

inference_models/inference_models/models/moondream2/moondream2_hf.py Outdated

grzegorz-roboflow added 19 commits

April 27, 2026 16:55

wip

d3fa498

wip

a76369c

wip

f0dfe81

wip

74f2e44

wip

93b6bb6


          Expose max batch size

d92ef7f

wip

5ec1f74

wip

392d1fe

wip

b35b91e

wip

d668c3a

wip

8ba2b13

wip

65a2633

wip

d52fd87


          enable pytest for inference and inference_models to be discovered by …

79a4e68

…vscode

wip

550dbb2


          expose batch and allow to instantiate model class without loading model

57777ce


          wip / enable loading the same model many times

2ac3ddf

wip

c5352ec

wip

f4e3286

grzegorz-roboflow added 19 commits

April 27, 2026 17:02


          Add ManagedModel to inference_models to expose interface used by infe…

e1e22b1

…renence_model_manager


          Add ManagedModel to inference_models to expose interface used by infe…

f271315

…renence_model_manager


          add resolve_task, get_supported_tasks to ModelManager

360e2e0


          infer -> process, add task to subproc backend

8f0fb3b


          task & interface discovery in inference_server

6bb83e2


          hot/cold eviction, idle timeout, request rate tracking, cache hit/miss

a798676


          reactive admission loop — OOM evict-retry, proper load failure handling


          code review

510b3d6


          code review

10c996d


          code review

1c255b3


          code review

d55c3e0


          code review

6d76d65


          propagate errors from subprocess backend to caller

032229b


          code review

a289df0


          code review

f3d7620


          unrelated fixes

cf99279


          code review

e6be8fd


          stubs

f5da973


          instrument tests

bf0f4d4

grzegorz-roboflow force-pushed the feat/new-model-manager branch from b9236ec to bf0f4d4 Compare

April 27, 2026 15:03

grzegorz-roboflow added 5 commits

April 27, 2026 20:32


          model registry

ed27b78


          add contributor guide

8eb9547

fix

04e65f0


          remove max_batch_size from inference_models

a1e997e


          batch size logic

bfa0fbe

github-advanced-security AI found potential problems

View reviewed changes

inference_server/inference_server/auth.py Fixed

grzegorz-roboflow added 2 commits

April 27, 2026 22:37


          Remove AutoModel resolve_class

c221c9f


          formatting

fb002eb

grzegorz-roboflow force-pushed the feat/new-model-manager branch from f67b160 to fb002eb Compare

April 27, 2026 20:38


          adjust README so it's factual

eb569a8

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

PawelPeczek-Roboflow PawelPeczek-Roboflow left review comments

yeldarby Awaiting requested review from yeldarby yeldarby will be requested when the pull request is marked ready for review yeldarby is a code owner

probicheaux Awaiting requested review from probicheaux probicheaux will be requested when the pull request is marked ready for review probicheaux is a code owner

hansent Awaiting requested review from hansent hansent will be requested when the pull request is marked ready for review hansent is a code owner

dkosowski87 Awaiting requested review from dkosowski87 dkosowski87 will be requested when the pull request is marked ready for review dkosowski87 is a code owner

rafel-roboflow Awaiting requested review from rafel-roboflow rafel-roboflow will be requested when the pull request is marked ready for review rafel-roboflow is a code owner

+1 more reviewer

github-advanced-security[bot] github-advanced-security[bot] left review comments

At least 1 approving review is required to merge this pull request.

Labels

None yet