refactor: migrate to Pydantic v2 and restructure API client architecture (among other changes)

[hassiebp]

Important

Major refactoring to upgrade Pydantic v1 to v2, reorganize the API client structure with lazy loading, remove deprecated methods, and update all client code to use new observation and trace APIs.

[!IMPORTANT]
Major refactoring to upgrade Pydantic v1 to v2, reorganize the API client structure with lazy loading, remove deprecated methods, and update all client code to use new observation and trace APIs.

Pydantic v2 Migration

• Updated pyproject.toml to require Pydantic v2 (^2) instead of v1
• Migrated all Pydantic models from pydantic_v1.BaseModel to UniversalBaseModel (Pydantic v2 compatible)
• Converted field definitions to use typing_extensions.Annotated with FieldMetadata for aliases instead of pydantic.Field(alias=...)
• Removed custom json() and dict() methods, replaced with Pydantic v2's model_dump() and model_rebuild()
• Updated serialization utilities in langfuse/api/core/pydantic_utilities.py to support both Pydantic v1 and v2
• Removed pydantic_v1 imports and replaced with direct Pydantic v2 imports throughout the codebase
• Updated langfuse/_task_manager/score_ingestion_consumer.py to use Pydantic v2 BaseModel and model_dump()
• Simplified langfuse/_utils/serializer.py to remove Pydantic v1 compatibility logic and added custom serialize_datetime() function

API Client Architecture Restructuring

• Reorganized langfuse/api/ directory structure: moved all client code from resources/ subdirectories to top-level module directories (e.g., annotation_queues/, blob_storage_integrations/, comments/, etc.)
• Implemented lazy loading in all __init__.py files using __getattr__ and dynamic imports dictionary to reduce startup time
• Renamed FernLangfuse to LangfuseAPI and AsyncFernLangfuse to AsyncLangfuseAPI in langfuse/api/client.py
• Added headers parameter support to client constructors for custom HTTP headers
• Converted client attributes from direct initialization to lazy-loaded properties using private attributes with underscore prefix
• Updated all client classes to use raw client implementations internally instead of direct HTTP handling
• Deleted entire langfuse/api/resources/ directory structure (300+ files) as part of restructuring

Deprecated Methods Removal

• Removed start_span() and start_as_current_span() methods from LangfuseSpan class
• Removed start_generation() and start_as_current_generation() methods from LangfuseSpan class
• Removed update_trace() method from LangfuseSpan class, replaced with set_trace_io() and set_trace_as_public()
• Removed update_current_trace() method from Langfuse client, replaced with set_current_trace_io() and set_current_trace_as_public()
• Removed deprecated start_span() and start_as_current_span() methods from Langfuse client
• Removed deprecated start_generation() and start_as_current_generation() methods from Langfuse client
• Removed warnings module usage for deprecation, replaced with typing_extensions.deprecated decorator

New Observation and Trace APIs

• Added start_observation() and start_as_current_observation() methods to replace deprecated span/generation methods
• Added set_trace_io() method to set trace input/output (replaces update_trace())
• Added set_trace_as_public() method to make traces publicly accessible
• Added set_current_trace_io() and set_current_trace_as_public() methods to Langfuse client
• Added propagate_attributes() context manager to langfuse/_client/propagation.py for setting trace attributes with proper context management

Client Updates

• Updated langfuse/_client/attributes.py to remove deprecated trace attribute parameters (name, user_id, session_id, version, release, metadata, tags)
• Updated langfuse/_client/client.py to use new observation and trace APIs, removed deprecated method imports
• Updated langfuse/_client/span.py to use new observation and trace APIs, removed deprecated methods
• Updated langfuse/_client/datasets.py to remove DatasetItemClient class and use DatasetItem directly
• Updated langfuse/_task_manager/media_manager.py to use LangfuseAPI instead of FernLangfuse
• Updated langfuse/_task_manager/resource_manager.py to use LangfuseAPI and AsyncLangfuseAPI
• Updated langfuse/langchain/CallbackHandler.py to use propagate_attributes() context manager instead of update_trace()

New API Modules

• Added 30+ new API modules with auto-generated client code from Fern API definition including:
  • annotation_queues/, blob_storage_integrations/, comments/, dataset_items/, dataset_run_items/, datasets/, health/, ingestion/, llm_connections/, media/, metrics/, metrics_v2/, models/, observations/, observations_v2/, opentelemetry/, organizations/, projects/, prompt_version/, prompts/, scim/, score/, score_configs/, score_v2/, sessions/, trace/, utils/
• Each module includes client classes, raw client implementations, and type definitions with lazy loading

Core API Utilities

• Added langfuse/api/core/enum.py with cross-version Python StrEnum compatibility
• Added langfuse/api/core/force_multipart.py for multipart form-data encoding
• Added langfuse/api/core/http_response.py with HttpResponse and AsyncHttpResponse wrapper classes
• Added langfuse/api/core/http_sse/ module for Server-Sent Events support
• Added langfuse/api/core/serialization.py with FieldMetadata and serialization utilities
• Updated langfuse/api/core/http_client.py with improved retry logic, jitter functions, and multipart handling
• Updated langfuse/api/core/client_wrapper.py to support custom headers and async token providers
• Updated langfuse/api/core/pydantic_utilities.py with Pydantic v2 support and new utility functions

Test Updates

• Updated all test files to use new observation and trace APIs instead of deprecated methods
• Replaced start_span() with start_observation() throughout test suite
• Replaced start_generation() with start_observation(as_type="generation") throughout test suite
• Replaced update_trace() calls with propagate_attributes() context manager throughout test suite
• Removed test_deprecation.py file as deprecated methods are no longer available
• Updated imports in test files to use new API module structure
• Updated tests/utils.py to return LangfuseAPI instead of FernLangfuse

Documentation

• Deleted langfuse/api/README.md and langfuse/api/reference.md documentation files
• Added .fern/metadata.json configuration file for Fern API code generation with Pydantic v2 settings

Other Changes

• Updated langfuse/batch_evaluation.py to import from new API module structure
• Updated langfuse/experiment.py to use DatasetItem instead of DatasetItemClient
• Updated langfuse/model.py to consolidate imports from new API structure
• Removed langfuse/types.py type definitions that are now in API modules
• Added Python 3.14 to CI workflow matrix in .github/workflows/ci.yml

^{This description was created by for 8869b29. You can customize this summary. It will automatically update as commits are pushed.}

^{This description was created by for b1e2624. You can customize this summary. It will automatically update as commits are pushed.}

[ellipsis-dev]

⚠️ This PR is too big for Ellipsis, but support for larger PRs is coming soon. If you want us to prioritize this feature, let us know at help@ellipsis.dev

---

Generated with ❤️ by ellipsis.dev

[greptile-apps]

Too many files changed for review. (621 files found, 100 file limit)