Skip to content

+ Add pytest db and migrations creation fixtures#83

Open
atin65536 wants to merge 4 commits intomasterfrom
feat/82-pytest-db-fixtures
Open

+ Add pytest db and migrations creation fixtures#83
atin65536 wants to merge 4 commits intomasterfrom
feat/82-pytest-db-fixtures

Conversation

@atin65536
Copy link
Copy Markdown

Closes #82.

@gemini-code-assist
Copy link
Copy Markdown

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the testing infrastructure by introducing a robust set of pytest fixtures and utilities for managing test databases and migrations. It refactors core SQL storage components to incorporate extensive type hinting and a flexible environment-based configuration system for database engines. These changes aim to provide developers with powerful tools for creating isolated and reliable testing environments, ultimately improving the overall quality and testability of database-dependent features.

Highlights

  • Pytest Integration: Introduced new pytest fixtures and utilities to simplify database and migration testing, including DBConfig, TestDBManagerCreator, and MigrationEngineCreator.
  • Type Hinting Enhancements: Significantly expanded type hinting across core modules like common/utils.py, storage/base.py, storage/sql/engines.py, storage/sql/migrations.py, storage/sql/orm.py, and storage/sql/sessions.py for improved code clarity and maintainability.
  • Environment-based Engine Configuration: Added a new env_config.py module and integrated it into engines.py to allow database engine configurations to be loaded dynamically from environment variables, supporting multiple named engines.
  • Improved Engine Management: Enhanced the EngineFactory with a using context manager and a close method for AbstractEngine and its implementations, ensuring proper resource cleanup for database connections.
  • Separate Engine Support for Models: Modified SQLStorableMixin and ObjectCollection to support associating models with specific named database engines, enabling more flexible and isolated data storage.
Changelog
  • pyproject.toml
    • Added restalchemy.testing.pytest.plugin as a pytest entry point.
  • restalchemy/common/utils.py
    • Added typing import and T = tp.TypeVar("T").
    • Made classproperty generic and added type hints to its __init__ and __get__ methods.
  • restalchemy/storage/base.py
    • Added typing import and defined several type aliases for filters and order-by clauses.
    • Made AbstractObjectCollection generic and added type hints to its methods.
    • Added _Self TypeVar to AbstractStorableMixin and type hints to its abstract methods.
  • restalchemy/storage/sql/engines.py
    • Added typing, os, dataclasses imports and imported env_config module.
    • Defined EngineNameOrInstance type alias.
    • Added type hints to various db_ properties and connection/session management methods in AbstractEngine.
    • Introduced an abstract close method to AbstractEngine and implemented it for PgSQLEngine and MySQLEngine.
    • Modified EngineFactory to include _engines_using list, added configure_factory_from_env method, and a using context manager.
    • Updated get_engine to handle _engines_using and destroy_engine/destroy_all_engines to call close on engines.
    • Added a module-level using function for convenience.
  • restalchemy/storage/sql/env_config.py
    • Added new file to define exceptions and dataclasses for environment-based database engine configuration.
    • Implemented EngineENVConfig for parsing engine details from environment variables.
    • Created EngineENVConfigs to manage multiple engine configurations, including setup from DATABASE_ENGINES.
  • restalchemy/storage/sql/migrations.py
    • Added typing import and imported the engines module.
    • Modified MigrationEngine.__init__ to accept an optional engine name.
    • Updated apply_migration, rollback_migration, and get_unapplied_migrations to use the engines.using context manager.
  • restalchemy/storage/sql/orm.py
    • Added typing imports for ClassVar, Generic, Optional, Type, TypeVar.
    • Imported AbstractSession, OptionalFilters, OptionalOrderBy from base and sessions.
    • Defined SQLStorableMixinType and OptionalAbstractSession.
    • Made ObjectCollection generic and updated its _engine property to use model_cls.get_engine_name().
    • Added _Self TypeVar to SQLStorableMixin and defined __tablename__ and __engine_name__ as ClassVar.
    • Added get_engine_name and updated _get_engine to use it.
    • Added type hints to restore_from_storage, insert, save, update, and delete methods.
    • Updated SQLStorableMixin.from_simple_type to use cls._get_engine().
  • restalchemy/storage/sql/sessions.py
    • Added typing and TracebackType imports.
    • Defined several type aliases for parameters, rows, and SQLStorableMixin.
    • Introduced abstract protocols for AbstractContextManager, AbstractCursor, AbstractConnection, and AbstractSession to standardize database interaction interfaces.
  • restalchemy/testing/pytest/init.py
    • Added new file to import and re-export fixtures from db and migrations submodules.
  • restalchemy/testing/pytest/db.py
    • Added new file defining DBConfig for test database configuration.
    • Provided setup_engine_from_env and database_config_from_env fixtures.
    • Introduced TestDBManagerCreator and db_manager_creator fixture for creating TestDBManager instances, including database creation/dropping and table clearing.
    • Added default_db_manager fixture.
  • restalchemy/testing/pytest/migrations.py
    • Added new file defining MigrationEngineCreator and migration_engine_creator fixture for creating TestMigrationManager instances.
    • Implemented logic for applying and rolling back migrations within a test context.
    • Added default_migration_engine fixture.
  • restalchemy/testing/pytest/plugin.py
    • Added new file to register pytest plugins: restalchemy.testing.pytest.utils, restalchemy.testing.pytest.db, restalchemy.testing.pytest.migrations.
  • restalchemy/testing/pytest/utils.py
    • Added new file providing xdist_worker_id fixture to detect pytest-xdist worker ID for parallel testing.
  • restalchemy/testing/typing.py
    • Added new file defining common type aliases used in testing utilities.
  • restalchemy/testing/utils/init.py
    • Added new file to import and re-export TestDBManagerConfig, TestDBManager, TestMigrationManagerConfig, TestMigrationManager.
  • restalchemy/testing/utils/db.py
    • Added new file defining utility functions get_database_uri and get_database_postfix.
    • Introduced ClearTableRecord and ClearTableRecords for managing table clearing operations.
    • Implemented TestDBManagerConfig and TestDBManager for setting up and tearing down test databases, including connection/session management, database creation/dropping, and table clearing.
  • restalchemy/testing/utils/migrations.py
    • Added new file defining TestMigrationManagerConfig and TestMigrationManager for managing migration application and rollback in a test environment.
  • restalchemy/tests/pytest/migrations/0000-init-cb5624.py
    • Added new file defining an initial migration step for a model table.
  • restalchemy/tests/pytest/models.py
    • Added new file defining Model and SecondModel classes, inheriting from SQLStorableMixin, with __tablename__ and __engine_name__ attributes.
  • restalchemy/tests/pytest/second_migrations/0000-init-1e68bf.py
    • Added new file defining an initial migration step for a second_model table.
  • restalchemy/tests/pytest/test_separate_engines.py
    • Added new file containing pytest tests demonstrating the use of separate database engines and migration managers for different models.
  • tox.ini
    • Added pytest to the envlist for tox environments.
    • Added pytest specific setenv and passenv configurations for DATABASE_URI, DATABASE_SECOND_URI, and DATABASE_ENGINES.
Ignored Files
  • Ignored by pattern: .github/workflows/** (1)
    • .github/workflows/tests.yml
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

Copy link
Copy Markdown

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a comprehensive testing framework using pytest, focusing on database and migration management. It adds support for multiple database engines configurable via environment variables, which is a significant enhancement. The changes include new pytest fixtures, testing utilities, and extensive type hinting across the storage layer, which greatly improves code quality and maintainability. I've identified a few issues, including a critical bug in path resolution for migrations, some potential crashes if configurations are incomplete, and some non-portable SQL in the testing utilities that could limit database compatibility. Addressing these points will make the new testing infrastructure more robust and reliable.

Comment thread restalchemy/testing/utils/migrations.py Outdated
Comment thread restalchemy/testing/utils/migrations.py
Comment thread restalchemy/storage/sql/env_config.py Outdated
Comment thread restalchemy/testing/utils/db.py Outdated
Comment thread restalchemy/testing/utils/db.py Outdated
Comment thread restalchemy/testing/utils/db.py Outdated
Comment thread restalchemy/tests/pytest/test_separate_engines.py Outdated
from restalchemy.storage.sql.dialect import mysql
from restalchemy.storage.sql.dialect import pgsql
from restalchemy.storage.sql import sessions
from restalchemy.storage.sql.env_config import (
Copy link
Copy Markdown
Author

@atin65536 atin65536 Mar 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This imports have that form especial for reexport.

from restalchemy.storage.sql import engines
from restalchemy.storage.sql import tables
from restalchemy.storage.sql.sessions import AbstractSession
from restalchemy.storage.base import OptionalFilters, OptionalOrderBy
Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changing this import to

from restalchemy.storage import base

and following usage like

base.OptionalFilters

Would not improve code readability and clearance.
Module name base is too common.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

from restalchemy.storage import base as base_storage

RowType = tp.Tuple[tp.Any, ...]
DictRowType = tp.Dict[str, tp.Any]

if tp.TYPE_CHECKING:
Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this case import module directly is impossible.

@@ -0,0 +1,7 @@
from .db import (
Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reexport for simplifying usage. It must be kept as is.

Comment thread restalchemy/testing/pytest/db.py Outdated
env_configs,
DEFAULT_NAME,
)
from restalchemy.testing.typing import SimpleGenerator, WorkerID
Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

from restalchemy.testing import typing as WHAT

Comment thread restalchemy/testing/pytest/db.py Outdated
DEFAULT_NAME,
)
from restalchemy.testing.typing import SimpleGenerator, WorkerID
from restalchemy.testing.utils.db import (
Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

from restalchemy.testing.utils import db  # `db` is too common

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

from restalchemy.testing.utils import db as db_utils

import pytest

from restalchemy.testing.typing import SimpleGenerator
from restalchemy.testing.utils.migrations import (
Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

from restalchemy.testing.utils import migrations  # `migrations` name is too common

@@ -0,0 +1,8 @@
from .db import (
Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reexport for simplifying usage. It must be kept as is.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Transfer DB fixtures from GC

2 participants