Add Centralized Logging and Monitoring Stack (Prometheus + Grafana) by berkal749 · Pull Request #44 · MicroClub-USTHB/mAIcro

berkal749 · 2026-03-23T19:13:53Z

I centralized the logger by creating the log.py file and importing the get_logger method
Updated the docker.compose.dev.yml script to add Grafana and the Prometheus database, and then created prometheus.yml
Monitoring stack setup (Prometheus + Grafana)
Prometheus config file was created prometheus.yml

Dev

…ocker setup with logging configuration

Copilot

Pull request overview

This PR introduces a basic observability stack (Prometheus + Grafana) via Docker Compose and provisions Grafana with a Prometheus datasource and a starter dashboard, alongside a small refactor to centralize logger creation in the FastAPI app.

Changes:

Add core.log.get_logger() and switch src/main.py to use it.
Add Prometheus + Grafana services to compose files, plus Prometheus and Grafana provisioning configs.
Add an initial Grafana dashboard JSON for service availability.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
src/main.py	Switch to centralized `get_logger()` usage.
src/core/log.py	New logging helper module with global logging configuration.
prometheus.yml	New Prometheus scrape configuration (mAIcro + qdrant).
grafana/provisioning/datasources/prometheus.yml	Provision Grafana Prometheus datasource.
grafana/provisioning/dashboards/dashboards.yml	Provision Grafana dashboard provider.
grafana/dashboards/maicro-overview.json	Add initial “mAIcro Overview” dashboard.
docker-compose.yml	Add Prometheus + Grafana services and a shared network/volumes.
docker-compose.dev.yml	Add Prometheus + Grafana to dev compose, plus shared network/volumes.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-03-23T19:17:57Z

+logging.basicConfig(
+    level=logging.INFO,
+    format="%(asctime)s | %(levelname)s | %(name)s | %(message)s",
+)
+
+
+def get_logger(name: str) -> logging.Logger:
+    """Get a logger instance with the app's default configuration."""


Configuring logging via logging.basicConfig(...) at import time is fragile under uvicorn (it typically installs handlers before importing the app, so basicConfig becomes a no-op). Consider moving configuration into an explicit configure_logging() that is invoked by the entrypoint, or configure a dedicated app logger/handler (or provide a uvicorn log_config) so formatting/level are reliably applied without global import side effects.

Suggested change

logging.basicConfig(

level=logging.INFO,

format="%(asctime)s | %(levelname)s | %(name)s | %(message)s",

)

def get_logger(name: str) -> logging.Logger:

"""Get a logger instance with the app's default configuration."""

_LOGGING_CONFIGURED = False

def configure_logging() -> None:

"""

Configure application logging.

This function is safe to call multiple times. If the root logger already

has handlers configured (for example, by uvicorn or another framework),

this function will not override that configuration.

"""

global _LOGGING_CONFIGURED

if _LOGGING_CONFIGURED:

return

root_logger = logging.getLogger()

# If handlers are already present (e.g., installed by uvicorn), respect

# the existing configuration and do not call basicConfig.

if not root_logger.handlers:

logging.basicConfig(

level=logging.INFO,

format="%(asctime)s | %(levelname)s | %(name)s | %(message)s",

)

_LOGGING_CONFIGURED = True

def get_logger(name: str) -> logging.Logger:

"""Get a logger instance with the app's default configuration."""

configure_logging()

Copilot · 2026-03-23T19:17:57Z

+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"


This dashboard references the Prometheus datasource via uid: "${DS_PROMETHEUS}", but there is no templating/input variable defined in the JSON (templating.list is empty) and the provisioned datasource file doesn’t set a fixed uid. This will usually make the dashboard fail to bind to the datasource on import/provisioning. Set an explicit datasource uid in Grafana provisioning and reference that literal uid in the dashboard JSON (or add the proper __inputs/templating variable wiring).

Suggested change

"uid": "${DS_PROMETHEUS}"

"uid": "prometheus"

Copilot · 2026-03-23T19:17:57Z

+    ports:
+      - "9090:9090"
+    volumes:
+      - ./prometheus.yml:/etc/prometheus/prometheus.yml


The Prometheus config bind-mount isn’t read-only here (unlike docker-compose.yml which uses :ro). Making it read-only helps prevent accidental in-container edits and keeps behavior consistent across environments.

Suggested change

- ./prometheus.yml:/etc/prometheus/prometheus.yml

- ./prometheus.yml:/etc/prometheus/prometheus.yml:ro

Copilot · 2026-03-23T19:17:58Z

@@ -12,13 +13,7 @@
 from core.ingestion import ingest_from_discord, run_startup_audit
 import logging


import logging is now unused after switching to get_logger; remove it to avoid ruff/linters failing and to reduce confusion about which logging config is active.

Suggested change

import logging

Copilot · 2026-03-23T19:17:58Z

+  - job_name: 'maicro'
+    static_configs:
+      - targets: ['maicro:8000']
+    metrics_path: '/metrics'
+


Prometheus is configured to scrape maicro:8000/metrics, but the FastAPI app currently doesn't expose a /metrics endpoint (no metrics/instrumentation found in src/). As-is, this scrape job will continuously 404; either add metrics instrumentation (e.g., Prometheus client/instrumentator) or adjust/remove this scrape config until the endpoint exists.

Suggested change

- job_name: 'maicro'

static_configs:

- targets: ['maicro:8000']

metrics_path: '/metrics'

Copilot · 2026-03-23T19:17:58Z

+      GF_SECURITY_ADMIN_USER: ${GRAFANA_ADMIN_USER:-admin}
+      GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_ADMIN_PASSWORD:-admin}


Grafana admin credentials default to admin/admin via ${...:-admin}. If this compose file is used beyond purely local dev, this creates an easily exploitable default. Prefer requiring these env vars (no insecure default), or inject via Docker secrets / a local-only override file.

Suggested change

GF_SECURITY_ADMIN_USER: ${GRAFANA_ADMIN_USER:-admin}

GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_ADMIN_PASSWORD:-admin}

GF_SECURITY_ADMIN_USER: ${GRAFANA_ADMIN_USER:?GRAFANA_ADMIN_USER must be set}

GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_ADMIN_PASSWORD:?GRAFANA_ADMIN_PASSWORD must be set}

Copilot · 2026-03-23T19:17:58Z

+      - GF_SECURITY_ADMIN_USER=${GRAFANA_ADMIN_USER:-admin}
+      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_ADMIN_PASSWORD:-admin}


Even in the dev compose, defaulting Grafana admin credentials to admin/admin can be risky (e.g., if ports are exposed on a shared network). Consider removing the defaults and documenting required env vars, or binding Grafana to localhost only when defaults are used.

Suggested change

- GF_SECURITY_ADMIN_USER=${GRAFANA_ADMIN_USER:-admin}

- GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_ADMIN_PASSWORD:-admin}

- GF_SECURITY_ADMIN_USER=${GRAFANA_ADMIN_USER:?GRAFANA_ADMIN_USER not set}

- GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_ADMIN_PASSWORD:?GRAFANA_ADMIN_PASSWORD not set}

Copilot · 2026-03-23T19:17:59Z

+datasources:
+  - name: Prometheus
+    type: prometheus
+    access: proxy
+    url: http://prometheus:9090
+    isDefault: true
+    editable: true


The provisioned Prometheus datasource doesn’t define a stable uid. Since dashboards often reference datasources by uid, provisioning without one can break pre-provisioned dashboards (including the one added in this PR that uses ${DS_PROMETHEUS}). Add an explicit uid here and keep dashboards aligned to it.

Copilot · 2026-03-23T19:17:59Z

+    metrics_path: '/metrics'
+
+  - job_name: 'qdrant'
+    static_configs:
+      - targets: ['qdrant:6333']


prometheus.yml includes a scrape job targeting qdrant:6333, but the non-dev docker-compose.yml doesn’t define a qdrant service/network alias. In that environment Prometheus will log repeated DNS/connection failures. Either add qdrant to docker-compose.yml (if intended), or split Prometheus config per environment so prod doesn’t reference a missing target.

Suggested change

metrics_path: '/metrics'

- job_name: 'qdrant'

static_configs:

- targets: ['qdrant:6333']

metrics_path: '/metrics'

YounesBensafia and others added 2 commits March 22, 2026 00:11

Merge pull request MicroClub-USTHB#41 from MicroClub-USTHB/dev

5fb904b

Dev

(idk bro but i did somthing) Add Prometheus and Grafana services to D…

27b2143

…ocker setup with logging configuration

Copilot AI review requested due to automatic review settings March 23, 2026 19:13

Copilot started reviewing on behalf of berkal749 March 23, 2026 19:14 View session

Copilot AI reviewed Mar 23, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Centralized Logging and Monitoring Stack (Prometheus + Grafana)#44

Add Centralized Logging and Monitoring Stack (Prometheus + Grafana)#44
berkal749 wants to merge 2 commits intoMicroClub-USTHB:devfrom
berkal749:main

berkal749 commented Mar 23, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Mar 23, 2026

Uh oh!

Copilot AI Mar 23, 2026

Uh oh!

Copilot AI Mar 23, 2026

Uh oh!

Copilot AI Mar 23, 2026

Uh oh!

Copilot AI Mar 23, 2026

Uh oh!

Copilot AI Mar 23, 2026

Uh oh!

Copilot AI Mar 23, 2026

Uh oh!

Copilot AI Mar 23, 2026

Uh oh!

Copilot AI Mar 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

-logging.basicConfig(
-    level=logging.INFO,
-    format="%(asctime)s | %(levelname)s | %(name)s | %(message)s",
-)
-def get_logger(name: str) -> logging.Logger:
-    """Get a logger instance with the app's default configuration."""
+_LOGGING_CONFIGURED = False
+def configure_logging() -> None:
+    """
+    Configure application logging.
+    This function is safe to call multiple times. If the root logger already
+    has handlers configured (for example, by uvicorn or another framework),
+    this function will not override that configuration.
+    """
+    global _LOGGING_CONFIGURED
+    if _LOGGING_CONFIGURED:
+        return
+    root_logger = logging.getLogger()
+    # If handlers are already present (e.g., installed by uvicorn), respect
+    # the existing configuration and do not call basicConfig.
+    if not root_logger.handlers:
+        logging.basicConfig(
+            level=logging.INFO,
+            format="%(asctime)s | %(levelname)s | %(name)s | %(message)s",
+        )
+    _LOGGING_CONFIGURED = True
+def get_logger(name: str) -> logging.Logger:
+    """Get a logger instance with the app's default configuration."""
+    configure_logging()

	- ./prometheus.yml:/etc/prometheus/prometheus.yml
	- ./prometheus.yml:/etc/prometheus/prometheus.yml:ro

		@@ -12,13 +13,7 @@
		from core.ingestion import ingest_from_discord, run_startup_audit
		import logging

		GF_SECURITY_ADMIN_USER: ${GRAFANA_ADMIN_USER:-admin}
		GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_ADMIN_PASSWORD:-admin}

		- GF_SECURITY_ADMIN_USER=${GRAFANA_ADMIN_USER:-admin}
		- GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_ADMIN_PASSWORD:-admin}

Conversation

berkal749 commented Mar 23, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants