TrainOps Observatory

Lightweight observability and cost optimization for ML training workflows

TrainOps Observatory provides real-time visibility into ML training workflows with minimal overhead. Add 5 lines of code to your training script and get instant insights into GPU utilization, bottlenecks, and cost optimization opportunities.

✨ Features

• 🚀 Minimal Integration - Add 5 lines to existing training code
• 📊 Real-Time Metrics - GPU/CPU utilization, throughput, memory
• 🔍 Bottleneck Detection - Automatically identify I/O, CPU, or GPU constraints
• 💰 Cost Tracking - Estimate and optimize training costs
• 🎯 Zero Overhead - < 1% impact on training time
• 🛠️ CLI & Dashboard - View metrics via command-line or web interface

🎬 Quick Demo

from trainops import TrainOpsMonitor

monitor = TrainOpsMonitor(run_name="my_experiment", project="research")

@monitor.track_training
def train_epoch(model, dataloader, optimizer):
    for batch in dataloader:
        # ... your training code ...
        monitor.log_step(loss=loss.item())

# Train
for epoch in range(10):
    train_epoch(model, train_loader, optimizer)
    monitor.log_epoch(epoch)

monitor.finish()

Add 5 lines of code. Save 30-40% on GPU training costs.

TrainOps Observatory provides real-time visibility into ML training workflows with minimal overhead. Automatically detect bottlenecks, get specific optimization recommendations, and track cost savings—all with less than 1% performance impact.

View results:

trainops runs show <run-id>

🚀 Quick Start

1. Start the Backend (30 seconds)

git clone https://github.com/nehadangwal/trainops-observatory
cd trainops-observatory

# Start services
docker-compose up -d

# Verify
curl http://localhost:5000/health

2. Install SDK (10 seconds)

cd sdk
pip install -e .

3. Run Example (2 minutes)

cd examples
python mnist_simple.py

See QUICKSTART.md for detailed setup.

📖 Documentation

• User Guide - Complete usage guide
• Technical Deep-Dive - Architecture and implementation
• Examples - Sample training scripts
• API Reference - REST API documentation

🎯 Use Cases

Identify Training Bottlenecks

# Run with different data loading configurations
python examples/resnet_cifar10.py --scenario baseline
python examples/resnet_cifar10.py --scenario optimized

# Compare results
trainops runs list --project cifar10_classification

Common Findings:

• 40-60% GPU utilization → I/O bottleneck (add num_workers)
• High CPU utilization → Data preprocessing bottleneck
• Low throughput → Batch size too small

Optimize Training Costs

Optimize Training Costs (Validated Impact) TrainOps tracks instance costs and identifies optimization opportunities.

Real-World Example: Fixing an I/O Bottleneck

Running a benchmark on Google Colab (T4 GPU) identified an I/O bottleneck (num_workers=0). By implementing the recommended fix (num_workers=4), the following measurable impact was achieved:

🚨 Bottleneck Detected: I/O Bound (32.7% GPU utilization)

Recommendation: Add num_workers=4 to DataLoader

💰 Estimated Impact: • Training Time: 3.14 min → 2.13 min (-32.1% Faster) • Throughput: 1653 samples/s → 2588 samples/s (+56.6% Increase) • Cost Savings: $0.026 per run → $0.018 per run (-32.1% Reduction)

🔑 Key Takeaway: Same compute resources, 32% faster results, enabling 1.5x more experiments in the same time.

🎯 Proven Results We validated TrainOps on real GPU training workloads. Here's what we found: ResNet-18 on CIFAR-10 (Google Colab T4 GPU) The Problem: Training was slower than expected due to I/O bottleneck The Fix: Single configuration change detected by TrainOps (num_workers: 0 → 4) Implementation Time: 1 minute (1 line of code) Results: ┌─────────────────────────────────────────────────────────────────┐ │ BEFORE → AFTER IMPROVEMENT │ ├─────────────────────────────────────────────────────────────────┤ │ Training Time 3.15 min → 2.14 min -32% ⬇️ │ │ Throughput 1,656 s/s → 2,564 s/s +55% ⬆️ │ │ Cost per Run $0.026 → $0.018 -32% ⬇️ │ └─────────────────────────────────────────────────────────────────┘

💡 Key Insight: Processing 907 more samples/second with same GPU What This Means for You: Your GPU SpendMonthly Savings (32%)Annual Savings$500/month (Individual)$160/month$1,920/year$5,000/month (Small Team)$1,600/month$19,200/year$50,000/month (Medium Team)$16,000/month$192,000/year$500,000/month (Large Team)$160,000/month$1,920,000/year

💰 ROI Example: For a team spending $50K/month on GPUs, TrainOps saves $16K/month. At $100/user/month for 10 users ($1,000/month), that's a 16x return on investment.

Track Team-Wide Metrics Bash

View all team experiments

trainops runs list --project team_research

Get platform statistics

trainops stats

🏗️ Architecture

┌─────────────┐     ┌──────────────┐     ┌─────────────┐
│  Training   │────▶│   Backend    │────▶│  Database   │
│   Script    │     │    (Flask)   │     │ (TimescaleDB│
│  (Python)   │     │              │     │             │
└─────────────┘     └──────────────┘     └─────────────┘
                            │
                            ▼
                    ┌──────────────┐
                    │     CLI      │
                    │   Dashboard  │
                    └──────────────┘

Components:

• Python SDK - Lightweight instrumentation (<1% overhead)
• Flask Backend - REST API for metrics ingestion
• TimescaleDB - Time-series optimized PostgreSQL
• CLI Tool - Command-line interface for viewing runs
• Dashboard (Coming Soon) - Web UI for visualization

🔧 CLI Usage

# List runs
trainops runs list
trainops runs list --project mnist
trainops runs list --status running

# Show run details
trainops runs show <run-id>

# View metrics
trainops runs metrics <run-id>
trainops runs metrics <run-id> --tail --limit 20

# List projects
trainops projects

# Platform statistics
trainops stats

# Delete runs
trainops runs delete <run-id>

⚙️ Configuration

Environment Variables

# API Configuration
export TRAINOPS_API_URL="http://localhost:5000"
export TRAINOPS_API_TIMEOUT=10

# Collection Settings
export TRAINOPS_COLLECT_INTERVAL=10  # seconds
export TRAINOPS_SEND_INTERVAL=30     # seconds

# Logging
export TRAINOPS_LOG_LEVEL=INFO

# Features
export TRAINOPS_ENABLE_GPU=true
export TRAINOPS_FAIL_ON_ERROR=false

Programmatic Configuration

monitor = TrainOpsMonitor(
    run_name="experiment_v2",
    project="research",
    api_url="http://custom:5000",
    instance_type="p3.8xlarge",
    tags={"team": "ml", "priority": "high"},
    collect_interval=5,
    send_interval=15
)

📊 Metrics Collected

Automatic System Metrics

• GPU utilization (%)
• GPU memory (used/total GB)
• CPU utilization (%)
• System RAM (%)
• Disk I/O (MB read/write)
• Training throughput (samples/sec)

Custom Metrics

Log any custom metrics via monitor.log_step():

monitor.log_step(
    loss=loss.item(),
    accuracy=acc,
    learning_rate=lr,
    custom_metric=value
)

🤝 Contributing

Contributions welcome! Please see CONTRIBUTING.md for guidelines.

📝 License

MIT License - see LICENSE for details.

🎯 Roadmap

Week 1 (Current): ✅

• Core SDK with decorator pattern
• Backend API with TimescaleDB
• CLI tool
• Example scripts
• Documentation

Week 2 (Next):

• Bottleneck detection engine
• Cost estimation with cloud pricing
• Next.js dashboard
• Real-time metrics visualization
• Run comparison view

Week 3:

• User validation with 3-5 ML practitioners
• Case studies with quantified savings
• Demo video
• Technical blog post

Future (v2):

• Multi-framework support (TensorFlow, JAX)
• Distributed training support
• Auto-optimization recommendations
• Team dashboards
• Slack/email alerts
• Carbon footprint tracking

📧 Contact

Neha Dangwal

• GitHub: @nehadangwal
• LinkedIn: linkedin.com/in/nehadangwal
• Email: dangwalneha2013@gmail.com
• Portfolio: nehadangwal.github.io

🙏 Acknowledgments

Built as part of a journey into ML infrastructure and AI safety research.

---

⭐ If you find this useful, please star the repo!# TrainOps Observatory