CLI Data Parser v2.0

A modular data pipeline system built as a CLI tool — not just a parser. Think of it like a mini Apache Spark / Airflow pipeline, or a backend data processing service.

---

What This Project Really Is

This is not just a parser.

It's a modular data pipeline system with:

• Separation of concerns
• Extendable architecture
• CLI usability

In industry terms:

• Like a mini Apache Spark / Airflow pipeline (simplified)
• Or a backend data processing service

---

Overall Flow (Burn This Into Your Brain)

Every file connects to this pipeline:

INPUT FILE → PARSER → CLEANER → TRANSFORMER → AGGREGATOR → WRITER → OUTPUT

Think of it like a factory assembly line — each stage does one job and passes the result forward.

---

main.py — THE CONTROLLER

This is the orchestrator (very important concept).

• Accepts CLI arguments
• Calls each stage in order
• Handles errors gracefully
• Controls the entire pipeline flow

---

Strengths

• Clean modular design
• Good separation of concerns (parser / processor / output)
• CLI-based → practical and reusable
• Supports multiple formats → CSV, JSON, XML
• Zero external dependencies (pure Python stdlib)

---

Project Structure

cli-data-parser/
├── parser/
│   ├── __init__.py
│   ├── csv_parser.py        ← chunked reading for large files (100MB+)
│   ├── json_parser.py       ← handles nested/flat JSON structures
│   └── xml_parser.py        ← namespace + attribute support
├── processor/
│   ├── __init__.py
│   ├── cleaner.py           ← clean, deduplicate, normalize nulls
│   ├── transformer.py       ← AND/OR multi-condition filter expressions
│   ├── aggregator.py        ← min, max, avg, sum, std_dev, top values
│   └── validator.py         ← schema validation (type, required, range, allowed)
├── output/
│   ├── __init__.py
│   └── writer.py            ← structured JSON with metadata + invalid records
├── utils/
│   ├── __init__.py
│   ├── logger.py            ← colored terminal output (Green/Red/Blue/Yellow)
│   └── batch.py             ← multi-file, directory, and glob pattern support
├── sample_data/
│   ├── sample.csv
│   ├── sample.json
│   ├── sample.xml
│   └── schema.json
├── main.py                  ← CLI entry point (orchestrator)
├── requirements.txt
└── README.md

---

Key Features

Feature	Description
Multi-format	CSV, JSON, XML
Batch processing	Directory or glob pattern input
Smart cleaning	Deduplication, null normalization, type cast
AND/OR filtering	Multi-condition filter expressions
Schema validation	Type, required, min/max, allowed values
Rich aggregation	min, max, avg, sum, std_dev, top values
Large file support	Chunked CSV reading (100MB+)
Colored output	Success/Error/Warning terminal colors
Zero dependencies	Python standard library only

---

⚡ Installation

git clone https://github.com/ganesh44we/cli-data-parser.git
cd cli-data-parser
python main.py --help

---

🚀 Usage Examples

Basic Parsing

python main.py --input sample_data/sample.csv --format csv --output result.json
python main.py --input sample_data/sample.json --format json --output result.json
python main.py --input sample_data/sample.xml --format xml --output result.json

Filter with AND / OR

python main.py --input sample_data/sample.csv --format csv --filter "age>25" --output result.json
python main.py --input sample_data/sample.csv --format csv --filter "age>25 AND status=active" --output result.json
python main.py --input sample_data/sample.csv --format csv --filter "status=active OR status=pending" --output result.json

Batch Processing

python main.py --input sample_data/ --output result.json
python main.py --input "sample_data/*.csv" --output result.json

Schema Validation

python main.py --input sample_data/sample.csv --format csv --schema sample_data/schema.json --output result.json

Full Pipeline

python main.py \
  --input sample_data/sample.csv \
  --format csv \
  --filter "age>25 AND status=active" \
  --schema sample_data/schema.json \
  --aggregate \
  --output result.json

Large File Support

python main.py --input big_data.csv --format csv --chunk-size 50000 --output result.json

---

⚙️ CLI Options

Option	Description
--input	File, directory, or glob pattern (required)
--format	csv / json / xml (auto-detected for batch)
--output	Output JSON file path (required)
--filter	Filter: "age>25" or "age>25 AND status=active"
--aggregate	Include min/max/avg/std_dev summary
--schema	Path to JSON schema for validation
--no-clean	Skip cleaning step
--no-transform	Skip transformation step
--chunk-size	Chunk size for large CSV files (default: 10000)

---

📊 Output Format

{
    "metadata": {
        "total_records": 6,
        "output_file": "result.json",
        "invalid_records": 1
    },
    "data": [...],
    "summary": {
        "total_records": 6,
        "fields": {
            "age": {
                "type": "numeric",
                "min": 25, "max": 40,
                "avg": 31.6, "sum": 190,
                "std_dev": 5.2,
                "null_count": 1
            },
            "status": {
                "type": "string",
                "unique_count": 2,
                "top_values": [
                    {"value": "active", "count": 5},
                    {"value": "inactive", "count": 1}
                ]
            }
        }
    },
    "invalid_records": [...]
}

---

🛠️ Tech Stack

• Python 3.10+
• Standard Library: csv, json, xml, argparse, re, glob, collections

---

Author

Ganesh Rayapati