edm

🌟 Architectural Vision & System Design

The Enterprise Data Manager (EDM) is architected as a hybrid ETL/ELT pipeline designed to bridge the gap between raw, unstructured/semi-structured data sources and highly optimized relational data warehouses. The system decouples compute (Python/Pandas) from storage (TSQL/SQL Server), allowing each layer to scale independently and perform operations where they are computationally cheapest.

[Raw Data Sources] ──> [uv-Managed Python Engine] ──> [TSQL Staging Layer] ──> [Production Data Warehouse]
                             (Pandas/PyArrow)             (Bulk Insert / ACID)          (Columnstore Indexes)

Core Data & System Flow

• Ingestion / Input: Data enters the system via memory-mapped PyArrow tables, minimizing serialization overhead. The ingestion layer is managed by a deterministic, ultra-fast uv virtual environment, ensuring zero-overhead dependency resolution and reproducible runtimes.
• Processing / Logic: Business logic, schema validation, and data cleansing are executed in-memory using vectorized Pandas operations. This avoids row-by-row iteration (RBAR) and maximizes CPU cache utilization.
• Persistence & Caching: Transformed data is streamed into TSQL staging tables using optimized bulk-copy protocols. Once staged, set-based TSQL stored procedures execute merge operations, manage surrogate key generation, and enforce referential integrity within strict transactional boundaries.

---

💻 Tech Stack & Engineering Decisions

Every technology in the EDM stack was selected to balance developer velocity, memory efficiency, and execution speed.

• Frontend / Control Plane: Headless, CLI-driven execution model. The runtime environment is managed via uv, which provides a 10-100x speedup in environment initialization and dependency resolution compared to traditional pip and poetry setups, making it ideal for ephemeral containerized deployments (e.g., AWS ECS, Kubernetes Jobs).