genai_training — Ankit Yadav

🌟 Architectural Vision & System Design

This repository serves as an interactive development and prototyping environment for Generative AI model training, fine-tuning, and evaluation. Rather than treating Jupyter Notebooks as isolated scratchpads, this system is architected as a modular, reproducible pipeline where notebooks act as the orchestrators for underlying high-performance PyTorch and Hugging Face workflows.

[ Raw Data Sources ] ──> [ Tokenization & Dynamic Padding ] ──> [ Quantized Base Model (NF4) ]
                                                                        │
[ Weights & Biases ] <── [ Evaluation & Validation ] <── [ PEFT / LoRA Adapter Training ]

Core Data & System Flow

Ingestion / Input: Raw instruction-tuning datasets are ingested, cleaned, and passed through a deterministic tokenization pipeline. Dynamic padding and smart batching are applied to minimize computational overhead during training.
Processing / Logic: The training loop leverages Parameter-Efficient Fine-Tuning (PEFT) with Low-Rank Adaptation (LoRA). Computation is optimized using mixed-precision (BF16/FP16) and gradient accumulation to simulate larger batch sizes on constrained hardware.
Persistence & Caching: Model checkpoints, adapter weights, and tokenizer configurations are serialized using the secure safetensors format. Intermediate states are cached locally using memory-mapped files to prevent redundant data processing.

💻 Tech Stack & Engineering Decisions

Every technology choice in this pipeline was made to balance developer velocity, resource constraints, and mathematical precision.

Execution Environment: Jupyter Notebooks were selected to provide an interactive, stateful interface for hyperparameter exploration, loss curve visualization, and rapid prototyping of model architectures.
Deep Learning Framework: PyTorch serves as the foundational tensor computation engine, offering low-level control over gradients and memory allocation.
Model Abstraction & Optimization: Hugging Face transformers and peft were chosen to standardize model loading, while bitsandbytes provides the 4-bit/8-bit quantization layers necessary to run large-scale models on commodity enterprise hardware.
Experiment Tracking: Weights & Biases (W&B) integration ensures that every training run is fully logged, versioned, and reproducible.

⚙️ Engineering Excellence & Best Practices

This codebase is built around production-grade machine learning engineering principles: