Shahida R. Khan

I designed and implemented an end-to-end cloud-native financial ledger engine that processes high-velocity trading data with zero data loss. I integrated AWS MSK (Kafka) stream ingestion with Apache Spark structured micro-batches. I also implemented atomic MERGE INTO operations on Delta Lake tables anchored on unique composite business hashes to guarantee deterministic recovery and pipeline idempotency. Finally, I substituted heavy network-wide shuffles with sub-millisecond DynamoDB lookups to seamlessly scale stream enrichment during intense market volatility.

I initiated a sweeping platform optimization track to remediate the Small File Problem across massive active data tiers. I programmed automated background file compaction and conditional VACUUM workflows on historical tables, cutting cloud compute waste on Databricks by 30% and saving $12,000 monthly. I also integrated Great Expectations into automated CI/CD pipelines to enforce mandatory runtime data quality assertions, reducing downstream staging-to-production data contract breaches to zero.

I architected a decoupled, high-throughput Lambda data platform to handle raw user clickstream and system performance tracking logs for millions of active daily users. I deployed distributed PySpark ETL pipelines on multi-node AWS EMR clusters. I also eliminated persistent data skew bottlenecks by implementing advanced, custom partitioning and broadcasting strategies, enabling the core analytical warehouse tables to meet a tight 15-minute data availability SLA for real-time recommendation engines.