Shahida R. Khan

I designed and implemented an end-to-end cloud-native financial ledger engine that processes high-velocity trading data with zero data loss. I integrated AWS MSK (Kafka) stream ingestion with Apache Spark structured micro-batches. I also implemented atomic MERGE INTO operations on Delta Lake tables anchored on unique composite business hashes to guarantee deterministic recovery and pipeline idempotency. Finally, I substituted heavy network-wide shuffles with sub-millisecond DynamoDB lookups to seamlessly scale stream enrichment during intense market volatility.

I initiated a sweeping platform optimization track to remediate the Small File Problem across massive active data tiers. I programmed automated background file compaction and conditional VACUUM workflows on historical tables, cutting cloud compute waste on Databricks by 30% and saving $12,000 monthly. I also integrated Great Expectations into automated CI/CD pipelines to enforce mandatory runtime data quality assertions, reducing downstream staging-to-production data contract breaches to zero.

I architected a decoupled, high-throughput Lambda data platform to handle raw user clickstream and system performance tracking logs for millions of active daily users. I deployed distributed PySpark ETL pipelines on multi-node AWS EMR clusters. I also eliminated persistent data skew bottlenecks by implementing advanced, custom partitioning and broadcasting strategies, enabling the core analytical warehouse tables to meet a tight 15-minute data availability SLA for real-time recommendation engines.

As a staff data engineer, I architected and single-handedly deployed a mission-critical proof of reserve (PoR) data pipeline from scratch to guarantee real-time cryptographic proof of funds for enterprise digital asset inventories. Operating as a strong individual contributor (IC), I owned the entire product lifecycle—from translating ambiguous financial requirements to executing the final production deployment.

I engineered a fault-tolerant ingestion framework that successfully aggregated high-velocity, fragmented data across Fireblocks vaults, centralized exchange APIs, and raw on-chain blockchain ledgers. To ensure 100% token coverage and zero-variance reporting, I built rigorous data reconciliation models with strict automated quality gates. I also authored comprehensive standard operating procedures (SOPs) to establish seamless operational governance for cross-functional teams.

This project demonstrated my ability to operate autonomously at a high level, maintaining absolute data reliability while executing at maximum speed. For architecting and deploying this fully production-ready pipeline in just two weeks, I was awarded the company-wide "Whale of the Quarter" honor for extraordinary performance.

As the lead data architect, I spearheaded the comprehensive security overhaul and governance restructuring of our enterprise data platform, successfully achieving ISO certification for 2026. Anticipating the strict regulatory and data sovereignty requirements of the MENA (Middle East and North Africa) region, I proactively designed and implemented robust security frameworks to ensure the platform was entirely audit-ready.

Operating as a highly autonomous IC and technical lead, I conducted rigorous pre-audit vulnerability assessments, identifying and resolving all critical security concerns prior to the external audit. I established strictly enforced data governance checklists, automated role-based access control (RBAC) validations, and comprehensive data encryption standards for data at rest and in transit.

By standardizing these compliance protocols and aligning our cloud infrastructure with MENA-specific enterprise security expectations, I ensured a flawless external audit with zero non-conformities. This project not only fortified the platform's security posture but also significantly elevated its commercial viability for enterprise and fintech clients operating in the Middle East.