Denis Arkhipov

I designed and developed a high-throughput, event-driven data processing service for an energy trading company. The service consumes fundamental time-series data critical to energy trading—including renewables generation actuals and forecasts, imbalance prices, energy auction results, ancillary services data, and other market indicators—from Azure Event Hub and routes it through multiple processing pipelines: Hot Storage (PostgreSQL/TimescaleDB for real-time queries) and Cold Storage (ADLS2 as Parquet archives).

Built in Python with async I/O (asyncpg, uvloop), the system employs a pluggable handler/consumer architecture with abstract base classes, Avro serialization, and a hospital/dead-letter pattern for failed message recovery.

Deployed on Kubernetes using Helm charts and KEDA for autoscaling based on Event Hub consumer lag, the system includes operational observability with Opsgenie alerting, lag monitoring, sequence number gap detection, and liveness probes.

I also developed supporting tooling, including a backfiller for historical data replay to the ETRM platform and a metadata bootstrap tool for managing product mappings across European energy markets such as EEX and RTE.

An end-to-end maritime data intelligence platform to predict port congestion and time-to-berth using AIS vessel position data. I architected a medallion data lakehouse using Dagster for orchestration, dbt and DuckDB for transformations, and Azurite (ADLS2-compatible) for storage, all containerised via Docker Compose for reproducible local development.

I ingested years of global AIS records from a REST API with async concurrency and in-memory Parquet conversion (PyArrow, ZSTD compression). I also built silver-layer canonical models (vessels, positions, port calls) with deterministic upsert/conflict detection and dbt snapshots for SCD Type 2, and gold-layer hierarchical forecast features.

I built a PyArrow Flight Server on top of DuckDB to serve low-latency analytical queries, powering a React and Kepler.gl geospatial heatmap UI for real-time and historical vessel tracking and for viewing congested areas. I set up time-series forecasting baselines using StatsForecast and hierarchical forecasting for port congestion metrics, and built a model performance dashboard tracking RMSE, MAE, and MAPE.

Added an AI chatbot (Chainlit, Claude LLM, and a DuckDB MCP server) to enable natural language querying.

I developed an AI-powered digital twin for the food and agriculture sector, allowing deal teams, portfolio managers, and strategists to query nutritional, competitive, and regulatory data using natural language.

I ingested the full USDA FoodData Central dataset (2+ million products, 400,000+ unique foods, and 26+ million nutrient records) into Databricks Unity Catalog through automated pipelines. Using dbt transformation models, I created pre-pivoted analytical views from raw EAV nutrient tables, reducing query complexity and enhancing AI accuracy.

I configured Databricks Genie rooms with SQL expressions and entity relationships, tailored to investment use cases like M&A due diligence, portfolio nutritional benchmarking, commodity substitution analysis, and regulatory risk assessments against WHO, UK, Chile, and GCC standards. Through a structured benchmark suite, I improved Genie query accuracy from 30% to 100% over several tuning cycles using systematic instruction engineering and EAV pivot patterns.

This tool empowered non-technical investment professionals to conduct data-driven competitive analysis, target screening, and portfolio-wide regulatory stress testing without requiring SQL knowledge.