Alejandro Olivares Benítez

A near real-time ETL to provide current reliable data to an application that helps to operate a flight network. It relied on an Aurora data warehouse for current data and a Redshift data warehouse for historical data. The ETL was built in Python and the process runs through Airflow which executes ECS tasks.

Designed and implemented a scalable data pipeline using AWS Lambda and Step Functions to efficiently process and load medium-sized files into a PostgreSQL database.

The solution included:
• A Lambda function to split large files into smaller chunks for parallel processing.
• Multiple Lambda functions, each handling specific processing logic based on file content.
• Step Functions orchestrating the execution flow, ensuring fault tolerance and efficient retries.
• Seamless integration with S3 for file storage and RDS (PostgreSQL) for structured data persistence

Developed a real-time streaming data pipeline using Apache Beam (Java) to process terabytes of adtech events per hour efficiently.

The solution involved:
• Apache Beam (Dataflow) for distributed stream processing, ensuring low-latency event handling.
• Protobuf for efficient message parsing, reducing serialization overhead.
• Google Cloud Storage (GCS) as a staging area for reprocessing when needed.
• BigQuery as the final destination for analytics and reporting.
• Terraform for infrastructure as code (IaC), automating deployment and scaling.
• Performance optimizations to handle high-throughput event ingestion while ensuring cost efficiency and fault tolerance.