Senior Data Engineer
2020 - 2021Arthur Intelligence- Designed and developed novel data pipelines and operations to collect data from heterogeneous databases running on third-party Windows servers.
- Designed and built self-updating persistent data extractors for Windows.
- Designed and built ETL and aggregation pipeline framework in Airflow on Kubernetes (GKE).
- Planned and executed the overall DevOps and automation.
Technologies: Python 3, Google Cloud Platform (GCP), Kubernetes, Google Kubernetes Engine (GKE), Docker, Docker Compose, JavaScript, ETL, Data Engineering, Google Cloud SQL, Ansible, Terraform, Poetry, invoke, Big Data Architecture, Pipelines, ComplianceSenior Data Engineer
2020 - 2021Precisely- Developed and enhanced ETL pipelines for mapping data from a variety of vendors.
- Investigated and fixed customer-reported issues with address data.
- Stabilized the existing pipelines, added monitoring and tests, and facilitated the knowledge transfer to the new team.
Technologies: Big Data, ETL Tools, ETL, Mapping, GIS, Amazon S3 (AWS S3), Amazon Web Services (AWS), Java 8, PostGIS, PostgreSQL, Jenkins, Jenkins Pipeline, GroovyCTO
2004 - 2021Visual Units- Made a full-fledged fleet management system from the ground up.
- Took the product through (so far) three successful major database migrations without downtime.
- Developed a product and infrastructure with less than an hour of downtime over the last three years, despite 24/7 traffic.
- Created tools to generate Excel reports for customers on demand.
- Developed a dead-reckoning system to track rail car speeds.
- Ended up buying out the company and its products.
Technologies: Celery, Debian, Linux, Django, Memcached, MongoDB, MySQL, C, Python, Java MESenior Data Engineer
2019 - 2020BCG GAMMA- Worked closely with the team of data scientists, enabling quick turnaround time for personalized offers and measurements at a chain of luxury retail stores.
- Designed and implemented ETL pipelines, CI/CD, and automation for provisioning EMR PySpark clusters on demand for a personalization project.
Technologies: Amazon Web Services (AWS), Amazon S3 (AWS S3), Spark, PySpark, Boto 3, Apache Airflow, EMR, PythonSenior Software Engineer
2018 - 2019Pitney Bowes- Reviewed and audited a 35 kLOC ETL implementation for geographical data processing.
- Helped to design and implement a new platform for ETL processing and normalization of geographical data from a variety of vendors.
- Created new environments for a complex set of ETL pipelines on AWS, including securing and arranging access to new VPCs and existing VPNs.
- Created the new ETL pipeline for one vendor of geographical data, loading and transforming large amounts of data, testing the resulting output, and producing intermediate and redistributable datasets.
- Created common/reusable stages for automating the database (SQL Server) loading/offloading using S3, bulk copies, and compression on EC2.
Technologies: Technical Leadership, Amazon Web Services (AWS), ETL, Amazon Virtual Private Cloud (VPC), Amazon ECS (Amazon Elastic Container Service), Amazon EC2, Apache Airflow, Terraform, Python, Pipelines, Jenkins, GroovyBack-end/CloudHSM Developer
2018 - 2018Sodlab- Built HSM-backed digital signatures for PDF from the ground up.
- Set up automated provisioning of CloudHSM identities.
- Integrated digital signatures and visual elements in PDF design.
- Set up a management and transaction API for the digital signing of legal documents using Dropwizard.
- Implemented synchronous and asynchronous workflows for signing using single or two-factor authorization.
Technologies: Compliance, Amazon Web Services (AWS), PDF, PDFBox, Java 8, Dropwizard, Amazon EC2, Amazon CloudHSMSenior Consultant
2017 - 2018Meniga- Planned, designed, and implemented real-time streaming ETL solutions using an AWS stack, lowering information lag from six hours to one minute.
- Coached technical leadership. Provided feedback and evaluations on the tech stack and work methods.
- Implemented new serverless services in an existing microservice architecture in Python and Go.
- Supported the sales organization with improved data modeling and performance improvements in the OLAP (Redshift) data store.
- Planned and set up CI integration with AWS Codebuild.
- Developed tools for identifying and analyzing performance issues in the analytics pipeline.
- Involved in planning and management of daily team activities for development teams.
Technologies: Leadership, Compliance, Technical Leadership, AWS Kinesis, Periscope, Redshift, PostgreSQL, AWS Systems Manager, Amazon ECS (Amazon Elastic Container Service), AWS Lambda, Python, GoSenior Cloud Architect
2017 - 2017QuantumBlack- Built a secure data ingestion system for sensitive data.
- Used S3, KMS, and CloudHSM to secure data and keys.
- Built safe provisioning where operations need to know no secrets.
- Built a Dropwizard/Angular/PostgreSQL application from scratch, including design, implementation, operations, and documentation.
- Integrated with multiple third-party authentication providers using SAMLv2 SP- and IdP-initiated authentication.
Technologies: Compliance, SAML, Angular, Amazon CloudHSM, Beanstalk, Terraform, Ansible, PostgreSQL, ECS, Amazon EC2, Dropwizard, Java 8Senior Developer
2016 - 2017Assa Abloy Mobile Services- Developed in Java 8 EE7 on JBoss/Wildfly and Dropwizard.
- Planned and implemented AWS migration and automation.
- Wrote custom Ansible modules for securely deploying secrets to AWS using Amazon KMS and S3.
- Fully automated deployments of CloudHSM clients.
- Planned and implemented zero-downtime deployments.
Technologies: Amazon Web Services (AWS), Linux Luna Client, Amazon CloudHSM, VPN, Relational Database Services (RDS), Amazon EC2, Ansible, Java 8Tech Lead
2015 - 2016DOOER- Planned and helped implement a new platform for automated bookkeeping.
- Assisted in building a development team from scratch.
- Planned and executed automation of deployments and provisioning for the new platform.
- Planned and set up data pipelines for accounting events feeding into accounting software and ML solutions.
- Managed teams' day-to-day work and priorities in close cooperation with the CEO and the head of engineering.
Technologies: Leadership, Apache Airflow, ETL, Technical Leadership, Amazon Web Services (AWS), Flask, Ansible, Node.js, PythonOps Consultant
2015 - 2015Wondersign- Helped migrate a product from physical servers to AWS.
- Did load tests and advised on provisioning.
- Together with others, performed final migration to the cloud.
Technologies: Amazon Web Services (AWS), Linux, PHPTechnical Lead
2014 - 2015Registry of Scotland- Worked as the technical lead of the Land Registry Act project.
- Worked to create modern web services for solicitors at the Registry of Scotland's Land Register.
- Deployed a fully redundant system of web applications, interfacing with new and legacy systems.
- Created a distributed logging and tracing system using Python logging and MongoDB.
- Supported releases and operations for an entirely new technology stack at the Registry of Scotland.
- Led the daily work of two development teams in cooperation with the Scrum Master, product owners, and other stakeholders.
Technologies: Leadership, Compliance, Web, Technical Leadership, Celery, RabbitMQ, RHEL, AngularJS, Flask, Python, MongoDBSenior Developer
2013 - 2014Ericsson- Built a new framework for non-functional testing of large distributed systems.
- Created data bridges between Java and Python.
- Enabled visualization of complex and heterogeneous test setups.
Technologies: Jython, Python, JavaConsultant
2005 - 2013Ericsson- Worked as the design team lead for a charging system node.
- Developed a custom business intelligence tool for site managers integrated with the customer's ERP system.
- Worked in a small team to create five complete and separate sales channels on a Django/AngularJS platform in six weeks.
- Led the modernization of a developer infrastructure, SCM tools, and source control for a customer site with 150 developers.
- Developed a non-functional test framework for telecom nodes from the ground up.
- Held courses and workshops in Python, Git, and Mercurial.
Technologies: Technical Leadership, Provisioning Tools, Online Charging Systems (OCS), Telecommunications, Integrated Database Management Systems (IDMS), ETL, MySQL, MongoDB, C++, C, .NET, Python, Java EE, Java SE, Java ME