Architect and Data Engineer2020 - PRESENTGoalcast
Technologies: Amazon Athena, Apache Airflow, Amazon QuickSight, Python
- Designed an architecture to capture, analyze, and dashboard social media posts and videos. Complex data and reporting requirements had to be met.
- Tracked features and bugs using Jira and created technical specifications based upon client requirements.
- Built complex ETL flows to manage the transformations using Athena.
- Built user friendly dashboards in AWS Quicksight to enable users to explore the data.
Technical Lead2018 - PRESENTMarketing Attribution Partners
Technologies: Data, Python, PyMC3, Data Warehouse Design, Django, SQL, PostgreSQL, NumPy
- Designed the architecture to meet requirements. Iteratively enhanced this as requirements increased in a managed way over time.
- Built the data model system service using PostgreSQL and Python (PyMC3), leading edge technology for marketing analysis.
- Built the data model and Django back-end API service for a SaaS solution to surface the model.
- Led the expansion of the team with processes to manage the quality and specification. Used Jira system to build an Agile CI/CD process. Managed team on a day-to-day basis.
- Enhanced the data processing with a 100x speed performance boost using low level Python (NumPy) to replace an intensive mechanism previously performed in SAS.
Lead Architect2020 - 2021Pharma Data Company
Technologies: Apache Airflow, Code Architecture, SQL, Redshift, Redshift Spectrum, AWS Glue, Data Lake Design, SAS
- Designed and communicated a novel ETL architecture to meet exacting requirements around data provenance, quality, security, and performance.
- Built the project plan and engaged with the development teams to ensure alignment. Managed the work alongside the program manager using Jira.
- Developed solutions to overcome complex edge cases to ensure smooth running of the system and to allow the project to be completed. Using SQL and PySpark.
- Supported implementation and quality assurance while the system was embedded.
MI Manager2013 - 2015Bet365
Technologies: Data Warehouse Design, ETL Tools, SQL, kognitio
- Led the development of analytics and business reporting (a team of six data engineers) and the micro-strategy reporting team.
- Led the enhancement of ETL processes to meet tighter timescales and more features.
- Worked on legal and compliance reporting across geographies to help roll out services worldwide.
Lead BI Architect2009 - 2013Capgemini
Technologies: Data Warehouse Design, SQL, Teradata, Hadoop, Cloudera
- Served as senior architect within Capgemini UK BIM (Business Information Management) practice. Communicated regularly with stakeholder management at partner (VP) level within Capgemini and at the senior executive level within the customer organization.
- Oversaw solution design for Capgemini customers, specializing in public sector. Significant customer stakeholder management and pre sales activity, including sales strategy and solution design work for large scale data solutions.
- Collaborated with partners including Cloudera to ensure optimal solution design. Fed back into open source community where possible.