Data Engineer and Developer in Meridian, ID, United States

I built a data model that unified Google Cloud Platform, Amazon Web Services, Packet, Digital Ocean cloud invoices into a star schema dimensional model.

I designed it from end to end and built all the data pipelines. This project was used internally to forecast the company's future cloud costs and to drill down into the specifics to see how money was being spent.

I architected, designed, and built a big data ecosystem nicknamed (BDP) for the United State's Social Security Administration. This was a Hortonworks stack on AWS 100% and automated by Ansible, CloudFormation, and Cloudbreak. The tech stack included Spark, Hadoop, Hive and more.

Out of this came an idea of dynamic templating so several clusters could be spun up in parallel and used for development or production.

The government had the most stringent requirements involving things being approved for FEDRAMPED and before using the cluster for official government use, we had to get the official "Authority to Operate" approval. This involved turning every security bell and whistle you could think of, including vormetric disk encryption, SSL on everything, security groups, firewalls, authorization, SSSD, Kerberos, RedHat IdM integration, cross-forest Active Directory trusts.

After getting the ATO, we helped several teams within the SSA get onboarded to the BDP to use Hive and Apache Spark for various ML and analytical workloads. Our work was so successful, the NIH wanted to use our cluster for some work too.

The ML work involved parameter tuning and installing TensorFlow on Spark, XGBoost, and other popular libraries.

I built a modern data lake on AWS for Time, Inc. This involved two Greenplum clusters and two Hadoop clusters and everything was automated with CloudFormation and Bash. Several clusters were also tied into the company's identity infrastructure.

I assisted in the migration of over 1PB from AWS to GCP. We migrated this data from S3 and Snowflake over to BigQuery and GCS. It took a total of ten engineers and three months. Dozens of data pipelines needed to be rewritten to work on Google's GCP Composer service (Google's hosted version of Airflow), tested, and deployed to production.

I migrated an entire web stack hosted in Azure on Cloud Services (Classic) over to a new Kubernetes cluster running on Azure AKS. This involved finishing up the migration from .NET to .NET Core and then getting .NET Core to build on Linux containers instead of on Windows successfully.

Finally, after getting the cluster pod autoscaling and horizontal pod autoscaling working correctly, I set up several CI /CD pipelines for production, staging, and development branches to automatically build and deploy the updated images to the production, staging, and development websites.

Out of all this, the company saved about $2,000 per month on their hosting costs as the autoscaling was much more resilient, and the new cluster was cheaper than paying for the raw VMs.

I built a modern web app for Empire Unmanned. This web app was responsible for allowing field pilot drones to upload their drone videos to the cloud to be processed for aerial drone inspections and mapping. In addition to being a back-end developer on the project, I was the engagement manager on the project and led a small team of engineers to complete the project.

I led a small team of software engineers to build out a small API written in Phoenix to replace a very old Perl-based system on its last legs. We got a basic API spec from the client and had to implement their needs while ensuring the API met the client's demands and was properly tested.