Data Engineer
2019 - 2020Hover- Migrated the codebase to a container-first, CI/CD system for hands-off automated testing and deployment using Docker, Python, Bash, and Codefresh.
- Built various CLI tools and libraries to manipulate segment.io plans, GitHub repositories, and Tableau resources. Written with Bash, SQL, and Python.
- Ran Airflow ETL hosted in Astronomer Cloud and coded the ETL in Python, SQL.
- Provided mentoring and training to team members on SQL performance tuning, collaborative coding, and docker container development.
- Advised the director and VP-level staff and helped drive cultural advance in the team's approach to collaborative development.
Technologies: Python, SQL, Bash, Airflow, PostgreSQL, DockerTest Automation Developer
2018 - 2019Endgame- Built concurrency into automated tests and CI systems. Mostly with Groovy and Python.
- Gave technical talks on TCP/IP networking, SQL, and database systems.
- Identified and fixed performance bottlenecks in Linux VMs.
- Coded features and bug fixes for a custom Python test integration system.
Technologies: TCP/IP Networking, SQL, Databases, Groovy, Python, Automated Systems, CISoftware Engineer
2016 - 2017PayPal- Coded cmd-line utilities, RESTful APIs, and client apps using Bash, Go, and SQL.
- Contributed enhancements to the Moby project. See Github.com/moby/moby/pull/27565.
- Wrote a software gateway in Go between Kubernetes and legacy network hardware.
- Advised directors and VPs on approaches to distributed database systems.
Technologies: Go, Bash, SQL, REST, KubernetesData Engineering Manager
2012 - 2015Lookout- Provided guidance, feedback, and leadership to a data engineering team. Set the tone for technical direction with engineering and analytics teams.
- Built Lookout's Hadoop cluster running Hive, Impala, HBase, and Map/Reduce jobs.
- Used my DBA, coding, and Unix sysadmin skills to improve performance, preserve data integrity, and maintain system availability.
- Coded ETL and automation tools using Go, C, Bash, and Python.
- Managed MySQL databases used for analytics, warehousing, and reporting.
Technologies: Python, Go, Hadoop, ETL, Jenkins, MySQL, Bash, CSoftware Engineer in Testing
2010 - 2012Cloudmark- Tested various C/C++ and Perl daemons that comprise both Cloudmark's anti-spam accuracy infrastructure and its enterprise MTA product.
- Searched, examined, and evaluated code for defects.
- Wrote code using C, Perl, Bash, and SQL to defend against regression, demonstrate correctness, and highlight defects.
- Assisted with Oracle administration and MySQL debugging.
Technologies: C/C++, Perl, Bash, SQL
Joseph Rothrock
Data Engineer and Developer in San Francisco, CA, United States
Member since April 28, 2020
Joseph designs and builds database systems as well as at the operational infrastructure to run them. He's led data engineering teams and has expertise in OLTP, data warehousing, and globally distributed NoSQL systems. Along the way, Joseph has made meaningful open-source contributions and developed a GitHub portfolio of database reference work.
Joseph is now available for hire
Experience
- Unix 20 years
- Databases 20 years
- SQL Performance 10 years
- Data Warehouse Design 10 years
- Data Warehousing 10 years
- ETL 10 years
- Bash 10 years
- Python 3 5 years
Availability
Part-time
Preferred Environment
Unix, Python, Bash, Go, PostgreSQL, C, SQL
The most amazing...
...project I've worked on won an Emmy in 2005 for our engineering effort bringing live TV to mobile phones.
Employment
Experience
- Roxanne (Development)https://github.com/rothrock/Roxanne
Roxanne is a very simple database server that allows a client to store and retrieve values by key. Keys are stored in a hash map of 64,000 buckets. Hash collisions are resolved by separate chaining onto linked lists at the end of the index file.
- Pipefish (Development)https://github.com/rothrock/pipefish
Have you ever wanted to send your MySQL query results directly to a file in HDFS? Well, now you can! Pipefish sends the result of your SQL statement to a tab-delimited file in HDFS. It doesn't create any intermediate or temporary files. It just reads the rows from the MySQL server and writes them as tab-delimited fields to a file in HDFS that you specify. At the end of the row-set, Pipefish flushes and closes the HDFS file and closes the MySQL connection.
Skills
Paradigms
ETLPlatforms
UnixStorage
Databases, SQL PerformanceOther
Data Warehousing, Data Warehouse DesignLanguages
C, Bash, Python 3, Golang, SQL
Education
- Bachelor of Business degree (magna cum laude) in Computer Information Systems1994 - 1996Georgia State University - Atlanta, GA, United States
