
Clayton Lemons
Verified Expert in Engineering
Software Developer
Ithaca, NY, United States
Toptal member since April 23, 2020
Clayton is a transformative software engineer and leader with over 15 years of experience in the software industry, innovating at the intersection of data, AI/ML, and cloud engineering. As both an individual contributor and a visionary leader, he is a dynamic force in transforming challenges into user-centric, high-quality software solutions. Renowned for his technical mastery, strategic foresight, and principled approach to software, Clayton elevates teams to collaborate and perform their best.
Portfolio
Experience
- Python - 14 years
- Back-end - 10 years
- Software Architecture - 10 years
- C++ - 10 years
- Machine Learning - 7 years
- Spark - 5 years
- Amazon Web Services (AWS) - 4 years
- Kubernetes - 4 years
Preferred Environment
Full-stack, TypeScript, PostgreSQL, Docker, Scala, Python, Kubernetes, Amazon Web Services (AWS), React, Machine Learning Operations (MLOps)
The most amazing...
...solution I've architected and developed is an internal AI/ML platform deployed on Kubernetes, leveraging EKS, JupyterHub, Spark, MLflow, Feast, and Pachyderm.
Work Experience
Principal Full-stack Engineer
AI-powered Legal Healthcare SaaS Startup
- Owned the end-to-end technical architecture and delivery for an early-stage AI SaaS startup, from infrastructure provisioning to production application deployment with a lean 3-person distributed team.
- Led the engineering team across two time zones, establishing development workflows while remaining hands-on with code and architecture.
- Debugged and resolved critical production issues, including environment-specific runtime failures, unblocking the team and maintaining platform stability.
- Built a complete cloud-native infrastructure from scratch: designed and implemented an entire Terraform IaC stack across Azure environments (development, staging, and production), establishing a foundation for a scalable AI platform.
- Architected a full-stack application using a modern tech stack (.NET back end, Angular front end, AKS/Kubernetes, and Azure services), making major technical decisions from library and tool choices to database design and deployment strategy.
- Architected an AI agent system providing personalized legal case assistance per organization, enabling attorneys to query case documents, receive strategic recommendations, automate task workflows, and identify relevant experts.
- Designed and prototyped a hybrid ranking vector store retrieval using Reciprocal Rank Fusion (RRF), combining direct field matching with semantic similarity search on medical terminology for improved recommendation accuracy of experts for cases.
- Established a production-grade GitOps deployment workflow using ArgoCD for continuous deployment, enabling the engineering team to self-manage and deploy multiple times each day.
Senior Cloud Engineer
U.S. Department of Veterans Affairs
- Architected and deployed a Kubernetes-based enterprise data governance stack integrating Starburst and Immuta for multi-source data curation and secure access controls across VA systems.
- Designed and implemented a Python SCIM client provisioner supporting Starburst and Immuta, solving a critical networking barrier preventing Entra ID user and group provisioning and unblocking enterprise-wide rollout of the data governance platform.
- Provisioned a custom Azure Application Gateway solution, enabling Databricks workspaces to connect to Starburst and Immuta within VNET, resolving a major connectivity blocker for enterprise analytics users.
- Built several custom Helm charts for deploying both commercial-off-the-shelf (COTS) products and in-house applications.
- Engineered a deployment automation system for containerized self-hosted integration runtimes (SHIRs) on Docker Swarm, enabling 200+ workgroups to access on-premises data through Azure Synapse and Data Factory.
- Established technical standards for multi-cloud scripting (PowerShell for cloud operations and Python for automation and data analysis), improving team efficiency and code consistency.
- Established Argo CD and GitOps as the deployment method for containerized applications, accelerating feature delivery cycles from weeks to days and virtually eliminating configuration drift across environments.
- Managed a production infrastructure across Azure and AWS using Terraform IaC, including Kubernetes orchestration, Azure Databricks, Synapse Analytics, and multi-database environments (SQL Server and PostgreSQL).
Co-founder and CTO
BlueFlag LLC
- Co-founded a cloud engineering consultancy, securing government and private sector contracts within two months of launch through strategic networking and delivering a 0-to-1 product for an AI SaaS startup.
- Built and led distributed technical teams totaling 10+ professionals across two concurrent engagements: an enterprise government contract (7-person team) and early-stage startup product development (3-person team).
- Define technical strategy, empower engineers, and architect solutions across multi-cloud environments (Azure, AWS), from enterprise-scale data platforms to greenfield AI-powered SaaS applications.
Senior Back-end Engineer
Farmstream LTD
- Designed and implemented a cost-optimized, serverless timelapse generation system for processing still images from thousands of farm cameras into streamable videos using AWS Batch, Step Functions, Lambda, and a containerized Python/FFMPEG script.
- Built a time-lapse system with end-to-end automation, including infrastructure provisioning using AWS CloudFormation and CI/CD using AWS Code pipeline and CodeBuild, securely integrated with GitHub via federated identity.
- Diagnosed and resolved a critical VPN connectivity issue blocking client access to customer cameras on the 4G provider network.
- Exceeded contract scope and client expectations by redesigning the VPC and subnet architecture as part of VPN connectivity resolution, establishing proper network segmentation (public, private, VPN-connected subnets) with secure resource isolation.
- Exceeded contract scope and client expectation by managing network resource provisioning using CloudFormation IaC.
- Enabled new product capabilities (camera streaming to web) that were previously blocked by VPN issues and network architecture limitations.
- Demonstrated full-stack cloud expertise and senior-level independent consultancy across a single engagement: requirements gathering, serverless architecture, video processing, network engineering, IaC, CI/CD, and security best practices..
Enterprise Strategy Architect
U.S. Department of Veterans Affairs
- Produced the technical strategy document for a large-scale, cloud-based data and analytics platform, helping stakeholders align on a governance framework, integration patterns, and analytics capabilities.
- Created a comprehensive test plan to validate a custom monitoring application for data integration pipelines in Azure Data Factory and Azure Synapse Analytics.
- Built a YAML-based declarative testing tool, enabling testers not familiar with Data Factory or Synapse to validate the application using simulated integration pipelines.
- Assumed a technical lead role when the lead engineer departed before a crucial deadline, quickly mastering C# to resolve project gaps and deliver on time, significantly contributing to an $80 million follow-on contract win.
- Identified scope reduction opportunities that saved several days of development effort on a tight deadline while preserving MVP functionality, enabling on-time delivery.
- Implemented back-end engine components for monitoring distributed data integration jobs.
- Improved codebase quality through systematic bug fixes, unit test coverage expansion by 85% for previously untested code, and implementation of software engineering best practices.
- Identified and remediated critical security vulnerabilities related to secret management, implementing Azure Key Vault integration with managed identity for secure secret access in all environments.
Senior Director of AI Technology
Elevance Health
- Managed an 8-member engineering team skilled in AI/ML, data science, back-end API development, and DevOps, fostering a diverse technical environment.
- Mentored eight engineers in technical and leadership skills, enabling two engineers to achieve technical leadership roles.
- Oversaw the maintenance of a cloud-native, elastic AI/ML platform built on JupyterHub and a data pipeline that processed HL7 FHIR resources for over 70 million patients, handling billions of claims and clinical records.
- Headed a cross-functional team to develop a research-focused data science platform on Google Kubernetes Engine, ensuring secure access to deidentified data for over 70 million patients.
- Designed a secure solution using Kasm to prevent data exfiltration from container-based JupyterLab workspaces running on the data science platform.
- Containerized a customized JupyterLab environment for the data science platform equipped with an extensive set of AI/ML tools.
- Architected a secure, multi-tenant persistent storage system to support both user and shared project data in JupyterLab workspaces, employing Google Cloud Storage buckets mounted with gcsfuse.
- Engineered a deployment strategy for an innovative synthetic data generation system.
- Served as the cloud engineering lead for an LLM-powered, internal AI chatbot for CSR queries, providing solutions for both the LLM's development and its deployment and integration with the tool's front-end application.
- Implemented Ray on Kubernetes with autoscaling and GPU support and showcased its ability to fine-tune a 70 billion parameter Llama 2 LLM using the Fully Sharded Data Parallel (FSDP) technique.
AI Software Engineering Lead
Elevance Health
- Led the architecture and team efforts to build and deploy a cloud-native AI/ML platform on AWS for enhanced data science on scalable infrastructure, integrating technologies such as Kubernetes, JupyterHub, Spark, MLflow, Feast, and Pachyderm.
- Secured cloud accounts for the AI/ML platform deployment, working in close collaboration with internal committees and DevSecOps teams to ensure compliance and governance alignment.
- Spearheaded the deployment of JupyterHub on Amazon EKS in close collaboration with DevSecOps engineers, leveraging EFS, EBS, S3, and custom Docker images to meet specific user environment needs.
- Developed a single sign-on (SSO) solution integrating JupyterHub and MLflow via Auth0, facilitating a seamless authentication experience for users.
- Developed a high-performance computing solution for users of the AI platform, seamlessly integrating on-demand, scalable GPU resources and elastic, Kubernetes-hosted Spark jobs with JupyterHub.
- Proposed and contributed a security solution to Pachyderm, improving the security posture of its "JupyterLab Pachyderm Mount Extension" and making it easier for users to integrate.
- Influenced Pachyderm's development roadmap by identifying numerous performance issues and suggesting new features and optimizations, several of which were implemented.
- Conducted comprehensive training sessions for data scientists and engineers on the effective use of the AI/ML platform, enhancing team capabilities.
- Directed a cross-functional team on the development and operationalization of a predictive model for type 2 diabetes on the AI/ML platform, successfully advocating for the use of Feast and MLflow to achieve MLOps best practices.
- Led the architectural design and development of a new FHIR generation pipeline to replace an old one, cutting processing time from three weeks to 24 hours for 70 million patients and substantially lowering operational costs.
AI Solutions Engineer Executive Advisor
Anthem
- Conducted comprehensive research to identify Pachyderm as an enterprise-grade COTS software solution that met specific needs for pipeline orchestration, distributed processing, incremental processing, and data tracking and provenance.
- Drove the procurement process for Pachyderm, successfully navigating licensing, negotiations, and acquisition.
- Directed and assisted a team of cloud engineers with the deployment of Pachyderm within the enterprise's AWS cloud infrastructure, specifically leveraging Amazon EKS.
- Oversaw the operationalization of Pachyderm, establishing robust processes and best practices for building and executing large-scale data pipelines.
- Standardized and documented data engineering best practices for the organization.
- Designed an internal, patient-focused health trajectory data structure, significantly enhancing data scientists' ability to rapidly analyze data and develop AI-driven health models.
- Led the design and development of a Pachyderm pipeline that hydrated the above health trajectory data structure with data from over 70 million patients, encompassing 2+ TB of data.
AI ETL Solutions Engineer via Toptal
Anthem AI - Telehealth/PIP
- Spearheaded the design and execution of a complex data pipeline that enables the seamless delivery of AI-driven health insights from an on-prem server to a cloud-based application.
- Engineered a Python-based API and storage framework for the storage and retrieval of AI-driven health insights on Amazon S3, utilizing compression, Base64 encoding, and indexing for flexibility and efficiency.
- Enhanced data science operations by providing expert ETL and ML pipeline engineering support to data scientists in the form of code reviews, debugging, pair programming, and performance optimization.
- Maintained on-prem ETL pipeline components built using Hive, PySpark, and Airflow.
- Advocated successfully to leadership for the transformation of an on-prem ETL pipeline to a cloud-native solution, leveraging Snowflake, Kubernetes, PySpark, and Pachyderm.
Research Software Engineer (Machine Learning)
GrammaTech
- Accelerated the static and binary analysis of a large-scale codebase by implementing a data and ML pipeline using Python, MongoDB, and JavaScript.
- Reduced computational costs for a binary analysis program by implementing a pupil-style ML model with scikit-learn.
- Crafted data analysis techniques in Python to detect security issues in JavaScript functions, such as swapped callback and error arguments in higher-order continuation-style functions.
- Leveraged the Doc2Vec model to vectorize function call sites and definitions, streamlining the detection of swapped arguments through semantic similarity of parameter and argument names.
- Developed the back end of a feature for binary scanning in a SaaS binary analysis tool.
Software Engineer I – III
National Instruments
- Earned recognition for outstanding performance, receiving the "Rookie of the Year" award, multiple fast-track promotions, and the opportunity to lead a key project.
- Standardized and streamlined the firmware downloading framework across multiple devices in the NI-DCPower and NI-DMM product families.
- Designed and implemented over 20 features in the NI-DCPower and NI-DMM driver APIs for Windows, with many improvements directly visible to the end user.
- Led the research and definition of three major features for a key product, collaborating with project managers across hardware teams and various stakeholders to ensure alignment and address technical requirements comprehensively.
- Designed and led the development of an internal programming language and compiler that targeted a proprietary instruction set for power supply output control, enabling flexible device behavior reconfiguration and complex output control.
- Implemented a client-server system that enables remote management of NI-DCPower API driver sessions, facilitating debugging and introspection.
- Improved developer workflow efficiency by implementing a Sublime Text plugin to integrate Perforce.
- Developed a VS Code extension that integrates NI's custom build system with Microsoft's C/C++ extension, enabling advanced features like semantic code completion.
- Researched the Tarantula fault localization technique, successfully created a prototype for select NI codebases, and showcased the findings at an internal engineering conference.
- Mentored more than 10 interns and junior engineers.
Web Developer
CleanTelligent Software
- Optimized several database queries and storage layouts, including the file storage system for customer photos, which reduced several API response times to just milliseconds.
- Implemented the back-end API for a customizable report generation tool.
- Applied a new UI theme to several parts of the website.
Software Engineer Intern
National Instruments
- Designed an essential kernel driver feature that streamlined driver communication with embedded storage devices on over 10 commercial products.
- Developed a code generation tool to support the driver feature, which automatically leveraged Python and Mako templates to generate C++ and LabVIEW code from metadata.
- Investigated and presented the advantages and disadvantages of various metadata schema formats for the code generation tool, then led a consensus meeting to select the most suitable one.
Web Developer
CleanTelligent Software
- Pinpointed and documented multiple user experience inconsistencies across related functionalities and addressed the corresponding issues successfully.
- Broadened the capabilities of a crucial job scheduling tool by integrating additional back-end queries and introducing new user interface elements on the front end.
- Resolved over 30 bugs throughout the CleanTelligent website's front end and back end, enhancing overall performance and user experience.
Experience
Genstrat
https://github.com/claytonlemons/genstratTo beat a strong NumPy/SciPy baseline, mapped one CUDA block per candidate solution, parallelizing solutions across SMs. Batches from PyGAD are written once to GPU memory and launched as one block per solution. Wrote class-counts once to global memory and kept them resident across all solution evaluations.
Histograms live in shared memory with modest bin caps to stay within the memory budget. Threads compute JS terms in parallel from histograms. A custom per-class tree reduction sums those terms, and thread 0 computes the final fitness. Only solution permutations and results cross the PCIe boundary.
Performance landed around 110x speedup on realistic workloads (500 solutions, 100k samples) versus the optimized CPU path, with smaller batches still delivering 20–40x gains.
Capstone Project for the Coursera Course "Functional Programming in Scala"
https://github.com/claytonlemons/fp-in-scala-capstoneNI-DCPower Soft Front Panel Debug
http://www.ni.com/en-us/innovations/white-papers/14/introducing-debug-driver-session-technology.htmlI implemented the project by creating a DLL that initiates an Apache Thrift server for command relay between remote and driver sessions over localhost, enabling seamless integration of debugging tools. Thrift's flexibility was key for future extensions to support connections from non-local processes.
I collaborated closely with LabVIEW and C# developers, who consumed the client API in order to implement the NI-DCPower Soft Front Panel Debug feature.
Subforce
https://github.com/claytonlemons/SubforceGMF Aquatics Website
Education
Master of Science Degree in Software Engineering
The University of Texas at Austin - Austin, TX, USA
Minor in Mathematics
Brigham Young University - Provo, UT
Bachelor of Science Degree in Computer Science
Brigham Young University - Provo, UT, USA
Certifications
Introduction to Parallel Programming with CUDA
Johns Hopkins University | via Coursera
Introduction to Concurrent Programming with GPUs
Johns Hopkins University | via Coursera
[NCA-AIIO] NVIDIA-Certified Associate in AI Infrastructure and Operations
NVIDIA
Getting Started with Accelerated Computing in Modern CUDA C++
NVIDIA
Fundamentals of Accelerated Computing with CUDA Python
NVIDIA
Managed Services on AWS and DevOps
Great Learning
Cloud Computing on AWS
Great Learning
Cloud Foundations
Great Learning
Managed Services on Azure
Great Learning
Azure Essentials
Great Learning
CKS: Certified Kubernetes Security Specialist
The Linux Foundation
CKA: Certified Kubernetes Administrator
The Linux Foundation
CKAD: Certified Kubernetes Application Developer
The Linux Foundation
Parallel Programming in Scala
Coursera
Functional Programming in Scala Capstone
Coursera
Functional Programming Principles in Scala
Coursera
Functional Program Design in Scala
Coursera
Big Data Analysis with Scala and Spark
Coursera
Advanced Python
Skills
Libraries/APIs
TensorFlow, Keras, NumPy, SciPy, Pandas, Scikit-learn, PySpark, React, PyTorch, Ruby ERB, Backbone.js, Mustache, Windows API, Azure Cognitive Services, Azure API Management
Tools
Sublime Text 3, Visual Studio, Gensim, Git, Perforce, GitLab, GitLab CI/CD, Claude, LabVIEW, Apache Airflow, Pachyderm, Amazon EKS, Pytest, Helm, Logging, AWS Step Functions, Bitbucket, Amazon Elastic Block Store (EBS), AWS ELB, AWS IAM, AWS Command Line Interface (CLI), Artifactory, Docker Hub, Docker Compose, Google Kubernetes Engine (GKE), Terraform, CVS, ChatGPT, GitHub, Azure Kubernetes Service (AKS), Azure Key Vault, Azure Application Gateway, Docker Swarm, Microsoft Copilot, Dynatrace, MSTest, Azure Logic Apps, NVIDIA Nsight Systems, Kubernetes Operators, NVIDIA Jetson, Named-entity Recognition (NER), Azure Machine Learning, Amazon Virtual Private Cloud (VPC), Amazon Elastic Container Service (ECS), AWS CloudTrail, AWS CodeCommit, AWS CodeBuild, AWS CodeDeploy, AWS CloudFormation, Amazon ElastiCache, AWS Cost Explorer, Azure Monitor, Azure Network Security Groups
Languages
Python, C++, Bash, TypeScript, PHP, CSS, Ruby, Java, Scala, GraphQL, HTML, JavaScript, Bytecode, Python 3, Embedded C, C, Bash Script, SQL, Snowflake, C#, C++17
Platforms
Kubernetes, Docker, Amazon Web Services (AWS), Cloud Native, Azure, NVIDIA CUDA, Windows, NetBeans, Software Design Patterns, Linux, AWS Lambda, Jupyter Notebook, OCI Artifact Registry, Director, Google Cloud Platform (GCP), Visual Studio Code (VS Code), Red Hat OpenShift, Azure Synapse Analytics, NVIDIA NeMo, Azure Event Hubs, Azure IaaS, Azure PaaS, Azure Data Lake Storage, Amazon EC2, AWS Elastic Beanstalk, Azure Functions
Frameworks
Spark, Flask, Hadoop, Jakarta Server Pages (JSP), Apache Struts 2, CakePHP, Ray, Bootstrap, Apache Thrift, .NET, Trino, Windows PowerShell, OAuth 2, Angular
Paradigms
Functional Programming, Concurrent Programming, High-performance Computing (HPC), Compiler Design, Distributed Programming, Software Testing, Parallel Programming, ETL, Pair Programming, Distributed Computing, DevOps, HL7 FHIR Standard, Agile, Scrum, ABAC, Parallel Computing, Azure DevOps, Serverless Architecture
Storage
PostgreSQL, MongoDB, Cloud Deployment, Databases, Apache Hive, Data Pipelines, Amazon S3 (AWS S3), Amazon EFS, Google Cloud Storage, Datadog, Microsoft Entra ID, SQL Server 2012, Azure SQL, Azure Cosmos DB, AWS Storage Gateway
Other
Windows Kernel Drivers, Software Architecture, Back-end, Distributed Systems, Web Development, Machine Learning, Artificial Intelligence (AI), Cloud, GPU Computing, Large Language Models (LLMs), Real-time Data, Development, Full-stack, Firmware, API Design, Concurrent Computing, Thread Scheduling, Processing & Threading, Code Validation, Data Mining, Software Project Management, Data Engineering, Programming, Operating Systems, Data Structures, Software Design, Security, Asymmetric Encryption, Server Development, Non-blocking I/O, Natural Language Processing (NLP), Compilers, Digital Signal Processing, Computer Engineering, Algorithms, Machine Learning Operations (MLOps), Statistical Analysis, CI/CD Pipelines, Big Data, Data Analytics, Data Modeling, Data Profiling, APIs, Frameworks, Storage, Code Review, Debugging, Advisory, Flake8, pre-commit, Orchestration, Data Science, Distributed Software, Negotiation, Procurement, COTS, Enterprise SaaS, Provenance, Data Lineage, IT Security, Technical Leadership, Functional Design, Kubernetes Operations (kOps), Open Source, Container Orchestration, Troubleshooting, Scheduling, Site Reliability Engineering (SRE), System Administration, Containerization, JupyterLab, MLflow, Feast, AWS Auto Scaling, Architecture, Single Sign-on (SSO), Optimization, Coaching, Career Coaching, Feedback Review, Software Engineering, Cross-functional Team Leadership, Strategic Planning & Execution, Idea Synthesization and Application, Staff Management & Development, Goal Management, Project Coordination, Business Requirements, Open-source LLMs, New Product Development, FastAPI, Cross-functional Collaboration, Llama 2, Fine-tuning, Data-level Security, Data Exfiltration Prevention, Mako, Version Control, Plugin Development, Client-server Model, DLL, Device Drivers, Full-stack Development, API Integration, ETL Tools, Data Management, IT Strategy, Leadership, Team Leadership, Remote Team Leadership, Protegrity, Retrieval-augmented Generation (RAG), Chatbots, Azure CLI, GitHub Runners, GitHub Actions, Active Directory (AD), Azure Databricks, Argo CD, GitOps, Semantic Kernel (SK), SCIM, Identity & Access Management (IAM), Azure AI Foundry, Azure Blob Storage, CUDA Kernel, Numba, Graphics Processing Unit (GPU), NVIDIA vGPU, NVIDIA Triton, NVIDIA TensorRT, Deep Learning, NVIDIA GPU Operator, NVIDIA NGC, NVIDIA Isaac, NVIDIA NVLink, NVIDIA AI Enterprise, NVIDIA MIG, NVIDIA NIM, NVIDIA DGX, Slurm Workload Manager, pygad, Genetic Algorithms, Python Extensions, Linear Algebra, Calculus, Number Theory, Cloud Computing, Microsoft Entra, Azure Service Bus, Azure Event Grid, Azure Virtual Machines, Load Balancers, AWS DevOps, AWS CodePipeline, AWS NAT Gateway, Amazon Route 53, Azure AD Connect, IT Governance, Azure Resource Manager (ARM), Azure Virtual Networks, Cursor AI, Deployment, Prompt Engineering, LangChain, AI Chatbots
How to Work with Toptal
Toptal matches you directly with global industry experts from our network in hours—not weeks or months.
Share your needs
Choose your talent
Start your risk-free talent trial
Top talent is in high demand.
Start hiring