Rajeshwar Agrawal, Developer in Jabalpur, Madhya Pradesh, India
Rajeshwar is available for hire
Hire Rajeshwar

Rajeshwar Agrawal

Bio

Rajeshwar is a staff-level software engineer and platform technical lead with 11+ years of experience turning ambiguous challenges in scale, reliability, and client delivery into back-end and platform systems. He leads technical direction while remaining hands-on across architecture, implementation, performance engineering, and production recovery. Rajeshwar's work creates leverage through shared platforms, engineering standards, open source contributions, mentoring, and cost-aware cloud design.

Portfolio

STEELEYE
Data, Big Data, Big Data Architecture, Data Management Platforms...
Motional
C#, ASP.NET, REST, REST APIs, Databases, MySQL...
Works Applications
Microservices, Spring Microservice, Cassandra, RESTful Development, Jenkins...

Experience

  • Software Engineering - 11 years
  • Back-end Development - 11 years
  • Distributed Systems - 11 years
  • Software Architecture - 11 years
  • Cloud Architecture - 11 years
  • Python - 11 years
  • Platform Engineering - 8 years
  • Technical Leadership - 5 years

Preferred Environment

Ubuntu, JetBrains IDE, Slack, Linux, MacOS

The most amazing...

...system I've built processes 3 million stock-market trades through 350,000 concurrent surveillance workflows, generating 600,000 alerts and 60 million queries.

Work Experience

Platform Tech Lead (Staff Software Engineer)

2021 - PRESENT
STEELEYE
  • Owned technical direction and delivery for a multi-tenant financial surveillance platform supporting enterprise SaaS and private-cloud deployments from solution design through long-term production support.
  • Defined an operating envelope covering 3 million trades per day, 350,000 concurrent surveillance workflows, 60 million trade queries within 20 hours, and 600,000 surveillance alerts, guiding load testing, capacity, and rollout decisions.
  • Led modernization of Conductor from PostgreSQL toward Redis and Elasticsearch 8, covering architecture, workflow compatibility, reconciliation, deployment tooling, load testing, sizing, and production stabilization.
  • Contributed 13 pull requests to Conductor OSS, with 10 merged, including Elasticsearch 8 persistence and improvements to sub-workflow reliability, recovery, and scalability.
  • Designed an Elasticsearch sharding strategy that improved effective performance by 2.7x without added compute or storage. Schema and archival changes also cut Elasticsearch costs by 4x and archive-storage costs by 2x.
  • Led cross-system production diagnosis across Conductor, Elasticsearch, Kafka, and workflow processing, tracing failures across component boundaries, and driving fixes through stabilization.
  • Shaped platform roadmaps, engineering standards, ownership boundaries, and delivery priorities while mentoring engineers and partnering with product, sales, support, and infrastructure teams.
  • Consolidated Python applications into a shared Pants monorepo, enabling more than 10 applications to launch within three months and reducing annual CI/CD costs from approximately $25,000 to $6,000.
  • Designed FastAPI services and Prefect/Parquet pipelines serving financial quotes, trades, and tick data, reducing ingestion and storage costs by approximately 80%.
Technologies: Data, Big Data, Big Data Architecture, Data Management Platforms, Event-driven Architecture, Prefect, Conductor, Netflix OSS, Orkes, Apache Airflow, Management, ETL, Refinitive API, Architecture, Cloud Architecture, Data Engineering, Parquet, Data Pipelines, Python 2, Distributed Systems, PostgreSQL, Algorithms, API Development, Kubernetes, FastAPI, GraphQL, Data Management, NumPy, Pytest, Containerization, Asyncio, Python Asyncio, Redis, Testing, GitHub Actions, Continuous Delivery (CD), Continuous Integration (CI), API Design, Debugging, Troubleshooting, Apache Spark, GitHub, Redis Cache, Finance, Back-end Development, Software as a Service (SaaS), Celery, Artificial Intelligence (AI), Jira, Large Language Models (LLMs), LangChain, OpenAI, Unix, Azure, Apache Arrow, Bash, Platform Engineering, Code Review, Claude, Claude Code, AI Agents, Agentic AI, Multitenancy, Leadership, Generative Artificial Intelligence (GenAI), Codex, OpenAI API, OAuth, LLM Integration, AI Automation, AI Tools, Claude API, Feature Analysis, Streaming, Algorithmic Trading, Trading, Real-time Data, Error Logging, AWS Secrets Manager, Cloud Run, Documentation, OpenAPI, Pub/Sub, Fintech, Data Architecture, Command-line Interface (CLI), CI/CD Pipelines, Leadership & Coaching, Communication, Database Design, Data Infrastructure, AWS Cloud Architecture, Pydantic, Data Modeling, System Design, Real-time Systems, SaaS, Solution Architecture, Systems Design, Technical Documentation, API Backwards Compatibility, XML, System Architecture, Temporal, Microsoft AI, Azure AI Services, Agentic Coding, Integration, Third-party Integration, Workflows, Technical Leadership, Low Latency, Cloud Infrastructure, Infrastructure, Infrastructure Architecture, Technical Architecture, On-premise, Monitoring, Observability, Observability Tools, Capacity Planning, Incident Response, Containers, Automated Deployment Scripts, SDKs, Asynchronous Programming, Distributed Applications, Distributed Architecture, Distributed Databases, SDK Development, Cloud Storage, Data Migration, Data Reconciliation, Event-driven Systems, Performance, Web Development, Integrations, Financial Services, Automation, AI-assisted Development, Anthropic, Data Extraction, Vibe Coding, Enterprise Architecture, AI-generated Code, Full-stack Development, Event-driven Design (EDD), Performance Optimization, Multi-tenant Architecture, Database Optimization, DevOps Automation, DevOps Engineer, Infrastructure Automation, Cloud Engineering, Site Reliability Engineering (SRE), Enterprise Integration, Middleware, Metadata, React, Financial APIs, SOC 2, Role-based Access Control (RBAC), Full-stack, Team Management, Project Management, Mentorship, Compliance, B2B, Financial Modeling, Back-end APIs, Single Sign-on (SSO), Workflow, Cloud Native, Orchestration, Email, Cron, Data Orchestration, Pipelines, Webhooks, API Connectors, Identity & Access Management (IAM), AWS ECS Fargate, Third-party APIs, OpenTelemetry, OAuth 2, JSON, Refactoring, Large-scale Projects, Solution Design, Systems, Model Context Protocol (MCP), Amazon EKS, Pandas, Software Engineering, Microsoft Azure, Performance Improvement, System Integration, Multi-tenant SaaS, Early-stage Startups, Startups, Auto-scaling Cloud Infrastructure, Caching, Distributed Computing, Search, Polars

Senior Software Engineer

2018 - 2020
Motional
  • Led architecture and delivery of an AWS data platform processing 20 TB of autonomous vehicle data per day from hundreds of test runs, using event-driven pipelines, AWS Step Functions, and Parquet storage.
  • Designed a high-throughput Python streaming pipeline that converted non-streamable logs into timestamp-seekable Avro, achieving 3.1 Gbps encoding and compression throughput while enabling low-latency access.
  • Built more than 25 PySpark, Pandas, and NumPy applications on Amazon EMR to transform ride data into Parquet and expose it through Athena, improving analytical access while reducing processing costs.
  • Created a self-service data-application platform with an SDK, base Docker image, reusable framework, and EC2 autoscaling, enabling 10+ applications in three months versus an estimated two to three months per application.
Technologies: C#, ASP.NET, REST, REST APIs, Databases, MySQL, Relational Database Services (RDS), Serverless, Terraform, Amazon EC2, Amazon S3 (AWS S3), Python, Python 2, Python 3, Flask, Flask-Marshmallow, SQLAlchemy, PyMySQL, Docker, Spring Boot, Distributed Systems, Scaling, .NET Core, AWS Lambda, Software Engineering, Linux, APIs, Datadog, Data Engineering, Relational Databases, Back-end, PySpark, .NET, Django, Messaging, Cloud, JDBC, Agile, Amazon Elastic Container Service (ECS), ECS, Amazon Kinesis, Amazon CloudWatch, AWS Step Functions, State Machines, Autoscaling, Scrum, Maps, SSH, Infrastructure as Code (IaC), Infrastructure Monitoring, Django REST Framework, Scalability, Software Architecture, Object-oriented Design (OOD), Object-oriented Programming (OOP), API Integration, WebSockets, Event-driven Architecture, Kubernetes, Software Implementation, Lambda Functions, RESTful Microservices, Unit Testing, SOLID Principles, Serverless Architecture, Architecture, Cloud Architecture, PostgreSQL, Algorithms, API Development, Data Management, NumPy, Pytest, Containerization, Asyncio, Python Asyncio, Testing, Debugging, Apache Spark, GitHub, Back-end Development, Unix, Spark, Bash, Platform Engineering, Code Review, Feature Analysis, Streaming, Cloud Run, Documentation, Pub/Sub, Command-line Interface (CLI), Database Design, Data Infrastructure, AWS Cloud Architecture, Data Modeling, System Design, Data Architecture, Real-time Systems, SaaS, Systems Design, Technical Documentation, Real-time Data, API Backwards Compatibility, XML, System Architecture, Integration, Low Latency, Terraform Cloud, Cloud Infrastructure, Containers, Automated Deployment Scripts, Asynchronous Programming, Distributed Applications, Distributed Architecture, Distributed Databases, SDK Development, Jupyter Notebook, Cloud Storage, Event-driven Systems, Web Development, Automation, Performance Optimization, Database Optimization, DevOps Automation, DevOps Engineer, Infrastructure Automation, Leadership, Cloud Native, Cron, AWS ECS Fargate, Third-party APIs, JSON, Refactoring, Large-scale Projects, Solution Design, Systems, Amazon EKS, Pandas, Performance Improvement, Early-stage Startups, Startups, Auto-scaling Cloud Infrastructure, Distributed Computing

Software Engineer

2015 - 2018
Works Applications
  • Designed and delivered an order-fulfillment platform integrating order, cart, payment, and shipping services, meeting an SLA of 1,000 fulfilled orders per hour through REST microservices and autoscaling.
  • Migrated a large Jakarta EE eCommerce monolith into Spring Boot REST microservices, helping establish clearer service boundaries and independently deployable back-end components.
  • Designed Kafka-based asynchronous workflows and optimized Cassandra and MySQL data models for distributed order processing across multiple back-end services.
  • Improved developer productivity by optimizing Jenkins CI/CD pipelines, tuning JVM parameters, and standardizing release artifact management through Artifactory.
Technologies: Microservices, Spring Microservice, Cassandra, RESTful Development, Jenkins, CI/CD Pipelines, Hadoop, REST APIs, Java, Microservices Architecture, Databases, SQL, Spring, NoSQL, Git, Amazon Web Services (AWS), Data Modeling, Apache Kafka, REST, Back-end, Distributed Systems, Scaling, Software Engineering, Linux, APIs, Relational Databases, eCommerce, Java 8, Java EE (Jakarta EE), Messaging, Cloud, JDBC, Autoscaling, Elasticsearch, SSH, Apache Cassandra, Scalability, Software Architecture, Hibernate, Object-oriented Design (OOD), Object-oriented Programming (OOP), Docker, MySQL, API Integration, Enterprise Resource Planning (ERP), Event-driven Architecture, Software Implementation, Lambda Functions, RESTful Microservices, Message Queues, Unit Testing, JUnit, SOLID Principles, Architecture, Cloud Architecture, Python 2, Algorithms, API Development, Testing, Back-end Development, Redis, Jira, Unix, Spark, Bash, Code Review, Feature Analysis, Documentation, Apache Tomcat, Jakarta EE (Java EE or J2EE), Spring Data JPA, Spring MVC, Command-line Interface (CLI), AWS Cloud Architecture, System Design, SaaS, Systems Design, Technical Documentation, API Backwards Compatibility, Cloud Infrastructure, HTML, Microsoft SQL Server, Performance Optimization, B2B, Cloud Native, Cron, Third-party APIs, Refactoring, SKUs, Retail Management, ERP Systems, System Integration

Software Engineer Intern

2013 - 2013
Canon Marketing Japan
  • Analyzed software projects using code metrics to establish a quantitative approach to software-quality assessment.
  • Designed and implemented a Python-based model that combined weighted metrics into a single software-quality score.
  • Received the computer science department’s best project award for my internship work.
Technologies: Python, Medical Software, Metrics, Bash, Back-end Development, Feature Analysis, Documentation, Command-line Interface (CLI), AWS Cloud Architecture, SaaS, Technical Documentation, Software Engineering

Experience

Enterprise Financial Surveillance Platform at Scale

I led architecture and production scaling for a multi-tenant financial surveillance platform. I translated expected demand into an operating envelope of 3 million trades per day, 350,000 concurrent workflows, 60 million trade-store queries, and 600,000 surveillance alerts, all completing within 20 hours. I designed asynchronous processing, workload isolation, horizontal scaling, storage strategy, load tests, and telemetry-driven capacity planning across Conductor, Kafka, Kubernetes, and Elasticsearch. This enabled reliable enterprise rollout without indiscriminate infrastructure overprovisioning.

Conductor Workflow Orchestration Platform Modernization

I led the modernization of a PostgreSQL-backed Conductor platform to Redis and Elasticsearch 8, covering architecture, workflow compatibility, data reconciliation, deployment tooling, load testing, capacity sizing, rollout, and production stabilization. I also contributed 13 pull requests to conductor-oss, with 10 merged, including Elasticsearch 8 persistence and improvements to sub-workflow reliability, recovery, and scalability. The result was a more scalable and resilient orchestration foundation for enterprise surveillance workloads.

Python Developer Platform and CI/CD Modernization

I led the modernization of a fragmented Python ecosystem into a shared Pants monorepo and standardized build and release platform. I introduced dependency management, remote build caching, automated security updates, Docker optimization, and ARM build support across multiple engineering teams. This reduced annual CI/CD costs from approximately $25,000 to $6,000 and enabled more than 10 applications to be delivered within three months. Beyond cost savings, the platform created reusable engineering standards and reduced delivery friction across teams.

AI-assisted Guardrail for Production Infrastructure Changes

I contributed to system design and developed prompts for an AI-assisted infrastructure review workflow. Running on self-hosted GitHub Actions runners, it analyzed Helm chart diffs generated by applying Flux changes and used a GPT-based model through AWS Bedrock to classify production risks as high, medium, or low. Findings were posted back to the originating GitHub pull request, making infrastructure risks visible earlier, standardizing risk-focused reviews, and reducing deployment failures across client environments.

Elasticsearch Performance, Capacity, and Cost Optimization

I led performance and cost optimization for high-load Elasticsearch clusters, using workload forecasts, load tests, and production telemetry to guide sizing, sharding, replication, and storage decisions. I designed a primary-sharding strategy that improved effective performance by approximately 2.7x without additional compute or storage. Schema and index changes reduced Elasticsearch costs by 4x, while compression and archival improvements reduced archive-storage costs by 2x.

Financial Market Data Services and Ingestion Platform

I designed and delivered FastAPI services and Prefect/Parquet pipelines serving financial quotes, trades, and tick data. I owned data flow, API design, batch orchestration, storage strategy, performance, and operational reliability. The platform improved access to Refinitiv market data while reducing ingestion and storage costs by approximately 80%, creating a sustainable foundation for downstream analytics and financial products.

Self-service Developer Platform for Data Applications

I designed and delivered a self-service platform that enabled engineering teams to build, deploy, and operate custom data applications on shared infrastructure. I created an SDK, a base Docker image, a reusable integration framework, and an AWS Step Functions execution model backed by EC2 autoscaling. Standardized deployment, access control, monitoring, and notifications enabled more than 10 applications to launch within three months, instead of the estimated two to three months per application.

Cloud Cost Governance Across Shared Platforms

I led a cross-platform AWS cost program spanning CI/CD, container registries, and production infrastructure. I identified the highest-cost areas, aligned owners across teams, and introduced build optimization, ECR lifecycle policies, automated image cleanup, and more efficient infrastructure usage. This delivered over $81,000 in savings, reduced CI costs by 47%, and established repeatable cost controls without compromising delivery speed or reliability.

Autonomous Vehicle Data Platform at 20 TB per Day

I led the design and delivery of an AWS data platform processing 20 TB of autonomous vehicle data per day from hundreds of test runs. I designed separate access paths for near-real-time log streaming and analytical workloads, using Avro with timestamp-based seeking, PySpark on EMR, Parquet storage, and Athena SQL access. AWS Step Functions orchestrated ingestion and transformation workflows. Concurrent Python processing achieved 3.1 Gbps encoding and compression throughput while improving data accessibility, operational reliability, and debugging.

Event-driven Order Fulfillment System

I built a back-end order fulfillment system for an eCommerce platform that integrated multiple APIs across orders, payments, shipping, and cart services. I developed microservices in Java and Spring Boot and designed an event-driven architecture using Kafka for asynchronous communication between distributed services. I also introduced autoscaling rules to support horizontal scaling, helping the platform meet an SLA of 1,000 fulfilled orders per hour while maintaining reliability under load.

Education

2010 - 2014

Bachelor's Degree in Computer Science

Indian Institute of Information Technology - Jabalpur, India

Skills

Libraries/APIs

REST APIs, PySpark, API Development, Pydantic, Back-end APIs, SQLAlchemy, Slack API, JDBC, Pandas, NumPy, Asyncio, Python Asyncio, Flask-Marshmallow, PyMySQL, Jenkins Pipeline, Refinitive API, OpenAI API, Claude API, OpenAPI, React

Tools

Git, Amazon Athena, Amazon Elastic MapReduce (EMR), Apache Maven, Kafka Streams, Pytest, GitHub, Claude, Codex, Cron, Amazon EKS, Terraform, AWS Glue, AWS Step Functions, Amazon Elastic Container Service (ECS), Jira, Claude Code, Observability Tools, Jenkins, Apache Avro, Amazon CloudWatch, Amazon Redshift Spectrum, GitLab, GitLab CI/CD, Jupyter, IPython Notebook, Amazon Virtual Private Cloud (VPC), Shell, PyPI, AWS Deployment, AWS SDK, Prefect, Apache Airflow, Amazon Simple Queue Service (SQS), Celery, Sentry, Apache Iceberg, Apache Tomcat, Microsoft AI, Helm

Languages

Python, SQL, Java, C#, Python 2, Python 3, Java 8, XML, C#.NET, Bash, Batch, GraphQL, HTML, JavaScript, ECMAScript (ES6)

Frameworks

Hadoop, Spark, Spring Microservice, Spring Boot, .NET Core, Apache Spark, ASP.NET, .NET, JUnit, OAuth 2, Flask, Spring, Django, Django REST Framework, Hibernate, Spring MVC, Flux

Paradigms

MapReduce, Microservices, Microservices Architecture, ETL, DevOps, Object-oriented Design (OOD), Object-oriented Programming (OOP), Event-driven Architecture, Unit Testing, Serverless Architecture, Testing, Database Design, Asynchronous Programming, Automation, Event-driven Design (EDD), B2B, Refactoring, Distributed Computing, REST, Agile, Real-time Systems, Role-based Access Control (RBAC), Continuous Integration (CI), Continuous Delivery (CD), RESTful Development, Scrum, Management, Model Context Protocol (MCP), Load Testing

Platforms

Docker, Apache Kafka, Amazon Web Services (AWS), Amazon EC2, AWS Lambda, Linux, Java EE (Jakarta EE), Kubernetes, Apache Arrow, Cloud Run, Cloud Native, Ubuntu, Unix, Azure, Jupyter Notebook, MacOS, Amazon, Jakarta EE (Java EE or J2EE)

Storage

Amazon S3 (AWS S3), Relational Databases, PostgreSQL, Elasticsearch, Redis Cache, On-premise, Distributed Databases, JSON, Auto-scaling Cloud Infrastructure, Databases, Data Pipelines, MySQL, Redis, NoSQL, Cassandra, Apache Hive, Datadog, Redshift, Microsoft SQL Server, MongoDB, Spring Data JPA, Database Replication, Apache Parquet

Industry Expertise

Project Management

Other

Software Development, CI/CD Pipelines, Parquet, Big Data, Data Modeling, Data Engineering, Back-end, Distributed Systems, Software Engineering, APIs, eCommerce, Data Warehousing, Cloud, SDKs, Containers, System Design, Software Architecture, API Integration, Concurrency, Data, Software Implementation, RESTful Microservices, Multithreading, SOLID Principles, FastAPI, Architecture, Cloud Architecture, Data Management, Containerization, API Design, Debugging, Troubleshooting, Finance, Back-end Development, Monorepos, Platform Engineering, Code Review, Leadership, Feature Analysis, Streaming, Documentation, Pub/Sub, Fintech, Data Architecture, Command-line Interface (CLI), Communication, Data Infrastructure, AWS Cloud Architecture, SaaS, Solution Architecture, Systems Design, Technical Documentation, API Backwards Compatibility, System Architecture, Integration, Third-party Integration, Workflows, Technical Leadership, Cloud Infrastructure, Infrastructure, Infrastructure Architecture, Technical Architecture, Incident Response, Automated Deployment Scripts, Distributed Applications, Distributed Architecture, SDK Development, Cloud Storage, Data Migration, Event-driven Systems, Performance, Web Development, Integrations, Financial Services, AI-assisted Development, Anthropic, Data Extraction, Vibe Coding, Performance Optimization, Multi-tenant Architecture, DevOps Automation, DevOps Engineer, Infrastructure Automation, Metadata, Financial APIs, Team Management, Workflow, Orchestration, Data Orchestration, Pipelines, Third-party APIs, Large-scale Projects, Solution Design, Systems, ERP Systems, Performance Improvement, System Integration, Multi-tenant SaaS, Startups, Data Warehouse Design, Relational Database Services (RDS), Scaling, Algorithms, Amazon RDS, Slackbot, Data Wrangling, Autoscaling, State Machines, Zstandard, ECS, SSH, Infrastructure as Code (IaC), Monitoring, Scalability, Enterprise Resource Planning (ERP), WebSockets, Lambda Functions, Message Queues, Artificial Intelligence (AI), Large Language Models (LLMs), AI Agents, Agentic AI, Multitenancy, Real-time Data, Error Logging, AWS Secrets Manager, Agentic Coding, Low Latency, Observability, Capacity Planning, Data Reconciliation, Enterprise Architecture, AI-generated Code, Database Optimization, Cloud Engineering, Site Reliability Engineering (SRE), Enterprise Integration, Middleware, SOC 2, Mentorship, Compliance, Financial Modeling, Email, API Connectors, Identity & Access Management (IAM), AWS ECS Fargate, SKUs, Retail Management, Microsoft Azure, Early-stage Startups, Search, Data Compression, Serverless, Amazon API Gateway, Autonomous Navigation, Self-driving Cars, Messaging, Amazon Kinesis, Maps, Jupiter, Infrastructure Monitoring, Apache Cassandra, Networking, Deployment, Memory Management, Memory Mapped Files, Processing & Threading, Benchmarking, Memory Profiling, Software Integration, Big Data Architecture, Data Management Platforms, Conductor, Netflix OSS, Orkes, GitHub Actions, Software as a Service (SaaS), pantsbuild, Schemas, Amazon Glacier, ARM, Cost Control, LangChain, OpenAI, Medical Software, Metrics, Generative Artificial Intelligence (GenAI), OAuth, LLM Integration, AI Automation, AI Tools, Algorithmic Trading, Trading, Leadership & Coaching, Temporal, Azure AI Services, Terraform Cloud, Full-stack Development, Full-stack, Single Sign-on (SSO), Webhooks, OpenTelemetry, Workflow Orchestration, Open Source, Reliability Engineering, Large-scale Distributed Systems, Developer Experience (DX), Build Automation, Security Automation, Sharding, Prometheus, Batch File Processing, Access Control, Prompt Engineering, Caching, Polars

Collaboration That Works

How to Work with Toptal

Toptal matches you directly with global industry experts from our network in hours—not weeks or months.

1

Share your needs

Discuss your requirements and refine your scope in a call with a Toptal domain expert.
2

Choose your talent

Get a short list of expertly matched talent within 24 hours to review, interview, and choose from.
3

Start your risk-free talent trial

Work with your chosen talent on a trial basis for up to two weeks. Pay only if you decide to hire them.

Top talent is in high demand.

Start hiring