Rajeshwar Agrawal, Developer in Jabalpur, Madhya Pradesh, India
Rajeshwar is available for hire
Hire Rajeshwar

Rajeshwar Agrawal

Bio

Rajeshwar is a staff-level software engineer who helps companies build scalable, reliable back-end and platform systems. With 10+ years of experience, he has delivered cloud-native services, data platforms, and distributed systems using Python, Kafka, Kubernetes, Elasticsearch, and AWS. He is known for solving complex technical problems, improving reliability and observability, reducing infrastructure costs, and helping teams deliver faster with confidence at scale.

Portfolio

STEELEYE
Data, Big Data, Big Data Architecture, Data Management Platforms...
Motional
C#, ASP.NET, REST, REST APIs, Databases, MySQL...
Works Applications
Microservices, Spring Microservice, Cassandra, RESTful Development, Jenkins...

Experience

  • Python - 11 years
  • Back-end Development - 11 years
  • System Design - 8 years
  • Software Architecture - 8 years
  • Cloud Architecture - 8 years
  • Event-driven Architecture - 7 years
  • Distributed Systems - 6 years
  • Elasticsearch - 5 years

Preferred Environment

Ubuntu, JetBrains IDE, Slack, Linux, MacOS

The most amazing...

...system I've built was a scalable trade and communications surveillance platform processing millions of records daily reliably and cost-efficiently.

Work Experience

Staff Software Engineer

2021 - PRESENT
STEELEYE
  • Delivered enterprise SaaS and private cloud integration projects for large clients, collaborating with executives, product leaders, and engineering teams from solution design through production rollout.
  • Integrated and deployed an AI-powered trade and communications surveillance product for enterprise clients, supporting high-value deals and production-scale compliance workflows.
  • Led the design and delivery of a scalable trade and communications surveillance platform processing millions of records per day using Conductor, Kafka, and Kubernetes.
  • Improved production reliability and troubleshooting by establishing end-to-end observability with Grafana, Kibana, Prometheus, Sentry, and Filebeat.
  • Improved developer productivity and reduced CI/CD costs by consolidating services into a Pants monorepo and adopting lower-cost ARM-based infrastructure.
  • Reduced annual infrastructure costs by over $80,000 through platform and architecture optimizations.
  • Reduced Elasticsearch storage costs through schema optimization and archival strategies for older indexes.
  • Built a schema-aware analytical storage layer using Apache Iceberg, Parquet, and Zstandard to support efficient downstream analytics and AI workloads.
Technologies: Data, Big Data, Big Data Architecture, Data Management Platforms, Event-driven Architecture, Prefect, Conductor, Netflix OSS, Orkes, Apache Airflow, Management, ETL, Refinitive API, Architecture, Cloud Architecture, Data Engineering, Parquet, Data Pipelines, Python 2, Distributed Systems, PostgreSQL, Algorithms, API Development, Kubernetes, FastAPI, GraphQL, Data Management, NumPy, Pytest, Containerization, Asyncio, Python Asyncio, Redis, Testing, GitHub Actions, Continuous Delivery (CD), Continuous Integration (CI), API Design, Debugging, Troubleshooting, Apache Spark, GitHub, Redis Cache, Finance, Back-end Development, Software as a Service (SaaS), Celery, Artificial Intelligence (AI), Jira, Large Language Models (LLMs), LangChain, OpenAI, Unix, Azure, Apache Arrow, Bash, Platform Engineering, Code Review, Claude, Claude Code, AI Agents, Agentic AI, Multitenancy, Leadership, Generative Artificial Intelligence (GenAI), Codex, OpenAI API, OAuth, LLM Integration, AI Automation, AI Tools, Claude API, Feature Analysis, Streaming, Algorithmic Trading, Trading, Real-time Data, Error Logging, AWS Secrets Manager, Cloud Run, Documentation, OpenAPI, Pub/Sub, Fintech, Data Architecture

Senior Software Engineer

2018 - 2020
Motional
  • Led the architecture of a data platform for ingesting and processing terabytes of autonomous vehicle data using event-driven AWS pipelines and Parquet-based storage.
  • Improved access to vehicle metrics data and reduced processing costs by building ETL applications with PySpark, Pandas, and NumPy on EMR.
  • Developed data transformation pipelines that converted dense ride data into efficient columnar formats, enabling scalable downstream analytics and faster access to operational data.
Technologies: C#, ASP.NET, REST, REST APIs, Databases, MySQL, Relational Database Services (RDS), Serverless, Terraform, Amazon EC2, Amazon S3 (AWS S3), Python, Python 2, Python 3, Flask, Flask-Marshmallow, SQLAlchemy, PyMySQL, Docker, Spring Boot, Distributed Systems, Scaling, .NET Core, AWS Lambda, Software Engineering, Linux, APIs, Datadog, Data Engineering, Relational Databases, Back-end, PySpark, .NET, Django, Messaging, Cloud, JDBC, Agile, Amazon Elastic Container Service (ECS), ECS, Amazon Kinesis, Amazon CloudWatch, AWS Step Functions, State Machines, Autoscaling, Scrum, Maps, SSH, Infrastructure as Code (IaC), Infrastructure Monitoring, Django REST Framework, Scalability, Software Architecture, Object-oriented Design (OOD), Object-oriented Programming (OOP), API Integration, WebSockets, Event-driven Architecture, Kubernetes, Software Implementation, Lambda Functions, RESTful Microservices, Unit Testing, SOLID Principles, Serverless Architecture, Architecture, Cloud Architecture, PostgreSQL, Algorithms, API Development, Data Management, NumPy, Pytest, Containerization, Asyncio, Python Asyncio, Testing, Debugging, Apache Spark, GitHub, Back-end Development, Unix, Spark, Bash, Platform Engineering, Code Review, Feature Analysis, Streaming, Cloud Run, Documentation, Pub/Sub

Software Engineer

2014 - 2018
Works Applications
  • Achieved an SLA of fulfilling 1,000 orders per second by creating an integrated order fulfillment pipeline using multiple RESTful microservices.
  • Migrated a large Jakarta EE monolith eCommerce web application to several small Java Spring boot RESTful API microservices.
  • Designed and optimized data models in Cassandra and MySQL for the eCommerce platform.
  • Improved developer productivity through DevOps tasks like optimizing CI/CD pipelines on Jenkins, tuning JVM parameters, and managing release artifacts on Artifactory.
Technologies: Microservices, Spring Microservice, Cassandra, RESTful Development, Jenkins, CI/CD Pipelines, Hadoop, REST APIs, Java, Microservices Architecture, Databases, SQL, Spring, NoSQL, Git, Amazon Web Services (AWS), Data Modeling, Apache Kafka, REST, Back-end, Distributed Systems, Scaling, Software Engineering, Linux, APIs, Relational Databases, eCommerce, Java 8, Java EE (Jakarta EE), Messaging, Cloud, JDBC, Autoscaling, Elasticsearch, SSH, Apache Cassandra, Scalability, Software Architecture, Hibernate, Object-oriented Design (OOD), Object-oriented Programming (OOP), Docker, MySQL, API Integration, Enterprise Resource Planning (ERP), Event-driven Architecture, Software Implementation, Lambda Functions, RESTful Microservices, Message Queues, Unit Testing, JUnit, SOLID Principles, Architecture, Cloud Architecture, Python 2, Algorithms, API Development, Testing, Back-end Development, Redis, Jira, Unix, Spark, Bash, Code Review, Feature Analysis, Documentation, Apache Tomcat, Jakarta EE (Java EE or J2EE), Spring Data JPA, Spring MVC

Software Engineer Intern

2013 - 2013
Canon Marketing Japan
  • Quantified software quality through software metrics.
  • Developed an equation that measures software quality through weighted software metrics.
  • Received best project award in the computer science department for the internship work.
Technologies: Python, Medical Software, Metrics, Bash, Back-end Development, Feature Analysis, Documentation

Experience

Scalable Financial Surveillance Data Platform

I built a real-time trade and communications surveillance platform capable of processing millions of records per day with strong reliability, observability, and cost efficiency. I led the design and implementation of the system using Conductor, Kafka, and Kubernetes, creating a scalable event-driven architecture that could handle variable workloads without compromising performance. I also improved multi-tenant fairness and operational visibility by introducing stronger workflow isolation, dynamic scaling, and end-to-end monitoring with Grafana, Kibana, Sentry, and Elasticsearch. The result was a platform that was easier to operate, more resilient in production, and better suited for long-term growth.

Python Monorepo and CI/CD Platform Modernization

I modernized an ecosystem of interdependent Python applications by consolidating them into a Pants monorepo and redesigning the CI/CD pipeline for faster, more reliable releases. I led the platform work needed to simplify dependency management across hundreds of projects, introduce build caching, automate security updates, and support ARM-based builds. The result was a development and release workflow that improved engineer productivity, strengthened release consistency, and reduced annual CI/CD costs from $25,000 to $6,000.

Elasticsearch Cost Optimization and Storage Efficiency

I led a cost-optimization initiative for Elasticsearch clusters by redesigning schemas, improving compression, and refining cluster configuration to improve scale and efficiency. I analyzed workload patterns, cluster sizing, sharding, and replication strategy to choose more effective storage and processing models, then implemented index design changes, Zstandard compression, cold storage, and ILM-based retention policies. The work reduced Elasticsearch costs by 4x, cut archive storage costs by 2x, and improved long-term operational efficiency without sacrificing reliability, performance, or resilience.

Cost-efficient Analytical Data Lake for Elasticsearch Data

I built a cost-efficient analytical storage platform for trade and communications surveillance data to support ML, AI, and large-scale analytics without the cost of duplicating Elasticsearch workloads. I designed a schema-aware data lake using Apache Iceberg, Parquet, and Zstandard to improve storage efficiency and query performance, and introduced distributed deduplication to reduce redundant processing and storage. The result was a lower-cost foundation for downstream analytics that remained scalable, efficient, and easier to extend over time.

Developer Platform for ETL Applications

I built a developer platform that allowed teams to create, deploy, and operate custom ETL applications on a shared data infrastructure. I designed an SDK, a base Docker image, and an execution framework backed by AWS Step Functions and EC2 auto-scaling groups, giving teams a scalable, monitored, and access-controlled way to run ETL workloads. The platform improved execution visibility through Slack, Datadog, and CloudWatch integrations and enabled more than 10 custom ETL apps to launch within 3 months, instead of the estimated 2-3 months per app.

AWS Cost Optimization Across CI/CD, ECR, and Infrastructure

I delivered over $81,000 in AWS cost savings by leading cost optimization efforts across CI/CD, container registries, and production infrastructure. I identified the highest-cost areas, aligned changes across teams, and implemented improvements such as build and release workflow optimization, ECR lifecycle policies, automated image cleanup, and more efficient infrastructure usage. The work reduced CI costs by 47%, significantly lowered ECR spend, and improved overall resource efficiency without compromising delivery speed or operational reliability.

Refinitiv Market Data Ingestion Pipeline

I built a market data ingestion pipeline for Refinitiv security quotes, using Parquet-based storage and Prefect batch workflows to improve processing efficiency and reduce storage overhead. As the main developer, I designed and implemented the pipeline to make ingestion more reliable, scalable, and easier to operate as data volumes grew. The result was an 80% reduction in data ingestion and storage costs while providing a more efficient foundation for downstream analytics and data access.

Scalable Data Warehouse for Autonomous Vehicle Data

I built a scalable data warehouse platform that made autonomous vehicle ride data easier to access and analyze through SQL. I designed the ingestion and transformation pipeline to convert terabytes of ride data from hundreds of daily test runs into Parquet and expose it through Amazon Athena for efficient querying. I also modeled the workflow as an AWS Step Functions state machine, which improved reliability, debugging, and operational visibility as the system scaled.

Scalable Streaming Pipeline for Autonomous Vehicle Logs

I built a scalable pipeline that made large autonomous vehicle test ride logs streamable and easier to access in near real time. I redesigned the data flow by converting non-streamable logs into a row-based Avro format with timestamp-based seeking, then implemented high-throughput ingestion and processing on AWS Step Functions. The system handled 20 TB of logs per day, achieved 3.1 Gbps of encoding and compression throughput, and enabled low-latency log access through a highly concurrent Python streaming and decoding service.

Event-driven Order Fulfillment System

I built a back-end order fulfillment system for an eCommerce platform that integrated multiple APIs across orders, payments, shipping, and cart services. I developed microservices in Java and Spring Boot and designed an event-driven architecture using Kafka for asynchronous communication between distributed services. I also introduced autoscaling rules to support horizontal scaling, helping the platform meet an SLA of 1,000 fulfilled orders per hour while maintaining reliability under load.

High-performance S3 File Transfer Tool

I built a high-performance Python desktop app for reliably downloading and uploading very large files to and from Amazon S3, including files of around 500 GB. I designed a concurrent network and storage I/O engine using threads, processes, queues, locks, semaphores, and chunk-level retries with backoff to maximize throughput while avoiding memory pressure and OOM failures. The tool achieved up to 3.1 Gbps on EC2, reached 125 Mbps on a 1 Gbps LAN, and delivered about 60% better performance than the official AWS Python SDK for the target workload.

Education

2010 - 2014

Bachelor's Degree in Computer Science

Indian Institute of Information Technology - Jabalpur, India

Skills

Libraries/APIs

REST APIs, PySpark, API Development, Slack API, JDBC, Pandas, NumPy, Asyncio, Python Asyncio, Flask-Marshmallow, SQLAlchemy, PyMySQL, Jenkins Pipeline, Refinitive API, OpenAI API, Claude API, OpenAPI

Tools

Git, Amazon Athena, Amazon Elastic MapReduce (EMR), Apache Maven, Kafka Streams, Pytest, GitHub, Codex, Terraform, AWS Glue, AWS Step Functions, Amazon Elastic Container Service (ECS), Jira, Claude, Claude Code, Jenkins, Apache Avro, Amazon CloudWatch, Amazon Redshift Spectrum, GitLab, GitLab CI/CD, Jupyter, IPython Notebook, Amazon Virtual Private Cloud (VPC), Shell, PyPI, AWS Deployment, AWS SDK, Prefect, Apache Airflow, Amazon Simple Queue Service (SQS), Celery, Sentry, Apache Iceberg, Apache Tomcat

Languages

Python, Java, C#, Python 2, Python 3, Java 8, C#.NET, SQL, Bash, Batch, GraphQL

Frameworks

Hadoop, Spark, Spring Microservice, Spring Boot, .NET Core, Apache Spark, ASP.NET, .NET, JUnit, Flask, Spring, Django, Django REST Framework, Hibernate, Spring MVC

Paradigms

MapReduce, Microservices, Microservices Architecture, ETL, Object-oriented Design (OOD), Object-oriented Programming (OOP), Event-driven Architecture, Unit Testing, Serverless Architecture, Testing, REST, Agile, DevOps, Continuous Integration (CI), Continuous Delivery (CD), RESTful Development, Scrum, Management

Platforms

Docker, Apache Kafka, Amazon Web Services (AWS), Amazon EC2, AWS Lambda, Linux, Java EE (Jakarta EE), Cloud Run, Ubuntu, Kubernetes, Unix, Jupyter Notebook, MacOS, Amazon, Azure, Apache Arrow, Jakarta EE (Java EE or J2EE)

Storage

Amazon S3 (AWS S3), Relational Databases, Elasticsearch, Redis Cache, Databases, Data Pipelines, MySQL, PostgreSQL, Redis, NoSQL, Cassandra, Apache Hive, Datadog, Redshift, Microsoft SQL Server, MongoDB, Spring Data JPA

Other

Software Development, Parquet, Big Data, Data Engineering, Back-end, Distributed Systems, Software Engineering, APIs, eCommerce, Data Warehousing, Cloud, System Design, Software Architecture, API Integration, Concurrency, Software Implementation, RESTful Microservices, Multithreading, SOLID Principles, FastAPI, Architecture, Cloud Architecture, Data Management, Containerization, API Design, Debugging, Troubleshooting, Back-end Development, Platform Engineering, Code Review, Feature Analysis, Streaming, Documentation, Pub/Sub, Data Warehouse Design, Relational Database Services (RDS), Scaling, Algorithms, Amazon RDS, Slackbot, Data Wrangling, Containers, Autoscaling, Zstandard, ECS, SSH, Infrastructure as Code (IaC), Scalability, Enterprise Resource Planning (ERP), WebSockets, Lambda Functions, Message Queues, Finance, Artificial Intelligence (AI), AI Agents, Multitenancy, Leadership, Error Logging, AWS Secrets Manager, Fintech, Data Architecture, CI/CD Pipelines, Data Compression, Serverless, Data Modeling, Amazon API Gateway, Autonomous Navigation, Self-driving Cars, Messaging, SDKs, State Machines, Amazon Kinesis, Maps, Jupiter, Infrastructure Monitoring, Apache Cassandra, Networking, Monitoring, Deployment, Memory Management, Memory Mapped Files, Processing & Threading, Benchmarking, Memory Profiling, Data, Software Integration, Big Data Architecture, Data Management Platforms, Conductor, Netflix OSS, Orkes, GitHub Actions, Software as a Service (SaaS), Monorepos, pantsbuild, Schemas, Amazon Glacier, ARM, Cost Control, Large Language Models (LLMs), LangChain, OpenAI, Medical Software, Metrics, Agentic AI, Generative Artificial Intelligence (GenAI), OAuth, LLM Integration, AI Automation, AI Tools, Algorithmic Trading, Trading, Real-time Data

Collaboration That Works

How to Work with Toptal

Toptal matches you directly with global industry experts from our network in hours—not weeks or months.

1

Share your needs

Discuss your requirements and refine your scope in a call with a Toptal domain expert.
2

Choose your talent

Get a short list of expertly matched talent within 24 hours to review, interview, and choose from.
3

Start your risk-free talent trial

Work with your chosen talent on a trial basis for up to two weeks. Pay only if you decide to hire them.

Top talent is in high demand.

Start hiring