Bala is available for hire
Hire BalaBala Ramachandran
Verified Expert in Engineering
Data Engineer and Developer
Location
Chennai, India
Toptal Member Since
November 14, 2022
Bala is a data engineer and business intelligence consultant with 20+ years of experience delivering innovative solutions to facilitate data-driven decisions. He is an Oracle Certified Professional with a master's degree in computer science and an MBA in systems and marketing. He has implemented multiple analytical solutions on-premises and on AWS and Azure cloud platforms. Bala has also built modern cloud data analytics systems and operational ERP analytics like PeopleSoft and Workday.
Portfolio
Anjane Technologies
Data Build Tool (dbt), Data Modeling, Data Architecture, Data Pipelines...
Tenna
JavaScript, Node.js, Internet of Things (IoT), PostgreSQL, Amazon DynamoDB...
Moka
Informatica Cloud, Data Build Tool (dbt), Azure, Informatica PowerCenter, SQL...
Experience
Availability
Part-time
Preferred Environment
Python 3, Data Build Tool (dbt), Snowflake, Data Warehousing, Data Engineering, ETL
The most amazing...
...thing I've built is a metadata-driven data lake solution—using Python, dbt, and the Jinja template engine—which ingested 250+ source systems with its metadata.
Work Experience
Senior Solutions Architect
2021 - PRESENT
Anjane Technologies
- Architected the metadata-driven data pipeline solution using Talend, data build tool (dbt), and Snowflake for Stanley Black & Decker, as part of the Caspian data lake.
- Created a dbt macro-based framework to auto-generate models using only metadata, such as column names and data types, and used the framework to successfully ingest data from over 200 systems, each having between 4-150 tables.
- Designed templatized dbt models for data marts, using custom materialization for merging data into target facts. The data marts used the data from the template-ingested data lake.
- Developed custom dbt packages—for data ingestion and load logging and monitoring—and a DataOps dashboard to assist the operations team across the various loads using Snowflake Snowsight.
- Built a custom solution for change-data-capture tables when a full refresh wipes out the target tables. I used Snowflake: task and time travel, dbt macros, and Qlik Replicate to automate and duplicate the solution for hundreds of bigger tables.
- Introduced and trained a team of SQL developers in dbt core and cloud with a custom-built bootcamp.
- Worked on Python-based data acquisition from external source systems—like Similarweb, Riversand, and Salsify—using templatized code, for reusability, with REST API.
- Acquired data from Excel sheets using Python Pandas and performed data wrangling before inserting them in Snowflake for further templatized ingestion.
- Used Snowflake SQL and metadata heavily for usage analytics and cost allocation across the usage teams.
Technologies: Data Build Tool (dbt), Data Modeling, Data Architecture, Data Pipelines, Snowflake, Talend, Collibra, Python 3, Python API, Jinja, Flask-RESTful, Flask, SQL, ETL, CDC, Qlik Replicate, Snowsight
Senior Engineer
2019 - 2021
Tenna
- Worked on an IoT-based asset management SaaS platform used in the US construction industry.
- Created a Node.js-based asset management system with microservices architecture and REST API, used in various construction industry assets.
- Built a prototype of Looker analytics on top of the SaaS data and later pivoted to an in-house React-based solution with a back end provided by materialized views in Postgres.
- Developed continuous integration testing and deployment of code into the AWS servers.
- Produced reports using React/Node.js and Postgres materialized views for the SaaS asset management system's internal and external customers.
- Designed an Athena-based service to query the messages received from IoT devices stored in Amazon S3 buckets.
Technologies: JavaScript, Node.js, Internet of Things (IoT), PostgreSQL, Amazon DynamoDB, Redis, Elasticsearch, RESTful Microservices, REST APIs, Looker, BI Reporting, Data Migration, Amazon Athena, MongoDB, Express.js, React, SaaS
Data Engineering Lead
2019 - 2019
Moka
- Engaged as the first data engineer hired to take over the external vendor-developed ETL code using Informatica PowerCenter.
- Set up the go-to data ingestion process and roadmap by building a prototype and proof of concept (POC), replacing Informatica PowerCenter and Intelligent Cloud with a more robust Airflow and dbt-based, extract-load-transform (ELT) solution.
- Worked with clients like Nestlé Waters North America and Vita Coco to onboard their financials, supply chain, and data relationship management data into the Moka analytics solutions.
- Used information cloud-based processes to ingest data from external providers like Nielsen, VIP, and TruePlanner for competitor, distributors' product depletion, and marketing spend analyses.
- Created mapping and integration algorithms to merge the client's internal and external data at the SKU level.
- Completed the full implementation of Vita Coco's onboarding into the SaaS application.
Technologies: Informatica Cloud, Data Build Tool (dbt), Azure, Informatica PowerCenter, SQL, Dedicated SQL Pool (formerly SQL DW), Azure SQL Data Warehouse, Apache Airflow, SaaS, Data Migration
Project Lead
2012 - 2019
New York University
- Worked with an enterprise data warehouse team across various projects involving PeopleSoft Campus Solutions, PeopleSoft Financials, and Workday HCM.
- Used IBM DataStage to extract and load data into an Oracle 11i-based data warehouse and RPC-based data extraction techniques to pull data from Workday with SOAP APIs.
- Created a service analytics data mart by pulling data from the ServiceNow cloud-based helpdesk software using REST API and built the data model to analyze tickets with other internal sources like projects and budgeting.
- Collaborated with external vendors to create the learning analytics solution by extracting data from external platforms like Sakai and merging the data with internal systems using Databricks Spark, Snowflake, and Tableau.
- Re-architected the solution to process data incrementally, managed the workflow using Apache Airflow, and rewrote the Spark DataFrame code for efficient incremental data processing and population into Snowflake.
- Developed a 16-year student-retention-analysis data model and ETL pipeline with complex logic to be used by the institutional research data scientist for planning and budgeting.
- Led the technical delivery of the Senior Leader Portal using Tableau dashboards, developed with data from structured sources across various domains and unstructured data like sustainability, affordability, and citations.
Technologies: PeopleSoft Campus Solutions (CS), PeopleSoft ERP Application, IBM InfoSphere (DataStage), Oracle Business Intelligence Enterprise Edition 11g (OBIEE), Python 3, Spark SQL, Databricks, Amazon Web Services (AWS), Tableau, Pandas, ETL Tools, Workday, Data Modeling, Data Engineering, Data Warehouse Design, ServiceNow REST API, Apache Airflow
Senior Manager
2008 - 2013
Cognizant
- Worked on establishing BNY Mellon's PeopleSoft Financials and enterprise performance management (EPM) account from scratch to 50+ resources onsite in New York and Pittsburgh, US, and offshore in Chennai and Pune, India.
- Executed multiple performance management projects for executive reporting, including fund transfer pricing, cost allocations, and spend analytics.
- Oversaw the global procurement analytics launch using the Oracle Spend Analytics product line, working across the business and IT teams and external vendors.
- Managed three projects and teams as a delivery manager and consulted for other teams across BNY Mellon on data warehouse technologies.
- Provided consulting services to other teams across Cognizant on various data warehousing tools and data modeling discussions.
Technologies: PeopleSoft, IBM InfoSphere (DataStage), Oracle Business Intelligence Enterprise Edition 11g (OBIEE), Oracle Business Intelligence Applications (OBIA), Informatica ETL, IBM Db2, Oracle 11i, PeopleSoft ERP Application, Enterprise Performance Management (EPM)
Team Lead
2004 - 2008
HSBC India
- Used various PeopleSoft applications and tools to implement the global general ledger, working for HSBC software development India, based out of Pune and Chicago.
- Worked on consolidations, fund transfer pricing, and allocation streams of the PeopleSoft EPM application, using PeopleCode, IBM DataStage server and parallel, and IBM Cognos for reporting.
- Oversaw the EPM application's POC 2 and 3 and pilot projects' releases, managing a team of 6-8 people.
- Coordinated the work executed onsite and offshore across multiple time zones.
- Managed the in-house bootcamp training and resource allocation across multiple projects as part of the recruitment and onboarding team.
Technologies: PeopleTools, IBM InfoSphere (DataStage), IBM Db2, IBM Cognos
Developer
2004 - 2004
Hexaware Technologies
- Worked on lease administration for the real estate management product development as part of the PeopleSoft Financials 8.9 release.
- Developed various enhancements for the PeopleSoft Financials 8.9 release, including Europe VAT and global components.
- Received Ascential DataStage training before IBM acquired the tool, then upgraded PeopleSoft Financials 8.9 and developed ETL jobs to migrate data from previous versions into the new version.
- Built a productivity tool to automate code review for PeopleSoft projects against configurable coding standards.
- Participated in the Hexaware–PeopleSoft India Services Center tie-up. After Oracle acquired PeopleSoft, this division was absorbed into Oracle India, Bangalore.
Technologies: PeopleSoft, PeopleTools, PeopleCode, SQR, Datastage, IIS SQL Server
Developer
2002 - 2004
Covansys Corporation
- Worked on the Public Employees Retirement System of Rhode Island, contributing to phases 1 and 2 of the project, covering payroll, benefits, and supplemental payroll requirements.
- Operated in the C++-based Forte development environment with MS SQL Server as the back end.
- Used the SQR SQL reporting tool for generating multiple batch-based reports for printing member letters and payroll checks.
- Developed various productivity tools to automate code generation with templates using Forte.
- Conducted workshops and training to onboard the new batch.
Technologies: Forte, C++, IIS SQL Server, SQR
Experience
Data Engineering for CPG Analytics
https://www.moka.nyc/contact.html#aboutLed the data engineering team for a consumer packaged goods (CPG) SaaS analytics company with marque customers like Nestle Waters and Vita Coco.
I led the customer data onboarding process for Vita Coco into the SaaS analytics platform. I also designed and developed the architecture to load the customer sales and financial data efficiently from the internal SAP systems, merged with the distributor depletion and marketing spend from third-party data providers.
Additionally, I created custom data pipelines for loading various master data into the Moka analytics system—including data from SAP, distributor depletion data from VIP, and marketing planner data—using Informatica Cloud running inside Microsoft Azure.
I developed the POC architecture for data pipelines to run across customers using the Apache Airflow workflow orchestration. I also worked on a POC for migrating from Informatica Cloud into dbt ELT solutions with Snowflake. Other technologies used in the project include Azure Blob Storage, Azure Virtual Machines, and Python for custom transformations.
I led the customer data onboarding process for Vita Coco into the SaaS analytics platform. I also designed and developed the architecture to load the customer sales and financial data efficiently from the internal SAP systems, merged with the distributor depletion and marketing spend from third-party data providers.
Additionally, I created custom data pipelines for loading various master data into the Moka analytics system—including data from SAP, distributor depletion data from VIP, and marketing planner data—using Informatica Cloud running inside Microsoft Azure.
I developed the POC architecture for data pipelines to run across customers using the Apache Airflow workflow orchestration. I also worked on a POC for migrating from Informatica Cloud into dbt ELT solutions with Snowflake. Other technologies used in the project include Azure Blob Storage, Azure Virtual Machines, and Python for custom transformations.
Caspian Data Lake and Data Mart
Architected, designed, and built a new metadata-driven, template-based data lake ingestion framework to perform ELT using tools such as Python, AWS, REST API, dbt, Snowflake, and Talend Cloud.
I designed and created various templates for ingesting data from flat CSV files, JSON, API, and Excel into the Snowflake data lake. I also architected and built a solution that automates dbt-based processes for ingesting data from the Qlik Replicate/Attunity replication tool, with data availability even when replication is restarted or fully refreshed.
In addition, I used the Snowflake zero-copy clone, time travel, and tasks features to monitor and redirect queries when source tables are truncated and recreated. I also re-platformed various data marts from Amazon Redshift to Snowflake and converted Redshift stored procedures to dbt models. I built and utilized dbt custom packages, macros, and materializations to efficiently migrate and populate the data marts.
Finally, I used Python to build the API ingestion from external systems to perform extract and load-related tasks and dbt for the transformations in Snowflake. I also managed a team of consultants to execute multiple data projects across various teams and other external vendors.
I designed and created various templates for ingesting data from flat CSV files, JSON, API, and Excel into the Snowflake data lake. I also architected and built a solution that automates dbt-based processes for ingesting data from the Qlik Replicate/Attunity replication tool, with data availability even when replication is restarted or fully refreshed.
In addition, I used the Snowflake zero-copy clone, time travel, and tasks features to monitor and redirect queries when source tables are truncated and recreated. I also re-platformed various data marts from Amazon Redshift to Snowflake and converted Redshift stored procedures to dbt models. I built and utilized dbt custom packages, macros, and materializations to efficiently migrate and populate the data marts.
Finally, I used Python to build the API ingestion from external systems to perform extract and load-related tasks and dbt for the transformations in Snowflake. I also managed a team of consultants to execute multiple data projects across various teams and other external vendors.
Education
2013 - 2013
Certificate in Business Analytics
New York University - New York, United States
2004 - 2009
Master of Business Administration (MBA) in Systems and Marketing
Sikkim Manipal University - Chennai, Tamil Nadu, India
2000 - 2002
Master's Degree in Computer Science
University of Madras - Chennai, Tamil Nadu, India
Certifications
MAY 2012 - PRESENT
Oracle Certified Professional
Oracle
Skills
Other
Data Build Tool (dbt), Data Warehousing, Data Engineering, PeopleSoft, Enterprise Performance Management (EPM), ETL Tools, Data Modeling, Data Warehouse Design, Data Architecture, Computer Science, Management Systems, Business Analysis, Forte, PeopleSoft ERP Application, Workday, Informatica Cloud, SaaS, Data Migration, Internet of Things (IoT), RESTful Microservices, BI Reporting, CDC, Snowsight, Azure Virtual Machines
Languages
Python 3, Snowflake, PeopleCode, SQL, C++, SQR, JavaScript, Python
Frameworks
Jinja, Express.js, Flask
Libraries/APIs
Pandas, REST APIs, Python API, ServiceNow REST API, Node.js, React, Flask-RESTful
Tools
PeopleTools, IBM InfoSphere (DataStage), GitHub, IBM Cognos, Oracle Business Intelligence Enterprise Edition 11g (OBIEE), Oracle Business Intelligence Applications (OBIA), Informatica ETL, PeopleSoft Campus Solutions (CS), Spark SQL, Tableau, Apache Airflow, Informatica PowerCenter, Looker, Amazon Athena, Collibra, Qlik Replicate
Paradigms
ETL
Storage
Datastage, IBM Db2, Oracle 11i, PostgreSQL, Data Pipelines, IIS SQL Server, Amazon DynamoDB, Redis, Elasticsearch, MongoDB, Azure Blobs, Azure SQL, Azure SQL Databases
Platforms
Databricks, Amazon Web Services (AWS), Azure, Azure SQL Data Warehouse, Talend, Amazon EC2, Dedicated SQL Pool (formerly SQL DW)
Collaboration That Works
How to Work with Toptal
Toptal matches you directly with global industry experts from our network in hours—not weeks or months.
1
Share your needs
Discuss your requirements and refine your scope in a call with a Toptal domain expert.
2
Choose your talent
Get a short list of expertly matched talent within 24 hours to review, interview, and choose from.
3
Start your risk-free talent trial
Work with your chosen talent on a trial basis for up to two weeks. Pay only if you decide to hire them.
Top talent is in high demand.
Start hiring