Renan Tomazoni Pinzon
Verified Expert in Engineering
Data Engineer and Developer
Porto Alegre - State of Rio Grande do Sul, Brazil
Toptal member since July 14, 2023
Renan is a skilled data engineer with a software engineering background and 15 years of experience. He has worked with Zenvia, PagoNxt, and Neogrid, leading the creation of a Payment Card Industry (PCI) compliance data platform and architecting new data solutions on AWS and Azure. Proficient in Scala, Java, Spark, Flink, Hadoop, Kafka, AWS, and Azure, Renan is a valuable asset in any organization.
Portfolio
Experience
- Java - 15 years
- Hadoop - 10 years
- Scala - 8 years
- Spark - 8 years
- Azure - 4 years
- Apache Kafka - 4 years
- Amazon Web Services (AWS) - 3 years
- Flink - 2 years
Work Experience
Data Engineer
Zenvia
- Created the new data platform's architecture on top of AWS, enabling new data solutions delivery.
- Built and developed Spark structured streaming jobs in Scala to ingest data from Kafka into S3 using Delta Lake as a storage layer.
- Architected and developed Spark batch jobs in Scala to generate other Delta tables using different file types as a source.
- Designed and developed Flink streaming jobs in Scala to process and correlate data from different Kafka topics and make them available in near real-time.
- Devised and developed microservices using Java to consume data from Kafka and store them on MongoDB.
Scala Developer
Freelance
- Developed microservices and libraries using Scala and Cats Effect.
- Created integration with external APIs using Scala and Cats Effect.
- Migrated microservices from Node.js to Scala with Cats Effect.
Freelance DevOps Engineer
Viewdeck Consulting
- Streamlined infrastructure provisioning by implementing Chef.
- Implemented Terraform, also streamlining infrastructure provisioning.
- Developed microservices in Node.js to integrate various CRMs.
Data Engineer
PagoNxt (a Santander company)
- Designed the architecture and defined the tech stack of the data platform following security standards like the PCI Data Security Standard.
- Created and developed core components and tools used by other data engineering teams to ingest, process, and deliver data.
- Devised and developed a custom change data capture (CDC) solution to capture data from MongoDB on top of MongoDB change streams.
Data Engineer
Neogrid
- Designed and developed solutions to process data using Spark with Scala on top of Hadoop to replace a traditional BI solution built on top of SQL Server Integration Services.
- Architected and developed solutions to process data using MapReduce, Apache Pig, and Apache Crunch on top of Hadoop.
- Created and developed APIs and libraries to work with Hadoop Distributed File System (HDFS) and HBase.
- Devised and developed pipelines to index data from HBase into Apache Solr in near real-time.
- Built and developed a logging solution for the entire company using the ELK stack.
- Helped the infrastructure team to automate the entire Hadoop infrastructure using Chef.
Software Architect
Neogrid
- Designed and developed enterprise and standalone applications in Java.
- Increased throughput by over 20% by enhancing the performance of a mission-critical electronic data interchange (EDI) application written in Java.
- Built and developed libraries to provide digital certification and cryptography capabilities using physical hardware security modules as underlying storage and OpenPGP standards through Java Cryptography Architecture and Extension (JCA/JCE).
- Created and developed a mission-critical EDI application to replace the previous system.
Project History
Content Repository
This repository's main requirements included high availability, fault tolerance, and low latency. This was built to replace a legacy solution that was using a traditional file system and database. The repository also offers schema management and evolution to allow each application to store the metadata relevant to its business.
Payment Cloud Platform
The data platform allows the ingestion of data using CDC on top of MongoDB, events from Kafka, and any file. It also allows processing data through streaming or batch processing with Spark running on Databricks or Kubernetes.
Supply Chain Management Analytics Platform
The platform's goal was to replace a legacy BI solution built on top of Microsoft Integration Services (SSIS), reducing our operational costs and providing more analytical capabilities to improve the replenishment time of our customers.
Education
Master of Business Administration (MBA) in Data Science (Big Data)
Instituto de Gestão e Tecnologia da Informação (IGTI) - Belo Horizonte, MG, Brazil
Master in Information Technology (MIT) in Software Engineering with Java
Instituto Infnet - Porto Alegre, RS, Brazil
Bachelor's Degree in Information Systems
Pontifícia Universidade Católica do Rio Grande do Sul (PUCRS) - Porto Alegre, RS, Brazil
Technical Diploma in Computer Science
Alcides Maya | Faculdade e Escola Técnica - Porto Alegre, RS, Brazil
Certifications
Databricks Certified Data Engineer Professional
Databricks
AWS Certified Solutions Architect - Associate
Amazon Web Services (AWS)
Databricks Certified Associate Developer for Apache Spark 3.0
Databricks
Databricks Certified Data Engineer Associate
Databricks
AWS Certified Cloud Practitioner
Amazon Web Services (AWS)
Academy Accreditation - Databricks Lakehouse Fundamentals
Databricks
DP-200 Implementing an Azure Data Solution
Microsoft
Microsoft Certified: Azure Fundamentals
Microsoft
DP-201 Designing an Azure Data Solution
Microsoft
Microsoft Certified: Azure Data Fundamentals
Microsoft
Microsoft Certified: Azure Data Engineer Associate
Microsoft
CCA Spark and Hadoop Developer
Cloudera, Inc.
Functional Programming in Scala
École Polytechnique Fédérale de Lausanne
Oracle Certified Professional, Java SE 6 Programmer
Oracle
Skills
Libraries/APIs
Apache Crunch, Node.js, Apache Lucene
Tools
Flink, RabbitMQ, Spark SQL, Dremio, JCA, JCE, ELK (Elastic Stack), Apache Ignite, AWS IAM, MongoDB Atlas, Chef, Cloudera, Azure Kubernetes Service (AKS), Terraform, Apache Maven, GitLab CI/CD, Apache Airflow, Shell, Apache Solr, Amazon EKS, Amazon ElastiCache, Amazon Elastic MapReduce (EMR), Java Amazon SDK, Amazon Simple Queue Service (SQS), Amazon Virtual Private Cloud (VPC), Docker Compose, Logstash, SBT, Flume, Apache ZooKeeper, Vagrant, Azure Application Insights, Azure Key Vault, HashiCorp, Git, Subversion (SVN)
Languages
Scala, Java, SQL, Java 6, R
Frameworks
Spark, Hadoop, Data Lakehouse, Apache Spark, Spring, Finagle, Delta Live Tables (DLT), Ant Design
Paradigms
MapReduce, ETL, Functional Programming, Parallel Computing, Azure DevOps
Platforms
Azure, Amazon Web Services (AWS), Databricks, Apache Pig, Spark Core, Apache Kafka, Java EE (Jakarta EE), Kubernetes, Azure Synapse, Java SE 6, Docker, Azure Event Hubs, Amazon EC2, Amazon Linux, Amazon Linux AMI, Linux
Storage
NoSQL, RDBMS, HBase, HDFS, Microsoft SQL Server, Data Pipelines, Amazon S3 (AWS S3), MongoDB, PostgreSQL, Data Lakes, Azure Cosmos DB, Azure Blobs, MySQL, MariaDB, Druid.io, Amazon DynamoDB, H2
Other
Big Data, Software Engineering, Computer Science, Distributed Systems, Programming, Delta Lake, Enterprise Java Beans (EJB), ELT, Data Modeling, Cloud, Cloud Infrastructure, Azure Databricks, Data Engineering, Data Architecture, Big Data Architecture, EMR, APIs, Architecture, AWS Certified Solution Architect, PCI DSS, WebLogic, Computer Repair, Networking, Cloud Services, User-defined Functions (UDF), Cloud Computing, Data Processing, Azure Data Lake, Azure Stream Analytics, Azure Data Factory (ADF), Blob Storage, CDC, CI/CD Pipelines, Azure Service Bus, Data Warehousing, Supply Chain, Supply Chain Management (SCM), Electronic Data Interchange (EDI), Amazon Glacier, Amazon RDS, OpenPGP, Cryptography, Filebeat, UKCloud, Cats Effect
How to Work with Toptal
Toptal matches you directly with global industry experts from our network in hours—not weeks or months.
Share your needs
Choose your talent
Start your risk-free talent trial
Top talent is in high demand.
Start hiring