Renan is available for hire

Renan Tomazoni Pinzon

Verified Expert in Engineering

Data Engineer and Developer

Location

Porto Alegre - State of Rio Grande do Sul, Brazil

Toptal Member Since

July 14, 2023

Renan is a skilled data engineer with a software engineering background and 15 years of experience. He has worked with Zenvia, PagoNxt, and Neogrid, leading the creation of a Payment Card Industry (PCI) compliance data platform and architecting new data solutions on AWS and Azure. Proficient in Scala, Java, Spark, Flink, Hadoop, Kafka, AWS, and Azure, Renan is a valuable asset in any organization.

Portfolio

Zenvia

Scala, Java, Spark, Flink, Hadoop, Databricks, Amazon Web Services (AWS)...

Freelance

Scala, Amazon Web Services (AWS), Amazon DynamoDB, SBT, Shell, Linux, H2...

Viewdeck Consulting

Chef, Terraform, Node.js, Docker, Vagrant, Amazon Web Services (AWS)...

Experience

Java - 15 years Hadoop - 10 years Scala - 8 years Spark - 8 years Azure - 4 years Apache Kafka - 4 years Amazon Web Services (AWS) - 3 years Flink - 2 years

Availability

Part-time

Preferred Environment

Spark, Flink, Scala, Java, Hadoop, Azure, Apache Kafka, NoSQL, SQL, Amazon Web Services (AWS)

The most amazing...

...thing I've done is lead the creation of a PCI compliance data platform for a cloud-based payment solution from scratch.

Work Experience

Data Engineer

2021 - PRESENT

Zenvia

Created the new data platform's architecture on top of AWS, enabling new data solutions delivery.
Built and developed Spark structured streaming jobs in Scala to ingest data from Kafka into S3 using Delta Lake as a storage layer.
Architected and developed Spark batch jobs in Scala to generate other Delta tables using different file types as a source.
Designed and developed Flink streaming jobs in Scala to process and correlate data from different Kafka topics and make them available in near real-time.
Devised and developed microservices using Java to consume data from Kafka and store them on MongoDB.

Technologies: Scala, Java, Spark, Flink, Hadoop, Databricks, Amazon Web Services (AWS), MongoDB, PostgreSQL, Kubernetes, Delta Lake, Amazon S3 (AWS S3), Apache Airflow, Amazon EKS, Amazon EC2, Amazon Glacier, Amazon ElastiCache, Amazon Elastic MapReduce (EMR), Java Amazon SDK, Amazon Linux, Amazon Linux AMI, Amazon Simple Queue Service (SQS), Amazon RDS, Amazon Virtual Private Cloud (VPC), Docker, Docker Compose, Shell, Linux, Git, AWS IAM, Druid.io, Data Engineering, Big Data, MongoDB Atlas, Data Architecture, Big Data Architecture, ETL, ELT, SQL, APIs, Architecture

Scala Developer

2021 - PRESENT

Freelance

Developed microservices and libraries using Scala and Cats Effect.
Created integration with external APIs using Scala and Cats Effect.
Migrated microservices from Node.js to Scala with Cats Effect.

Technologies: Scala, Amazon Web Services (AWS), Amazon DynamoDB, SBT, Shell, Linux, H2, Cats Effect, Git, APIs

Freelance DevOps Engineer

2019 - 2021

Viewdeck Consulting

Streamlined infrastructure provisioning by implementing Chef.
Implemented Terraform, also streamlining infrastructure provisioning.
Developed microservices in Node.js to integrate various CRMs.

Technologies: Chef, Terraform, Node.js, Docker, Vagrant, Amazon Web Services (AWS), Amazon EC2, UKCloud, Linux, Git

Data Engineer

2019 - 2021

PagoNxt (a Santander company)

Designed the architecture and defined the tech stack of the data platform following security standards like the PCI Data Security Standard.
Created and developed core components and tools used by other data engineering teams to ingest, process, and deliver data.
Devised and developed a custom change data capture (CDC) solution to capture data from MongoDB on top of MongoDB change streams.

Technologies: Spark, Scala, Java, Azure, Databricks, MongoDB, Kubernetes, Dremio, Delta Lake, PCI DSS, SBT, Docker, Docker Compose, Azure Event Hubs, Azure Service Bus, Azure Data Lake, Azure Databricks, Azure DevOps, Azure Application Insights, Azure Kubernetes Service (AKS), Azure Key Vault, HashiCorp, Git, Data Engineering, Big Data, Data Architecture, Big Data Architecture, ETL, ELT, SQL, Architecture

Data Engineer

2013 - 2019

Neogrid

Designed and developed solutions to process data using Spark with Scala on top of Hadoop to replace a traditional BI solution built on top of SQL Server Integration Services.
Architected and developed solutions to process data using MapReduce, Apache Pig, and Apache Crunch on top of Hadoop.
Created and developed APIs and libraries to work with Hadoop Distributed File System (HDFS) and HBase.
Devised and developed pipelines to index data from HBase into Apache Solr in near real-time.
Built and developed a logging solution for the entire company using the ELK stack.
Helped the infrastructure team to automate the entire Hadoop infrastructure using Chef.

Technologies: Spark, Scala, Hadoop, Java, HBase, Solr, MapReduce, Apache Pig, Apache Crunch, RabbitMQ, Spring, Chef, ELK (Elastic Stack), HDFS, Finagle, Apache Ignite, Microsoft SQL Server, Logstash, Cloudera, Linux, Apache Lucene, SBT, Apache Maven, Flume, Apache ZooKeeper, Docker, Docker Compose, Vagrant, Git, Data Engineering, Big Data, Data Architecture, Big Data Architecture, ETL, ELT, SQL, APIs, Architecture

Software Architect

2009 - 2013

Neogrid

Designed and developed enterprise and standalone applications in Java.
Increased throughput by over 20% by enhancing the performance of a mission-critical electronic data interchange (EDI) application written in Java.
Built and developed libraries to provide digital certification and cryptography capabilities using physical hardware security modules as underlying storage and OpenPGP standards through Java Cryptography Architecture and Extension (JCA/JCE).
Created and developed a mission-critical EDI application to replace the previous system.

Technologies: Java, Java EE, WebLogic, Microsoft SQL Server, Apache Maven, Spring, ELK (Elastic Stack), JCA, JCE, Enterprise Java Beans (EJB), RabbitMQ, Ant Design, Shell, OpenPGP, Cryptography, Logstash, Filebeat, Git, Subversion (SVN), SQL, APIs, Architecture

Experience

Content Repository

An active repository to store content and metadata related to it. As the software architect, I developed most features of the solution using Java. The solution relies on HBase and HDFS to store the data and uses Apache Solr to index data and allow low-latency searches.

This repository's main requirements included high availability, fault tolerance, and low latency. This was built to replace a legacy solution that was using a traditional file system and database. The repository also offers schema management and evolution to allow each application to store the metadata relevant to its business.

Payment Cloud Platform

A cloud-native payment platform built on top of Azure. I was the main data platform architect. I defined the technologies and developed frameworks, libraries, and tools used by other data engineers.

The data platform allows the ingestion of data using CDC on top of MongoDB, events from Kafka, and any file. It also allows processing data through streaming or batch processing with Spark running on Databricks or Kubernetes.

Supply Chain Management Analytics Platform

An analytical platform for supply chain management. As the main architect, I did the environment dimensioning, defined the technology stack, and developed numerous critical components of the solution.

The platform's goal was to replace a legacy BI solution built on top of Microsoft Integration Services (SSIS), reducing our operational costs and providing more analytical capabilities to improve the replenishment time of our customers.

Skills

Languages

Scala, Java, SQL, Java 6, R

Frameworks

Spark, Hadoop, Data Lakehouse, Apache Spark, Spring, Finagle, Ant Design

Libraries/APIs

Apache Crunch, Node.js, Apache Lucene

Tools

Flink, RabbitMQ, Spark SQL, JCA, JCE, Solr, ELK (Elastic Stack), Apache Ignite, AWS IAM, MongoDB Atlas, Chef, Cloudera, Azure Kubernetes Service (AKS), Terraform, Apache Maven, GitLab CI/CD, Apache Airflow, Shell, Apache Solr, Amazon EKS, Amazon ElastiCache, Amazon Elastic MapReduce (EMR), Java Amazon SDK, Amazon Simple Queue Service (SQS), Amazon Virtual Private Cloud (VPC), Docker Compose, Logstash, SBT, Flume, Apache ZooKeeper, Vagrant, Azure Application Insights, Azure Key Vault, HashiCorp, Git, Subversion (SVN)

Paradigms

MapReduce, ETL, Functional Programming, Parallel Computing, Azure DevOps

Platforms

Azure, Amazon Web Services (AWS), Databricks, Apache Pig, Spark Core, Apache Kafka, Java EE, Kubernetes, Azure Synapse, Java SE 6, Docker, Azure Event Hubs, Amazon EC2, Amazon Linux, Amazon Linux AMI, Linux

Storage

NoSQL, RDBMS, HBase, HDFS, Microsoft SQL Server, Data Pipelines, Amazon S3 (AWS S3), MongoDB, PostgreSQL, Data Lakes, Azure Cosmos DB, Azure Blobs, MySQL, MariaDB, Druid.io, Amazon DynamoDB, H2

Other

Big Data, Software Engineering, Computer Science, Distributed Systems, Programming, Delta Lake, Enterprise Java Beans (EJB), ELT, Data Modeling, Cloud, Cloud Infrastructure, Azure Databricks, Data Engineering, Data Architecture, Big Data Architecture, EMR, APIs, Architecture, AWS Certified Solution Architect, Dremio, PCI DSS, WebLogic, Computer Repair, Networking, Cloud Services, User-defined Functions (UDF), Cloud Computing, Delta Live Tables, Data Processing, Azure Data Lake, Azure Stream Analytics, Azure Data Factory, Blob Storage, CDC, CI/CD Pipelines, Azure Service Bus, Data Warehousing, Supply Chain, Supply Chain Management (SCM), Electronic Data Interchange (EDI), Amazon Glacier, Amazon RDS, OpenPGP, Cryptography, Filebeat, UKCloud, Cats Effect

Education

2017 - 2018

Master of Business Administration (MBA) in Data Science (Big Data)

Instituto de Gestão e Tecnologia da Informação (IGTI) - Belo Horizonte, MG, Brazil

2014 - 2015

Master in Information Technology (MIT) in Software Engineering with Java

Instituto Infnet - Porto Alegre, RS, Brazil

2006 - 2011

Bachelor's Degree in Information Systems

Pontifícia Universidade Católica do Rio Grande do Sul (PUCRS) - Porto Alegre, RS, Brazil

2006 - 2007

Technical Diploma in Computer Science

Alcides Maya | Faculdade e Escola Técnica - Porto Alegre, RS, Brazil

Toptal matches you directly with global industry experts from our network in hours—not weeks or months.

Share your needs

Discuss your requirements and refine your scope in a call with a Toptal domain expert.

Choose your talent

Get a short list of expertly matched talent within 24 hours to review, interview, and choose from.

Start your risk-free talent trial

Work with your chosen talent on a trial basis for up to two weeks. Pay only if you decide to hire them.

Top talent is in high demand.

Start hiring