Joao Cotrim
Verified Expert in Engineering
Data Engineer and Developer
Lisbon, Portugal
Toptal member since December 20, 2022
Joao has been working as a data engineer since 2009. In the last two years, he has mainly worked on energy and utilities projects with different responsibilities, from developer to technical lead on a 10-member team. Joao is keen on problem-solving and constantly searches for the best solution. He works well independently but enjoys working in a group, learning, and sharing knowledge. He is very familiar with Agile-Scrum and ServiceNow, not only as an end user but also as a data source.
Portfolio
Experience
- SQL - 13 years
- Data Engineering - 13 years
- Informatica PowerCenter - 9 years
- ServiceNow - 6 years
- Jira - 6 years
- Azure Data Factory (ADF) - 3 years
- Databricks - 3 years
- PySpark - 3 years
Preferred Environment
Databricks, Informatica PowerCenter, Databases, SQL, Analytics, Business Intelligence (BI), Data Engineering, Azure, Azure Databricks, Azure Data Factory (ADF)
The most amazing...
...promotion I've had is to tech lead on a 10-member team, building analytics over IT Sources like ServiceNow, Jira, SAP SuccessFactors, and other data sources.
Work Experience
Data Engineer
HCL Technologies
- Handled extract, transform, and load (ETL) processes proficiently using Informatica PowerCenter.
- Managed XML target files from various sources, including databases, files (XML, delimited, fixed-width, and multi-line files).
- Resolved efficiently incidents related to data processing, system errors, or performance issues.
- Managed Unix-based systems, including scripting and automation.
Tech Lead
DXC Technology
- Developed Databricks notebooks over thousands of tables between IT JSON data sources, including ServiceNow, Jira, IBM Tivoli Composite Application Manager (ITCAM), Kiuwan, and SonarQube, into Azure Data Lake Storage Gen2.
- Established hundreds of processes in PowerCenter to extract data from ServiceNow (web services consumer transformation), Jira (Open Database Connectivity (ODBC)), and ITCAM (ODBC) to an Oracle database on-premise.
- Used the data stack with Microsoft Azure cloud services with Databricks, Data Factory, Azure Synapse, Azure Key Vault, and CI/CD with Azure DevOps.
- Identified new business data as the IT product grew by creating a new instance for corporate data with corporate data from SAP SuccessFactors tables on Azure Delta Lake.
- Managed a team of ten data engineers as a tech lead, emphasizing security compliance, code quality, and promoting and maintaining best practices. Handled welcoming and giving formation to new elements.
- Worked with Agile and Scrum using Jira and Xray. Used ServiceNow's workflow to create service and change requests, including incident management.
- Integrated and maintained minimum-value projects developed by external contractors on Azure Databricks and Data Factory.
- Analyzed requirements and designed technical solutions with source integration and target database models to answer and deliver business needs.
Data Engineer
KCSIT
- Developed incident resolution and corrective and evolutive application maintenance for energy and utility products with Informatica PowerCenter and SQL Developer, which also involved using ServiceNow to manage tasks.
- Extracted data in a voice-of-the-customer (VoC) project from different SAP sources and crossed them into a small Oracle database model with four tables to give an analytic overview of customer satisfaction.
- Monitored and enhanced performances on ETL processes with Informatica PowerCenter and SQL and PL/SQL with SQL Developer.
- Performed ETL developments as a developer in an ongoing marketing analytics project with PowerCenter over source files (CSV or XLS) and Oracle sources, resulting in a target database to overview marketing data.
ETL Developer
CGI
- Performed technical and functional analysis and design and implementation of ETL processes and database structures for data integration and analytics over a customized Oracle E-Business Suite environment that served over 20 water supply companies.
- Facilitated application management resolving incidents and service requests of different business areas such as contracts, invoicing, debt, billing, and reads.
- Conducted performance analysis of daily ETL processes, Oracle Database tables and indexes, and SQL tuning on reporting queries as key methods to provide detailed and aggregated information on time.
- Developed and maintained PL/SQL functions and procedures to framework and control process execution.
Experience
Data Analytics Over IT and Corporate Data
BI over Oracle EBS Customization for Water Utilities
Education
Engineer's Degree in Computer Engineering
Iscte - University Institute of Lisbon - Lisbon, Portugal
Certifications
Microsoft Certified: Azure Data Engineer Associate
Microsoft
Skills
Libraries/APIs
PySpark
Tools
Informatica PowerCenter, Jira, Excel 2016, Informatica ETL, Oracle Warehouse Builder (OWB), Microsoft Excel, Synapse, Microsoft Power BI
Languages
SQL, Transact-SQL (T-SQL), SQL DDL, SQL DML, Java, Python, XML
Paradigms
Scrum, Business Intelligence (BI), ETL, Database Design
Platforms
Databricks, Azure, Azure Synapse, Unix, Linux, Azure SQL Data Warehouse
Storage
Databases, Oracle SQL, SQL Performance, Azure SQL Databases, PL/SQL, PL/SQL Developer, PostgreSQL, JSON, Oracle SQL Developer, Microsoft SQL Server, Data Lakes, Data Pipelines, Azure SQL, Database Administration (DBA)
Other
Development, Programming, Azure Data Lake, Azure Databricks, Data Processing, Data Engineering, Azure Data Factory (ADF), ServiceNow, Analytics, CSV, Data Queries, Medallion Architecture, PL/SQL Tuning, PySQL, Performance Tuning, Data Warehousing, Data Management, Team Leadership, ETL Pipelines, Physics, Electrodynamics, APIs, Leadership, Unix Shell Scripting
How to Work with Toptal
Toptal matches you directly with global industry experts from our network in hours—not weeks or months.
Share your needs
Choose your talent
Start your risk-free talent trial
Top talent is in high demand.
Start hiring