Data Engineering Developer in Berlin, Germany

I redesigned the ETL as well as the database system to improve the performance of loading the data. It went from loading in more than eight hours to two hours. I designed the SQL server architecture to be a highly scalable system using various approaches including partitioning and partitioned views.

Other Roles and Responsibilities:
• Designed, modeled, and architected new database system using Sybase ASE, MS SQL server, and Oracle for scalability and performance.
• Optimized and performance-tuned existing and new procedures using SQL DMVs Sybase Monitoring Tables and Oracle performance views to reduce queries running in hours to minutes.
• Used SQL server trace, profiler, and extended events to troubleshoot the performance root causes (using analysis and fixes).
• Used Sybase ASE’s XML show plans, trace flags, abstract query plans and statistics for performance root cause analysis.
• Designed and developed stored procedures, functions, triggers, views, and indexes in Sybase as well as the SQL server.

I worked on the McAfee Mobile Security application which used by various mobile devices for security and other activities. The data was being stored in a SQL Server 2008/2012. I worked on scaling the system from 4,000 transactions per minute to around 70,000 transactions per minute by removing various performance issues. I redesigned the HA and DR as well as the reporting systems using clustering, and database mirroring as well as replication.

Other Roles and Responsibilities:
• Used SQL Server trace, profiler, and extended events to troubleshoot the performance root cause (using analysis and fixes).
• Designed and developed tables, stored procedures, indexes for new development and enhancement.
• Worked on test-driven development and development using Agile.
• Implemented data modeling for changes and new development.
• Monitored the production server performance using DMVs and Perfmon; depending on requirements for tuning the system, also the application and existing queries/objects.
• Designed, tested, and tuned extensively on big data and NoSQL technologies like Cassandra and Hadoop, Hive, and Pig stack to test different scenarios using Python to migrate the existing application onto a big data platform.

I worked as a Sybase database designer as well as a developer for J.P. Morgan's PB credit application. I mainly worked with business analysts and gathered the requirements for new changes and reports. With this material, I then redesigned the existing database system as well as developed using SQL, TSQL, stored procedures views, and more.

Other Roles and Responsibilities:
• Designed and developed stored procedures, functions, triggers, views, and indexes in Sybase.
• Used Sybase ASE’s XML show plans, trace flags, abstract query plans, and statistics for a performance root cause analysis.
• Optimized and query performance-tuned existing and new procedures using monitoring tables to reduce queries running time by up to 2 to 30 times.
• Developed Unix shell, Python, and Perl scripts for ETL and data analytics.
• Implemented data modeling for changes and new development.
• Wrote SQL and T-SQL code using Sybase.

I worked as the database designer and developer and administrator to the Credit Suisse Swap database system—I redesigned the database for new reports and migrated some of the reports from a SQL server to Sybase IQ. I also migrated the SQL server from 2000 to 2008. The main highlight for me was redesigning a C# FIFO trade matching in a SQL server for ad-hoc rematching.

Other Roles and Responsibilities:
• Created new stored procedure, functions, triggers, and views in a SQL Server.
• Optimized and query performance-tuned existing and new procedures using DMVs.
• ETL development using SSIS 2008 and report development using SSRS 2008.
• Using SQL server trace, profiler, and extended events to troubleshoot the performance root cause analysis and fixes.
• Developed Unix Shell, Python, and Perl scripts.
• Conducted an impact analysis for the migration of SQL server 2000 to the 2008 version.
• Defined the capacity planning and designed the migration to SQL server 2008 for performance improvement.
• Changed DTS packages to SSIS packages as well changed SQL and T-SQL code to be compatible with SQL Server 2008.

I have designed data pipelines for risk and PNL teams for an investment bank using technologies such as SQL, Sybase, Python, HDFS, Spark, and Kafka. The pipeline uses Kafka to move data from Sybase IQ to HDFS. Then we run a computing-intensive job against the HDFS and again move the output to Kafka and then it is sent to our reporting server in Sybase IQ and HDFS. Real-time data is then directly fed from Kafka to Spark and then processed and sent to reporting servers.

I designed a data warehouse and data lake for transparency using AWS services like ETL, S3, Lambda, EMR, Redshift, SQL, Python, and Spark. The data is used to land in S3 buckets in CSV files and then a lambda used to launch an EMR Spark job to clean up and process data in parallel.

I cleaned up and enriched the data that was to be moved to Redshift and where we will run our reporting queries. I have used PySpark and Python.