Maksym Karashchuk

Designed and assisted in implementing a cloud-based application in Azure for data unification purposes in the role of a data architect. I gathered all client requirements regarding network, security, role, responsibilities, backup, and recovery strategies. All these details were effectively documented and visualized with unified modeling language (UML) tools, enabling more than 12 data engineers involved in the project to know what exactly was expected to be built in each phase. In the end, the solution unified data from around 13 sources with different structures in one Azure Synapse data warehouse was ready to be used for further analysis.

Built an advanced Python application for data flow monitoring between over ten distributed identity management systems from different regions. The main difficulty was dealing with various source formats and staying compliant with country or regional laws for data processing, such as the General Data Protection Regulation (GDPR) and the California Consumer Privacy Act of 2018 (CCPA). Furthermore, the whole distributed environment was successfully visualized for the first time during this project, which brought significant value to the client.

Built a Java application that automatically translates information about support shifts provided in an Excel file on SharePoint to Outlook calendar events. This application has resolved many problems with shift attendance by supporting team members and improving their project performance. The solution is flexible enough to recognize new people in a team and add them to the list of support team members for further usage.

Implemented a Python application for real-time road sign recognition. The solution can recognize a type of sign with a maximum speed of 24 frames per second. The approach transforms an incoming image with a signaling gateway (SGW) method and then applies a maximally stable extremal regions (MSER) algorithm to identify interest areas. Each selected location is sent to the trained model to get the road sign's name and probability.

Implemented a metadata-driven solution for the dynamic transformation of hundreds of different file sources for daily, monthly, and yearly reports. The solution increased the speed of data analysis from days to hours in the client's organization. The challenge was that each file had a completely different format and required a lot of cleansing before uploading to the core of the data warehouse (DWH). Furthermore, the solution had to be bulletproof to ensure it couldn't fail at any moment in case of minor changes in the file—since files were received directly from business users.

A solution that had to prove the concept of real-time data transfer from the source Oracle DB to Synapse DWH for further advanced data analysis. I participated as the primary data engineer who built the main data flow across the platform up to the Power BI reporting. The real-time data replication from Oracle DB was achieved using Oracle GoldenGate, which was connected to the Event Hub and was constantly propagating changes on the source DB. The idea was to take the data from Azure Event Hub, join it with some reference data in Databricks, and put it to Synapse using streaming datasets. Along the way, the performance had to be measured and optimized to achieve the best possible cost-to-value ratio.