J Chan

Much of the data directly crawled from the major Chinese eCommerce platforms have dirty business-relevant master attributes such as category or brand. Because these are the primary segments clients rely on to conduct effective business intelligence, these master attributes must be intelligently mined and standardized to allow effective downstream analyses and reap the benefits of proper Master Data Management.

We found that relying on regex patterns to tag crawled products to the nutritional health category was not accurate, so I collaborated with a data science colleague to deploy text classification in the form of MXNet's Transformer model to perform more intelligent product name tagging.

To further improve the accuracy of category mapping, I managed the further implementation of fast.ai and PyTorch's VGG-16 image classifier to utilize product images as well, as sometimes the product name sounded like a nutritional health item, but the image would indicate otherwise. Consolidating the two results as an enriched tagging workflow increased our average accuracy of category mapping to 93.5% and passed the client quality requirement to publish to the production environment and launch the final digital product.

Originally our ETL data pipelines of millions of rows of crawled eCommerce product and category-level data were written as Talend jobs, but given the instability of the weekly loads, I was tasked to transfer all of the ETL logic to another technology. After deliberate review, I decided Spark SQL was the best choice, given the large amount of data and the relative ease those pipelines could be run on AWS. Other teams were increasingly reliant on AWS' superior technology, which greatly reduced the learning curve in such an endeavor. Finally, I was also skilled in SQL and its flexible functionality, which made Spark SQL an even more logical choice.

After painstakingly reviewing the logic, documenting the logic, and monitoring topline statistics of the existing load, I slowly rewrote all nine workflows and made sure to compare my test load with the production load to make sure none of the columns were missing any crucial logic. After several months, the last workflow was written, and I rounded out the endeavor by packaging the scripts to s3 with a slew of shell scripts that allowed the entire ETL workflow to run on Amazon EMR. Not only did the new workflow run more stably, it also ran approximately 40% faster and was easier to QC.

Due to a rehaul of the existing infrastructure from optimization roadblocks, I was tasked with transferring Talend logic of mapping crawled eCommerce data to master brands to another technology. Because I had previous experience with Spark SQL and realized its great data mining capacity given its distributed nature, I started documenting the existing logic by poring through the Talend workflows with the intention of using Spark SQL for the final code.

The biggest challenge by far was utilizing the existing keyword look-up table used to do keyword mapping, as merging the look-up tables effectively as a cross-join with no way to partition efficiently greatly reduced the distributed computing potential of Spark SQL. After days of fervent research, I came across the concept of using another column to force certain partitions to travel together so that they were processed together, thereby allowing Spark's distributed power to take over as expected. From this experience, I gained not only more confidence in my Spark code, but also as a developer holistically.