Steven Rouk

A dataset that used network device configuration files to determine the correct services that should be available to a customer. The project involved combining data from five different systems—several of which had significant data quality issues—into a single consumable dataset with 99%+ data quality accuracy.

As the sole developer on the project, I worked closely with end users to design the desired data schema, come up with data logic to meet the business needs, and validate the accuracy of the dataset. As the requirements continually evolved, I brought together disparate teams at each step to reach a common understanding of the required data and data logic. In the end, we increased the data accuracy by 25%+ in a customer-facing production environment.

A process I created to automatically determine the characteristics of a new dataset in a Snowflake database, including nulls, distinct values, correlation with other columns, trends over time, data types, and intersections with other datasets. This allowed data engineers to do in minutes what used to take days or weeks to accomplish. After the initial data profiling, data engineers could conduct additional ad hoc analysis as needed using Python, SQL, and Jupyter Notebooks. They used this analysis to answer questions that arose from the automated data profiling process.

An NLP data science project using arXiv.org research paper metadata to study the evolution of machine learning over the last 20 years. I pulled metadata for 1.6 million research papers, filtered the dataset down to 50,000 machine learning-focused research papers, and then used topic modeling to study shifts in the topics discussed over time. I used the Python scikit-learn package for topic modeling, Pandas for storing and manipulating data, Seaborn and Matplotlib for plotting, and custom text visualizations to demonstrate the topic modeling algorithm.

I worked with the Stanford Social Network: Reddit Hyperlink Network dataset made available through the Stanford Network Analysis Platform (SNAP). This dataset catalogs hyperlinks between subreddits over 2.5 years, from January 2014 through April 2017.

Key Activities
• Created a graph of the data using the NetworkX Python graph library.
• Experimented with creating my own graph data objects to load and traverse the graph.
• Analyzed connections (in-degree and out-degree), distinct networks (component analysis), sharing reciprocity, centrality, and PageRank.
• Experimented with ways to visualize large graph structures by randomly sampling neighbor nodes.