Michael Sendker

A Flask-based web application for collaboratively constructing a definitive and exhaustive repository of annotations on world literature. It's basically designed to be the Wikipedia of annotations.

I designed, developed, and maintained (which I continue to the present day) the project solo with occasional help when I can rope in friends. It consists of about 14 KLOC, and I am currently redesigning it from the ground up using React as the front end and a better data structure on the back end.

This is the project that roped me into programming. I learned every aspect of the web application development lifecycle, including conception, design, development, deployment, and maintenance.

I wanted to build a crawler with Scrapy, use it to generate sentiment analyses of news articles, and see which news agencies were happy, sad, or neutral.

I wrote about 30 scrapers—some with Selenium, some with just Scrapy's request system—and used them to run them at varying intervals. I used SQLAlchemy to store the articles and agencies and then build a Flask site to display the information, with Materialize as a basic framework to make it look semi-spiffy. Then I added in a word cloud for each news agency for the day. The front page shows what spider is running at any given time.

The initial deployment was on a Raspberry Pi 4, but it became too much and I moved up to a bedroom server. It's up to 55,000 articles.

One of the most interesting things in this project was figuring out how to represent the sentiment as a color. I wanted neg/pos sentiments to be displayed as a gradient between red and green. Turns out, you can literally just represent any two points in hex color space as points on a graph and to find the point in the gradient you just take the distance. The code's at Github.com/mas-4/maudlin, if you want to see it.

I developed—and redeveloped over several successive weeks—a rather large web scraping and API scraping system that gleans information about Amazon Marketplace offers on products the client is selling from multiple data sources.

Due to increasing project complexity, it ended up becoming rather large with a decent amount of features:
• Multithreading capabilities
• A token tracking system to make sure to stay within the rate limit header spoofing and proxy rotation
• Near real-time updates for various settings including which ASINS to track and notification endpoints using cloud file storage
• A PostgreSQL database for tracking data (using SQLAlchemy as the ORM, of course).
• Database synchronization with an Elasticsearch instance for a Kibana dashboard

The software was deployed to an Arch Linux VPS and used a systemd unit to run. I've done automated data gathering (mostly API and web scraping) before but never on this level. I had a lot of fun on this project, building it from the ground up. I'm especially grateful to learn multithreading at a decent level.