Najam Ahmed Ansari

Shared Shelf Commons is an open-access library initiative by Artstor and their subscribing organizations. Artstor wanted their media to be openly accessible through IIIF. I engineered a solution that uses the SS Commons API to extract their collections and their metadata and then generates IIIF manifests that can then be viewed in Mirador or any other IIIF viewing environment.

This is the project that I am most fond of: a multimedia-ingestion-and-processing system that was built using Python. The API layer was written in Flask and we used Celery workers for the heavy lifting and background tasks. We used Nginx in front of the Flask servers for load balancing, handling the media uploads, computing the checksums, and serving the static media. This solution replaced a legacy system written in Erlang which had quite a few shortcomings.

The new solution was designed to be massively scalable. The web and background processes could be run on separate servers and all communications between the two were handled via RabbitMQ. The process was optimized so that Nginx would handle the file uploads inside a TMPFS mount, then run checksum calculations, and forward the request to Flask. Flask would then run some initial tests to ensure that the files weren't corrupted, create relevant database entries for the files, move them to our NAS drive, and create a background job for processing the files. Nginx was also used for serving static and external media, thus freeing up the Flask workers. We also used the ELK stack (Elasticsearch, Logstash, and Kibana) for generating insights from our logs.

As the founding engineer, I wrote all of our back-end code and created our cloud infrastructure. Since our initial MVP, I have been involved in scaling up the product and the team. We are now a post-Series A company with more than ten software engineers on the team.