- Senior Software EngineerMoz2011 - PRESENT
Technologies: Python, C++, Ruby, Java, Elasticsearch, HBase, qless, NSQ, gevent
- Rewrote a service recursively crawling customer sites and analyzing and reporting SEO issues.
- Wrote a queueing system (qless) that has been widely adopted internally for externally for production systems.
- Designed and implemented a service for crawling and indexing pages discovered through important RSS feeds.
- Helped to implement an algorithm to remove navigation, headers, and footers from web content for the purposes of indexing (eventually published).
- Wrote a number of web crawlers for different purposes, contributing many well-used open source projects along the way to the state of the art of web crawling.
- Crawled and processed tens of billions of pages across all my various crawlers.
- Worked to support our next generation of backlinks indexing infrastructure.
- Graduate ResearcherIBM Research2010 - 2010
Technologies: Python, WebSockets, C++
- Collaborated between KAUST's supercomputing department and IBM Research.
- Augmented a computational steering library to work with WebSockets.
- Included work with Lawrence Berkeley National Lab to eventually support streaming visualization.
- Targeted KAUST's supercomputing infrastructure, an IBM BlueGene/P.
- Worked to enable researchers to examine, monitor, and update parameters of running simulations.