Scroll To View More
Dan Lecocq

Dan Lecocq

Seattle, WA, United States
Member since May 6, 2014
Dan is an engineer and cowboy coder with a background in big data and distributed systems. He has extensive experience with profiling, optimization, asynchronous network I/O, and getting huge amounts of work pushed through a pipeline reliably and efficiently.
Dan is now available for hire
  • Moz
    Python, C++, Ruby, Java, Elasticsearch, HBase, qless, NSQ, gevent
  • IBM Research
    Python, WebSockets, C++
  • Distributed Programming, 7 years
  • Python, 5 years
  • Concurrent Programming, 4 years
  • C++, 3 years
  • Test-driven Development (TDD), 2 years
  • HBase, 1 year
Seattle, WA, United States
Preferred Environment
Linux, Git, Python, C++, Ruby, JavaScript
The most amazing...
...thing I've coded is a system to crawl and index hundreds of millions of tweeted URLs within 10 minutes of being tweeted.
  • Senior Software Engineer
    2011 - PRESENT
    • Rewrote a service recursively crawling customer sites and analyzing and reporting SEO issues.
    • Wrote a queueing system (qless) that has been widely adopted internally for externally for production systems.
    • Designed and implemented a service for crawling and indexing pages discovered through important RSS feeds.
    • Helped to implement an algorithm to remove navigation, headers, and footers from web content for the purposes of indexing (eventually published).
    • Wrote a number of web crawlers for different purposes, contributing many well-used open source projects along the way to the state of the art of web crawling.
    • Crawled and processed tens of billions of pages across all my various crawlers.
    • Worked to support our next generation of backlinks indexing infrastructure.
    Technologies: Python, C++, Ruby, Java, Elasticsearch, HBase, qless, NSQ, gevent
  • Graduate Researcher
    2010 - 2010
    IBM Research
    • Collaborated between KAUST's supercomputing department and IBM Research.
    • Augmented a computational steering library to work with WebSockets.
    • Included work with Lawrence Berkeley National Lab to eventually support streaming visualization.
    • Targeted KAUST's supercomputing infrastructure, an IBM BlueGene/P.
    • Worked to enable researchers to examine, monitor, and update parameters of running simulations.
    Technologies: Python, WebSockets, C++
  • Shovel (Development)

    Simple command-line dispatch of Python functions. Users find themselves regularly wanting to invoke small, simple Python functions from the command line, so I wrote what has become one of Moz's most popular repos.

  • qless (Development)

    A rich queueing system for Redis, used for production services both at Moz and elsewhere. It utilizes Redis's Lua script support to implement complex atomic operations for queueing. It consists of a Lua core ( and Ruby ( and Python ( bindings.

  • simhash-py (Development)

    Fast simhash in Python. It supports maintaining and finding near-duplicates in a set of documents with extreme speed. It consists of our underlying library simhash-cpp ( and the surrounding Python bindings.

  • pyreBloom (Development)

    Extremely fast bloom filter manipulations in a Redis instance. While Redis itself does all the persistence, this library implements a highly-efficient Python C extension.

  • dragnet (Development)

    Web page content extraction. This is the implementation supporting some published work ( where we separate the main content of web page articles and blog posts from the other components (navigation, headers, footers, etc.).

  • Languages
    Python, C++, JavaScript, Ruby, Lua
  • Paradigms
    Concurrent Programming, Distributed Programming, Test-driven Development (TDD)
  • Platforms
  • Storage
    Redis, AWS S3, Elasticsearch, HBase, MySQL
  • Other
    Open Source
  • Libraries/APIs
  • Master's degree in Applied Mathematics and Computational Science
    2009 - 2010
    King Abdullah University of Science and Technology - Thuwal, Saudi Arabia
  • Bachelor's degree in Computer Science
    2004 - 2009
    Colorado School of Mines - Golden, CO
I really like this profile
Share it with others