Júlio César Batista

This was my undergraduate capstone project, an experiment to use Leap Motion and machine learning to classify signs of the Brazilian Sign Language. Within this project's scope, I directly worked with education professionals to validate the idea and identify potentially required refinements.

I contributed to Scrapy, an open-source project maintained by Zyte, with code changes and reviews of pull requests and issues.

This project required adding support to data export to the Google Cloud Storage (GCS), github.com/scrapy/scrapy/pull/3608. In this pull request, I followed the framework practices to easily export scraped data to the GCS, as it happens with the AWS S3. This allowed future developers to use the framework to GCS off the shelf, only requiring some configuration without any explicit line code.

It also involved moving the ItemLoader to a standalone library, github.com/scrapy/scrapy/pull/4516. I removed the ItemLoader code from Scrapy in a new library, allowing developers to use it without Scrapy.

I worked on a broad crawler project to make it customizable by the customer. Through a configuration file in Amazon S3, the customer should be able to add or remove seed URLs, decide the scraping depth (how many links to follow), and use JavaScript to render the page or go with raw HTML. This project was scraping thousands of seed URLs daily through a discovery and extraction strategy in which a set of crawlers were finding links in the seed URLs, and other processes were collecting the data out of these discovered links.