Miguel Saraiva, Developer in Lisbon, Portugal
Miguel is available for hire
Hire Miguel

Miguel Saraiva

Verified Expert  in Engineering

Web Scraping Developer

Location
Lisbon, Portugal
Toptal Member Since
September 26, 2016

Miguel has twenty years and counting of programming and software architecture experience, from super fast multithreaded engines to ultra distributed crawling services, market research, online news gathering, and game development. He works with technologies including Assembly, (already forgotten) ALGOL 68, ZBuffers, the C family, and Azure.

Portfolio

Toptal Client
Web Scraping, Web Crawlers, Amazon EC2, Amazon DynamoDB, Python, Scrapy, Node.js
Vosdal Studios
Protocol Buffers, Parse, XNA, OpenStreetMap, Unity, Azure, C++, C#, Unity3D
Toptal Client
Transport Layer Security (TLS), Electron, Node.js

Experience

Availability

Part-time

Preferred Environment

Visual Studio, C++, C#

The most amazing...

...thing I've built is a system on Azure Cloud with dynamically distributed physics engine calculations over new nodes based on client entrance/exit.

Work Experience

Developer

2016 - PRESENT
Toptal Client
  • Developed a system from the ground up to crawl and scrape Australian sites, gathering information and making it available to customers, using a map front end. Some information is downloaded in real time from open APIs so as to ease the users' work.
  • Integrated the work of developers from around the world into our own system.
  • Developed APIs for external clients so they could access our collected data from their own services and applications. Those APIs are being served by Amazon EC2 machines.
Technologies: Web Scraping, Web Crawlers, Amazon EC2, Amazon DynamoDB, Python, Scrapy, Node.js

Chief Technology Officer

2010 - PRESENT
Vosdal Studios
  • Developed a distributed physics engine, using Azure and C# for the back end. The relevance here is the MMO needs, using distributed computing to cope with any number of simultaneous players, all in a single server farm.
  • Developed frameworks for ease of working with online data storage, Facebook's Parse.com, Azure Storage, and SQL.
  • Developed several algorithms to fuse vector and metadata information from OpenStreetMap with satellite altimetry maps to generate realistic terrain for in-game use.
  • Used the above developments, in house, on several desktop, Android, and iOS games using Unity3D mainly as the front end. The same services were used with XNA for WIndows Phone.
Technologies: Protocol Buffers, Parse, XNA, OpenStreetMap, Unity, Azure, C++, C#, Unity3D

Developer

2020 - 2020
Toptal Client
  • Developed a script for automating the buying process of Shopify and demandware-based stores.
  • Adjusted the script to to mimic human behavior and overcome CAPTCHA-based protections.
  • Adjusted TLS security handshakes to overcome bot protections.
Technologies: Transport Layer Security (TLS), Electron, Node.js

Back-end Developer

2019 - 2020
Toptal Clients
  • Developed bots to work with Shopify and other shop frameworks.
  • Worked with several APIs to deliver a transparent experience to the end user.
  • Dealt with several bot protection schemes, overcoming them.
Technologies: Electron, Node.js

Freelance Research and Development Consultant

2012 - 2014
GfK
  • Consulted on matters of audience research methodology.
  • Optimized database access and data generation processes with C#, T-SQL, and SQL database technologies.
  • Consulted for requirements gathering for new markets.
Technologies: SQL, C#

Chief Technology Officer

2005 - 2010
Markdata
  • Architected and developed an online information gathering framework (web scraping) for internal news clipping processes, using C#, Azure Compute and Storage, XPath parser, and SQL.
  • Architected and developed the multimedia data fusion engine framework and algorithm. Created it in C# for added modularity, expansion, and flexibility.
  • Architected a web service for audience and planning engine, for both client and internal use.
  • Accepted clients' requests and designed the necessary architecture, deciding on best tech stacks to use to help multi-disciplinary teams built to fit the expectation of the clients.
Technologies: Azure, XPath, HTML, SQL, C#, C++

Senior Software Developer

1996 - 2005
Markdata
  • Developed next-generation television audience analysis software using graphic UI, MFC, and OWL.
  • Built fast calculation engines for several clients and products. Used both internally and for some selected client's apps. Made integrations using extended stored procedures for SQL server.
Technologies: Microsoft Foundation Class Library (MFC), C++

Senior Software Developer

1993 - 1995
Marktest
  • Designed and developed DOS-based solutions for television audience reporting.
  • Designed and developed Windows-based solutions for television audience reporting.
  • Spearheaded the use of C++ in the development team.
Technologies: Windows, OWL, C++, C

RIVERZ

A solution for the architecture of the physics simulations of “Rivers Under Roar,” a single shard, single instance, persistent massively multiplayer online (MMO) game, using a commercial cloud computing service. Presented in the Game Development Conference Videojogos2014, on the IPCA campus.

This was the back-end service for a multi-platform game working on desktop (macOS and Windows) and mobile machines (Android and iOS)

Languages

C++, XPath, C, C#, JavaScript, T-SQL (Transact-SQL), BASIC, Pascal, CSS, HTML, OWL, SQL, Python

Frameworks

.NET, Windows Presentation Foundation (WPF), Unity3D, Unity, Electron, Express.js, Scrapy

Libraries/APIs

Sockets, Google Maps, Node.js, Google Maps SDK, Xamarin.Android, Amazon Product Advertising API, Bing Maps API, OpenStreetMap API, LINQ, XNA, Microsoft Foundation Class Library (MFC)

Tools

Microsoft Visual Studio, Fiddler, Blender, TFS, Subversion (SVN), CVS, Git, Visual Studio, Skype, Hyper-V, Bitbucket

Paradigms

Distributed Programming, Waterfall Development, Agile Software Development, Object-oriented Programming (OOP), Concurrent Programming, Object-oriented Design (OOD), REST

Platforms

Windows, Visual Studio Code (VS Code), Amazon Web Services (AWS), Azure, Parse, MacOS, Linux, Amazon EC2

Storage

JSON, Amazon DynamoDB, Amazon S3 (AWS S3), Microsoft SQL Server, MySQL, Azure SQL Databases

Other

Web Crawlers, Back-end, Architecture, Scalability, Multithreading, Game Development, Web Scraping, TCP/IP, Return Path Data, Xamarin.Forms, Shader Technologies, Unity Shaders, OpenStreetMap, Transport Layer Security (TLS), Protocol Buffers

1988 - 1993

Master's Degree in Computer Science

Faculdade de Ciências e Tecnologia da Universidade Nova de Lisboa - Lisbon, Portugal