Miguel Saraiva
Verified Expert in Engineering
Web Scraping Developer
Miguel has twenty years and counting of programming and software architecture experience, from super fast multithreaded engines to ultra distributed crawling services, market research, online news gathering, and game development. He works with technologies including Assembly, (already forgotten) ALGOL 68, ZBuffers, the C family, and Azure.
Portfolio
Experience
Availability
Preferred Environment
Visual Studio, C++, C#
The most amazing...
...thing I've built is a system on Azure Cloud with dynamically distributed physics engine calculations over new nodes based on client entrance/exit.
Work Experience
Developer
Toptal Client
- Developed a system from the ground up to crawl and scrape Australian sites, gathering information and making it available to customers, using a map front end. Some information is downloaded in real time from open APIs so as to ease the users' work.
- Integrated the work of developers from around the world into our own system.
- Developed APIs for external clients so they could access our collected data from their own services and applications. Those APIs are being served by Amazon EC2 machines.
Chief Technology Officer
Vosdal Studios
- Developed a distributed physics engine, using Azure and C# for the back end. The relevance here is the MMO needs, using distributed computing to cope with any number of simultaneous players, all in a single server farm.
- Developed frameworks for ease of working with online data storage, Facebook's Parse.com, Azure Storage, and SQL.
- Developed several algorithms to fuse vector and metadata information from OpenStreetMap with satellite altimetry maps to generate realistic terrain for in-game use.
- Used the above developments, in house, on several desktop, Android, and iOS games using Unity3D mainly as the front end. The same services were used with XNA for WIndows Phone.
Developer
Toptal Client
- Developed a script for automating the buying process of Shopify and demandware-based stores.
- Adjusted the script to to mimic human behavior and overcome CAPTCHA-based protections.
- Adjusted TLS security handshakes to overcome bot protections.
Back-end Developer
Toptal Clients
- Developed bots to work with Shopify and other shop frameworks.
- Worked with several APIs to deliver a transparent experience to the end user.
- Dealt with several bot protection schemes, overcoming them.
Freelance Research and Development Consultant
GfK
- Consulted on matters of audience research methodology.
- Optimized database access and data generation processes with C#, T-SQL, and SQL database technologies.
- Consulted for requirements gathering for new markets.
Chief Technology Officer
Markdata
- Architected and developed an online information gathering framework (web scraping) for internal news clipping processes, using C#, Azure Compute and Storage, XPath parser, and SQL.
- Architected and developed the multimedia data fusion engine framework and algorithm. Created it in C# for added modularity, expansion, and flexibility.
- Architected a web service for audience and planning engine, for both client and internal use.
- Accepted clients' requests and designed the necessary architecture, deciding on best tech stacks to use to help multi-disciplinary teams built to fit the expectation of the clients.
Senior Software Developer
Markdata
- Developed next-generation television audience analysis software using graphic UI, MFC, and OWL.
- Built fast calculation engines for several clients and products. Used both internally and for some selected client's apps. Made integrations using extended stored procedures for SQL server.
Senior Software Developer
Marktest
- Designed and developed DOS-based solutions for television audience reporting.
- Designed and developed Windows-based solutions for television audience reporting.
- Spearheaded the use of C++ in the development team.
Experience
RIVERZ
This was the back-end service for a multi-platform game working on desktop (macOS and Windows) and mobile machines (Android and iOS)
Skills
Languages
C++, XPath, C, C#, JavaScript, T-SQL (Transact-SQL), BASIC, Pascal, CSS, HTML, OWL, SQL, Python
Frameworks
.NET, Windows Presentation Foundation (WPF), Unity3D, Unity, Electron, Express.js, Scrapy
Libraries/APIs
Sockets, Google Maps, Node.js, Google Maps SDK, Xamarin.Android, Amazon Product Advertising API, Bing Maps API, OpenStreetMap API, LINQ, XNA, Microsoft Foundation Class Library (MFC)
Tools
Microsoft Visual Studio, Fiddler, Blender, TFS, Subversion (SVN), CVS, Git, Visual Studio, Skype, Hyper-V, Bitbucket
Paradigms
Distributed Programming, Waterfall Development, Agile Software Development, Object-oriented Programming (OOP), Concurrent Programming, Object-oriented Design (OOD), REST
Platforms
Windows, Visual Studio Code (VS Code), Amazon Web Services (AWS), Azure, Parse, MacOS, Linux, Amazon EC2
Storage
JSON, Amazon DynamoDB, Amazon S3 (AWS S3), Microsoft SQL Server, MySQL, Azure SQL Databases
Other
Web Crawlers, Back-end, Architecture, Scalability, Multithreading, Game Development, Web Scraping, TCP/IP, Return Path Data, Xamarin.Forms, Shader Technologies, Unity Shaders, OpenStreetMap, Transport Layer Security (TLS), Protocol Buffers
Education
Master's Degree in Computer Science
Faculdade de Ciências e Tecnologia da Universidade Nova de Lisboa - Lisbon, Portugal