Russell Stuart

I took over the maintenance of iBCS (Intel Binary Compatibility System) to allow my employer to run an SCO Unix application running under Linux. iBCS supports running many different Unix personalities under Linux. Initially, I packaged it for Red Hat (produced a .rpm) and later for Debian (a .deb package).

I ported the original 32-bit kernel modules to the 64 bit Linux kernel. Later, when the ever-changing kernel ABI made continued maintenance of the kernel modules difficult, I figured out how to intercept the foreign syscalls and emulate them purely in userspace. As I had no way of testing most of the personalities the kernel modules emulated, their code had to remain largely untouched, so iBCS-US had to reproduce the Linux 2.6 kernel environment they thought they were running in, in userspace. They required replicating many kernel data structures such as inodes, file descriptors, and functions such as kernel mode to user mode copying, the kernels memory management, and loading executables into RAM.

Native Linux user authentication uses an API/library called PAM (Pluggable Authentication Modules). PAM is middleware enabling programs such as login, XDM, and SSH to authenticate a user or session without knowing how the administrator wants that authentication and session management done. The administrator is free to select one or more of the canned authentication and session modules supplied with PAM such as passwords, FIDO2, Kerberos, or write their own.

Until Pam-python arrived, a custom PAM module had to be written in C, and that C module would then be loaded into the most security-sensitive programs in the system—the ones that authenticated and authorized the users. With Pam-python, these modules could be written in Python, thus eliminating whole classes of security problems endemic in C programs such as buffer overruns, dangling pointers, and since Python destroys all objects on exit—most resource leaks.

Pam-python works by dynamically loading the Python interpreter library into the program when a Python PAM module is referenced and mediates the interface between the Python program and the rest of the system, which is typically written in C. Pam-python is written in C itself and comes with 100% unit test coverage.

Python-lrparsing is an LR(1) parser generator, augmented so some tasks LR(1) makes difficult, like processing lists, associative operators, and prioritizing some productions, are easy to do.

To the best of my knowledge, it produces the faster parsers out of all the Python parser generators, beating most of the competition by an order of magnitude. It hijacks the Python expression syntax so that LR productions can be written in native Python, with syntax checked by the Python interpreter, rather than demanding its input be supplied by strings. It includes error recovery, a tokenizer that can be replaced, and very atypically for a parser generator user-friendly error messages. It comes with extensive documentation, I suspect the most extensive of any parser generator out there. Examples provided include a Lua 5.2 interpreter and a parser for all SQLite3 syntax. It has 100% unit test coverage.

Solidify turns dynamic websites into static ones. The process is sometimes called "flat filing" and is often done with a tool like Wget. Solidify was written to flat-file a poorly maintained WordPress site that was becoming a security risk.

Unlike simple tools like Wget, Solidify converts all URLs within a site to be relative, and it also processes URLs discovered in CSS and JavaScript. It can also edit the site as it is downloaded to remove parts that make it difficult to handle, for example, the "compare revisions" feature on Wikis.
Allowing you to compare any two revisions of a single page with 100 revisions, the number of ways of viewing that page explodes from 100 to 9900. It can also handle local copies of assets the original site loaded from foreign servers, so random internet outages are less likely to cause the result to break.

To speed the process up, Solidify typically downloads six pages concurrently in background threads and processes the results in another thread when gives it another advantage over Wget and similar tools: it's much faster.

LaCrosse WS-2300 was a weather station with a proprietary windows driver to collect the data. I wanted my home Debian web server to have a page displaying the local weather, both current and historical, so with the aid of other open-source developers, I reverse-engineered the RS232 protocol and wrote a Python driver for it.

The driver could be run interactively to see current readings and program the device. It could also be run as a background daemon, continuously collecting data and summarising it into snapshots of what happened over configurable time intervals like five minutes, an hour, or whatever, writing the summary to disk.

The web front end was a GUI for that summaries data, presenting it in various ways. It handled unexpected interruptions such as power failures by saving partially collected data and resuming from where it left off when next started.

The driver can be imported as a standard Python module, so the protocol deciphering and data collection could be used without the rest of it. I ceased work on the driver when the LaCrosse was retired, but its modular nature has meant other open-source projects such as WeeWX could use it as their driver, so it lives on.

I wrote scraper-helper in Python 2 and 3 when I discovered that most websites don't supply APIs for the data they house.

Scraper-helper is a web proxy that records every aspect of interactions of a website, including every detail in the HTTP requests and responses, the details of SSL certificates, and the timing. It writes every asset it sees to a separate file. It also generates X.509 certificates needed to perform the necessary man-in-the-middle (MITM) attack on SSL sites to make this possible.

A novel feature of scraper-helper is it can playback to the website the browser interactions recorded earlier. In other words, it can send the post's and other interactions and record the site's responses just as if the browser sent them, including the timing between requests. The developer can edit the recorded script, reducing it to a minimal set of steps and eliminating timing pauses if they aren't relevant. The recorded script can then be translated to code to produce the web scraping application.

DotGnu is an open-source C# implementation, now defunct.

I used it as an open-source version of C# under Linux to interface to a new device that was being developed using Windows CE's C# implementation.

Using C# meant the protocol implementation could be shared, saving time for both sides. The vendor's protocol driver used multiple threads and timers and unfortunately had to be rewritten and the rewrite was given back to the vendor so their side would work too. Sadly, it wasn't just the driver implementation that was buggy, DotGnu's mutex's and locking deadlocked or didn't lock at all on rare occasions.

My contribution to DotGnuwas was rewriting the CLI's threading, mutex, and timer implementations which was done in C.

Rdiff-image is a backup solution aimed at open-source VM's. It creates full and differential backups to Amazon S3 and local storage. It splits the backup into two sections—secret and public. Used together, a restore from the secret and public backup completely restores the VM. The public portion contains the bulk of the VM image—everything from the bar, the keys, passwords, and private information.

The public version can be restored without the secret part. You get a working VM with all passwords and keys set to a well-known value like "x" when you do that. The objective is to let open-source developers download the VM and maintain it collaboratively.

Rdiff-image handles when to automatically create full and differential backups and prunes older backups as they become less relevant. It does it in a way that optimizes both safety, like the differential is always off a full, never another differential, and space. A backup of a 500GB VM is currently handling email, blogs, and a website for a club, with hourly backups during a week, weekly backups for a few months, and monthly backups for a couple of years; going back seven years costs around $0.20 per month.