Charles Grady

Various summaries and explorations of large-scale biodiversity analyses. I collaborated with multiple research biologists to perform, analyze, and summarize large biodiversity analyses to answer questions that previously could not be computed.

I collected and pre-processed raw input data, performed hundreds of thousands of single-species workflows, and generated large-scale multi-species data structures—including hundreds of thousands of species by hundreds of thousands of geographic sites—and some additional generated data structures that were then used to generate global analyses. This required significant data engineering, involving 5+ billion input records, and new data structures and methods for computing global analyses.

The portions of these projects that were possible in the past would have taken several months to years of computational time, but I had reduced the time required to several days. Other portions of the analyses were only possible at smaller scales—hundreds of sites by hundreds of species at most—and I was able to perform them 10,000 times for each analysis in matters of hours at up to 10 to the 12th power in data sizes. I then co-authored manuscripts with the researchers describing the methods.

A suite of tools for analyzing biological collection holdings against all publicly available collection data. I created a package of Python-based tools for analyzing specimen records and where they fit among all published specimen records for each species. This project requires a large amount of data ingestion and data engineering to compare records that are served in various formats and may need updating; for example, taxonomic names can change over time.

The total input data set is approximately 5 billion records from 6 data sources, but it can be expanded as data becomes available. I wrote tools to process these inputs and standardize them so they can be grouped and assessed one species or collection at a time.

Each specimen record undergoes many single- and multi-variate analyses across various dimensions to determine how likely each is to be an outlier, representative, unique, a duplicate, among others. Data is summarized at various taxonomic and phylogenetic levels to assess larger patterns and the collection density and distribution. The end goal is to create actionable items for a collection to improve for funding reviews, public presence, and research value.

Lmpy is a library of open source tools used for biodiversity analyses. These tools originated as part of the Lifemapper back end for various high-performance and high-throughput computations and were thought to be potentially valuable for collaborating biologists. The data structures and methods can be used locally, and any extensions developed using these objects and tools can be directly integrated into the Lifemapper computational back end without additional modification.

The motivation behind this project came from the need to modify and reimplement various scripts and software written by biologists that could not scale to their needs. My involvement in this project started from those re-implementations, as I developed the library so that the biologists we collaborated with could use code, tools, and scripts designed with computational performance in mind. I set up the repository, including CI/CD for testing, PyPI packaging, Docker builds, and documentation builds, created new tools, and reimplemented others, including my binary matrix randomization algorithm. I contributed to nearly all of the code and documentation until version 3.1.21.

A parallelized implementation of Dijkstra's algorithm was designed to take advantage of parallelized resources to compute coastal inundation height, flood modeling, or other shortest path computations. The project was written in Python using WorkQueue to compute worker management.

The project splits raster files into manageable-sized chunks to be worked on in parallel. These grid chunks can be of heterogeneous sizes and allow data sizes and scales that would not be feasible without specialized hardware. The critical points of the method are that individual chunks, or tiles, can be operated on in parallel and isolation and that the computations can be repeated as necessary when new information is determined. This results in an algorithm that is not necessarily as work efficient as standard Dijkstra's algorithm but runs in significantly less real time given the ability to utilize parallel resources, including massively parallel resources and supercomputers while requiring a fraction of the memory resources needed. In addition, heterogeneous data scales can be used for regions with higher resolution data.

I designed and implemented this project for my master's thesis and received honors for it.

A Python client for accessing the Lifemapper project's various web services, such as OGC, RESTful, and authentication. I created a Python client library, including most of the web services it interacted with, to access the web services our project provided.

This client was aware, as dynamically as possible, of all of the services and data models that were currently exposed and was able to construct workflow requests for the computational back end. The client library was then distributed to our collaborating researchers, the general public, and additional clients, such as a QGIS plugin, that used this library as a liaison for interacting with the Lifemapper web services.

A plugin for the VisTrails workflow software. I created this plugin that reads an ecological metadata language file, pulls the input data locations (web or local), and reads the prescriptive process steps to generate the same workflow used to create the original data.

This project allows researchers to include their input data and processes in a standardized format with their publications. Consumers of those publications can then use these data or metadata packages to recreate the researcher's findings and tweak those workflows as desired for different datasets or to perform additional analyses.

I presented this work at the 2011 Environmental Information Management Conference, and a paper covering this work was published in the proceedings.

A web interface add-on for the Java-based Specify 5, a biological collection management software. I wrote the web interface for exposing an institution's biological collections.

The software was written using Java Server Pages and could be installed with Microsoft IIS or Apache. My portion of this project was to expose data from a full-text index of the collection database. This software was distributed and enabled by hundreds of institutions to expose their collections for public consumption. It was also highly configurable and provided interfaces for collection managers to see what data was accessed.

This was an early-career project, and—while I believe there are a couple of installations that are still live—there are many software packages available now that have made this project obsolete.