Karthik Narayan

With a fellow graduate students and two undergraduate students, I built a 3D scanner using commodity hardware.

A fellow graduate student and I collaborated on constructing the physical 3D scanner. Together, we decided on employing a turntable, an arm, and 5 entry-level DSLR cameras and 5 Carmine PrimeSense devices (depth sensors). I developed and wrote a bundle adjustment-based algorithm to calibrate these sensors with respect to each other. Working with another graduate student, we automated the full scanning process—a single click is all it takes to collect all the data associated with a single object: 600 12 MP images, 600 registered RGB-D point clouds, and intrinsic and extrinsic camera poses for each image and point cloud.

I wrote the bundle adjustment algorithm using Google Ceres, a large-scale non-linear optimization library; this library is also used for calibration at Google StreetView. We wrote an end-to-end pipeline in Python that allowed reproducibility of scans and calibration results.

Using this 3D scanner, we published a dataset, BigBIRD, consisting of 125 objects.

We published and presented this work at ICRA 2014.

Using the data collected from the BigBIRD dataset, I created a novel 3D reconstruction algorithm capable of scanning objects to within 2 mm of accuracy. The reconstruction algorithm is able to scan objects that are traditionally difficult or impossible to scan by existing scanners; samples objects could be transparent, translucent, or have heavy specularities.

To the best of our knowledge, this is the first paper that fuses visual hull information gathered from RGB cameras with KinectFusion cues from depth sensors; this paper demonstrates that combining these two data modalities can produce 3D scans that outperform scans using only one of these approaches.

At a high level, this approach (1) computes object segmentations from the high-resolution RGB images, (2) computes a visual hull using the segmented, calibrated RGB images, (3) computes a KinectFusion model using the calibrated depth data (Section III-C), (4) refines the original raw depth maps using the visual hull and KinectFusion models and merges these refined depth maps into a point cloud, and (5) forms an object mesh by fusing this merged point cloud with the visual hull.

This work was published at ICRA 2015.

Using the BigBIRD dataset and the meshes computed in our ICRA 2014 paper, I constructed a novel algorithm capable of producing photorealistic mesh colorings. Other existing approaches simply compute mesh colorings via averaging: to compute the color of a mesh vertex, simply average the color of that vertex across all images that display that vertex.

Unfortunately, this method doesn't work well because: (1) cameras may not be perfectly calibrated, which may produce heavy artifacts and (2) highly non-Lambertian objects (e.g., soda cans) may produce ghosting effects due to not having the same color observation from all views.

In this paper, I detailed a non-linear least squares problem to solve these issues. Demonstrating that solving this problem naively using Gauss-Newton traditional techniques, I proposed a hybrid coordinate descent and Levenberg-Marquardt algorithm to solve this problem. Ultimately, solving this problem yielded 3D color meshes of very high quality.

I conducted a user study demonstrating that our method outperformed the state of the art substantially; our technique was ultimately able to recover extremely fine details, e.g., lines on barcodes.

At Hudson River Trading, I constructed a number of deep learning-based strategies that currently trade more than 5% of the US equities markets. These strategies currently contribute towards a large portion of HRT's P&L. This project was written in Python and heavily optimized C++.

Please contact me directly for questions regarding this project.

At an internship at Citadel LLC, I created and deployed the first deep learning-based alpha models at Citadel's Global Quantitative Strategies arm. These strategies produce $xx million/year in live P&L. The models were written in Python and heavily optimized C++.

Please contact me directly for questions.