James Irie

I led an end-to-end churn analysis for a legal and professional services SaaS company where early customer churn directly affected recurring revenue. With no centralized analytics database and only a 2-month timeline, I built an interpretable churn model across product usage and tier signals, achieving roughly 80% ROC-AUC and 70% PR-AUC.

The strongest predictor was prolonged login inactivity, which identified a high-risk segment but did not fully explain why customers churned. To make the results actionable, I combined model findings with exit survey review and manual case analysis of inactive churned users across firm sizes and practice areas.

Two consistent drivers emerged: a lack of effective product training and unresolved support issues prior to churn. Based on these findings, Sales and Customer Success launched an outreach campaign to about 50 recently acquired but inactive users, achieving a response rate above 50% and validating the need for improved onboarding, training, and support.

I developed and deployed production models supporting student outcome prediction across 30+ partner institutions. I contributed to rebuilding the team's ML infrastructure, migrating from a Spark-based system to a Scikit-learn and Metaflow stack running on AWS Batch, which improved pipeline efficiency and reduced computational cost while enabling scalable batch training and scoring workflows.

I built and maintained the training and scoring workflows for production models, including Redshift feature and scoring tables under dedicated data science schemas, using a strict SQL and YAML-based framework with GitHub review and historical prediction logging for model traceability. I also played a key role in retraining 30+ institution-specific models during the infrastructure migration, managing retrain scheduling, collaborating with the data quality team on performance validation, and overseeing configuration updates to ensure a smooth rollout within a two-month timeline.

Beyond the technical work, I partnered closely with data engineers, software engineers, and product managers in an agile environment and worked directly with the DQA team to ensure they could operate the new retraining process independently.

I led a large language model (LLM) evaluation project to determine whether transcript-based AI outputs were reliable enough for business use. The project used hour-long customer onboarding call transcripts, in which I applied NLP and a Llama-based LLM to extract structured insights such as sentiment, evidence presence, and feature-interest signals. I created human-labeled ground truth, designed repeatable prompts, and ran the model 1,000 times per transcript to measure consistency, reliability, and failure patterns across different analytical tasks. The evaluation showed that some tasks were reliable enough for summarization and evidence extraction, while others were too inconsistent for automated reporting. My work helped define safe-use boundaries for LLM-assisted analytics and provided non-technical stakeholders with a clear framework for understanding when LLM outputs could support decision-making and when human review was still required.

I contributed to developing an automated AWS QuickSight model-monitoring dashboard that helped the data science team track production model health across partner institutions. My main responsibility was building the Jupyter notebook and Python analysis scripts that ran weekly anomaly detection, bias checks, feature value analysis, and model score performance analysis. The outputs were written to tables used by QuickSight, allowing the dashboard to refresh automatically and replace manual notebook-based reviews. I also partnered with teammates on dashboard design and worked with Data Engineering and DevOps partners to support the setup of Athena, Glue, and IAM. This work improved visibility into data quality issues, feature drift, score shifts, and segment-level model behavior, making it easier for the team to diagnose whether changes reflected true model degradation or upstream data/behavioral changes.