Offline Reinforcement Learning for ICU Insulin Dosing: A Reproducible Benchmark and Honest Off-Policy Evaluation

An offline reinforcement learning benchmark for insulin dosing in a general intensive care unit population, built from the MIMIC-III critical care database. Unlike prior work that models a single intravenous infusion rate on a narrow cohort, this study covers the three insulin delivery routes actually used at the bedside and treats scheduled basal insulin as patient context rather than as a per-decision action.

The central contribution is honesty of evaluation. The learned policy is compared against clinicians using four off-policy estimators with patient-level bootstrap confidence intervals, against both a glucose time-in-range proxy and real in-hospital mortality, and under a reward-sensitivity analysis. The finding is a well-powered negative result: under careful evaluation, offline RL matches but does not outperform clinician dosing, and the positive advantages reported in earlier work do not survive the same scrutiny.

Motivation

Glucose control in the ICU is a repeated decision under a hard safety limit: too little insulin leaves patients hyperglycemic, too much causes hypoglycemia, which can be fatal. Several studies claim reinforcement learning can dose insulin better than clinicians, but most model only an intravenous infusion rate and rely on optimistic off-policy value estimates. This project asks whether such claims hold up on a realistic, general-ICU problem under evaluation designed to resist the known failure modes.

Data and linkage

The study uses a curated glucose-insulin dataset derived from MIMIC-III v1.4 (MetaVision records, 2008–2012): 9,518 patients, 12,210 ICU stays, and roughly 604,000 glucose and insulin events, with insulin doses paired to a preceding glucose reading by clinician-defined timing rules. Using the shared patient, admission, and ICU-stay identifiers, the events were linked to the MIMIC-III ADMISSIONS, PATIENTS, and ICUSTAYS tables, with complete coverage, to recover in-hospital mortality, age, sex, and care unit.

Method

• Decision process: each ICU stay split into 4-hour bins; state summarizes recent glucose and trend, insulin on board, trailing 24-hour basal exposure, and demographics; action is the reactive short-acting correction (none, or subcutaneous bolus / intravenous bolus / intravenous infusion at a low, medium, or high dose).
• Basal as context: scheduled long- and intermediate-acting insulin is encoded in the state rather than the action, matching clinical practice and removing a failure mode in which the policy abandoned basal coverage.
• Policy: discrete Conservative Q-Learning with a double-DQN target and Polyak-averaged target network.
• Evaluation: four off-policy estimators (clinician value, fitted Q evaluation, weighted importance sampling, weighted doubly robust) with 200-sample patient bootstrap confidence intervals, run against a glucose proxy and real mortality, plus a reward-sensitivity analysis across four reward definitions.

Results (test split, 95% patient-bootstrap intervals)

Findings

The doubly-robust estimate sits at or below the clinician value on the proxy, and every mortality interval includes the clinician value, so the policy is statistically indistinguishable from clinicians on real outcomes. An apparent proxy-reward advantage under a thin state disappeared once age, sex, and care unit were added, and the direction of any proxy difference was not stable across reward definitions. No benefit over clinicians is claimed.

Contribution

A reproducible, general-ICU benchmark with a realistic multi-route action space, a basal-as-context design, and an honest evaluation protocol, together with a well-powered null result that serves as a corrective to overclaiming in clinical reinforcement learning.