Robust Reinforcement Learning for Dynamic Treatment Regimes under Model and Measurement Uncertainty

arXiv:2610.22336v1 | 2026-10-18

Problem

In dynamic treatment regimes (DTRs), such as adaptive cancer therapy, clinicians need to adjust treatments over time based on patient responses. However, measurement errors in biomarkers and uncertainties in disease dynamics can undermine the reliability of learned policies, leading to suboptimal or unsafe treatment decisions.

Method

The authors model the patient as a hidden context and formulate the problem as a robust contextual Markov decision process (MDP). This approach jointly addresses generalization across patients and robustness to measurement errors and shifts in disease dynamics. The framework is applied to adaptive therapy for metastatic prostate cancer in a simulator calibrated to clinical data.

Interactive explanation: hidden patient context, noisy PSA, and treatment decisions

This figure is an explanatory toy walk-through based on the paper's setup, not a reproduction of the paper's experiments. It uses deterministic synthetic values to show how a history-dependent policy can react to a noisy biomarker while the underlying patient state remains hidden.

Toy adaptive treatment timeline with hidden burden, noisy PSA, and on/off treatment decisions A deterministic six-month example compares hidden tumor burden, measured PSA, and treatment decisions. Buttons advance or play the months one by one. Toy monthly decision loop Synthetic illustration; not reported patient data Hidden patient state Underlying burden The paper models the patient as hidden context: the policy never sees this directly. Observed biomarker Measured PSA (toy) pretreatment level Noise in the biomarker can shift what the policy thinks is happening. Policy action Abiraterone this month The learned controller can use the whole PSA/action history, not just one threshold. Month 1 of 6 Month 1: treatment starts on, PSA is high, and the hidden burden is still large.

Paused ยท month 1 of 6.

Fallback summary: In the paper, the hidden patient state determines how cancer burden evolves. Clinicians only observe PSA, which may be noisy. The robust RL policy is trained to choose monthly treatment actions that still work when PSA is perturbed and when disease dynamics shift.

  1. Month 1-2: treatment is on while PSA is still near pretreatment level.
  2. Month 3-4: the measured PSA drops, so treatment may pause, but hidden burden can differ from the measurement.
  3. Month 5-6: PSA rises again, which may inform the next treatment decision in this toy example.

Findings

Results show that the learned policies are generalizable and robust, substantially outperforming the existing adaptive therapy method. In particular, the policy attains almost no loss of performance when the biomarker is accurate and the disease evolves as anticipated. Across a diverse array of scenarios, the resulting policy Pareto-dominates the existing method, meaning it performs at least as well in all scenarios and better in some.

Limitations

The evaluation is conducted in a simulator calibrated to clinical data; real-world clinical validation is not reported in the abstract. The full paper also makes clear that robustness depends on how observation perturbations and dynamics shifts are modeled, and the results remain tied to the abiraterone simulator and its fitted patient populations.

Practical Significance

This work provides a principled reinforcement learning methodology for learning reliable adaptive treatment policies in medical settings where uncertainty is prevalent. It could improve the safety and effectiveness of AI-driven treatment planning for diseases like cancer, ensuring policies remain effective despite imperfect measurements and evolving disease conditions.