Volume 2 - Issue 3, May - June 2026

Model-based reinforcement learning (MBRL) with diffusion world models can capture complex, multimodal environment dynamics, but pure offline training frequently leads to model exploitation: the agent learns to take advantage of model inaccuracies, while naive online fine-tuning erases the useful offline behavior. We propose the Bayesian Diffusion World Model (BDWM), a framework that equips diffusion world models with epistemic and aleatoric uncertainty estimates via a lightweight ensemble of diffusion decoders, without sacrificing generative expressiveness. During offline training, model rollouts are filtered through an uncertaintypenalized buffer that discards transitions with high epistemic uncertainty, limiting model exploitation. For offline-to-online transfer, a dynamic scheduler adjusts the mixture of real and imagined data according to current model confidence, accompanied by a decaying behavioral cloning regularizer that keeps early online behavior safe. On MuJoCo locomotion and Adroit dexterous manipulation tasks, BDWM reaches up to 3x better sample efficiency than model-based and model-free baselines, with no performance drop at the offline-to-online transition. BDWM is the first framework to integrate Bayesian uncertainty into diffusion world models across the full offlineto-online pipeline.

Ayush Agrawal, Anshul Sharma, Madhur Sahu, Dr. S.K Sharma,"Uncertainty-Guided Diffusion World Models for Sample-Efficient Offline-to-Online Reinforcement Learning" International Journal of Advanced Multidisciplinary Research and Educational Development, V2(3): Page(965-969) May-June 2026. ISSN: 3107-6513. www.ijamred.com. Published by Scientific and Academic Research Publishing.