CLBF-Constrained Curious Actor-Critic: Safe and Exploratory Reinforcement Learning for Embodied Robotic Systems

Abstract

We propose a novel reinforcement learning framework for embodied robotic systems that integrates safety guarantees with curiosity-driven exploration through a Control Lyapunov-Barrier Function (CLBF) constrained actor-critic architecture. The proposed method, termed C3AC, addresses the fundamental tension between safe operation and exploratory behavior in physical robots. The critic network is augmented to output three distinct value components: a standard action-value function, a Lyapunov certificate that enforces convergence to goal states, and a barrier certificate that maintains forward invariance of a predefined safe set. These certificates are learned jointly with a forward dynamics model, and their satisfaction is enforced through differentiable penalty terms in the critic loss. To encourage exploration of novel yet safe regions, an ensemble of five forward dynamics models is maintained, and the epistemic uncertainty across this ensemble serves as an intrinsic reward signal. A dual Lagrangian network adaptively weights this curiosity bonus based on the current safety margin, ensuring that exploration is only encouraged when Lyapunov constraints are satisfied. The actor network outputs nominal actions that are subsequently projected onto a provably safe set via a differentiable quadratic program solved in real time at 50 Hz. This projection step explicitly enforces the Lyapunov decrease condition and barrier constraint, thereby guaranteeing safety during execution. The entire system operates in a closed loop with off-policy training using Soft Actor-Critic updates. The primary contributions of this work are threefold: first, the embedding of dual safety certificates directly into the critic architecture; second, the coupling of ensemble-based intrinsic motivation with safety-constrained action projection; and third, the real-time differentiable optimization that backpropagates through the safety projection to update the actor. Experimental evaluations on a UR5 robotic arm demonstrate that C3AC achieves superior task completion rates while maintaining zero safety violations during training, outperforming both standard SAC and prior safety-constrained methods in terms of sample efficiency and exploration quality.