This project is a PyTorch re-implementation of IQCRNN, a method that enforces strict stability guarantees on Recurrent Neural Networks used in control systems.

Overview

Standard Reinforcement Learning agents can behave unpredictably in unseen states. This approach forces the agent’s weights to satisfy Integral Quadratic Constraints (IQC) via a projection step. Effectively, it solves a convex optimization problem (Semidefinite Program) inside the gradient descent loop to ensure the controller never violates Lyapunov stability criteria.

The method bridges classic Robust Control Theory (1990s) with Deep Reinforcement Learning (2020s), providing mathematical certificates of safety for neural network controllers.

Features

  • Hybrid Optimization: Interleaved standard Gradient Descent (PyTorch) with Convex Optimization (cvxpy + MOSEK) to project weights onto the “safe” manifold after each training step.
  • Complex Constraints: Implemented the “Tilde” parametrization from the original paper to convexify the non-convex stability conditions of the RNN dynamics, transforming an intractable problem into a solvable Linear Matrix Inequality (LMI).
  • Safety-Critical Domains: Applied the controller across six control systems (cartpole, inverted pendulum, nonlinear pendulum, pendubot, power grid, and vehicle dynamics), including unstable plants where “crashing” during training is unacceptable.

Usage

The repository includes train and run scripts for all six control systems.

Results

The project measures the tension between safety guarantees and training throughput.

  • The Bottleneck: Solving an SDP at every few steps of training is computationally expensive (interior-point SDP solvers scale steeply, roughly $O(n^6)$ in the matrix dimension). While it provided mathematical certificates of safety, it highlighted why these methods haven’t yet overtaken standard PPO/SAC in production: the “safety tax” on training time is steep.
  • Known rough edge: the codebase carries artifacts of its research origins, including a reqs.txt dependency dump.
  • The Lesson: theoretical guarantees come with engineering fine print. The route past this bottleneck is differentiable convex optimization layers (cvxpylayers), which would make the projection end-to-end differentiable.

Citation

Credit to the original authors:

@misc{gu2021recurrentneuralnetworkcontrollers,
      title={Recurrent Neural Network Controllers Synthesis with Stability Guarantees for Partially Observed Systems},
      author={Fangda Gu and He Yin and Laurent El Ghaoui and Murat Arcak and Peter Seiler and Ming Jin},
      year={2021},
      eprint={2109.03861},
      archivePrefix={arXiv},
      primaryClass={eess.SY},
      url={https://arxiv.org/abs/2109.03861},
}