cdc2019tutorial.html

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
  "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">
<head>
<meta name="generator" content="jemdoc, see http://jemdoc.jaboc.net/" />
<meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
<link rel="stylesheet" href="jemdoc.css" type="text/css" />
<title>CDC 2019 Tutorial - From Self-Tuning Regulators to Reinforcement Learning and Back Again </title>
</head>
<body>
<table summary="Table for page layout." id="tlayout">
<tr valign="top">
<td id="layout-menu">
<div class="menu-category">Nikolai Matni</div>
<div class="menu-item"><a href="index.html">home</a></div>
<div class="menu-item"><a href="publications.html">publications</a></div>
<div class="menu-item"><a href="people.html">people</a></div>
<div class="menu-item"><a href="https://github.com/unstable-zeros">unstable&nbsp;zeros</a></div>
<div class="menu-item"><a href="cdc2019tutorial.html" class="current">CDC&nbsp;2019&nbsp;Tutorial</a></div>
<div class="menu-category">Teaching</div>
<div class="menu-item"><a href="./courses/ese6050-spring2024/index.html">ESE&nbsp;6050</a></div>
<div class="menu-item"><a href="./courses/ese2040-fall2023/index.html">ESE&nbsp;2040</a></div>
<div class="menu-item"><a href="./courses/ese618-fall2021/index.html">ESE&nbsp;618</a></div>
<div class="menu-item"><a href="./courses/ese680-fall2019/index.html">ESE&nbsp;680</a></div>
</td>
<td id="layout-content">
<div id="toptitle">
<h1>CDC 2019 Tutorial - From Self-Tuning Regulators to Reinforcement Learning and Back Again <br /></h1>
<div id="subtitle"><a href="https://nikolaimatni.github.io">Nikolai Matni</a>, <a href="https://people.kth.se/~alepro/">Alexandre Proutiere</a>, <a href="http://www.control.lth.se/personnel/anders-rantzer.html">Anders Rantzer</a>, <a href="https://stephentu.github.io/">Stephen Tu</a></div>
</div>
<h2>Abstract</h2>
<p>Machine and reinforcement learning (RL) are increasingly being applied to plan and control the behavior of autonomous systems interacting with the physical world. Examples include self-driving vehicles, distributed sensor networks, and agile robots. However, when machine learning is to be applied in these new settings, the algorithms had better come with the same type of reliability, robustness, and safety bounds that are hallmarks of control theory, or failures could be catastrophic. Thus, as learning algorithms are increasingly and more aggressively deployed in safety critical settings, it is imperative that control theorists join the conversation. The goal of this tutorial paper is to provide a starting point for control theorists wishing to work on learning related problems, by covering recent advances bridging learning and control theory, and by placing these results within an appropriate historical context of system identification and adaptive control.</p>
<h2>Slides</h2>
<ul>
<li><p><a href="./files/1912CDCintro.pdf">Introduction</a> by Anders Rantzer.</p>
</li>
<li><p><a href="https://www.dropbox.com/s/nf350an518bth5u/cdc2019-rl-tutorial-clean.pptx?dl=0">From Self-Tuning Regulators to Reinforcement Learning and Back Again</a> by Nikolai Matni.</p>
</li>
<li><p><a href="./files/slidesAP.pdf">Optimal Adaptive Control in Discrete Systems</a> by Alexandre Proutiere.</p>
</li>
<li><p><a href="./files/1912CDCexplore.pdf">Dual Control: Optimization Based Exploration/Exploitation</a> by Anders Rantzer.</p>
</li>
<li><p><a href="./files/cdc19_sysid_tutorial_Tu.pdf">Concentration Inequalities for System Identification</a> by Stephen Tu.</p>
</li>
</ul>
<h2>Papers</h2>
<h3>Tutorial Papers</h3>
<ul>
<li style="line-height: 17px; padding-left: 20px; text-indent: -15px; " class="full-width" value="1">
                        <p style="text-indent: -15px; " class="paragraph_style_1"><span style="font-size: 14px; " class="Bullet"></span><span style="width: 3px; " class="inline-block"></span> N. Matni, A. Proutiere, A. Rantzer and S. Tu, <a href="https://arxiv.org/abs/1906.11392"> From Self-Tuning Regulators to Reinforcement Learning and Back Again</a>, IEEE Conference on Decision and Control, 2019.<br /></p>
                    </li> 
                    <li style="line-height: 17px; padding-left: 20px; text-indent: -15px; " class="full-width" value="1">
                        <p style="text-indent: -15px; " class="paragraph_style_1"><span style="font-size: 14px; " class="Bullet"></span><span style="width: 3px; " class="inline-block"></span> N. Matni, and S. Tu, <a href="https://arxiv.org/abs/1906.11395"> A Tutorial on Concentration Bounds for System Identification</a> IEEE Conference on Decision and Control, 2019.   <br /></p>
                    </li> 
</ul>                    
<h3>Additional References</h3>
<h4>From Self-Tuning Regulators to Reinforcement Learning and Back Again:</h4>
<ul>
<li><p>S. Dean, H. Mania, N. Matni, B. Recht, and S. Tu, <a href="https://arxiv.org/abs/1710.01688">On the sample complexity of the linear quadratic regulator</a>, Journal of Foundations of Computational Math, 2019.</p>
</li>
</ul>
<ul>
<li><p>S. Dean, H. Mania, N. Matni, B. Recht, and S. Tu, <a href="https://arxiv.org/abs/1805.09388">Regret bounds for robust adaptive control of the linear quadratic regulator</a>, NeurIPS, 2018.</p>
</li>
</ul>
<ul>
<li><p>S. Dean, S. Tu, N. Matni, B. Recht, <a href="https://arxiv.org/abs/1809.10121">Safely learning to control the constrained linear quadratic regulator</a>, IEEE American Control Conference, 2019.</p>
</li>
</ul>
<ul>
<li><p>Y.-S. Wang, N. Matni, and J. C. Doyle, <a href="https://arxiv.org/abs/1610.04815">A system level approach to controller synthesis</a>, IEEE Transactions on Automatic Control, 2019.</p>
</li>
</ul>
<ul>
<li><p>H. Mania, S. Tu, B. Recht, <a href="https://papers.nips.cc/paper/9205-certainty-equivalence-is-efficient-for-linear-quadratic-control.pdf">Certainty Equivalence is Efficient for Linear Quadratic Control</a>, NeurIPS, 2019</p>
</li>
</ul>
<h4>Optimal Adaptive Control in Discrete Systems:</h4>
<ul>
<li><p>J. Ok, A. Proutiere, D. Tranos, <a href="https://arxiv.org/abs/1806.00775">Exploration in structured RL</a>, NeurIPS, 2018.</p>
</li>
</ul>
<ul>
<li><p>A. N. Burnetas, M. N. Kathehakis, <a href="https://www.jstor.org/stable/3690147">Optimal Adaptive Policies for Markov Decision Processes</a>, Mathematics of Operations Research, 1997.</p>
</li>
</ul>
<ul>
<li><p>T. Jaksch, R. Ortner, P. Auer, <a href="http://www.jmlr.org/papers/volume11/jaksch10a/jaksch10a.pdf">Near-optimal Regret Bounds for Reinforcement Learning</a>, NeurIPS, 2009.</p>
</li>
</ul>
<ul>
<li><p>S. Agrawal, R. Jia, <a href="https://papers.nips.cc/paper/6718-optimistic-posterior-sampling-for-reinforcement-learning-worst-case-regret-bounds">Optimistic posterior sampling for reinforcement learning: worst-case
regret bounds</a>, NeurIPS, 2017.</p>
</li>
</ul>
<ul>
<li><p>Y. Jedra, A. Proutiere, <a href="https://arxiv.org/abs/1903.10343">Sample complexity lower bounds for linear system identification</a>, IEEE Conference on Decision and Control, 2019.</p>
</li>
</ul>
<ul>
<li><p>Y. Abbasi-Yadkori et al., <a href="https://papers.nips.cc/paper/4975-online-learning-in-markov-decision-processes-with-adversarially-chosen-transition-probability-distributions">Online Learning in Markov Decision Processes with Adversarially Chosen Transition Probability Distributions</a>, NeurIPS, 2013.</p>
</li>
</ul>
<h4>Dual Control: Optimization Based Exploration/Exploitation:</h4>
<ul>
<li><p>A. Rantzer, <a href="https://arxiv.org/abs/1912.03550">Minimax Adaptive Control for State Matrix with Unknown Sign</a>, arXiv:1912.03550, 2019.</p>
</li>
</ul>
<h4>Concentration Inequalities for System Identification:</h4>
<ul>
<li><p>M. Simchowitz et al., <a href="https://arxiv.org/abs/1802.08334">Learning Without Mixing: Towards a Sharp Analysis of Linear System Identification</a>, Conference on Learning Theory, 2019.</p>
</li>
</ul>
<ul>
<li><p>T. Sarkar, A. Rakhlin, <a href="https://arxiv.org/abs/1812.01251">Near optimal finite time identification of arbitrary linear dynamical systems</a>, ICML 2019.</p>
</li>
</ul>
<ul>
<li><p>A. Rantzer, <a href="https://ieeexplore.ieee.org/document/8431891">Concentration Bounds for Single Parameter Adaptive Control</a>, IEEE American Control Conference, 2018. </p>
</li>
</ul>
<ul>
<li><p>Y. Abbasi-Yadkori et al., <a href="https://arxiv.org/abs/1102.2670">Online Least Squares Estimation with Self-Normalized Processes: An Application to Bandit Problems</a>, Conference on Learning Theory, 2011.</p>
</li>
</ul>
<ul>
<li><p>B. Bercu, A. Touati, <a href="https://arxiv.org/pdf/0707.3715.pdf">Exponential inequalities for self-normalized martingales with applications</a>. Annals of Applied Probabability, 2008.</p>
</li>
</ul>
<ul>
<li><p>H.B. Mann, A. Wald, <a href="https://www.jstor.org/stable/pdf/1905674.pdf">On the Statistical Treatment of Linear Stochastic Difference Equations</a>, Econometrica, 1943.</p>
</li>
</ul>
<ul>
<li><p>J.S. White, <a href="https://projecteuclid.org/euclid.aoms/1177706450">The Limiting Distribution of the Serial Correlation Coefficient in the Explosive Case</a>,  Annals of Mathematical Statistics, 1958.</p>
</li>
</ul>
<!-- Global site tag (gtag.js) - Google Analytics -->
<script async src="https://www.googletagmanager.com/gtag/js?id=UA-49484039-2"></script>
<script>
  window.dataLayer = window.dataLayer || [];
  function gtag(){dataLayer.push(arguments);}
  gtag('js', new Date());

  gtag('config', 'UA-49484039-2');
</script>
<div id="footer">
<div id="footer-text">
Page generated 2025-10-03 12:16:28 EDT, by <a href="http://jemdoc.jaboc.net/">jemdoc</a>.
</div>
</div>
</td>
</tr>
</table>
</body>
</html>