Autonomous robots have been a vision of robotics, artificial intelligence, and cognitive sciences. An important step towards this goal is to create robots that can learn to accomplish a multitude of different tasks triggered by environmental context and higher-level instruction. Early approaches to this goal during the heydays of artificial intelligence research in the late 1980s showed that handcrafted approaches do not suffice and that machine learning is needed. However, off the shelf learning techniques often do not scale into real-time or to the high-dimensional domains of manipulator and humanoid robotics. In this book, we investigate the foundations for a general approach to motor skill learning that employs domain-specific machine learning methods. A theoretically well-founded general approach to representing the required control structures for task representation and execution is presented along with novel learning algorithms that can be applied in this setting. The resulting framework is shown to work well both in simulation and on real robots.