This book offers a unique pathway to methods of parallel optimization by introducing parallel computing ideas and techniques into both optimization theory, and into some numerical algorithms for large-scale optimization problems. The presentation is based on the recent understanding that rigorous mathematical analysis of algorithms, parallel computing techniques, and "hands-on" experimental work on real-world problems must go hand in hand in order to achieve the greatest advantage from novel parallel computing architectures. The three parts of the book thus bring together relevant theory, careful study of algorithms, and modelling of significant real world problems. The problem domains include: image reconstruction, radiation therapy treatment planning, transportation problems, portfolilo management, and matrix estimation. This text can be used both as a reference for researchers and as a text for advanced graduate courses.