Catering to the current interest in increasing the spectral efficiency of optical fiber networks by the deployment of high-order modulation formats, this monograph describes transmitters, receivers and performance of optical systems with high-order phase and quadrature amplitude modulation. In the first part of the book, the author discusses various transmitter implementation options as well as several receiver concepts based on direct and coherent detection, including designs of new structures. Hereby, both optical and electrical parts are considered, allowing the assessment of practicability and complexity. In the second part, a detailed characterization of optical fiber transmission systems is presented, regarding a wide range of modulation formats. It provides insight in the fundamental behavior of different formats with respect to relevant performance degradation effects and identifies the major trends in system performance.