Calcium fluoride is a key lens material in deep-ultraviolet microlithography. However, exposure to high fluences of the application wavelength, 193 nm, results in coloring of the material and degrading of its optical properties. Based on theoretical modeling and with the support of experimental evidence, the damage process is discussed from the initial matter-light interaction to the formation of complex radiation damage structures. Understanding laser damage - or radiation damage in general - is crucial for the development of future high-performance optical materials.