"Color science is a fundamental field of science dedicated to understanding the creation of colored stimuli, sources of illumination, and ultimately the human perception of color. It builds upon, and crosses the disciplinary boundaries of, chemistry, physics, life sciences, mathematics, and psychology."
-- from the Rochester Institute of Technology's Program of Color Science, https://www.rit.edu/science/pocs/.
The science of color is not a single field but a genre of inquiry that encompasses a range of disciplines, including but not exclusive to: physics (which looks at how colors come to be from the reflection of light on objects), psychology (which looks at what affects our perception of and psychological reaction to color), physiology (such as in the study of synesthesia, in which color perception intertwines with other sensory data), as well as colorimetry (the measurement and quantification of the color spectrum). These disciplines will also often overlap and contribute to each other's findings.
Runge and Munsell Color Spheres (via Wikimedia Commons)
There are many more books related to color science at NYPL. You can find some of them by browsing by these subject areas in the NYPL catalog:
"The Dress" is a well-known example of the way our individual perceptions of color can differ. This video explains the science behind that phenomenon.