Review of colour constancy in human visual system

Introduction

Object surfaces differ in how they can absorb and reflect light [Brainard, 2003]. The spectral distribution of the light that reaches the eye under specific lighting conditions is known as luminance. In the simplest model, luminance is determined by the spectral power distribution (colour) and angle of the light reaching the object, and the reflectance properties of the object in the scene. More complex models also account for the atmosphere the light travels through.[Adelson, 1999] Human beings are able to see the same scene under different illumination conditions, and are still able to maintain a perceptual representation of scenes that remains stable against these changes in illumination [Kraft and Brainard, 1999]. This phenomenon is known as colour constancy. Colour constancy is essential if colour information is to be used as a useful variable in the identification of an object, as if something appears to be a different colour under differential lighting conditions, colour cannot be a useful perceptual indicator of object properties.

Human vision traditionally exhibits good colour constancy across changes in illuminant spectral power distribution [McCann 1976, Kraft and Brainard 1999]. The human colour constancy mechanism must cope with two distinct situations in order to be effective: changes in scene illuminant over time (termed successive colour constancy), and areas of the same scene which are lit by illuminants of differing spectral power distribution (known as simultaneous colour constancy) [Brainard, 2003]. The mechanisms that govern these two phenomena need neither be the same nor mutually exclusive. In order to be colour constant, the visual system must be able to discount the variations in illumination across scenes so as to determine the reflectance of a given surface with a degree of constancy. The visual system is composed of low, mid and high level processing centres that may all have role to play in this image analysis. This review looks at experiments conducted to test varying hypothesis on the mechanisms behind this area of colour vision, and then relates these to the project work to be undertaken. The project is essentially a study into the ability of the visual system to discriminate illuminant boundaries, tested by measuring the degree of simultaneous colour constancy across two simulated illuminants.

The first section of this review looks at the models proposed for the colour constancy phenomenon. Work into lightness perception is also introduced, and its relevance discussed. Mechanistic and computational models are compared, and their successes and failures are examined. The second section describes the experiment and the aims for the project.

Colour constancy mechanisms

Visual information about a scene is detected by the retina, which soon after a great amount of image compression through various mechanistic models passes the information to the visual system via the optic nerve and the thalamus [Kandell, 2000]. The retina is capable of discriminating differing wavelengths of light by virtue of its three classes of cone photoreceptors [Kandell, 2000]. This raw data does not however provide direct clues about the spectral power distribution of the illuminant. This is explained in figure 1.

The colour signal (luminance) reflected back to the observer is not useful by itself; to achieve colour constancy the illuminant variances need to be discounted in order to normalise the information about object reflectance.[Helmholtz, 1962] Initially this may seem a paradox, as the raw data cannot be decomposed once confounded by the retina. Indeed it is possible to simulate conditions where two images may appear entirely alike to an observer, but in actuality comprise alternate illuminants and surfaces respectively. [Brainard, 2003]. Models attempting to explain colour and lightness constancy propose systems that can use information across the whole scene to overcome this problem, and make an educated guess as to what the illuminant may have been. This is known as the Illumination Estimation hypothesis, and is generally considered the broad method by which the visual system maintains constancy. However recent work by Rutherford et al (2002) into lightness constancy showed that the Illumination estimation hypothesis failed to perform as expected, potentially calling this pillar of research into doubt. However doubts remain with the experimenters as to the experimental design, and further work needs to be done to address this issue further.

In searching for potential mechanisms to explain the phenomenon, colour constancy has been approached from two key directions, that of a mechanistic approach, and that of a computational one. Mechanistic theories attempt to explain constancy phenomena by ascribing data analysis and processing to simple visual mechanisms at the cellular level. One such theory was proposed by Von Kries (Brainard, 2003), and is called the Von Kries adaptation model. This model takes information from the initial encoding of the colour signal at the cones, and ascribes each type of cone (L, M and S) a quantal absorption rate for a given luminance. The photons that make up a luminance are therefore grouped into their relative stimulation of 3 classes of cone. The model demonstrates that when the signals are adapted through a vector transformation, quantal absorption rates can be held relatively constant across illumination changes by controlling changes in signal gain. Standardisation can be potentially achieved by applying a linear model [Wandell, 1995] to the luminance data that takes into consideration the standard spectral composition of daylight. Judd et al [1964] measured the spectral composition of daylight around the world by recording reflected light off a surface of known achromatic reflectance. They discovered that daylight is relatively constant. This finding provides a reference databank for the most likely illuminant to be encountered by the visual system.

This is a preview of the whole essay

This is evidence for a mechanistic system behind successive colour constancy. This gain control model gives a good approximation of constancy across illumination variance across scenes, but lacks an explanation of the factors that control this adaptation. Gilchrist et al, 1999 has applied this model to the lightness constancy phenomena closely associated with colour constancy (where the perceived reflectance of a surface remains constant over luminance changes). He found that such a model would need to find a reference lightness, and “anchor” all other measurements to this reference. This is referred to as an anchoring rule. How this happens is unclear. The adaptation gain control models proposed therefore lack sufficient understanding of the way in which the assumptions necessary for their validity arise. Such questions may only be truly addressed when more defined physiological models of the neural processing in the visual system are understood.

The second approach towards understanding the basis of colour constancy calculation has been to develop image processing algorithms able to achieve colour constancy by analysing raw data received by the retina (luminance), and then testing these data against measured constancy in human observers. This is known as the computational approach. Computational models often adopt a 2 step model (Brainard, Kraft et al, 2003). Firstly the image is analysed, and an estimate of the illuminants spectral power distribution is ascertained. Secondly, with the information from the first step, the surface properties in the image are described independently of illuminant variations. Computational models differ in the scene conditions under which they are applicable, and the methodology by which each step is achieved. Below several theories are considered, and their applicability and failings highlighted.

Buchsbaum (1980) proposes a model that functions in a scene consisting of flat matt images of uniform colour and arranged in the same plane (ie 2D). This representation is known as the Mondrian world. As we have seen from figure 1, luminance is a product of the equation:

E(x,y) * R(x,y) = L (x,y)

This relationship can be applied to any surface in the scene, and a matrix generated of all available surfaces in the image. The illuminant is then estimated based on the spatial mean of the cone quantal absorption values off each surface. The model assumes that colour constancy is achieved by assuming that if one adds up all the reflectances in a scene under an illuminant, the average will be grey. This “grey world hypothesis” [Kraft, Brainard, 1999 Wandell, 1995] assumes an illuminant spectral power distribution based on the lightness of the average grey value determined by the model. However experiments show this not to be the case in real world situations, and so the model is flawed in that it is only capable of functioning within the constraints of the Mondrian world.

Other models have sought to expand upon the proposals also introduced in work on anchoring in lightness [Gilchrist, Agostini et al 1999]. Anchoring (as described above) refers to mapping lightness values to a known benchmark of lightness from which all others can be compared. Conclusions from work into anchoring have shown that Anchoring obeys two rules. The first condition, the highest luminance rule, dictates that the surface of highest luminance in a scene is perceived as white (highest luminance), thus anchoring the rest of the scale to this maximum [Li and Gilchrist, 1999]. The second feature of anchoring noted was that the largest area tends to appear white [Gilchrist and Cataliotti, 1995]. An area deemed to be white by the visual system is also of interest in colour constancy, as an ideal white surface will reflect the whole visible spectrum uniformly. If the visual system recognises an area which should be white (based on the anchoring rules) and applies a transformation to the raw cone data such that luminance data from this area shows uniform spectral response, then illuminant variances will have been discounted due to the anchoring to a known reflectance. Lee (1986) demonstrates how the lightest points in an image (specularities) provide clues as to the spectral composition of the illuminant. However the paper adds the proviso that the visual system is also capable of delivering colour constancy across scenes in the absence of specularities, so this is not the only method by which illuminant information may be collected, rather specularities may be used to glean information as part of a larger system encompassing multiple mechanisms.

One important model to consider is that developed by Land and McCann (1971). Land and McCann proposed a model that looked at lightness variations in scenes, and mathematically decomposed them into changes in reflectance (from one surface to another), and changes in illuminance (illuminant change). This is of particular interest in studying simultaneous colour constancy, where two or more illuminants are involved. The model, named Retinex (after Retina – Cortex) assumes that a reflectance change will exhibit a large change in luminance (step edge) around the boundaries of the surface and neighbouring surfaces. Illuminance changes will however manifest as much more gradual changes in luminance. By taking spatial derivatives of the luminance changes, and recognising high derivatives as reflectance changes, and low derivatives as illuminance changes, illuminance variation can be removed from the scene luminance statistics and a re-integrated model generated with illuminant changes discounted. The Retinex model assumes that there are only finite possibilities as to how luminance changes in an image can occur, and that the visual system is inherently aware of these properties in real world scenes. However when Retinex is applied to 3D situations, the model fails. When the image in figure 2 is applied to the Retinex model, the model fails to predict the human perception of the top of the shape being lighter. Retinex classifies both light-dark edges as being reflectance steps. To explain this effect, it has been suggested that the visual system uses what are known as junctions.

Junctions are areas within an image where surfaces with differing luminancies meet. They exist in both 2D and 3D representations, and depending on the junctions’ make-up, provide the visual system with information on the dimensions of the object. They offer clues as to the shading and reflectance of a surface (Sinha 1993). This allows the visual system to segregate surfaces, and treat them discretely within the larger context of the image. In simultaneous colour constancy, such segregation may be very important when looking for illumination boundaries.

One junction of note is the Ψ junction. It places strong constraints upon the limits of possible illuminance and reflectance combinations. (Adelson, 1999). These junctions may be utilized by colour constancy models (none as yet have done so) to determine illuminant boundaries, and hence treat areas of a scene discretely. X junctions are also discussed by Adelson, and their usefulness in determining the transparency or opacity of objects or atmospheres is another direction in which colour constancy research could potentially look toward. It is precisely such mid level processing in the visual system that may yield clues toward the mechanisms behind colour constancy.

Junctions are merely one avenue meriting closer attention in the design of future models of the illuminant variance detection systems necessary for lightness and colour constancy. Junctions themselves are merely potential tools which allow the visual system to apply spatial filtration onto a scene. The nature of this spatial filter and the parameters which define it are also worthy of further investigation. The Retinex model distinguishes different surfaces with distinct reflectances, while newer models such as the Bayesian model of colour constancy (Brainard, 1997) also analyse the probability of a particular illuminant/reflectance combination occurring in a potential space. This allows a decision to be made as to the probability that another illuminant is present in the scene. Gilchrist and Cataliotti (1994) discussed the need for local and global frameworks within a scene, in which objects could be grouped and treated discretely form the rest of the scene, or as part of the whole scene. Adelson (1999) describes a potential model for these frameworks as taking the shape of what he describes as an “adaptive window”. The image luminance statistics within the window are processed separately, allowing for luminance variances within a scene.

The use of computational models and mechanistic theories in colour and lightness constancy has received extensive interest from scientists in the visual neurosciences. These models are however all limited by the circumstances in which they define themselves, and none as yet have come close to mirroring the adaptability and versatility of the human visual systems own in built mechanism.

The mechanisms underlying colour constancy may well be a combination of low level mechanistic adaptation models which begin with simple centre-surround inhibition at the retinal level, moving up to so called mid-level mechanisms such as analysis of luminance changes at junctions, contours, and grouping via spatial filtering, and finally high level processing which may analyse information in the context of past experiences and application of Bayesian decision theory to the likelihood of a reflectance/illuminance combination.

Project investigation

The second section on this literature review outlines the theory and literature related specifically to the project. Detailed discussion of the project methods etc will be addressed in the final submission.

The project is an investigation into simultaneous colour constancy. An observer will be presented with several stimulus patterns under a uniform illuminant which can also be in one of two conditions. Figure 3 gives an example of one of the stimulus patterns.

Looking at the image, one can clearly see a well defined area which is predominantly blue. This area is a simulated illumination boundary, and should be treated by the visual system as having an illuminant of differing spectral power distribution to that of the rest of the stimulus pattern. The centre of the blue simulated illuminant region contains a diamond shaped hole (shaded white with a black dot in figure 3). Behind this will be a CRT computer monitor which can display any chromaticity in the RGB colour-space. This is referred to as the test patch. The observer is asked to adjust the chromaticity of the test patch until it appears achromatic (somewhere on the perceptual continuum from black to grey to white, and having no colour cast) [Kraft and Shannon et al, 2002]. This technique is known as achromatic adjustment. When the test patch is achromatic, the chromaticity of the test patch is recorded, and serves as a surface of known luminance under a particular (perceived) illumination. If the illumination boundary is correctly perceived, then the achromatic setting should reflect the fact that the test patch is under a simulated blue illuminant. If the visual system fails to perceive the illumination boundary and takes into account the red surround of the stimulus pattern, the achromatic setting will be different. This method of testing illuminant perception is well established (Helson and Michels, 1948; Werner and Walraven, 1982) and will hopefully provide accurate confirmation of the perception (or not) of an illumination boundary.

There are 12 stimulus patterns in total. They are designed so as to test the flexibility of the spatial filter (or adaptive window)s ability to grow and shrink by adjusting the size of the area under the simulated illuminant. The patterns are also varied in the definition of the boundary between the simulated illumination differences. Figure 4 shows a stimulus pattern without the continuous border present in figure 3. Such a pattern relies on the spatial filter being able to detect the illumination boundary even though it is not explicitly defined. This is achieved by introducing a ‘jittered edge’ to the boundary. This jitter pattern can be seen in figure 5, and is clearly visible when compared to the un-jittered pattern present in figure 3.

The jitter pattern is a randomisation of the illumination border, and jitter of varying degrees will be introduced. This topic will be addressed in more detail in the final project.

The determination of illumination boundaries is addressed in Gilchrist, Annan (2002). They state that the articulation (which they define as the levels of lightness within the illuminant surface) allows for greater articulation of the illuminant boundaries. This is echoed in colour research by Kraft and Brainard (2002), who also found scene complexity to have a bearing (albeit in certain circumstances) on colour constancy. It is therefore possible that because of the greater number of coloured stimulus within the larger annuli, the spatial filter will be able to discriminate the simulated illumination boundary more readily. Figure 6 shows an example of a stimulus pattern with a larger annulus. The adaptability of the spatial filter is a topic which is also addressed by the varying annulus sizes presented. If there is a limit to the area that a spatial filter can discriminate, then it is possible that results from stimuli with the smallest annuli will show achromatic settings that are affected by the surrounding shapes of alternate simulated illumination. We can therefore potentially estimate a size for any spatial filter in degrees of the visual field.

The pattern stimuli may be inverted to their complementary colours by changing the illuminant. Figure 7 shows the effect this would have.

The theory behind this returns to the initial dogma of colour science stated at the beginning of this review, that every luminance is a product of the spectral power distribution of the illuminant and the reflectance properties of the surfaces in question. There are multiple combinations of illuminant and reflectance that can produce the same scene, and additionally there are conditions where a surface with specific properties can display its complementary colours under a differential illuminant.

In conclusion, colour constancy is a vital aspect of a humans ability to accurately derive a meaningful percept of an objects surface reflectance (in essence to tell what colour something is), with lightness constancy a close cousin. In many ways, the two phenomena are related, both dealing with the analysis of visible light detected by the eye, both interested in the illumination boundaries of objects, and how different surfaces reflect incident illumination. It appears that the problems faced in deriving an explanation for the visual systems constancy systems may be harder to solve than anticipated, due to the number of factors which appear to play a role in appraising a scene and determining the details. Systems ranging from low to high level processing are potentially implicated, with everything from memory to cellular inhibition in the retina potentially affecting the final visual percept. Kraft and Brainard (1999) show that it cannot be merely simple mechanisms that underlie this system, and clearly something quite elaborate is occurring. This concise review has attempted to unearth some key areas of this expansive topic, specifically in the areas related to simultaneous colour constancy and the determination of illumination boundaries. The final project report will contain more information about the set up and theory behind specific areas of the experiment.

References

Adelson, E.H. (1999). Lightness perception and lightness illusions. In The Cognitive Neurosciences, M.S. Gazzaniga, ed. (Cambridge, MA: MIT Press).

Brainard D. H. and W. T. Freeman, 1997, ‘Bayesian color constancy,’ J. Opt. Soc. Am. A 14, 1393–1411

Brainard, D. H. 2003 Color constancy, The Visual Neurosciences, Eds Chalupa. L.M , Werner J.S, p.948-956, The MIT Press, London

Brainard, D. H., Longère, P., Delahunt, P. B., Freeman, W. T., Kraft, J. M., & Xiao, B. (2006). Bayesian model of human color constancy. Journal of Vision, 6(11), 1267 1281

Cataliotti, J. and A. L. Gilchrist (1995). Local and global processes in surface lightness perception. Perception and Psychophysics 57(2): 125-135.

Gilchrist A, Kossyfidis C, Agostini T, Li X, Bonato F, Cataliotti J, Sephar B, 1999, An Anchoring Theory Of Lightness Perception, American Physiology Journal, Vol106 No. 4 pages 795-834

Helmoltz. H., von. (1962). Treatise on physiological optics. New York: Dover

Helson, H., & Michels, W. C. (1948). The effect of chromatic adaptation on achromaticity. Journal of the Optical Society of America, 38, 1025-1032

Judd, D. B., MacAdam, D.L., and Wyszecki, G.W. 1964. Spectral distribution of typical daylight as a function of correlated color temperature. J. Op. Soc. Am., 54:1031-1040

Kraft. J M, Maloney. S I., & Brainard D H., 2002, Surface-Illuminant Ambiguity and Color Constancy: Effects of Scene Complexity and Depth Cues, Perception, vol 31, pages 247-263

Kraft, J. M., & Brainard, D. H. (1999). Mechanisms of color constancy under

nearly natural viewing. Proc. Nat. Acad. Sci. USA, 96(1), 307-312.

Land, E. H and J.J McCann, 1971. Lightness and Retinex Theory, J. Opt oc Am, 61:1-11

Lee, H. 1986. Method for computing the scene-illuminant chromaticity from specular highlights, J. Opt Soc. Am. A, 3:1694-1699

Li, X. and A. Gilchrist (1999). Relative area and relative luminance combine to anchor surface lightness values. Perception and Psychophysics 61(5): 771-785.

McCann J. J., McKee S. P. and Taylor T. H., 1976 Quantitative studies in retinex theory a comparison between theoretical predictions and observer responses to the "color mondrian" experiments, Vision Research, Volume 16, Issue 5, , , Pages 445-448.

Rutherford, M. D., & Brainard, D. H. (2002). Lightness constancy: A direct test of the illumination-estimation hypothesis. Psychological Science, 13(2), 142–149.

Sinha, P., and Adelson, E. H. (1993). Recovering Reflectance in a World of Painted Polyhedra. Proceedings of Fourth International Conference on Computer Vision (pp.156-163) Berlin; May 11-14, 1993.

Wandell B.A, 1995, Color Constancy, Foundations Of Vision, p.287-314, Sinauer Associates Inc, Sunderland, Massachusetts, USA

Werner, J. S., & Walraven, J. (1982). Effect of chromatic adaptation on the achromatic locus: the role of contrast, luminance and background color. Vision Research, 22(8), 929-944.

Wurtz, R. and Kandel, E. (2000a). Central visual pathways. In Kandell, E., Schwartz, J., and Messel, T., editors, Principles of Neural Science (4th edition), pages 523.547