A material gets its colour from regions of its molecular structure known as ‘chromophores’, which absorb photons of visible light at particular wavelengths. Any photons that aren’t absorbed are re-emitted, and the wavelength of these photons determines the colour that we see.
Over time, exposure to the higher-energy photons in sunlight can damage the structure of a material’s chromophores, affecting their ability to emit photons at certain wavelengths. For example, red materials are particularly susceptible to fading in sunlight. The chromophores in these materials emit red light by mopping up photons of all the other wavelengths.
Among these absorbed photons are those toward the blue – and higher-energy – end of the spectrum. Red materials are also good absorbers of photons of ultraviolet (UV) light, which have higher energies yet. The result is that the red-emitting chromophores degrade at a faster rate, triggering fading. So simply put, the colour doesn’t ‘go’ anywhere when materials fade – it’s just not emitted so well by the chromophores.