Why aren't black, white or grey colours?

Well, that's tricky. There are a lot of technicalities involved, but as far as pigments go, black, white and grey are colors. And that is where the technicalities start. The way we observe pigments have to do with light, and so one could consider even pigment blacks, greys and whites as not colors...

In any case, a "color" where light is concerned must appear on the spectrum, which black and white do not. Because white is all the colors of light combined and black is no light. Grey, on the other hand, is less well defined, as it is only percieved. In other words grey is not a part of the light color wheel. Grey is how we perceive objects in dim light. Cones perceive photon energy and are responsible for our perception of color. Rods perceive light intensity. In dim light, where not enough photon energy is present to see in color, our rods can still function to perceive differences in light intensity and we see the world in shades of grey.