Are there any countries or territories in the world whose flag does not contain any of the Olympic flag colours?

The short answer is no. Here's the long answer:

the colors simply came from those colors that appeared on all the national flags of the world at that time.

http://wiki.answers.com/Q/What_do_the_Olympic_rings_represent

This means that all countries who compete should have at least one colour appearing in their flag. According to the above website, Thialand, Vietnam, Cambodia, Lao, Iraq, Haiti, Malaysia and the Philipines don't compete. I have checked all those countries' flags and all the flags contain at least one of the Olympic colours.

Here's a list of countries' flags:
https://en.wikipedia.org/wiki/List_of_countries_by_colors_of_national_flags

All the subheadings have at least one color from the five Olympic ones so it looks like the answer is no.

It appears that the ones in all territories do as well:
http://commons.wikimedia.org/wiki/Dependent_territory_flags