Is there a unit of measurement smaller than a nanometer?

Yes
Scientists often measure lengths even smaller than a nanometer—the width of an atom, for instance, or the wavelength of a light ray. For this purpose, they use the angstrom (Å or A), equal to 0.1 nanometers.
http://www.answers.com/topic/measurement

1 angstrom (Å) = 0.1 nm

You mean the smallest distance we can measure?

There are measurements smaller than 1 Angstrom - 1 picometer is 100 times smaller, and 1 femtometer (also known as a fermi) is 100,000 times smaller, and is about the size of an atomic nucleus. But there are not any technologies available to practically do anything on that kind of length scale, so
the smallest useful scale is about an Angstrom.

there is actually something smaller then a femi
planck length

"If a particle or dot about 0.1mm in size (which is at or near the smallest the unaided human eye can see) were magnified in size to be as large as the observable universe, then inside that universe-sized "dot", the Planck length would be roughly the size of an actual 0.1mm dot, that is, about the size of smallest object the naked human eye can see"
https://en.wikipedia.org/wiki/Planck_length