Reading through the book, "Smart Moves: Why Learning is Not All in your Head," by Carla Hannaford. I came across the answer.
"As we go through puberty, the endolymph fluid in the semicircular canals thickens in response to reproductive hormones. This thickening causes the hair cells to be bent for a longer time, thus causing the whole system to take a longer time to return to a comfortable equilibrium."
She goes on to stay that staying active with things like bike riding and yoga help to keep the balance and equilibrium from becoming to bad.
Having started doing some recent somersaults with moves incorporated into Animal Flow type work, I found it pretty unnerving that at first a simple tumbling activity could bring a bit of dizziness. After a few sessions these simply vanished as my body adapted to it.
You only get rusty when you quit moving it. Somersaults I think, should be a mainstay in life. How many of us actually ever train equilibrium. Simple, but effective.