Where you want to!
Dr. Mercer is a professor here at BYU who just started exercising 8 years ago in his mid 20s I'm assuming and now he runs marathons. That was what he found fun and he did it!
That's pretty much the same way you need to be. If you get a ton of satisfaction out of certain types of exercise (even if you're not good at them) that's what you should do. It's fun and challenging as you learn something new, and gives you another reason to wake up in the morning. If you don't like what you're doing, it will become a chore and you'll give up.
One of my older brothers said there was a 56 year-old woman in one of his workout classes that couldn't even do a push-up when she started. I think Dr. Mercer and she are great examples of how far you can come regardless of when you start. You'll never be as good as an Olympian athlete in their prime, but you can still kick a lot of booty and never give up on yourself!