Once or twice a week I go on the lunch ride with randomdreams and one or two other guys. The weather today is gorgeous: sunny skies, 70°F (21°C), very light breeze and dozens of cyclists on the road. We rode 18.8 miles in 54 minutes. For an old guy like me, that 21 mph average pace is smokin' fast -- I'm usually a fast C or slow B rider on the weekend club rides.
When I'm solo, my average is more like 16 mph, but a great group pushes me and encourages me to go beyond what I can normally do.
If you're interested in getting better, pushing yourself beyond your normal limits is a must. If your normal pace is 9 mph, push yourself to go 12 mph. If you normally go about 15 mph, go 18 mph.
I'm weird in that I like coming back to the office with rubbery legs and on the verge of blacking out because I've expended my blood sugar on the final sprint. I don't race and I only rarely participate in organized rides so there's no real goal for me, but I do like to improve myself physically and get better. If I did have a goal, I would be much more methodical in my training.
I realize there are several people who aren't that interested in going fast and there's nothing wrong with that. You enjoy just getting out to enjoy the scenery and the smells and the sounds and feel the wind.