You aren't being silly.

I will say that one of my kids was an early reader, one a late reader (for Montessori school, anyway) and they both came out fine. Sometimes kids just don't quite make the developmental leap until later. It really was like that with our son -- one day the light just went ON over his head and he went from not quite understanding sounds and letters to reading.