We were at the track basically all week because the Matech team had the hospitality available for us. I'd say you could realistically be there Wed-Sun and see all you need to see, but if there's a GT racing, the whole week was cool. For example, a couple of us dumbasses walked the whole track. :lol The actual city of Le Mans is close enough to other stuff that your wife can go do other things. You can get to Normandy, some wine areas, and Paris isn't that far by TGV (1 hour). My wife and I stayed in Paris Saturday-Sunday-Monday and showed up in Le Mans on Tuesday. I think it's preferable to be near the track and go out, that way you're close during the 4 days of significant activity in Le Mans.
We stayed about 30 minutes from the track, which was reasonable and there were really nice countryside and roads between the house we rented and the track. Paris is a bit of a hike (200 km) and traffic can be a pain. We rented a car at the Le Mans train station.
If the Matech team is racing the GT at Le Mans again next year, I suspect we'll be back with another large group.