I went last year and it was similar to how you describe. It did feel really low key although quite a few of the spectators were watching on the race a big screen inside the "Wing" which to me seemed to defeat the purpose of attending it - why not just watch it at home?
I really enjoyed the racing though and I'd would definitely go again (unfortunately I couldn't make it this year).
In terms of attendance I don't think the time of year helps and apart from Le Mans (with all the associated history), endurance racing is a pretty tough sell as a spectator event to the casual motor racing fan.