When I first got into watches and spent four figures on a very understated vintage Rolex, I wore it much less than I should have. It was in amazing condition and I was clearly paranoid about scratching or stretching it, and it seemed potentially a bit showy around my frequently broke artist friends. Now it's a watch I reach for when I'm looking for something subtle and easy to wear. Clearly the goal posts move over the years.
In the same way, I used to prefer to have a choice of watches in the £1-2k range give or take, plus the odd vintage cheapie, rather than consolidating with something higher value. They just seemed easier to wear without worrying about the watch, or feeling showy. Now the psychological barrier has moved (and ten years have passed), and I'm inclined to agree with the OP on where the comfortable middle ground lies. So a bit of trading up is in order, something I could have done years ago but chose not to. The point, I guess, is that 'it's all relative', though you can also pick a point that feels comfortable and seems to be 'in the middle'.
As it stands though, my most pricey piece is the one that gets the most wrist time. It's a Grand Seiko - not only the most confortable watch I own, but also comfortably under the radar. I assume you're average Joe has no idea what it is, so I never think twice about wearing it anywhere. I bought it with the very clear intention of wearing it often, if it needs re-finishing one day then so be it. So my most expensive watch is also my daily beater!