I had a similar size collection to yours at one point. It was a big mess to be honest, nice watches but lots of stuff bought on a whim due to a good sale etc. Over time I've kept whittling it down and adding some upgrades, and I'm now at 5 watches, with 2 earmarked to go and 1 earmarked to come in (if it ever turns up!). That'll leave me with 4 just like you're proposing, and personally that feels just right. Others on here no doubt will recoil in horror at such a limited choice, but I just think you get more time to appreciate the ones you've got and less time worrying about it.
So my vote is go for it - you've made your choices!