Yes they do care.
Take your Rolex example. You can buy a lot of watches that look like a Rolex but are very clear with their branding that they are not a Rolex. Seen across a bar room they might look identical.
That's fine nobody is kidding anybody.
Now consider a fake Rolex pretending to be what it is not. For a start that is illegal and it's done to deceive. There are people in the world who will try to pass off the better fake Rolex as if they are real. The first thing the buyer knows is when he walks into the Rolex shop to get it fixed.