It is not your imagination.....
However, I personally do not find it irritating but many people do.
I think it stems from people communicating a lot more nowadays, with internet, mobile telephones, Facebook, whatsapp etc. etc. and there is usually a conversation going on in their head and when the discussion comes to conversation, the 'So' is part of the beginning of the end of a conversation, if that makes sense?
Example:
Someone has sent a message out to friends asking what time to go out, and the general consensus is whenever. The same person then approaches their immediate company and starts the sentence:
So, what time shall we go out?
Just a theory perhaps...