Looking at the list (link) leads me to think that the FIA is trying to avoid criticism of nationalistic bias by having representatives from many countries (all those holding a GP?). I take that they are nominated by their national organisations.
Having any more than four, as at present, for any one event would probably be unwieldy, if not unmanageable.
While the idea of having a smaller, permanent team is appealing I doubt that it would be without its own problems. Who would choose them...the FIA's decisions are frequently criticised and an election by drivers or teams would similarly be open to accusations of bias. However, it shouldn't be an insoluble problem, but...?