minhag - Speaking while wearing Tefillin

Is there a real halachic source for the widespread practice not to speak while wearing tefillin? Of course, one is required to keep his mind on the tefillin, and may not get drawn into conversations that will distract him. But what about short, non-frivolous statements? Very often these fall into the category of gemillus chasadim; e.g., wishing someone good morning, giving directions, informing what time mincha is, etc. I wonder whether there is a basis to forgo these chesed opportunities on account of what seems to be merely a safeguard rather than halachah.