halacha - Are we allowed to ask someone if he's a convert?

I remember learning that we are forbidden to "remind a convert of his past". The implication, as I remember it, is reminding someone that he used to be an idolater. (Though not all converts were.) As a result, I was taught, we can't ask somebody if he is a convert unless there really is a specific need to know (like investigating a marriage prospect for a kohein, who is forbidden to marry a convert).

If this is correct, what is the source?

If this is not correct and we are in fact allowed to ask someone if he is a convert, are there any specific situations where, per halacha, we shouldn't? I could imagine, for example, that there might be "after the fact" cases where we shouldn't, e.g. doing so could invalidate an existing marriage.