signal analysis - Reduce the "glare" effect of a receiver within a system

I'm working this summer on a very interesting, albeit difficult to understand at 100 %, project and it turns around reducing the glare effect than could be applied on a receiver when he gets signals coming from two emitters sending $X(t)$ (more energy for this signal compared to the other) and $s(t)$ (much weaker).

Let me put it in other words.

The receiver gets a signal mixture $Y(t)=H(t)*X(t)+g(t)*s(t)$ but we don't know $H$ nor $X$ nor $g$ nor $s$ nor the frequencies of the emitted signals. The only thing we know is $Y(t)$ and the fact that their spectra are dissociated in the best cases.

A picture speaks a thousand words, so...

Basically, the signal $X(t)$ overpowers $s(t)$. So I was thinking of a way to get the $s(t)$ and the $X(t)$ and with the few info I've gotten, I couldn't implement the generic solutions. However, I still have one card which is the FASTICA but that would suppose having many samples of "mixed " signal to extract the two important signals. What do you think about it? I'm open to any software and/or mechanic suggestion.

I've read an interesting paper on this but they were talking about the near-far effect for GPS signals. Thanks for the help!!

Answer

If you want to separate two sources you need some kind of diversity. Your know almost nothing about these signals so the only diversity you can exploit is frequency one. So first both spectrum should not be overlapped. And the second: as you know not all transmitted energy of the strong source (that you called $X(t)$) is allocated in the band you draw. Some out of band emission exists and its power depends on shape filter and power amplifier. So if even $X(t)$ and $s(t)$ are not overlaped at all, $s(t)$ may be totally shaded by out of band emission of $X(t)$. And you can't filter it out. You may refer to the problem of multiple access interference (MAI) problem of multi-user systems for details.

Now suppose frequency diversity is big enough and no out of band emission of $X(t)$ interferes with $s(t)$. As you were told FFT can be used to allocate both of the signals and further we can do bandpass filtering to completely distinguish them. But you probably need a large amount of data to achieve high resolution with FFT approach. Though you can improve FFT performance by windowning data set. The main advantage of FFT in this case is its ability to allocate signal even at very bad SNR (0 or less). You only need to increase its size.

The alternative is using some correlation properties and second order statistics. As I understand Blind source separation rely on it (PCA, ICA, but I can mistake). You may estimate input signal autocorrelation matrix as $R = E[xx^H]$. Further if you make $svd(R)$ you'll get singular eigen values of the signal mixture. Using these methods you can separate sources only if their eigenvalues are distinguishable from each other and noise ones (noise subspace). If the diversity is low or signal is too weak its eigen value decreases and it becomes "invisible" to the algorithm. The main advantage is much less amount of data is needed to allocate sources in comparison with FFT approach.

It is also important how large band of search is. It restricts complexity of the computations required. Furthermore $svd$ based methods (e.g. spectrum estimation) are good for narrowband sources (as I've seen, can mistake) while FFT is well enough for both narrow- and broadband cases.