Verifying Linear Time Invariance

I have a system of the form:

$$T(x(n))=x(n)+3x(n-2)-5x(n-3)x(2n)$$

I claim that

$$T(x(n-k))=x(n-k)+3x(n-k-2)-5x(n-k-3)x(2n-2k),$$ $$y(x(n-k)= x(n-k)+3x(n-k-2)-5x(n-k-3)x(2n-2k)$$

and

$$T(ax_1(n)+bx_2(n))=ax_1(n)+bx_2(n)+3\left( ax_1(n-2)+bx_2(n-2)\right) - 5 \left(ax_1(n-3)+bx_2(n-3) \right)\left( ax_1(2n)+bx_2(2n)\right),$$ $$aT(x_1(n))+bT(x_2(n))) = ax_1(n)+3x_1(n-2)-5x_1(n-3)x_1(2n) + ax_2(n)+3x_2(n-2)-5x_2(n-3)x_2(2n)$$

Hence, the system is not linear, but is time-invariant. I'm I applying the formulas correctly?

Answer

The term $x(2n)$ is not time invariant: it gives the even samples of $x$. Shift the input by 1 sample and you'll get the odd terms, which are different from the even terms shifted 1 place.

Your reasoning above is not correct. You should compare $T(x(n-k))$ to $T(x(n))|_{n-k}$ (that is, the original output, shifted by $k$ samples).

Edit: These type of exercises are a common source of confusion because of the following abuse of notation: $x(n)$ is not a signal, but the nth sample of the signal, and having $n$ inside of the expressions is formally wrong. Even Oppenheim's book makes this mistake when working with LTI systems and impulse response.

The correct way for expressing transforms between signals is as follows:

Let $x$ and $y$ be signals; $T\{\cdot\}$ be the transformation (system) under scrutiny; $R_k\{\cdot\}$ be a delay of $k$ samples.

So, in general you would write the relation between input and output signals as $y = T\{ x \}$, where the whole signals are involved. Using $n$ inside these expressions collapses the signal to a single sample and this is wrong. And having the same variable $n$ on both sides of the equation only leads to confusion.

Then, a system is time-invariant if $R_k\{T\{ \cdot \}\} = T\{ R_k\{ \cdot \}\}$ for all integer $k$.

This is why I gave the example above with even-odd samples, in order to avoid the usual confusion with $n$ inside the expressions.

Now, going back to the system $T$ that evaluates to $y[n] = x[2n]$ for each sample $n$:

Let $x_k = R_k\{x\}$ and $y_k = R_k\{y\} = R_k\{T\{x\}\}$.

Evaluating all these signals at sample $n$: $x_k[n] = x[n-k]$; $y_k[n] = x[2n - 2k]$.

Now, $T\{x_k\}$ at sample $n$ is $x_k[2n] = x[2n - k]$ which is different from $y_k$.