acid base - Why doesn't a buffer solution change ph(Appreciably?)

Consider a buffer solution containing weak acid $\ce{CH3COOH}$ and its salt $\ce{CH3COONa}$: \begin{align} \ce{CH3COOH &<=> CH3COO- + H+} \tag{1}\\ \ce{CH3COONa &-> CH3COO- + Na+} \tag{2} \end{align}

These two reactions take place and now we are majorly left with $\ce{H+}$, $\ce{CH3COOH}$, $\ce{CH3COO-}$, $\ce{Na+}$ in the solution. Since, $\ce{Na+}$ is a spectator ion, we can ignore it.

We know that buffers resist change in pH on adding small quantities of $\ce{H+}$ and $\ce{OH-}$. Let us add $\ce{H^+}$ to this solution.

A buffer solution must contain some things that could remove these $\ce{H+}$ ions, otherwise the pH would change. As we add $\ce{H+}$ to this buffer solution, it will react with $\ce{CH3COO-}$ to form $\ce{CH3COOH}$.

But here's the catch.

$\ce{CH3COO-}$ is getting removed and $\ce{CH3COOH}$ is getting formed. According to Le Chatelier's principle (LCP), the equilibrium should shift towards in the forward reaction (in reaction (1)). Shifting of equilibrium in the forward reaction should increase the concentration of $\ce{H+}$ and hence decrease the pH of the solution.

Now, one might say that $\ce{CH3COOH}$ is a weak acid and it would not decompose much. So, we can actually ignore the amount of $\ce{H+}$ so generated.

Let's "assume" that's true for a while and move ahead.

Let us add $\ce{OH-}$ ions to the solution.

Remember there's $\ce{H+}$, $\ce{CH3COOH}$, $\ce{CH3COO-}$, $\ce{Na+}$ present in the solution. $\ce{OH-}$ can react with either (a) $\ce{H+}$, or (b) $\mathrm{CH_3COOH}$.

Here comes the tricky part.

Let us only consider hydroxide ions reacting with $\ce{H+}$

If $\ce{OH-}$ reacts with $\ce{H+}$, water will be formed.

Here although hydroxide ions are getting removed, $\ce{H+}$ are also being removed from the solution. Now, because of the removal of $\ce{H+}$ shouldn't the solution become less acidic and the pH should increase.

Now, some might say as soon as $\ce{H+}$ is removed, the equilibrium in the first reaction shift towards the right.

But wait, didn't we just ignore that (in the case when we added $\ce{H+}$ in the solution)?

Can anyone explain why is there a contradiction?

P. S.: I have ignored the $\ce{H+}$ and $\ce{OH-}$ from water because it is irrelevant to the discussion.