Almost similar looking sentences with and without に particle

I am having confusion with the follow two sentences. Currently I am preparing JLPT N5 so please answer accordingly (I can't understand very advanced explanations).

私は日曜日国へ帰ります

私は誕生日に国へ帰ります

So there are expressions of time where we should use に and some where we shouldn't . Can anyone classify for which expressions of time we should に particle and where we shouldn't. I read that for specific "time" we should use に but then also it seems like 日曜日 is a specific time. So why are we not using に particle here. These examples are from a book.