rishonim - Which sins are beyond punishment?

Yishai said in a comment elsewhere on this forum :

It has been pointed out in Rishonim that some sins are so bad they can't get a punishment - because the punishment creates forgiveness, but when the sin is too great, forgiveness is not an option.

1. Which sins are these and who are the Rishonim that said this?


2. How could a sin be so great so as to be beyond punishment, when even the gravest sins are not [they are punished by capital punishment]?

Answer

I found a case attributed to the Semag (one of the last ba'alei Tosfos) where a sin is so heinous that not only is forgiveness not an option, but we don't administer capital punishment since putting this person to death won't serve as atonement (ie we let him live).

But first some background.

There's a prohibition of offering one's children to Molech, a type of avodah zara. Leviticus 18:21 states:

וּמִֽזַּרְעֲךָ֥ לֹא־תִתֵּ֖ן לְהַעֲבִ֣יר לַמֹּ֑לֶךְ וְלֹ֧א תְחַלֵּ֛ל אֶת־שֵׁ֥ם אֱלֹהֶ֖יךָ אֲנִ֥י יְהוָֽה׃

Do not allow any of your offspring to be offered up to Molech, and do not profane the name of your God: I am the Lord.

Sanhedrin 64a discusses this in more depth, lots of rishonim weigh in, and Sefer Chinuch, Mitzvah 208 codifies this. The punishment for passing one's child to Molech is death by stoning.

Continuing the discussion about Molech, the gemara (Sanhedrin 64b) proceeds to make a shocking(!) inference:

אמר רב אחא בריה דרבא העביר כל זרעו פטור שנאמר מזרעך ולא כל זרעך

Rav Aḥa, son of Rava, says: [One who] passed all his offspring [through fire to the priests of Molekh is] exempt, as it is stated: “of your offspring,” [indicating:] But not all your offspring.

The Semag (סמ"ג · לאו · מ) offers a WILD rationale for this reasoning.

העביר כל זרעו פטור שנאמר כי מזרעו נתן למולך מקצת זרעו ולא כל זרעו ויש טעם בזה לתשובת המינים מפני שבמיתת ב"ד מתכפרין המומתין וזה עשה כל כך עבירה גדולה שאין הקב"ה רוצה שיהא לו שום כפרה

If one passed all of his children [to Molech] he is exempt [from being put to death via beis din], because it says [a person who gives] from his seed to Molech. ie some of his seed, not all of his seed, and there is a reason for this to answer the heretics, namely that death via beis din redeems the said person who was put to death, and this person [who passed ALL of his children to Molech] did such a huge aveirah that Hashem doesn't want him to receive any redemption at all [ie via being put to death via beis din]

Bottom line: the Semag says the punishment of stoning is only for a person that gives "some" of his children to Molech. But if that person gave "all" of his children to Molech, it's such a big aveirah that there is no forgiveness and we don't give him a kaparah by putting him to death.