Shalom,
I would like to know if Jews consider it acceptable to pray to deceased ancestors, forefathers, or "saints" like some Christians in the Catholic tradition do. Deuteronomy 18:11 would seem to prohibit speaking with the dead or praying to them, and several pagan religions have "ancestor cults" where they pray to their predecessors. Therefore I would expect it to be explicitly forbidden in Judaism to pray to any of the dead, whether they were righteous or wicked.
However, I was reading in Sotah 34b and I noticed that it reads:
Raba said: It teaches that Caleb held aloof from the plan of the spies and went and prostrated himself upon the graves of the patriarchs, saying to them, 'My fathers, pray on my behalf that I may be delivered from the plan of the spies'. (As for Joshua, Moses had already prayed on his behalf; as it is said: And Moses called Hoshea the son of Nun Joshua,17 [meaning], May Jah save thee [yoshi'aka] from the plan of the spies.)
This seems like mere speculation on the Rabbi's part and is an interpretation that appears to conflict with Deuteronomy 18:11. Do Jews take Sotah 34b as an example of legitimate prayer to deceased ancestors? And if so is there any teaching in Judaism that the deceased can hear when someone calls to them?
If Judaism allows praying to ancestors then how would that be distinguished from the forbidden practice of necromancy, and the episode of King Saul consulting the sorceress at Endor to speak with the prophet Samuel?
For a long time there have been Jews who have indeed beseeched the dead. However, numerous sources state this is is prohibited Some state that it is permitted if the request is not directly from the dead, but just that the dead beseech God.
Forbidden when the request is directly of the deceased
Maharam Shikk writes in a responsum (OH 293) that such practices raise both the issue of doresh el hametim (Deut. 18:11) and of serving God through an intermediary (apparently the prohibition of avoda zara; idolatry). He concludes that if one relates one's problems to the dead hoping that they will intercede with God it is permissible, but if one wants help from them directly, it would be forbidden. He emphasises that even using them as intermediaries is forbidden according to many authorities.
Similarly, the Ben Ish Hai writes in a responsum (Rav P'alim Vol II YD 31) that it is forbidden to make requests of a dead person directly. Doing so constitutes doresh el hametim. One may only ask that the dead intercede with God. He writes this in explanation of the Zohar (Acharei Mot: 71) which is quoted by some as a source to permit praying to the dead:
נמצא לא הותר לאדם להתפלל על קברי הצדיקים, לשאול בקשתו מן הצדיקים עצמן, כי זה חשיב דורש אל המתים, אלא הוא מתפלל ושואל בקשתו מהקב"ה, ורק מבקש מנפש הצדיק שתתפלל עליו לפני הקדוש ברוך הוא, שישמע תפלתו ויעשה בקשתו
Always forbidden
Others, take an even harsher stance; that it forbidden even if the request from the deceased is just that it intercede with God. R. Moshe Tsuriel notes here that Rambam (Hilkhot Ta'aniyot 4:17) emphasises that prayers in a graveyard are not meant to be directed to the dead. In this, Rambam rules like Rabbi Levi Bar Hamma, in Ta'anit 16a against a contrary opinion recorded there. R. Tsuriel (there) is of the opinion that those who were accustomed to pray in various forms to the dead are following the contrary view in Ta'anit, but that halakha is not in accordance with that view.
The Sefer Hameorot (Ta'anit 16a) similarly explains that according to the latter Talmudic view (who he himself follows) addressing petitions to the dead is forbidden:
שאין לנו לבקש מן המתים שיבקשו רחמים על החיים, שהרי לא מצינו בכל המקרא בכמה צרות שאירעו לישראל, ליחידים או לרבים, שיבקשו מן המתים לבקש רחמים עליהם
Similarly, the Bach (YD 217:51) implies that prayer in a cemetery can only be permitted if it is addressed to God; not the dead. (although he cites views that even this would be prohibited, see there for more):
ודוקא להשתטח על קברי אבות ולהתפלל לפניו יתברך
Additionally, the Hokhmat Adam (Issur V'heter 89:7) writes negatively about people who discuss their problems with the dead:
ז איסור דורש אל המתים זה שמרעיב עצמו ולן בבית הקברות כדי שתשרה עליו רוח הטומאה (סימן קע"ט סעיף י"ג) ואותן נשים וכן עמי הארצות שהולכין על קברי מתים וכאילו מדברים עם המתים ואומרים להם צרותיהם קרוב הדבר שהם בכלל זה
And [regarding] those women and ignoramuses who go to graves and speak as though to the dead and tell them their woes, and it seems likely that they are included in this prohibition. (My translation of the bolded portion)
Maharam Shikk understands that he is saying that it is doresh el hametim.
Maharam Mintz (Responsa:79) similarly cites the opinion that making requests of the dead; even just that they intercede with God is forbidden as doresh el hametim:
כי בלאו הכי יש גדולים קראו תגר ע"ד זה וקראו ודורש אל המתים, דרוב עמי הארץ ונשים עבידי מליצים בינם ובין קונם ב"ה.
The Minhagei Maharil (Hilkhot Ta'anit 18), quoted by the Elya Rabba (581), Ba'er Hetev (OH 581:17), and the Mishna Berurah (581:27) similarly writes that one shouldn't address the dead at all:
והמשתטח על קברי הצדיקים ומתפלל אל ישים מגמתו נגד המתים השוכבים שם, אך יבקש מאת השם יתברך שיתן אליו רחמים בזכות הצדיקים שוכני עפ
That is, one mustn't address the dead or even pay attention to them, but only address God.
The Hayei Adam (138:5), and Kitsur (128:13) paraphrase the Maharil and add that addressing the dead directly would seem to violate doresh el hametim:
ויעשה הקדוש ברוך בו חסד בזכות הצדיקים אבל אל ישים מגמתו נגד המתים השוכנים שם, כי קרוב הדבר שיהיה בכלל ודורש אל המתים
Do Jews take Sotah 34b as an example of legitimate prayer to deceased ancestors
As mentioned, like non-Jews, many Jews throughout the ages have beseeched the dead. Some invoked this story in Sotah as justification. However, those authorities who prohibit such activities clarify that the passage is not a proof. For example, R. Hayim Paltiel of Magdeburg cited by the Bach (ibid), writes that the passage in Sota refers to praying to God, and merely using the grave-site as a holy location which at which it is advantageous to pray:
דלא מצינו רק גבי כלב שנשתטח על קברי אבות- היינו שמתפללים לה', ומקום קדוש גורם שתהא תפילתם נשמעת
[It is not clear to me exactly how he read the passage in Sotah; perhaps he had a variant text.]
Most sources saying it is forbidden do not mention the passage. Perhaps this is because it is aggadic and does not carry the same authority as halakhic passages. Additionally, even were the passage in Ta'anit halakhic, not every halakhic position in the Talmud is accepted. In this matter, for example, we have already noted commentators who understand the matter of addressing prayers to the dead to be the subject of a Talmudic dispute. Accordingly, the passage in Sotah could be aligned with the view in Ta'anit that permits, while the aforementioned authorities could nevertheless rule against it, in accordance with the other view in Ta'anit.
And if so is there any teaching in Judaism that the deceased can hear when someone calls to them
Although you are only asking for sources that the deceased can hear one praying according to those who invoke the passage in Sotah as a source to beseech the dead, in reality, the question exists in a vacuum; that is, even if it is forbidden to beseech the dead, they may still be able to hear the living.
An aggadic passage in Berakhot 18b indicates that the dead are aware of the goings on in the world, and can hear people talking to them. However, others are of the opinion that the dead are not aware of what is happening in their vicinity. (See R. Qafih's explanation of Rambam's Hilkhot Evel (14:13). One verse that indicates the contrary is Ecclesiastes (9:5) which states:
הַחַיִּים יוֹדְעִים, שֶׁיָּמֻתוּ; וְהַמֵּתִים אֵינָם יוֹדְעִים מְאוּמָה
The living know they will die; but the dead don't know a thing.
In a similar vein, R. Yehuda ibn Bil'am writes in his commentary to Isaiah (8:19):
הנפש אין לה שום ידיעה אחר פרידתה מן הגוף
The soul has no knowledge after its separation from the body.
Radak similarly implies that he understands the verse in Ecclesiastes simply, and that for this reason, any prayers to or through the dead, are absurd in his commentary to Isaiah (8:19):
אבל זה שאתם אומרים לנו שנדרוש אל האובות והידעונים זהו דבר שאין הדעת סובלתו כי הם מתים, ואיך נדרוש בעד החיים אל המתים, כי המתים אינם יודעים מאומה
But this that you say to us, that we should seek out the ovot and yidonim, this is something that the intellect does not accommodate, for they are dead! And how can we beseech the dead on behalf of the living, for the dead don't know anything.
If Judaism allows praying to ancestors then how would that be distinguished from the forbidden practice of necromancy
At this point I will mention some of the opinions of those who permit such practices. R. Eliezer of Metz writes in his Sefer Yereim (334-335) that the prohibition of doresh el hametim only includes involvement with the body of the deceased; involvement with the spirit of the deceased, however, would be permitted. This is one of the sources cited in justification of the practice to distinguish it from forbidden practices.