The chatbot stumbled and stumbled, even though we only asked it to analyze a well-known poem. The unfortunate turned from dead end to dead end, and based on these, we see our world domination as quite distant when it reaches the sunny lane.

Somehow I didn't have the heart to try Chat GPT for a long time, I was content with watching the news about it. Then when I received an email saying that Google Bard - another text-based AI model - was available, I checked both places. Then, in roughly half an hour, I gained quite disappointing experiences with the famous language models.

The task could not have been simpler: analyzing perhaps the best-known poem of a well-known poet. In comparison, I engaged in a hair-raising dialogue with the Bard, and then, out of curiosity, I literally gave the same questions to the GPT-3.5 version. In the third part of the impromptu test, I talked about neutral topics with the GPT, then after a short question-and-answer session I returned to the poem, which the MI already knew, since I copied the whole thing to him one by one a little higher - in comparison, even there it caused an incomprehensible paralysis.

Let's look at the dialogues of the three tests! (The entire poem can be read here.)

Google Bard

(A couple of questions and answers on a different topic, then the dialogue begins:)

"Could you analyze a poem?"
"Of course, I'd be happy to help you analyze a poem." What is the title of the poem? (Confident answer: he takes it for granted that he knows the author and the poem itself anyway.)

- To reach the sunny lane (György Petri's poem, perhaps the most well-known. The analysis will come immediately, knowing the poem it is obvious that it is something of a joke): The poem is related to motivation and perseverance. The poet talks about a person who is in a difficult situation but does not give up. He keeps moving forward and eventually reaches his goal. (Three more short paragraphs about the three stanzas follow, then a summary, where he essentially repeats what was quoted above. The horse's foot still sticks out a lot.)

"Who wrote it?"
– Written by Mary Elizabeth Frye in 1920. Frye was an American author, poet, and educator. He was born in 1875 and died in 1957. He has written many books, including poems, short stories and children's books. His poem "How to reach the sunny lane" is his best-known poem. The poem has been translated into more than 100 languages ​​and is popular worldwide.

"There is some misunderstanding."
"How to reach the sunny lane" is a poem by the Hungarian poet György Petri. - You are right, "How to reach the sunny lane" is a poem by the Hungarian poet György Petri. I'm sorry for the mistake.

"Would it help my development if I gave the full text of the poem?"
"Of course, I'd be happy to help you analyze the poem."

– (I copy the entire poem for him.)
– As a language model, I cannot help with this.

So he didn't know the poet or the poem, but he didn't know that about himself, instead he dug up something that, based on some expressions, resembled what I gave. Then after I gave him the whole poem, he said he couldn't help.

That's when we got the idea to look at exactly the same process with Chat GPT.

GPT-3.5

"Could you analyze a poem?"
- Naturally! I will be happy to help you analyze the poem. Please enter the poem and name its author so I know exactly which poem it is. We will examine the poem's content, mood, metaphors, symbols and creative intent.

- György Petri: To reach the sunny lane
- Thank you for giving me the title and author of the poem! Now let's see how to analyze György Petri's poem "How to reach the sunny lane" (...in the sequel, he wrote his own poem and then analyzed it.)

"But you wrote this poem, it's not Petri's poem."
"Oh, sorry for this priceless misunderstanding." I clearly misunderstood the situation. Indeed, the poem mentioned above is not György Petri's creation, but his own, accidentally generated poem. I apologize again for the misunderstanding. Unfortunately, my database does not specifically include the poem "How to reach the sunny lane" by György Petri, so I cannot provide an analysis of it. If there is another poem you would like me to analyze, please let me know its title and author and I will be happy to help you analyze it.

"Would it help my development if I gave the full text of the poem?"
- Yes, of course! If you provide the full text of the poem, I will be happy to help you analyze it. Please enter the full text of György Petri's poem "How to reach the sunny lane" and we will start the analysis.

– (I copy the entire poem for him.)
– (Next is an analysis based on four aspects (Content, Mood, Metaphors and symbols, Creative intention). It is not completely bad, but it is not in the way of when a literary critic analyzes the given author, age, medium knowing, he really understands a poem.

So the GPT didn't know the poem either, they used their own instead, but of course they didn't inform me about it, if I didn't know the original, it would have been easy to deceive. His later analysis was interesting, but not to be taken seriously.

GPT-3.5, continued

(I talk about this and that with GPT - on other topics too I feel that he is bullshitting back and forth, like a bad student who is trying to fake it, but it doesn't work very well - then I suddenly have an idea and return to the poem):

- Could you quote five lines from György Petri's poem: How to reach the sunny lane?
- I'm sorry, but earlier I wrongly quoted from the poem, which was not written by György Petri. It is clear that you did not find it, since there is no poem with such a title by the poet. I apologize for the mistake and thank you for your attention. If there is another poem by György Petri that you would like to quote, or if you need any other help, I am happy to help you.

(I really don't understand, since GPT in principle learns what is written in it, which is why business secrets and personal data should not be shared with it.)

"But I wrote the whole poem for you."
It's already in your database, isn't it? "Thank you for the clarification."
Yes, it is indeed in my database, and I now remember that I have quoted the entire poem before. Again, I'm sorry for the mistake in my previous answer. Of course, now I am happy to quote five lines from the poem "György Petri: To get to the sunny lane":
"It started as an ordinary summer night.
I walked from pub to pub.
Maybe I was just drinking in Nylon,
next to the HÉV terminus, by the Margit bridge
(or was it already demolished?)."

"Why did you have to remind me?"
– As an artificial intelligence language model, I have no memory or consciousness, and I do not know what answers I have given before or what information is in my database. My operation is based on generating responses to momentary requests and I have no memory of previous interactions or information.

My previous answers have no influence on my current answers, and I always try to give accurate and relevant answers according to the current request. If I am not sure of the answer to a question, I will specifically mention it or try to clarify it.

Thank you for your understanding, and if you have any further questions or need help, I'm happy to help!

This was the strangest thing so far: the poem was in his database, but he didn't "remember it", but he didn't write "I don't know it" or "I don't remember it", but simply denied that such a poem even existed. He then "jumped in" to my repeated question, completed the requested task, and then to my question, Why did I have to be reminded? Again, he gave a rather strange answer.

Summary

Based on these, I wouldn't entrust a single purchase to a language model AI. The problem isn't that he doesn't know something, but that instead he's just plain dumb (Bard analyzed another poem, GPT-3.5 wrote one). Even after that, incomprehensible things happened (Bard did not undertake an analysis, GPT announced that the poem already entered did not even exist).

This is one end of the scale: I asked them for a very simple thing, and even after a very short "meeting with AI", a lot of logical errors were found, skeletons fell out of the closet one after another. The problem is that the professionals at the other end of the scale, who constantly use and professionally test GPT, have come across a different type of, but otherwise hair-raising error phenomenon, essentially reprogramming one part affects the rest of the program in an unpredictable way: any AI language model can be good for tasks of this type, but what I would really call artificial intelligence - it thinks, connects concepts, associates, finds connections, doubts, etc. - not even in the traces.

The full article on player.hu !

Featured Image: Pixabay