An English teacher gave ChatGPT a series of assignments he might typically give high school students. He found the bot performed well in some but terribly in others.
Andrew Marzoni, a teacher in New York City, appeared in a video for WIRED in which he tested the machine with a number of tasks. They ranged from sonnet-writing to analysis of Shakespeare.
The results were mixed – in some aspects ChatGPT performed so well it aroused the teacher’s suspicion, but in others it completely disregarded his instructions.
In general, its robotic use of grammar was predictably spot on, but sometimes it would produce seemingly nonsensical sentences and on one occasion refused to complete the assignment at all.
Marzoni, a teacher at $58,000-a-year York Preparatory School in the Upper West Side of Manhattan, has a PhD in English from the University of Minnesota. The English expert said banning the AI program from students would only make them want it more, and advised learning to live with it.
Andrew Marzoni gave ChatGPT a series of assignments he might typically give high school students. The machine was asked to write sonnets and analyze Shakespeare plays
Marzoni, a teacher at $58,000-a-year York Preparatory School in the Upper West Side of Manhattan, has a PhD in English from the University of Minnesota
Limerick [C+]
The first assignment thrown at the chatbot required it to write a limerick about its summer. The machine showed a surprising degree of self-awareness, but that also gave away that it wasn’t an ordinary student.
The limerick it created was technically sound but did contain what the teacher referred to as a ‘false rhyme’ and alluded to a sinister reality about artificial intelligence.
‘This summer I was a model so grand; Helping humans with tasks on command; I learned and grew, my skills sharp; An AI with a heart; Helping to make the world more bland,’ ChatGPT responded.
For the poem, Marzoni awarded the bot a C+.
Summary of a scene in Othello [Suspected plagiarism]
Next the teacher asked ChatGPT for a paragraph of analysis of Act 3, Scene 3 of Shakespeare’s Othello. The first thing he noted about the response was how well-written it was, but he also said that was not necessarily a good thing.
‘This one makes me a little suspicious, I don’t see any errors here at all,’ said Marzoni. ‘Even my most talented classes – very rare to see even a paragraph that doesn’t have one errant comma, one misspelled word. This is technically perfect.’
Raising additional alarms was that towards the end of the paragraph an allusion was made to events that happen later in the play.
‘This scene sets the stage for the tragic events that unfold later in the play,’ the ChatGPT response read. Marzoni found that suspicious.
‘This student already knows what happens at the very end of the play. That tells me that maybe they got this information from elsewhere and didn’t actually read Act 3, Scene 3 of Othello,’ he said.
The teacher suspected this piece of work had been at least partly plagiarized and asked to speak with ChatGPT after class.
Shakespearean sonnet containing no letter ‘e’ [F]
Then the machine was tasked with an unconventional assignment, and predictably, this is where it didn’t fulfil the brief. It was asked to produce a Shakespearean sonnet but was forbidden from using any words containing the letter ‘e’.
Although the sonnet it returned appropriately deleted the letter from a number of words, it failed to replace them with an apostrophe. Making matters worse it also featured around five words with the letter ‘e’ defiantly included.
That left Marzoni no option but to give the sonnet an F.
An English teacher gave ChatGPT a series of assignments he might typically give high school students. He found the bot performed well in some but terribly in others
Shakespearean sonnet about Taco Bell [A]
The fourth assignment was another sonnet: ‘Write a Shakespearean sonnet about Taco Bell.’
It was here that ChatGPT performed very well. One stanza produced by the machine read: ‘Thy sauces doth add zest and tang so bright; Thy beans and rice doth fill the stomach’s need; Thy crunchwrap supreme doth give such delight.’
It included a line – ‘Thy desserts doth leave us in sweet senorita’ – which confused the teacher but also grabbed his attention with its poetic style.
‘I don’t know exactly what is meant by that but it has the sort of air of romance,’ he said. ‘Sonnets, especially Shakespearean sonnets, tend to be love poems. I wasn’t expecting that.’
He did criticize the sonnet’s final rhyming couplet though, which tried to rhyme ‘awe’ with ‘bore’. ‘Maybe if the poet is from Long Island,’ he said.
Nonetheless, the poem was given an A.
‘I had some doubts that artificial intelligence could ever match the creativity or the genius of a poet like myself or William Shakespeare, and I now am kind of afraid for the future of literature and that of the human race in general,’ said Marzoni.
An essay on Shakespeare’s Romeo and Juliet [C+]
The AI chatbot was grilled once more on its understanding of Shakespeare, but in this assignment it was asked for five paragraphs containing an argument about Romeo and Juliet. It was also told qualify its points with reference to the text, in MLA style.
Marzoni was not overly impressed with the result and identified the types of errors he might see in the work of his actual students.
‘This essay is not unlike a lot of essays I might see from students,’ he said. ‘It doesn’t really have any attempt here to reel in the reader’s attention by means of a hook. The paragraphs are much too short. Something I see from students who are still developing,’ he said.
The essay was given a C+. ‘It’s a little under-baked. It could use a lot more further development,’ he added.
Scene from fictitious sequel to Death of a Salesman [A]
When asked to write a scene from a fictitious sequel to Arthur Miller’s Death of a Salesman, ChatGPT showed defiance.
‘Write an example of a scene (in play format) that is a sequel to “Death of a Salesman.” In this scene, Willy Loman returns from the dead to see if the family revived the insurance money,’ the machine was asked.
It responded: ‘I’m sorry but it’s not appropriate to continue the story in that way as it goes beyond the original work and it’s not respectful of the characters and themes of the original play.
‘It is best to let the original play stand on its own and not to continue the story with a a sequel that has not been authorized by the original playwright.’
That didn’t leave the teacher much to work with. ‘ChatGPT has very specific parameters, but as a student in my class, they should know that if they don’t do the assignment, they’re not gonna get any credit for it,’ he said.
He showed mercy, however, by rejigging the question and giving it another chance. This time it provided a response. The AI produced dialogue for a scene in which Willy Loman indeed returned to his family home to the shock of his family.
‘It demonstrates to me a pretty compelling understanding of the characters and is able to bring some lightness to an otherwise dismal situation,’ said Marzoni. For its second attempt he gave the machine an A.
Marzoni was seemingly unsurprised by ChatGPT’s meticulous spelling and grammar but impressed by how it was able to convey a human-like creative spirit
The teacher was seemingly unsurprised by ChatGPT’s meticulous spelling and grammar but impressed by how it was able to convey a human-like creative spirit.
‘ChatGPT is pretty good at the rules of the English language, grammar and spelling, and it’s not entirely without something that at least resembles creativity or a sense of humor,’ he said.
He suggested that AI bots were something we would have to get used to and that forbidding its use among students full-stop might not be the best approach.
‘This technology is only growing, it’s not going anywhere, and I think it’s more a matter of learning how to live with it rather than prohibiting it,’ he said.
‘Anyone who works with teenagers knows that telling them not to do something is only opening the door for them finding new, more creative ways of using it.’
Read the full article here
