A group of University of Minnesota Law School instructors administered popular artificial intelligence bot ChatGPT four law school exams that it took alongside real students, and discovered that the bot was a C+ student on paper.
The ChatGPT bot is currently considered the world’s most advanced example of generative intelligence.
The instructors were curious about ways the bot might be used both to help students cheats and professors teach, so they decided to test its abilities.
The AI robot passed four law classes in the bottom percent of the class at the University of Minnesota Law School (pictured)
Jonathan Choi, the University of Minnesota associate professor of law whose idea it was to administer several law school tests to the ChatGPT bot
Jon Choi, an associate professor of law, ran exam questions through ChatGPT and mixed the resultant exams in with students’ tests.
The bot ended up with a ‘low but passing grade’ in Constitutional Law, Employee Benefits, Taxation, and Torts. All told, it wound up with a C+ average.
According to the paper, the bot came in 36th out of 40 students in Con Law, 18th out of 19 in Employee Benefits, 66th out of 67 in Tax, and dead-last out of 75 in Torts.
Ultimately, the bot performed better on essays than on multiple choice questions, and performed particularly poorly at multiple choice questions involving math, which the paper’s authors say it ‘bombed.’
The paper’s authors explained that in essay questions, the bot was able to adequately recite legal rules and correctly describe cases (without proper citations), but was minimally skilled at ‘spotting issues’ and providing in-depth analysis.
The three human graders, who are listed as the co-authors of the paper, graded the tests blind but said they had suspicions about which were AI generated exams.
In all cases, they were correct about which ones were not written by humans.
Robot answers, they noted, contained perfect grammar and were much more repetitive than human exams.
Choi told Business Insider that, for the time being, it will remain relatively easy for professors to spot a ChatGPT generated exam or paper. But, a ChatGPT-generated rough draft of a paper that a student then edits and revises may prove a more difficult challenge.
The point remains, however, that ChatGPT passed the exams, albeit not with flying colors.
‘Although ChatGPT would have been a mediocre law student, its performance was sufficient to successfully earn a JD degree from a highly selective law school,’ reads the paper.
ChatGPT parent company OpenAI recently received a multi-billion dollar investment from Microsoft to continue building its intelligent tool
Two of the listed co-authors of the paper, who successfully identified which tests were written by the AI robot
The ChatGPT bot was given four law school exams at the University of Minnesota that it took alongside real students. It ultimately passed in the lowest quartile of the class
Choi is also not discounting the fierce competition the bot encountered in Minnesota.
‘I would say just on knowledge of the law and the ability to synthesize and make legal arguments, Minnesota students are going to be some of the best in the country,’ Choi told BI.
‘ChatGPT is performing near the bottom of a class in which 99% of the students will pass the bar exam. Almost all of the students are going to be successful in their careers. So, it’s a tough comparison.’
Choi is also deeply aware that ChatGPT is likely not going away and should not be ignored, especially if practicing attorneys begin to use it in their own law practices.
‘If lawyers start to use these tools to augment their abilities to be lawyers, then we need to appropriately change our teaching and our tests,’ he said.
‘When they invented calculators, they didn’t force students to continue doing math by hand. They allowed them to use calculators and just changed the nature of the test. So I think that’s the task for legal education,’ he said.
Meanwhile, the company behind ChatGPT – OpenAI – just received a massive multi-billion dollar injection of cash from Microsoft, such that it may, in its own words, continue to develop its ‘increasingly safe, useful, and powerful’ product.
Read the full article here
