Discover more from Econ Soapbox
Econ Soapbox vs ChatGPT
A professor and a machine enter, only one leaves
Last month I wrote about the 2015 almost-closure of Sweet Briar College (read it here). The attempted closure of Sweet Briar is one of my favorite academic anecdotes. The absolute lunacy and dereliction of duty by the Board of Directors will be retold in academia for decades. My Substack post is also based entirely on media sources. I don’t know anyone who was working at Sweet Briar at that time and have no inside knowledge. My only “edge” is my insight into how academia works in general.
My most popular post over the last few months was when I had the AI software ChatGPT take the final exam I gave in my environmental economics class (read that post here). Overall, ChatGPT did fantastic on that exam. Most answers were perfect, and all were well-written. Limitations on answering graph-based questions aside, ChatGPT appeared to be an A-student. The only oddity was an AI “hallucination”, where the software made up an entire person and narrative that would be a correct answer if only it wasn’t totally fictional.
This post will be a combination of sorts of the Sweet Briar and ChatGPT Final Exam posts: what happens when you ask ChatGPT to write an article about the 2015 Sweet Briar controversy? More importantly, which article is better, mine or the machine’s?
Econ Soapbox is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.
This should be right in ChatGPT’s wheelhouse. The Sweet Briar almost-closure got a lot of media attention, from both traditional outlets as well as on online forums. There were a lot of narratives at play: the future of women’s colleges, the future of higher education in general, an opaque board against the university community, etc. Thus, there should be plenty of information for ChatGPT to draw from. Additionally, nothing in my post was about underlying economic theories or concepts. It was just a retelling of a story. In other words, this challenge should play to ChatGPT’s strengths.
I gave ChatGPT two prompts. You can see both prompts and full responses here, but I’ll provide highlights below. The first prompt was, “Please write a 1,000-word post on how Sweetbriar almost closed in 2015 but then didn't.” ChatGPT’s response started pretty good:
In the annals of higher education, Sweetbriar College's tale stands as a testament to resilience and the power of collective determination. In 2015, this prestigious women's liberal arts college, nestled in the serene hills of Virginia, faced an unprecedented crisis that threatened its very existence. Faced with financial challenges and declining enrollment, the institution was on the brink of closure. However, against all odds, Sweetbriar managed to avoid this fate, emerging stronger and more determined than ever before. This article delves into the story of Sweetbriar's near-closure in 2015 and the remarkable turnaround that saved the college.
This sends shivers down my spine as a college professor. It is well written and coherent. Easily superior to the average essay I read from a student. The interesting error, from an AI standpoint, is that I incorrectly called the college Sweetbriar (one word), rather than Sweet Briar (two words) in the prompt, and ChatGPT repeated the mistake throughout the response.
Most of the rest of the response is also accurate. It correctly identifies the Board as the group seeking to close the school and Saving Sweet Briar as the organization of alumnae devoted to saving it. However, this part of the response is completely wrong:
Amidst mounting pressure, a significant breakthrough came when the Virginia Attorney General, Mark Herring, sided with the Saving Sweetbriar movement. Herring argued that the board of directors had violated its fiduciary duty by attempting to close the college without exhausting all possible alternatives. The court's subsequent ruling further bolstered the movement's resolve, allowing Sweetbriar to remain open and suspending any actions toward its closure.
Mark Herring was the attorney general, but he did not take the side of Saving Sweet Briar. Instead, he filed an amicus brief that agreed that the Board of Directors had the right to close the school. Herring did eventually help broker the deal that would lead to members of the Board resigning and members of Saving Sweet Briar taking control of the school, but it would be inaccurate to say that he “sided” with the Saving Sweet Briar movement. Also, there was not a court ruling that gave Saving Sweet Briar control of the Board, but a mediation agreement.
So ChatGPT spit out a largely correct response that has some errors. Along with those stated above, ChatGPT also said the school now has rising enrollment. This needs context; enrollment has risen since the 2015-2016 academic year when many students left in the wake of the controversy, but is still below the pre-attempted-closure 2014-2015 enrollment.
ChatGPT, like Google, is a tool that needs to be used appropriately. I decided to try the prompt again, but this time ask for a longer response (to match my own Substack post) and with more detail. I asked, “Write a 1,500-word engaging narrative on how Sweet Briar College almost closed in 2015 but then didn't. Explain who tried to close it, why it was surprising, and what happened instead.”
Unsurprisingly, ChatGPT’s response has Sweet Briar as two words in response to the second prompt. The introduction is different from the first but even more impressive:
Nestled in the rolling hills of Virginia, Sweet Briar College had long been a bastion of women's education. However, in 2015, the college faced an unimaginable threat. The very institution that had nurtured generations of brilliant minds and empowered women was on the verge of closure. The shocking decision to shutter Sweet Briar College came from an unexpected source, leaving the college community and supporters around the world stunned. Yet, what transpired in the aftermath of this announcement would prove to be a remarkable tale of resilience, determination, and the unyielding spirit of Sweet Briar.
Again, shivers. This is easily college-level writing. A bit melodramatic, but better than most of the writing I read as a professor. The rest of the response, again, is mostly accurate. ChatGPT identifies the fight as being between the Board and Saving Sweet Briar. The financial issues are discussed. That said, the legal aspect is still completely wrong:
In a surprising turn of events, the Virginia Attorney General, Mark Herring, threw his support behind the Saving Sweet Briar movement. Herring contended that the board had indeed failed in its fiduciary responsibilities and that alternative measures should have been explored. This unexpected ally bolstered the movement's resolve, injecting renewed hope into their battle for survival.
The courtroom became the battleground where the fate of Sweet Briar College hung in the balance. Ultimately, the court ruled in favor of the Saving Sweet Briar movement, delivering a resounding victory that echoed through the halls of the college and beyond.
So, which is better? Obviously, I’m not an objective observer, but I think I beat the machine. To my knowledge, there are no factual errors in my post, while ChatGPT has some humdingers.
Additionally, ChatGPT responses lack context and bigger-picture type thinking. ChatGPT can retell a narrative but is going to answer the question being asked in a direct manner. Humans are better (for now) and generating unprompted linkages and providing loose context. That said, it’s alarming how good ChatGPT is at providing a direct response.
As a college professor, this software is going to necessitate a big rethink in how I ask essay questions. It also leads to a more existential question: should ChatGPT be primarily viewed as a way to cheat or a tool that saves time? In other words, with the availability of ChatGPT, does this fundamentally diminish the importance of being able to write well? After all, being able to multiply 43*58 in your head is a cool party trick, but isn’t in any way a necessary skill in today’s world. All of our teachers were wrong when they told us, “You have to learn this because you won’t always have a calculator in your pocket as an adult!” Is ChatGPT a new version of the calculator?
So far I think (and hope) not. Being able to express yourself well, whether in writing or speaking, will always be a valuable skill. The key will be to use ChatGPT to enhance your writing, not replace it. Just as a calculator doesn’t make you a good mathematician, ChatGPT isn’t going to make you a great writer.