MTC: ⚖️ ChatGPT and the Supreme Court: Two Years of Progress in Legal AI ⚖️
/What can we learn about the evolution of generative aI in its ever growing analysis of the supreme court?
Ed Bershitskiy’s recent SCOTUSblog article, “We’re not there to provide entertainment. We’re there to decide cases,” offers a compelling analysis of how ChatGPT has evolved since its launch in 2023, particularly in its application to Supreme Court-related questions. The article highlights both the successes and shortcomings of AI models, providing valuable insights for legal professionals navigating this rapidly advancing technology.
In 2023, the original ChatGPT model answered only 42% of Supreme Court-related questions correctly, often producing fabricated facts aka “hallucinations” and errors. Fast forward to 2025, newer models like GPT-4o, o3-mini, and o1 have demonstrated significant improvements. For instance, o1 answered an impressive 90% of questions correctly, showcasing enhanced accuracy and nuanced understanding of complex legal concepts such as non-justiciability and the counter-majoritarian difficulty. Krantz’s analysis also underscores the importance of verifying AI outputs, as even advanced models occasionally produce mistakes or hallucinations.
Always Check Your Work When Using Generative AI - It Can Create Hallucinations!
🚨
Always Check Your Work When Using Generative AI - It Can Create Hallucinations! 🚨
The article compares three distinct AI models: GPT-4o is detail-oriented but prone to overreach; o3-mini is concise but often incomplete; and o1 strikes a balance between depth and precision. This comparison is particularly relevant for legal professionals seeking tools tailored to their needs. For example, GPT-4o excels at generating detailed narratives and tables, while o1 is ideal for concise yet accurate responses.
Lawyers are not going to be replaced by ai but those lawyers who do not know how to use ai in their practice and mindful of its constant changes will be left behind!
Krantz also explores how the line between search engines and AI-powered tools is blurring. Unlike traditional search engines, these AI models analyze queries contextually, offering more comprehensive answers. However, legal practitioners must exercise caution when relying on AI for research or drafting to ensure ethical compliance and factual accuracy - in other words, always check your work when using AI!
As AI continues to evolve, its role in legal practice is becoming indispensable. By understanding its strengths and limitations, lawyers can leverage these tools effectively while safeguarding against potential risks. Krantz’s article provides a detailed roadmap for navigating this technological transformation in law.
PS: I can’t stress enough to always check your work when using AI!
Happy Lawyering!
MTC