Thursday, 4 September 2025
27.9 C
Singapore
29.6 C
Thailand
19.4 C
Indonesia
28.3 C
Philippines

New research highlights ChatGPT’s struggles in helping with coding tasks

New research from Purdue University reveals significant errors in ChatGPT's programming assistance, emphasising caution and calling for further study.

According to recent research, ChatGPT is still grappling with effectively assisting with programming issues despite becoming an overnight sensation. While many developers have turned to generative AI tools like GitHub’s Copilot to streamline their workflow and free up time for more productive tasks, a new study from Purdue University sheds light on significant shortcomings in ChatGPT’s performance.

Study reveals widespread errors

Researchers at Purdue University analysed 517 questions from Stack Overflow, comparing ChatGPT’s answers to those provided by human experts. The findings were startling: more than half (52%) of the responses generated by ChatGPT were incorrect. The breakdown of errors is as follows: 54% were conceptual misunderstandings, 36% were factual inaccuracies, 28% were logical mistakes in code, and 12% were terminology errors.

The study also highlighted that ChatGPT often produced unnecessarily lengthy and complex responses. This overabundance of detail can lead to confusion and distractions for developers seeking straightforward answers. Despite these issues, an ultra-small-scale poll involving 12 programmers revealed that one-third preferred ChatGPT’s articulate, textbook-like responses. This preference underscores how easily the AI’s seemingly authoritative tone can mislead coders.

Implications for the coding community

The implications of these findings are significant. Errors in coding can cascade, potentially causing problems across multiple departments or even entire organisations. The researchers emphasise the importance of caution when using ChatGPT for programming tasks.

They state, “Since ChatGPT produces many incorrect answers, our results emphasise the necessity of caution and awareness regarding the usage of ChatGPT answers in programming tasks.” This caution is vital to prevent minor coding errors from escalating into more significant, complex issues.

Call for further research and transparency

Beyond urging caution, the researchers advocate for further studies to identify and mitigate these errors. They also call for greater transparency and communication regarding the potential inaccuracies in ChatGPT’s responses. This openness is crucial for developers to make informed decisions about when and how to use AI tools in their workflows.

As the coding community continues to integrate AI into its practices, these findings serve as a reminder of the limitations and risks associated with relying too heavily on automated tools. While ChatGPT and similar technologies offer exciting possibilities, their current capabilities require scrutiny and responsible use to ensure they genuinely enhance productivity without introducing significant errors.

Hot this week

Microsoft AI launches first in-house models, MAI-Voice-1 and MAI-1-preview

Microsoft launches MAI-Voice-1 and MAI-1-preview, its first in-house AI models, focusing on advanced voice and text-based capabilities.

Thinking Machines partners with OpenAI to accelerate AI adoption in Asia Pacific

Thinking Machines partners with OpenAI to expand enterprise AI adoption across Asia Pacific with training, app design, and leadership programmes.

Kobo introduces Instapaper integration to replace Pocket on e-readers

Kobo replaces Pocket with Instapaper on its e-readers through a free firmware update, ensuring users maintain a seamless read-it-later experience.

Plaud.ai introduces Note Pro with smarter recording and AI-powered features

Plaud.ai unveils the Note Pro, a smarter AI note-taking device with improved recording, automation, and app integration, launching in October 2025.

OpenAI and Anthropic conduct cross-company AI safety evaluations

OpenAI and Anthropic evaluated each other’s AI systems, revealing safety gaps and stressing the need for stronger safeguards in the industry.

Singapore pushes for AI fluency to boost workforce competitiveness

Singapore is urging workers to become fluent in AI, with new training pathways and government support to boost workforce skills.

Amazon launches new AWS region in New Zealand

Amazon launches its first AWS infrastructure region in New Zealand, investing NZ$7.5b to boost jobs, cloud services, and sustainability.

Global Anti-Scam Summit Asia 2025 launches major initiatives to fight online fraud

Global Anti-Scam Summit Asia 2025 in Singapore unveils new initiatives to fight scams with technology, funding, and cross-border collaboration.

Google Play Games to introduce new profiles with stats and social features

Google is introducing new Play Games profiles on Android, featuring gaming stats, achievements, and social tools, rolling out from 23 September.

Related Articles

Popular Categories