Tuesday, 16 September 2025
28.2 C
Singapore
32.1 C
Thailand
29.4 C
Indonesia
27 C
Philippines

New research highlights ChatGPT’s struggles in helping with coding tasks

New research from Purdue University reveals significant errors in ChatGPT's programming assistance, emphasising caution and calling for further study.

According to recent research, ChatGPT is still grappling with effectively assisting with programming issues despite becoming an overnight sensation. While many developers have turned to generative AI tools like GitHub’s Copilot to streamline their workflow and free up time for more productive tasks, a new study from Purdue University sheds light on significant shortcomings in ChatGPT’s performance.

Study reveals widespread errors

Researchers at Purdue University analysed 517 questions from Stack Overflow, comparing ChatGPT’s answers to those provided by human experts. The findings were startling: more than half (52%) of the responses generated by ChatGPT were incorrect. The breakdown of errors is as follows: 54% were conceptual misunderstandings, 36% were factual inaccuracies, 28% were logical mistakes in code, and 12% were terminology errors.

The study also highlighted that ChatGPT often produced unnecessarily lengthy and complex responses. This overabundance of detail can lead to confusion and distractions for developers seeking straightforward answers. Despite these issues, an ultra-small-scale poll involving 12 programmers revealed that one-third preferred ChatGPT’s articulate, textbook-like responses. This preference underscores how easily the AI’s seemingly authoritative tone can mislead coders.

Implications for the coding community

The implications of these findings are significant. Errors in coding can cascade, potentially causing problems across multiple departments or even entire organisations. The researchers emphasise the importance of caution when using ChatGPT for programming tasks.

They state, “Since ChatGPT produces many incorrect answers, our results emphasise the necessity of caution and awareness regarding the usage of ChatGPT answers in programming tasks.” This caution is vital to prevent minor coding errors from escalating into more significant, complex issues.

Call for further research and transparency

Beyond urging caution, the researchers advocate for further studies to identify and mitigate these errors. They also call for greater transparency and communication regarding the potential inaccuracies in ChatGPT’s responses. This openness is crucial for developers to make informed decisions about when and how to use AI tools in their workflows.

As the coding community continues to integrate AI into its practices, these findings serve as a reminder of the limitations and risks associated with relying too heavily on automated tools. While ChatGPT and similar technologies offer exciting possibilities, their current capabilities require scrutiny and responsible use to ensure they genuinely enhance productivity without introducing significant errors.

Hot this week

Ulanzi OA-14 Camera Cage for Osmo Action 3/4/5 review: Rugged protection with creative flexibility

Ulanzi OA-14 adds rugged protection and accessory mounts to the DJI Osmo Action 5 while keeping battery swaps quick and easy. It is also compatible with the Osmo Action 3 and 4.

Samsung could launch two Galaxy Z Fold8 models in 2026

Samsung may release two Galaxy Z Fold8 models in 2026, including one with a square-like screen, alongside the Galaxy Z Flip8.

Agora expands OpenAI partnership to strengthen conversational AI offerings

Agora expands its partnership with OpenAI, integrating the Realtime API into its platform to power more natural multimodal conversational AI.

Samsung may launch a super-sized foldable phone in 2026

Samsung is reportedly planning a new super-sized foldable phone for 2026, joining the Galaxy Z Flip and Fold series.

OpenAI signs reported US$300 billion cloud deal with Oracle under Project Stargate

OpenAI has reportedly signed a US$300 billion cloud deal with Oracle under Project Stargate, one of the largest agreements of its kind.

Biwin unveils Mini SSD, a tiny storage device that could replace microSD cards

Biwin launches Mini SSD, a tiny yet powerful storage device that could replace microSD cards if industry standards are adopted.

Apple brings major upgrades to Powerbeats Pro 2 with iOS 26

Apple adds heart rate, fitness, and smart usability upgrades to Powerbeats Pro 2 with iOS 26, launching on 15 September.

UltraGreen.ai secures US$188 million anchor investment at US$1.3 billion valuation

UltraGreen.ai secures US$188 million anchor investment led by 65EP, Vitruvian, and August, valuing the firm at US$1.3 billion.

ConnectingDNA launches AI-powered DNA wellness marketplace in Singapore

ConnectingDNA launches the world’s first AI-powered DNA wellness marketplace in Singapore, offering personalised health insights and secure data protection.

Related Articles

Popular Categories