Evaluating research quality with Large Language Models:An analysis of ChatGPT’s effectiveness with different settings and inputs

作     者：Mike Thelwall Mike Thelwal

作者机构：Information SchoolUniversity of SheffieldSheffield S102TNUK 

出 版 物：《Journal of Data and Information Science》 (数据与情报科学学报(英文版))

年 卷 期：2025年第10卷第1期

页      码：7-25页

摘      要：Purpose:Evaluating the quality of academic journal articles is a time consuming but critical task for national research evaluation exercises,appointments and *** is therefore important to investigate whether Large Language Models(LLMs)can play a role in this ***/methodology/approach:This article assesses which ChatGPT inputs(full text without tables,figures,and references;title and abstract;title only)produce better quality score estimates,and the extent to which scores are affected by ChatGPT models and system ***:The optimal input is the article title and abstract,with average ChatGPT scores based on these(30 iterations on a dataset of 51 papers)correlating at 0.67 with human scores,the highest ever *** 4o is slightly better than 3.5-turbo(0.66),and 4o-mini(0.66).Research limitations:The data is a convenience sample of the work of a single author,it only includes one field,and the scores are *** implications:The results suggest that article full texts might confuse LLM research quality evaluations,even though complex system instructions for the task are more effective than simple ***,whilst abstracts contain insufficient information for a thorough assessment of rigour,they may contain strong pointers about originality and ***,linear regression can be used to convert the model scores into the human scale scores,which is 31%more accurate than ***/value:This is the first systematic comparison of the impact of different prompts,parameters and inputs for ChatGPT research quality evaluations.

主 题 词：ChatGPT Large Language Models LLMs Scientometrics Research Assessment 

学科分类：0502[文学-外国语言文学类] 050201[050201] 05[文学] 

核心收录：

D　O　I：10.2478/jdis-2025-0011

馆 藏 号：203156501...