Methods for Evaluating AI Tools in Journalism

Introduction

This briefing is a bit different from others developed by CNTI’s Global AI and Journalism Research Working Group. Rather than exploring a particular AI use, our goal is to understand what the current research says — and misses — about how news providers evaluate AI tools across different uses.

Despite the breadth of research on the uses of AI in journalism, little attention has been given to methods for evaluating AI tools. Further, the research that does exist does not address the business (and revenue) aspects of journalism. This briefing starts with research from fields outside of journalism, and identifies practices that are transferable to journalism, highlights journalism-specific research, and concludes with recommendations for further research.

Research we reviewed outside journalism suggests:

• Evaluating AI tools is more complex than feature comparison across fields.² Even technologists and engineers widely acknowledge the difficulties.
  • Most practices for evaluating AI tools start by looking at outputs because it is functionally impossible to examine the inner workings of the AI tools or the underlying models.
  • By design, AI systems are nondeterministic: the same input can lead to different outputs ³ which means evaluations often cannot be replicated.
  • Superficial differences between prompts can have outsize effects on outputs.⁴
  • Many AI tools were developed by and for a relatively small user base, and do not work equally well across the full spectrum of human variation.⁵
    
• Standardized evaluations (benchmarks and scorecards) aren’t as helpful as one might hope. While these evaluations exist across fields, they tend to be abstract and infrequently align well with specific use cases.⁶
  • Metrics like accuracy and speed vary between different specific tasks such as developing headlines, captioning images, summarizing meetings or categorizing datasets.
  • Computer scientists have attempted to create journalism benchmarks. Unfortunately, several of these efforts suffer from limited understanding of day-to-day journalistic work.⁷
    
• Decision-makers benefit from starting with goals, not with tools.
  • Methods for evaluating an AI tool are stronger when they align with the goals for using the tool.⁸ The evaluation measures must also tie to specific institutional needs. In the journalism context, that would mean starting from journalistic values.⁹
  • It’s easier for people to suggest solutions than to reflect on how they do things, but both are important sources of data. Instead of relying solely on the words people use to describe the tool they want, observing actual user behaviors will enrich evaluation. ¹⁰ That’s just as true for staff as for audiences. Including user researchers in the process from the beginning can help.
    
• Evaluating AI tools needs to be an ongoing practice, not a one-time decision.¹¹ Small changes to underlying models can impact outputs in unpredictable ways.
  

Journalism-specific research we reviewed suggests:

• Several recent studies are converging on a set of dimensions that are relevant across many different journalism uses of AI tools.¹² How to weigh them will depend on the intended use of the AI tool, and none of these dimensions are binary. This remains a baseline, not an exhaustive list.
  • Accuracy: How frequently is the tool accurate? How much work is required to verify its outputs?
  • Neutrality & Objectivity: Are there underlying biases in results? How often does this occur, and how can users ensure that it doesn’t?
  • Transparency: Does the tool explain the logic behind its outputs, and are users aware of the limitations?
  • Autonomy & Independence: How will third-party updates impact journalists’ work? Would losing access to particular models or APIs impact their process? How can they ensure they are accountable for their work product?
  • Efficiency: Does it save time and energy, increase scale or both? Do those gains offset any extra work required?
    

Lessons from Technical Fields

What is AI evaluation, and why is it difficult?

Evaluation is the practice of determining whether a tool is good enough for a specific purpose in a specific context.

We know more about developers than users.

Developers building their own AI tools evaluate them using a range of methods, and there is research on those methods. At the same time, anyone considering adopting new AI tools also needs to evaluate them to decide if they will work in their specific context. To date, there has been much less research on the evaluation practices of end users and institutional decision-makers, compared with developers.¹³

It’s difficult to evaluate the tools directly.

Across fields, people evaluate AI tools by examining outputs because it is almost never possible to examine the inner workings of the tools directly. Evaluating AI tools is especially hard because they are nondeterministic by design.¹⁴ That means they won’t give the same results every time, and errors may not follow a predictable pattern. Large language models (LLMs) in particular are sold as “general-purpose” technologies that can be adapted for a wide range of use cases that may not have a single correct answer.¹⁵ That makes context-specific evaluation all the more important. In practical contexts, evaluation should mean identifying the gap between what technology can do and what people actually need.¹⁶

Even major technology companies haven’t reached a consensus.

One might expect large, well-resourced technology companies to set the evaluation standards for everyone else. After all, these companies have been working with AI tools longer than anyone else. But even they struggle to evaluate whether the quality of LLM tools is “good enough,” especially for outward-facing work. A mixed-methods study of multiple product teams at Microsoft¹⁷ corroborates major challenges to setting evaluation standards identified in other research, many of which are relevant to journalism:

• “Correct” or “good” answers are subjective and underspecified.
• Non-deterministic technology means results cannot be replicated exactly.
• Small model tweaks can create big changes to outputs.
  

The same study found several emerging solutions to these challenges that may be relevant to journalism. In response to changing evaluation metrics, Microsoft employees defined custom metrics through collaboration with domain experts, combined qualitative and quantitative metrics, and laid out clear rubrics. To counter nondeterminism, they rolled out small-scale releases and ran A/B tests so they could identify unanticipated issues before full release. They responded to changing models by automating scheduled evaluation at regular frequencies and establishing internal standards.

The study also cautions that while these evaluation practices are becoming standardized, there is little data about whether they are actually effective.

There are two approaches with complementary strengths and weaknesses.

Across fields, there are two primary evaluation methods in use:

• Benchmarks are, essentially, standardized tests. A benchmark consists of a large dataset, a series of tasks and a scoring procedure. Typically, benchmarks are simple so that outputs can be scored automatically, often by other machine learning models.
• Real-world testing, or human annotation, requires experts to review outputs manually and qualitatively. This form of evaluation measures the AI tool’s judgments against human ones, but there are limits to the applicable scale.
  

The problems with benchmarks and scorecards

Computer science and engineering were the first fields to evaluate AI tools, and they offer lessons for other fields. However, the way they frame the problem can seem opaque and/or irrelevant to other fields.

Benchmarks are inexpensive and easy to compare.

These fields rely heavily on benchmarks. This approach has some clear advantages. Most obviously, benchmarks make it easy to compare different models.¹⁸ Many benchmarks can be scored automatically, which is relatively inexpensive.¹⁹

Benchmarks don’t translate well to the real world.

But benchmarks have many weaknesses. First, they don’t always measure what they’re supposed to measure (construct validity).²⁰ Second, the results don’t always translate to real-world performance (ecological validity).²¹ For example, benchmarks can be sensitive to minor differences in prompts, such as formatting.²² This is especially important in real-world settings. Third, benchmarks can be overly positive: it is possible for developers to optimize for benchmark performance rather than real-world performance. Many benchmarks remain static over time and rely on multiple-choice items with a single correct answer. That means models can be trained directly on test items, and they often are.²³ Fourth, few benchmarks attempt to measure bias or safety issues.²⁴

Current benchmarks assume people are culturally homogeneous. We’re not.

Fifth, benchmarks tend to favor particular cultural and linguistic assumptions that are not directly related to task performance. A recent audit of 51 publicly released multilingual benchmarks found that nearly all of them fall short in terms of language coverage, representativeness, rigor and transparency.²⁵ Specifically, they overrepresent Indo-European languages and Latin script. That’s important because language family and script type impact how AI models break up text, which in turn affects computation costs and speed. In other words, efficiency gains may vary based on the language being used. The same study also found that more than half the non-English content in the benchmark datasets was translated from English, not written in the language being tested.²⁶ That means benchmarks are less likely to be culturally representative. Moreover, only a few of the benchmarks are transparent about the linguistic composition of the training set.

Benchmarks don’t explain why AI tools fail or how to improve them.

Sixth, typical benchmarks don’t provide much insight into the reasons for success or failure.²⁷ They can’t explain why models perform poorly or what changes could help them perform better. That also means benchmarks may have more value for academic publication than applied value. In fact, several studies have suggested that researchers are more likely to rely exclusively on benchmarks, compared with people making product or policy decisions.²⁸

Decision-makers tend to see human evaluation as a necessary complement to benchmarks, given the shortcomings of benchmarks on their own.

The value of real-world testing

Real-world testing can be combined with benchmarks for more robust evaluation.

In contrast to benchmarking, real-world testing can help identify the maximum capabilities of new AI tools.²⁹ However, it cannot provide a sense of average performance or the range of typical behavior. That makes this approach on its own insufficient for decision-making.

A few additional studies have explored institutional decision-making about AI tools outside technical industries, largely through interviews.³⁰ These studies identify similar considerations and pitfalls. Domain expertise and context-sensitive tests complement benchmarks.³¹ Across sectors, the following considerations came up repeatedly:³²

• Sociocultural concerns. There are major disparities in linguistic performance, even within single models.³³ In some cases, customizing AI tools for particular languages and cultural contexts requires incorporating multiple models or providing extensive customization.³⁴ Domain expertise is critical to achieving this.
• Institutional concerns. Policies and compliance structures vary between organizations and sectors. Decision-makers should consider the impact on the institution and on other internal and external stakeholders.³⁵ If there is insufficient capacity to incorporate and verify domain expertise on an ongoing basis, deployment may be unsustainable.³⁶
• Technological concerns. Flexibility and ongoing evaluation are essential so that workflows can accommodate third-party technical changes. Small-scale rollouts can serve as an additional form of evaluation to identify real-world issues.³⁷
  

When decision-makers see AI use as the goal itself, it creates risk.

Two additional interview studies of organizational decision-makers suggest that they may not always be aware of their own limitations. The first study, which included participants from journalism alongside law, medicine and the public sector, found a need for increased AI literacy to ensure that decision-makers have more accurate perceptions of model capabilities and weaknesses, as well as tools to ensure that adoption does not create overreliance.³⁸ The second study included 25 interviewees. Only one was from the media, but the findings were consistent across sectors.³⁹ These researchers found that leaders across sectors overestimate the impact of adopting AI tools while underestimating the technical effort and risks involved. They also identified a lack of awareness of methods for identifying user needs and a failure to iterate and ideate before moving forward with AI tools. These challenges are related: spending more time ideating would help organizations identify costs and risks before building out a solution that isn’t viable. Leaders may also be deceived by the ease of building prototypes, assuming this speed translates to full implementation, which is often untrue.

Both studies suggest that additional ideation and literacy about risks could improve AI evaluation and adoption practices across industries.

Core Values: Journalism-specific Decision-making

Three bodies of literature attempt to address the difficulties of evaluating LLMs and other AI tools for journalism use cases. Unfortunately, they are largely siloed and do not benefit from one another’s insights.

Computer science researchers have tried to do this work without journalism domain experts, which has been flawed. 

First, several computer science researchers have tried to develop LLM benchmarks specifically for journalism tasks. One study, which measured writing proficiency and adherence to journalistic ethics across five editorial tasks in written Chinese, found considerable ethical shortfalls.⁴⁰ For example, all the models they tested failed to identify headlines with potential social harm. Another study measured factuality, logical consistency, importance, readability, objectivity and journalistic style for English-language journalism.⁴¹ But these two studies include only limited participation from domain experts, so they may not fully reflect real-world use cases. For example, one of these studies describes journalists’ work as “complex real-world web mining workflows.”⁴² It is far from clear whether journalists would agree.

Researchers have shared practices from large news outlets with in-house data science teams, but it may be difficult for smaller organizations to apply their lessons. 

At the other extreme, a small number of large news outlets have in-house data science teams that build custom AI tools. For example, Bloomberg News is applying its guidelines for rules-based automation to new tools, including an internal LLM trained to perform specific tasks, including headline suggestions and identifying sentiment in financial stories.⁴³ But even organizations of this size and complexity need to establish specific thresholds for “good enough” performance that balance a range of business needs. An ethnographic study of the BBC⁴⁴ found that new AI tools that are built or adapted in-house are first tested offline against current products, then evaluated by editors and finally A/B tested with end users. There is no single threshold or metric that determines “good enough” in those cases; instead, evaluation is always negotiated between data science, editorial and business stakeholders.

The most valuable studies to date are interdisciplinary and participatory, with a focus on eliciting values to reflect on, rather than specific metrics or thresholds. 

A promising approach to evaluation includes journalists in the design process without assuming they have in-house technical expertise.⁴⁵ In these cases, technologists collaborate closely with domain experts to understand their needs and concerns. Two separate groups of researchers identified similar core values through this method, as well as similar questions to ask during evaluation:⁴⁶

• Accuracy: AI tools can introduce falsehoods. How frequently is the AI tool accurate, and how much work is required to verify its outputs? The goal is not necessarily perfect accuracy, but “accurate enough” to be useful.
  
• Neutrality & Objectivity: AI tools may have underlying biases or may output language that breaks with journalistic norms of avoiding value judgments. How frequently does this occur, and how can users ensure that it doesn’t?
  
• Transparency: Most AI tools do not provide fully explainable or replicable results. Does the AI tool explain the logic behind its outputs, and are users aware of the limitations? If so, is it relatively easy for users to edit the final product to meet their overall standards?
  
• Autonomy & Independence: Most use cases depend on access to third-party models. If models change or journalists lose access, will their work or processes be at risk? Moreover, the lack of transparency may mean that AI tools are subtly steering users’ decisions. How can journalists ensure they are fully accountable for their work product? And how much control do journalists have over their data?
  
• Efficiency: Organizations adopt AI tools to save time and energy, increase scale or both. But using AI tools also requires additional time to prompt, verify and edit their outputs. Do the efficiency gains offset the additional labor?
  

These core values overlap and reinforce one another. For example, a tool that saves time in other ways may still be valuable at a relatively lower accuracy threshold. We have spoken with news organizations that avoid creating a seamless user experience because friction in the process encourages journalists to critically evaluate key decisions and produce higher-quality work.

Nascent efforts to develop journalism benchmarks may eventually make it easier to compare different tools’ or processes’ adherence to these core values.⁴⁷

Evaluation is a process, not an answer.

These same researchers also identified the following best practices:

• Evaluations should be grounded in real-world practice and specific tasks.
• Practitioners should be involved at multiple stages.
• Evaluation procedures that build end-user literacy would be especially valuable. In particular, users may struggle to understand how LLMs work conceptually and why they provide false information. ⁴⁸ Making it easier to understand the uncertainty of outputs could pay off.
• Customizable templates or “cookbooks” can help create replicable practices.
  

A third interdisciplinary research team⁴⁹ offers twelve questions that can be used to evaluate both external and in-house AI tools. We reproduce them here verbatim.

Concern	Question(s)
Generalizability	Is the AI tool applicable to multiple contexts?
Costs	Are there open-source datasets, AI models, or software provided by, for instance, journalistic organizations that can be used to build or enhance the AI tool? Can a crowdsourced approach be applied to enhance the quality and quantity of the data used for training the AI tool?
User-friendliness	Is the AI tool designed in a user-friendly way, making it useful for users with different levels of technical skills?
Accuracy	Does the AI tool consistently produce reliable and accurate results in line with journalistic standards? Are there technical or human safeguards in place for the AI tool that minimize erroneous results, and, to the extent possible, mitigate their impact?
Journalistic values	Does the AI tool reflect journalistic values such as reporting accuracy, relevance, and diversity? Were journalists involved and consulted in the development process?
Independence	To what extent do journalists and newsrooms own the AI tool they use (data, code, infrastructure)? How are journalists affected by changes to components that they do not own?
Transparency	Is the AI tool’s source code publicly available and well-documented, making it easy to understand and open to scrutiny?
Data privacy	Have precautions been taken to prevent or mitigate the risk of the model sharing sensitive information from users with actors external to the journalistic organization?

Where more research would be helpful

Evaluation remains an emerging area of research and practice. Some of the most important gaps we identified for the field of journalism are:

• Operational uses: There is very little research exploring uses of AI tools in journalism organizations for revenue, staffing and other operational uses. In some cases, research from other fields may be transferable. For example, using AI tools to target advertisements may have fewer field-specific constraints than newsgathering uses. However, more research in these areas would provide a more complete understanding of AI use in journalism.
  
• Evaluating vibe coding tools and other support uses: To date, there is a near-total absence of research on vibe coding in journalism. Vibe coding tools may be more straightforward to evaluate than other kinds of tools: they need to consistently output executable code that passes standard quality assurance checks. But in the absence of research, it remains difficult to understand how vibe coding may be changing the types of outputs that journalists can produce.
  
• Additional journalistic values and concerns: The emerging consensus lays out a baseline, but it is not yet comprehensive.
  
• Specific methods to evaluate journalistic values: While there is a growing consensus about which journalistic values need to be evaluated, there is still a great deal of work to do on how to best evaluate them.
  
• Scalable evaluation methods: There remains a need to develop scalable methods that can measure best-case performance, average performance and variation along a range of context-specific metrics (e.g., accuracy, transparency, etc.).
  
• Understanding smaller organizations: Much of the research on AI decision-making in journalism has looked at larger organizations with dedicated data science or AI teams. But the majority of news organizations are small enough that they need to rely primarily on off-the-shelf tools. Additional research could shine light on how they decide which AI tools to use and when.
  
• Multilingual evaluation and multilingual use cases: Our 2025 briefing on translation and transcription identified downstream effects of gaps between languages which remain relevant here.
  
• Efficacy of decision-making practices: Several of the included papers describe current decision-making practices, but do not address whether these practices are actually effective.⁵⁰
  
• More collaboration: Research on AI evaluation remains largely siloed. Collaboration between researchers from different disciplines and industries could greatly benefit practitioners.
  

Footnotes

1.  Mari, 2024. ↩︎
2. Nahar et al., 2025; van der Maden et al., 2026; Wirth et al., 2025; Xie et al., 2026. See Schjøtt & Blanke (2026) and C. Li et al. (2026) for journalism in particular. ↩︎
3.  Both Abdurahman et al., 2025 and Nahar et al., 2025 consider strategies to address non-determinism in different fields. ↩︎
4.  Abdurahaman et al., 2025; Hardy et al., 2025; McIntosh et al., 2026. ↩︎
5.  Atari et al., 2023; Cifliku & Heuer, 2026; C. Li et al., 2026; McIntosh et al., 2026; Uppadhyay et al., in review; Varuvel Dennison et al., 2026 ↩︎
6.  Hardy et al., 2025; van der Maden et al., 2026 ↩︎
7.  M. Li et al., 2024; Chien et al., 2025. ↩︎
8.  Abdurahaman et al., 2025; C. Li et al., 2026; McIntosh et al., 2026; Varuvel Dennison et al., 2026 ↩︎
9.  Cordero et al., 2025; C. Li et al., 2026; Schjøtt & Blanke, 2026; Wirth et al., 2025 ↩︎
10.  Xie et al., 2026 ↩︎
11.  Abdurahaman et al., 2025; Laire, 2025; Lucas, 2026; Nahar et al., 2025; Wirth et al., 2025 ↩︎
12.  C. Li et al., 2026; Rønning Pedersen et al., 2026a; see also Cordero et al., 2025 and Sibinga, 2026. ↩︎
13.  See Jaźwińska & Chandrasekar, 2025 on the gap in journalism. ↩︎
14.  Both Abdurahman et al., 2025 and Nahar et al., 2025 consider strategies to address non-determinism in different fields.7 ↩︎
15.  Liao & Xiao, 2025; Orr & Kang, 2024. ↩︎
16.  See Liao & Xiao, 2025. ↩︎
17.  Nahar et al., 2025 ↩︎
18.  Chang et al., 2024; Guo et al., 2023. In practice, however, most benchmarks are used by a single developer, making comparison difficult (Baack et al., 2026). ↩︎
19.  Chang et al., 2024; Guo et al., 2023 ↩︎
20.  McIntosh et al., 2026; Orr & Kang, 2024; C. Li et al., 2026 ↩︎
21.  Hardy et al., 2025; Liao & Xiao, 2025; C. Li et al., 2026 ↩︎
22.  Abdurahman et al., 2025; Chang et al., 2024; Guo et al., 2023; Hardy et al., 2025 ↩︎
23.  Chang et al., 2024; Guo et al., 2023; Orr & Kang, 2024; Uppadhyay et al., in review. ↩︎
24.  Baack et al., 2026; Eriksson et al., 2025; McIntosh et al., 2026 ↩︎
25.  Uppadhyay et al., in review. ↩︎
26.  Uppadhyay et al., in review. ↩︎
27.  Eriksson et al., 2025; Guo et al., 2023 ↩︎
28.  Eriksson et al., 2025; Hardy et al., 2025; Nahar et al., 2025; Wirth et al., 2025 ↩︎
29.  Kapoor et al., 2026 ↩︎
30.  Hardy et al., 2025; Varuvel Dennison et al., 2026; Xie et al., 2026; Yu et al., 2025 ↩︎
31.  Hardy et al., 2025; Varuvel Dennison et al., 2026 ↩︎
32.  Varuvel Dennison et al., 2026 provide this framework. ↩︎
33.  CNTI, 2025; Uppadhyay et al., in review ↩︎
34.  Varuvel Dennison et al., 2026 ↩︎
35.  Yu et al., 2025 ↩︎
36.  Varuvel Dennison et al., 2026 ↩︎
37.  Varuvel Dennison et al., 2026 ↩︎
38. Yu et al., 2025 ↩︎
39.  Xie et al., 2026 ↩︎
40.  M. Li et al., 2024 ↩︎
41.  Chien et al., 2025 ↩︎
42.  Chien et al., 2025, p. 2 ↩︎
43.  Quinonez & Meij, 2024 ↩︎
44.  Schjøtt & Blanke, 2026 ↩︎
45.  Cordero et al., 2025; C. Li, 2026; C. Li et al., 2026; Rønning Pedersen, et al., 2026a ↩︎
46.  C. Li et al., 2026; Rønning Pedersen, et al., 2026a. ↩︎
47.  Jaźwińska & Chandrasekar, 2025. ↩︎
48.  C. Li, 2026; Rønning Pedersen, et al., 2026b. ↩︎
49.  Cordero et al., 2025, p. 16 ↩︎
50.  Nahar et al., 2025; Yu et al., 2025 ↩︎