Addressing “Lawful but Awful” Content

CNTI’s Assessment

The “lawful but awful” dilemma describes a longstanding tension between legality and morality in which systems, practices and speech may comply with legal standards but cause social harm. Platform companies struggle to apply fragmented, often U.S.-centric “marketplace of ideas” frameworks across diverse cultural and legal contexts. The challenge is not simply one of improving moderation tools or determining liability. It is about defining the desired end state of the information environment itself. Is the goal to reduce harm, preserve maximum speech, redistribute control over visibility, or make platform power more transparent and contestable? At times, these goals may conflict, and different governance approaches may implicitly prioritize one over the others.

Policy deliberation

Balancing free speech with mitigating systemic risk is a challenging task for policymakers. There is no standard or universal definition of “awful” or harmful speech, and there is little consensus on the specific points at which such harm necessitates action. A major legislative concern is the transatlantic divide between the U.S. Section 230 model, which protects platform editorial discretion, and the EU’s Digital Services Act, which mandates transparency and risk assessment. These debates involve competing concerns about free expression, user safety and well-being, and institutional power. Existing measures designed to address harmful content can shift significant authority either to private platforms or to governments, each carrying its own risks for public discourse. Debates also persist over whether platforms should be treated as neutral “common carriers” or as providers of protected editorial rights. These tradeoffs also highlight the practical reality that context-sensitive, large-scale moderation remains inherently difficult — if not impossible — to execute consistently.

Journalistic concerns

While journalists are not the only professionals — or even individuals — navigating these tensions, they operate under uneven legal protections across jurisdictions, which can both enable critical reporting and, at times, prevent harmful content from being held accountable. Some journalists’ workflows are shaped by the need to document human rights violations, which often requires sharing graphic or disturbing material. This creates a near paradox: platforms can serve as essential archives that preserve evidence and enable accountability, yet the same content can also cause harm to audiences. More profoundly, journalism itself faces significant challenges as the line between legacy journalists and content creators continues to blur, complicating who qualifies for legal protections. At the same time, journalistic business models and audience reach are increasingly affected by “visibility moderation,” in which platforms’ content moderation quietly limits how widely content is distributed. This can reduce the impact of important but sensitive reporting without a clear explanation or meaningful opportunity to challenge those decisions. Journalists must also navigate failures in algorithmic moderation that misinterpret political speech or protest slogans as violations, underscoring the need for more context-sensitive, human-led moderation to protect legitimate news coverage.

Technology governance

As technology firms do not operate in isolation from governments and institutions, developers are tasked with designing mechanisms that are transparent and democratic, while avoiding extreme openness that could be exploited by malicious actors. In addition, current algorithms have a limited ability to parse complex, context-dependent intent, such as satire. Multi-stakeholder alliances — made up of developers, civil society organizations and governments — must also grapple with the dual role of generative AI as both a detection tool and a source of harmful content. There are also issues around the lack of regional and linguistic expertise needed to prevent Euro-American cultural biases in global systems, as well as platforms’ economic incentives to amplify controversial content. Looking ahead, governance may shift toward user-controlled filtering rather than removal to make platform power more legible and contestable.

The Issue

“Lawful but awful”¹ describes a longstanding tension between legality and morality in which systems, practices and speech may comply with legal standards but cause profound social harm.

In the digital age, scholars have been applying the “lawful but awful” concept to online content. They are especially focused on content that is available to the public on digital platforms.

That said, debates around “lawful but awful” forms of speech predate the digital era. To name just one example, years before the 1994 genocide, licensed Rwandan radio stations were broadcasting dehumanizing propaganda, such as calling Tutsis “cockroaches.” It is now widely accepted that this propaganda, while legal, helped make the genocide “socially acceptable.” Now, in the digital age, the challenges around “lawful but awful” have only become more acute.

Platformization increases the reach of harmful content. It also makes it less clear who is legally responsible because distribution is shared among many entities including the original content creators, platforms that host and amplify that content, and other account holders who may share the content for a variety of reasons. For example, “Great Replacement” narratives linked to extremist violence continue to circulate and misleading health information rapidly spreads on social media, demonstrating that societal standards for acceptable discourse are constantly evolving, which complicates the policymaking process.

The “lawful but awful” dilemma is often framed as a question of how to moderate such content, which risks obscuring more fundamental questions: What should the goal of intervention actually be, and what would it mean to fully resolve the issue? As this primer shows, the challenge is not simply one of improving moderation tools but of defining the desired end state of the information environment itself. Is the goal to reduce harm, maximize free speech, redistribute control over visibility, or make platform power more transparent and contestable? At times, these goals may conflict, and different governance approaches may implicitly prioritize one over the others. Without clarity on this underlying question, debates over content moderation strategies risk becoming technical fixes applied to a problem that is not clearly defined. The core argument of this primer is that harm prevention and free speech cannot be reconciled definitively, which means solutions must focus on procedures over specific outcomes.

Much of the current research and policy debate takes a platform-centric view, focusing on content moderation tools such as removal, downranking and account sanctions. But these questions — and the challenge of how best to address them — extend to spaces where standard platform content moderation tools do not easily translate, such as private messaging, email and live public events. In these decentralized and/or private environments, the moderator’s toolkit is often technically or legally inaccessible. Forcing public-square policies onto private correspondence — like SMS messages or encrypted chats — risks significant violations of privacy and individual rights. As noted by digital rights groups like the Electronic Frontier Foundation, policy mandates designed for public feeds often fail to account for the unique protections of private communications, leaving uncertainty as to whether content in this area should be subject to law, in addition to the overarching question of whether harmful content should be regulated at all.

Since much of the current debate and research focuses on how to manage this form of content on digital platforms, it is also the central focus of this primer (particularly in the research assessment section); in addition, we point to ways in which “lawful-but-awful” content presents a broader governance challenge.

What Makes It Complex

I. There is no universal standard for defining or governing “awful,” harmful speech. The category of “lawful but awful” exemplifies a fundamental challenge in efforts to govern speech: the tension between what is legally protected and what is socially harmful. Every government places some limits on speech, though the boundaries vary considerably across legal systems and cultural contexts. The challenge is that many forms of speech may cause emotional, social or political harm without violating the law, leaving no clear consensus about where intervention is appropriate. Because laws are rooted in specific political, historical and cultural contexts, what is protected as free speech in one country may be criminal in another.

These divergences are reinforced by competing philosophical frameworks about what speech is, what harm it can cause and who should regulate it. Much of contemporary platform governance draws on a “marketplace of ideas” logic, deeply rooted in U.S. legal and cultural traditions, which assumes that more speech is a corrective to bad speech and that platforms should function as open conduits rather than editorial gatekeepers. The largest platform companies have generally applied this U.S.-centric framework globally, even though free speech principles vary from country to country. This can create a “digital manifest destiny” in which U.S.-centric frameworks are exported worldwide. The result is a diverse global landscape without a consistent definition of harmful speech, operating under one dominant governing approach. At the same time, major U.S.-based platforms routinely prohibit many forms of lawful speech even within the U.S., including nudity, graphic violence, bullying and certain forms of inauthentic or hateful behavior.

II. Meaning is context-dependent. Language is complex and, in many cases, meaning depends on context, which makes identifying harmful speech especially difficult. Fundamentally, harm cannot be objectively or universally determined from content alone. Linguistic research demonstrates that the meaning of a word, phrase or image can shift depending on speaker characteristics, the speaker’s intent and other aspects of a social situation, leading to multiple interpretations of a piece of content in different contexts. This creates a significant challenge, since something that appears harmful in one context could be harmless — or even positive — in another.

As platform companies turn to automated content moderation approaches, context becomes a pressing issue. With billions of pieces of content circulating across platforms, automated systems (whether deterministic/rules-based or probabilistic/AI) are unavoidable, even though they are often unable to interpret context. Meta’s Oversight Board, for example, has emphasized that satire and political speech (such as the Iranian protest slogan “Marg Bar Khamenei,” translated as “Death to Khamenei”) require contextual analysis to avoid wrongful takedowns. At the same time, fully manual reviews would be impossible to achieve given the enormous volume of content.  

III. The same content can have simultaneous social harms and social benefits. Digital platforms have emerged as vital tools for documenting and archiving history. For instance, visual evidence circulating on digital platforms plays a crucial role in documenting abuse and supporting accountability. At the same time, research links various types of social harm to the viewing of violent material. For example, repeated exposure to traumatic events through the media is associated with heightened stress and anxiety, even among indirect witnesses. Violent imagery also has the potential to desensitize the public, re-traumatize communities targeted by the documented injustices, and teach and facilitate the imitation of violent behaviors through digital exposure.

The result is “lawful but awful” content functioning as both informal archives of social injustice and environments that expose users to ongoing psychological harm. This leaves platform companies to perform multiple, sometimes conflicting, roles without a clear prioritization framework.

Some legislation protects specific categories of documenters (such as journalists or human rights defenders), but this can introduce additional complexity. The U.K., for example, is considering how to apply proactive protections for public-service media to digital platforms. Other countries may be exploring “must-carry” provisions for journalism online. However, these protections raise a key question: what qualifies as journalism?

Finally, human moderators — some underpaid and overworked — must make rapid decisions on millions of flagged posts with little transparency about their decision-making processes. With this in mind, there is no shortage of evidence that moderation work is actively harmful to human moderators. This labor is frequently outsourced through global subcontracting networks, including to workers in the Global South, who are routinely exposed to graphic and traumatic material while often receiving inadequate psychological support. These working conditions place moderators at heightened risk of secondary trauma, PTSD, anxiety and other adverse mental health outcomes.

IV. The platform economy creates financial incentives to amplify controversial content. It has long been established that content that evokes emotion, including outrage, anxiety and awe, drives engagement. In the digital era, this has led to its amplification from recommendation systems, content creators and platform users as well as to its under-moderation. For example, propaganda and extremism have gained wide reach on algorithmic platforms. As a result, some argue that addressing “lawful but awful” content should target the digital advertising market that creates these incentives in addition to content moderation and fact-checking.

Platform companies and advertisers offer similar economic incentives to individual platform users, rewarding “us-against-them” narratives and disinformation tools such as conspiracy theories. Both creators of this type of content and users who amplify it can be financially rewarded for doing so.

V. Current approaches to “lawful but awful” content (specifically, platform moderation) disproportionately affect marginalized communities. There is substantial empirical evidence that marginalized communities are among those most negatively affected by current platform governance practices in two interrelated ways: Their speech is frequently over-moderated, while abuse directed at them is frequently under-moderated.

This manifests as systematic erasure for entire groups. For instance, Meta’s use of lower AI “confidence thresholds” for Arabic content, particularly compared to content in Hebrew, resulted in the removal of more than 1,000 documented instances of human rights reporting in Palestine. This is part of a larger phenomenon known as epistemic injustice, where marginalized communities are denied equal recognition as credible knowers and speakers. As a result, their ability to document and communicate experiences of violence is systematically undermined.

Moreover, members of marginalized communities face a safety gap. Unlike the swift, automated removal of their own political speech, the dehumanizing abuse they endure often requires slow manual reporting to get it removed. In the Global Majority, this latency often translates into real-world violence, as seen when TikTok failed to moderate high-reach hate speech toward LGBTQ+ communities in Ethiopia until after the harm had escalated to online harassment and physical attacks.

Such examples demonstrate how different forms of structural violence can normalize hostility toward targeted groups, contributing to stochastic or cumulative extremism that increases the likelihood of offline harassment and violence. They also illustrate that while content moderation approaches may incorrectly remove lawful speech by members of marginalized communities, they frequently leave lawful yet harmful harassment, dehumanizing rhetoric, and coordinated intimidation online. This asymmetry shows how current moderation systems frequently reproduce and reinforce existing social inequalities.

VI. There are multiple methods of content moderation, which makes it hard to measure their consistency, impact and effectiveness. Platform companies’ moderation systems are now employing subtler forms of visibility control in addition to outright removal of “problematic” content. For instance, content moderation sometimes relies on indirect interventions like shadowbanning, in which a user’s reach is reduced without notifying them. To illustrate, interviews with TikTok creators show how the experience of shadowbanning produces algorithmic gaslighting: a platform appears to function normally from the content creator’s perspective while their content is made invisible to others. This leaves any content creator uncertain about whether the issue lies with the algorithm, their content or their behavior. The lack of transparency and absence of concrete rules also enable platform companies to maintain plausible deniability about having made any editorial decision regarding “lawful but awful” content. Recent digital reduction policies capture how this logic is carried out.

State of Research

Current research overwhelmingly focuses on “lawful but awful” as a content moderation and platform governance problem, overlooking the complex infrastructural and societal factors at play. These factors are also relevant in private, offline and decentralized communication spaces. Among the research focused on content moderation and digital governance, a strong consensus has emerged around the need for a balanced approach to safety and speech. Researchers and journalists both draw the conclusion that current global moderation policies are Euro- and U.S.-centric, leaving the rest of the world without resources for culturally and politically relevant oversight. There is also a broad consensus that platforms are “black boxes” — public-facing messages about how companies perform content moderation omit details of how content is actually hidden or promoted. Research also shows that current practices often moderate marginalized and/or underrepresented voices while amplifying those promoting controversial content for monetary gain. 

While there is consensus on exclusionary and opaque moderation policies, deep disagreements persist regarding the platform companies’ fundamental legal responsibility for filtering content, whether it is legal or illegal, harmful or not. Scholars and policymakers remain divided over whether these entities should act as neutral common carriers or as active stakeholders with the right to curate the online environment. This highlights a clear gap in existing research: Although the social consequences of moderation are well documented, there is a distinct lack of technical frameworks to ensure these systems are democratic, transparent and genuinely sensitive to context. The current academic and policy focus remains largely reactive, creating a significant research gap in proactively defining the qualities of a healthy information environment — one that prioritizes sustainability and pluralism over harm mitigation. 

To address these gaps, future studies should expand the scope beyond content moderation and platform governance to investigate broader approaches to “lawful but awful” content across communication environments. A pivot toward investigating globalized and content-sensitive frameworks that move beyond a “one-size-fits-all” Euro-American approach would be valuable to the field. Specifically, emerging user-centric management models have yet to be evaluated. These studies will be vital in determining how technical designs can effectively balance the tension between protecting users from harm and preserving freedom of expression.

State of Legislation

A transatlantic dispute over accountability for “lawful but awful” online content characterizes today’s regulatory environment. In this “duty of care” period, global policies have shifted from primarily prohibiting such content to requiring platform companies to proactively address its broader risks. While the U.S. has largely relied on litigation and reform proposals without enacting comprehensive federal platform legislation, the European Union and other jurisdictions, such as the United Kingdom, have adopted a broader statutory approach.

Given that most major social media platform companies are based in the United States and that the companies fall under U.S. legal jurisdiction, the U.S. regulatory context remains central, with respect to its legal deference to the First Amendment. It follows a model with a high threshold for restricting speech, generally protecting it unless it constitutes a “clear and present danger” or “direct incitement to violence.” Since 1996, Section 230 of the U.S. Communications Decency Act has shielded platform companies from liability for third-party content while allowing them to moderate content as they see fit. This arrangement has enabled platform companies to disclaim editorial responsibility while exercising editorial power, granting them vast discretion in defining and enforcing content moderation policies. Increasingly, however, plaintiffs and state attorneys general have sought to establish liability by arguing that harms arise not only from third-party content but also from platform companies’ own design choices, recommendation systems and algorithms. Civil rights and youth safety cases reflect a broader effort to hold platform companies accountable for their addictive design and for disseminating “lawful but awful” content, while testing the limits of Section 230’s protections.

Conversely, the European Union takes a more interventionist approach. Rather than creating a general obligation to remove lawful content, the Digital Services Act (DSA) establishes due diligence obligations, including risk assessments, audits, researcher access and algorithmic transparency. Notably, more recent regulations, such as the European Media Freedom Act (EMFA), further complicate these debates by granting certain procedural protections to recognized media providers. These protections raise concerns about unequal treatment between legacy journalists, independent creators and general users.

While Section 230 was designed to support a free and open internet, the DSA was designed to make platform power more legible and contestable through governance and transparency requirements. Both frameworks face political backlash. Section 230 has become a bipartisan target for reform. It has already been narrowed through legislation such as FOSTA-SESTA (targeting sex trafficking crimes) and is increasingly being tested through platform-design litigation. Meanwhile, the DSA has sparked geopolitical tensions such as the U.S. labeling it a “foreign censorship threat” and critics warning it may constrain innovation and encourage over-censorship.

Outside the United States and the EU, content moderation regulation is fragmented and evolving, often combining legislative proposals with judicial intervention.

Notable Legislation