Skip to content

Chapter 5 of 7

Integrity, Moderation, and the Misinformation Problem

Some content is engineered to win at engagement metrics even when it does not genuinely help users. Engagement bait is a clear example: posts that ask readers to "comment YES" or "type AMEN" produce low-quality interactions. Platforms often downrank this content because it pollutes feeds without adding value. A related pattern is clickbait in short videos, where the first seconds promise something the rest of the video does not deliver. It may earn initial views but tends to produce swipes and negative signals, dragging down the post's overall performance.

Other content is not just low-quality but actively harmful. Moderation systems try to detect and act on rule-breaking content, using a mix of automated classifiers and human reviewers. Actions range from removal, which deletes or restricts access, to demotion, which reduces distribution while keeping the content available. Many platforms also operate recommendation eligibility rules: a piece of content may be allowed to exist on the platform but blocked from being widely recommended, which limits its spread without removing it entirely. The popular term "shadow ban" usually refers to perceived or real reductions in distribution; in practice, what users experience as a shadow ban is often the result of policy limits, weak quality signals, or poor audience fit.

Misinformation gets a structural advantage in engagement systems. Novel, emotionally charged claims travel fast through shares and outrage, even when they are false, because the metrics that drive virality reward exactly those qualities. Outrage farming exploits this by deliberately provoking anger and comments to boost reach. To counter the spread of misinformation without blocking all sharing, platforms add friction to resharing in some cases, for example inserting prompts like "read before sharing." Downranking misleading ads follows the same logic: even when such ads generate clicks, platforms protect long-term trust by limiting their reach, because short-term click wins can drive complaints and churn.

All chapters
  1. 1Ads, Auctions, and the Economics of Attention
  2. 2How Feeds Learn from Your Behavior
  3. 3Virality: How Content Spreads and Stops Spreading
  4. 4Crafting Short-Form Video That Retains Viewers
  5. 5Integrity, Moderation, and the Misinformation Problem
  6. 6Platform Differences in Discovery and Sharing
  7. 7The Attention Economy and Healthy Use

Drill it

Reading is not remembering. These come from the Social Media Scrolling Virality Addon deck:

Q

SM: What is an ads auction (high level) on social platforms?

When there’s a chance to show an ad, advertisers compete to win that impression; the platform picks a winner based on bid and predicted performance/value.

Q

SM: What is CPM?

CPM is cost per 1,000 impressions—an ad pricing metric focused on views rather than clicks.

Q

SM: What is CPC?

CPC is cost per click—an ad pricing metric focused on paying for clicks.

Q

SM: What is CTR and why do platforms care about it?

CTR is click-through rate (clicks ÷ impressions); it indicates how compelling an ad/content is and affects optimization decisions.