Building Web Reputation Systems

Table of contents

1. Preface
   1. What Is This Book About?
   2. Why Write a Book About Reputation?
   3. Who Should Read This Book
   4. Organization of This Book
      1. Part I: Reputation Defined and Illustrated
      2. Part II: Extended Elements and Applied Examples
      3. Part III: Building Web Reputation Systems
      4. Role-Based Reading (for Those in a Hurry)
   5. Conventions Used in This Book
   6. Safari® Books Online
   7. How to Contact Us
   8. Acknowledgments
      1. From Randy
      2. From Bryce
2. I. Reputation Defined and Illustrated
   1. 1. Reputation Systems Are Everywhere
      1. An Opinionated Conversation
      2. People Have Reputations, but So Do Things
      3. Reputation Takes Place Within a Context
      4. We Use Reputation to Make Better Decisions
      5. The Reputation Statement
         1. Explicit: Talk the Talk
         2. Implicit: Walk the Walk
         3. The Minimum Reputation Statement
      6. Reputation Systems Bring Structure to Chaos
      7. Reputation Systems Deeply Affect Our Lives
         1. Local Reputation: It Takes a Village
         2. Global Reputation: Collective Intelligence
         3. FICO: A Study in Global Reputation and Its Challenges
         4. Web FICO?
      8. Reputation on the Web
         1. Attention Doesn’t Scale
         2. There’s a Whole Lotta Crap Out There
         3. People Are Good. Basically.
            1. Know thy user
            2. Honor creators, synthesizers, and consumers
            3. Throw the bums out
         4. The Reputation Virtuous Circle
         5. Who’s Using Reputation Systems?
         6. Challenges in Building Reputation Systems
         7. Related Subjects
         8. Conceptualizing Reputation Systems
   2. 2. A (Graphical) Grammar for Reputation
      1. The Reputation Statement and Its Components
         1. Reputation Sources: Who or What Is Making a Claim?
         2. Reputation Claims: What Is the Target’s Value to the Source? On What Scale?
         3. Reputation Targets: What (or Who) Is the Focus of a Claim?
      2. Molecules: Constructing Reputation Models Using Messages and Processes
         1. Messages and Processes
         2. Reputation Model Explained: Vote to Promote
         3. Building on the Simplest Model
      3. Complex Behavior: Containers and Reputation Statements As Targets
      4. Solutions: Mixing Models to Make Systems
         1. From Reputation Grammar to…
3. II. Extended Elements and Applied Examples
   1. 3. Building Blocks and Reputation Tips
      1. Extending the Grammar: Building Blocks
         1. The Data: Claim Types
            1. Qualitative claim types
               1. Text comments
               2. Media uploads
               3. Relevant external objects
            2. Quantitative claim types
               1. Normalized value
               2. Rank value
               3. Scalar value
         2. Processes: Computing Reputation
            1. Roll-ups: Counters, accumulators, averages, mixers, and ratios
               1. Simple Counter
               2. Reversible Counter
               3. Simple Accumulator
               4. Reversible Accumulator
               5. Simple Average
               6. Reversible Average
               7. Mixer
               8. Simple Ratio
               9. Reversible Ratio
            2. Transformers: Data normalization
               1. Simple normalization (and weighted transform)
               2. Scalar denormalization
               3. External data transform
         3. Routers: Messages, Decisions, and Termination
            1. Common decision process patterns
               1. Simple Terminator
               2. Simple Evaluator
               3. Terminating Evaluator
               4. Message Splitter
               5. Conjoint Message Delivery
            2. Input
               1. Typical inputs
               2. Reputation statements as input
               3. Periodic inputs
            3. Output
               1. Return values
               2. Signals: Breaking out of the reputation framework
               3. Logging
      2. Practitioner’s Tips: Reputation Is Tricky
         1. The Power and Costs of Normalization
         2. Liquidity: You Won’t Get Enough Input
         3. Bias, Freshness, and Decay
            1. Ratings bias effects
            2. First-mover effects
            3. Freshness and decay
         4. Implementer’s Notes
      3. Making Buildings from Blocks
   2. 4. Common Reputation Models
      1. Simple Models
         1. Favorites and Flags
            1. Vote to promote
            2. Favorites
            3. Report abuse
         2. This-or-That Voting
         3. Ratings
         4. Reviews
         5. Points
         6. Karma
            1. Participation karma
            2. Quality karma
            3. Robust karma
      2. Combining the Simple Models
         1. User Reviews with Karma
         2. eBay Seller Feedback Karma
         3. Flickr Interestingness Scores for Content Quality
      3. When and Why Simple Models Fail
         1. Party Crashers
         2. Keep Your Barn Door Closed (but Expect Peeking)
            1. Decay and delay
            2. Provide a moving target
      4. Reputation from Theory to Practice
4. III. Building Web Reputation Systems
   1. 5. Planning Your System’s Design
      1. Asking the Right Questions
         1. What Are Your Goals?
            1. User engagement
            2. Establishing loyalty
            3. Coaxing out shy advertisers
            4. Improving content quality
         2. Content Control Patterns
            1. Web 1.0: Staff creates, evaluates, and removes
            2. Bug report: Staff creates and evaluates, users remove
            3. Reviews: Staff creates and removes, users evaluate
            4. Surveys: Staff creates, users evaluate and remove
            5. Submit-publish: Users create, staff evaluates and removes
            6. Agents: Users create and remove, staff evaluates
            7. Basic social media: Users create and evaluate, staff removes
            8. The Full Monty: Users create, evaluate, and remove
         3. Incentives for User Participation, Quality, and Moderation
            1. Predictably irrational
            2. Incentives and reputation
            3. Altruistic or sharing incentives
               1. Tit-for-tat and pay-it-forward incentives
               2. Friendship incentives
               3. Crusader, opinionated incentives, and know-it-all
            4. Commercial incentives
               1. Direct revenue incentives
               2. Incentives through branding: Professional promotion
            5. Egocentric incentives
               1. Fulfillment incentives
               2. Recognition incentives
               3. Personal or private incentives: The quest for mastery
         4. Consider Your Community
            1. What are people there to do?
            2. Is this a new community? Or an established one?
            3. The competitive spectrum
      2. Better Questions
   2. 6. Objects, Inputs, Scope, and Mechanism
      1. The Objects in Your System
         1. Architect, Understand Thyself
            1. So…what does your application do?
            2. Perform an application audit
         2. What Makes for a Good Reputable Entity?
            1. People are interested in it
            2. The decision investment is high
            3. The entity has some intrinsic value worth enhancing
            4. The entity should persist for some length of time
      2. Determining Inputs
         1. User Actions Make Good Inputs
            1. Explicit claims
            2. Implicit claims
         2. But Other Types of Inputs Are Important, Too
         3. Good Inputs
            1. Emphasize quality, not simple activity
            2. Rate the thing, not the person
            3. Reward firsts, but not repetition
            4. Use the right scale for the job
            5. Match user expectations
         4. Common Explicit Inputs
            1. The ratings life cycle
               1. The interface design of reputation inputs
            2. Stars, bars, and letter grades
               1. The schizophrenic nature of stars
               2. Do I like you, or do I “like” like you
            3. Two-state votes (thumb ratings)
            4. Vote to promote: Digging, liking, and endorsing
            5. User reviews
         5. Common Implicit Inputs
            1. Favorites, forwarding, and adding to a collection
               1. Favorites
               2. Forwarding
               3. Adding to a collection
            2. Greater disclosure
            3. Reactions: Comments, photos, and media
      3. Constraining Scope
         1. Context Is King
         2. Limit Scope: The Rule of Email
         3. Applying Scope to Yahoo! EuroSport Message Board Reputation
      4. Generating Reputation: Selecting the Right Mechanisms
         1. The Heart of the Machine: Reputation Does Not Stand Alone
         2. Common Reputation Generation Mechanisms and Patterns
            1. Generating personalization reputation
            2. Generating aggregated community ratings
               1. Ranking large target sets (preference orders)
            3. Generating participation points
               1. Points as currency
            4. Generating compound community claims
            5. Generating inferred karma
      5. Practitioner’s Tips: Negative Public Karma
      6. Draw Your Diagram
   3. 7. Displaying Reputation
      1. How to Use a Reputation: Three Questions
      2. Who Will See a Reputation?
         1. To Show or Not to Show?
         2. Personal Reputations: For the Owner’s Eyes Only
         3. Personal and Public Reputations Combined
         4. Public Reputations: Widely Visible
         5. Corporate Reputations Are Internal Use Only: Keep Them Hush-hush
      3. How Will You Use Reputation to Modify Your Site’s Output?
         1. Reputation Filtering
         2. Reputation Ranking and Sorting
         3. Reputation Decisions
      4. Content Reputation Is Very Different from Karma
         1. Content Reputation
         2. Karma
            1. Karma is complex, built of indirect inputs
            2. Karma calculations are often opaque
            3. Display karma sparingly
            4. Karma caveats
      5. Reputation Display Formats
      6. Reputation Display Patterns
         1. Normalized Score to Percentage
         2. Points and Accumulators
         3. Statistical Evidence
         4. Levels
            1. Numbered levels
            2. Named levels
         5. Ranked Lists
            1. Leaderboard ranking
            2. Top-X ranking
      7. Practitioner’s Tips
         1. Leaderboards Considered Harmful
            1. What do you measure?
            2. Whatever you do measure will be taken way too seriously
            3. If it looks like a leaderboard and quacks like a leaderboard…
            4. Leaderboards are powerful and capricious
            5. Who benefits?
      8. Going Beyond Displaying Reputation
   4. 8. Using Reputation: The Good, The Bad, and the Ugly
      1. Up with the Good
         1. Rank-Order Items in Lists and Search Results
         2. Content Showcases
            1. The human touch
      2. Down with the Bad
         1. Configurable Quality Thresholds
         2. Expressing Dissatisfaction
      3. Out with the Ugly
         1. Reporting Abuse
            1. Who watches the watchers?
      4. Teach Your Users How to Fish
         1. Inferred Reputation for Content Submissions
            1. Just-in-time reputation calculation
         2. A Private Conversation
         3. Course-Correcting Feedback
      5. Reputation Is Identity
         1. On the User Profile
            1. My Affiliations
            2. My History
            3. My Achievements
         2. At the Point of Attribution
         3. To Differentiate Within Listings
      6. Putting It All Together
   5. 9. Application Integration, Testing, and Tuning
      1. Integrating with Your Application
         1. Implementing Your Reputation Model
         2. Rigging Inputs
         3. Applied Outputs
         4. Beware Feedback Loops!
         5. Plan for Change
      2. Testing Your System
         1. Bench Testing Reputation Models
         2. Environmental (Alpha) Testing Reputation Models
         3. Predeployment (Beta) Testing Reputation Models
            1. Performance: Testing scale
            2. Confidence: Testing computation accuracy
            3. Application optimization: Measuring use patterns
            4. Feedback: Evaluating customer’s satisfaction
            5. Value: Measuring ROI
      3. Tuning Your System
         1. Tuning for ROI: Metrics
            1. Model tuning
            2. Application tuning
         2. Tuning for Behavior
            1. Emergent effects and emergent defects
               1. Defending against emergent defects
            2. Keep great reputations scarce
         3. Tuning for the Future
      4. Learning by Example
   6. 10. Case Study: Yahoo! Answers Community Content Moderation
      1. What Is Yahoo! Answers?
         1. A Marketplace for Questions and Yahoo! Answers
         2. Attack of the Trolls
            1. Time was a factor
            2. Location, location, location
         3. Built with Reputation
         4. Avengers Assemble!
      2. Initial Project Planning
         1. Setting Goals
            1. Cutting costs
            2. Cleaning up the neighborhood
         2. Who Controls the Content?
         3. Incentives
         4. The High-Level Project Model
      3. Objects, Inputs, Scope, and Mechanism
         1. The Objects
         2. Limiting Scope
         3. An Evolving Model
            1. Iteration 1: Abuse reporting
               1. Inputs
               2. Mechanism and diagram
               3. Analysis
            2. Iteration 2: Karma for abuse reporters
               1. Inputs
               2. Mechanism and diagram
               3. Analysis
            3. Iteration 3: Karma for authors
               1. Inputs
               2. Mechanism and diagram
               3. Analysis
            4. Final design: Adding inferred karma
               1. Inputs
               2. Mechanism and diagram
               3. Analysis
      4. Displaying Reputation
         1. Who Will See the Reputation?
         2. How Will the Reputation Be Used to Modify Your Site’s Output?
         3. Is This Reputation for a Content Item or a Person?
      5. Using Reputation: The…Ugly
      6. Application Integration, Testing, and Tuning
         1. Application Integration
         2. Testing Is Harder Than You Think
         3. Lessons in Tuning: Users Protecting Their Power
      7. Deployment and Results
      8. Operational and Community Adjustments
      9. Adieu
   7. A. The Reputation Framework
      1. Reputation Framework Requirements
         1. Calculations: Static Versus Dynamic
            1. Static: Performance, performance, performance
            2. Dynamic: Reputation within social networks
         2. Scale: Large Versus Small
         3. Reliability: Transactional Versus Best-Effort
         4. Model Complexity: Complex Versus Simple
         5. Data Portability: Shared Versus Integrated
         6. Optimistic Messaging Versus Request-Reply
            1. Performance at scale
      2. Framework Designs
         1. The Invisible Reputation Framework: Fast, Cheap, and Out of Control
            1. Requirements
            2. Implementation details
            3. Lessons learned
         2. The Yahoo! Reputation Platform: Shared, Reliable Reputation at Scale
            1. Yahoo! requirements
            2. Yahoo! implementation details
               1. High-level architecture
               2. Messaging dispatcher
               3. Model execution engine
               4. External signaling interface
               5. Reputation repository
               6. Reputation query interface
            3. Yahoo! lessons learned
      3. Your Mileage May Vary
         1. Recommendations for All Reputation Frameworks
   8. B. Related Resources
      1. Further Reading
      2. Recommender Systems
      3. Social Incentives
      4. Patents
5. Index
6. About the Authors
7. Colophon
8. Copyright