Media Effects Research Lab - Research Archive
Does content matter? The effects of type and number of user-generated comments on news stories
Student Researcher(s)
Erin Ash (Ph.D Candidate);
Andrew Peeling (Ph.D Candidate);
Faculty Supervisor
INTRODUCTION
As of 2006, approximately 80 percent of newspaper Web sites provide the opportunity for users to post comments on news stories or submit user feedback of some kind (Beyers, 2006). A previous study by one of the authors revealed that, for readers with a high online sense of community, the presence of comments on news stories led to lower ratings of story liking. This study was designed to test aspects of the modality and agency components of Sundar’s (2008) MAIN model in order to develop a theoretical explanation for that counter-intuitive finding. Specifically, this research investigates the influence of the type of comments and the number of comments on judgments of journalistic quality.
RESEARCH QUESTIONS
RQ1: For individuals, controlling for story type and participation in online forums, what is the relationship between the type and number of comments appearing on an online news story and perceived level of journalistic quality of the news site?
RQ2: How does online sense of community act as a separate predictor and/or a moderator in this relationship?
RQ3: How do MAIN model heuristics, namely bandwagon, expertise judgments and invasiveness mediate this relationship?
METHOD
A mixed-factorial experiment was designed to answer the research questions by determining the effects of the content and amount of user-generated comments on online newspaper Websites. Two between-subjects factors, substance of the comments and number of comments, and one within-subject factor, type of story, were included in the design, creating four conditions. Participants (n = 95, age M = 20.84, SD = 1.84) were exposed to one of the four conditions by reading an online news story and the accompanying comments from a fictitious news site created for this study. Participants then completed an online questionnaire which measured perceived levels of journalistic quality and each of the heuristics regarded as a possible mediating variable.
RESULTS
Results demonstrated that individuals with a high sense of online community perceived the news stories as possessing higher quality F (1, 171) = 7.28, p < .01 and newsworthiness F (1, 171) = 11.88, p < .001, as Figure 1 and Figure 2 below illustrate. The analysis revealed a main effect for number of comments on perceived quality, with “many” condition reporting significantly higher ratings of quality (M = 6.18; SE = 1.89) than did those in the “few” condition (M = 5.70; SE = 1.79), F (1, 171) = 4.94, p < .05, as illustrated in Figure 3. This relationship was mediated by judgments of the expertise of those who commented on the stories.
A number of comments X online sense of community interaction for objectivity ratings, F (1, 171) = 3.95, p < .05, revealed that for those with a high online sense of community, many comments led to higher objectivity ratings; whereas those with a low online sense of community perceived stories with many comments to be less objective.
Figure 4: Number of comments X sense of community on objectivity
A 3-way interaction for number of comments X type of comments X online sense of community on objectivity ratings, F (1, 171) = 3.76, p = .05, was also found. When the comments were substantive, a greater number of comments led to greater story liking for those with a high sense of community; whereas the opposite was true for those with a low sense of community, such that many comments led to lower judgments of story liking. However, when the comments were banal, few comments led to higher liking ratings for those with a high sense of community than for those with a low sense of community; and sense of community had no effect on liking for many comments.
Figure 5: Type of comments X number of comments X sense of community on liking
Finally, the type of comments had a significant effect on the heuristics expertise and invasiveness; but, with the exception of expertise and the factor quality, these heuristic judgments did not lead to differences in journalistic quality perceptions.
CONCLUSION
The results of this experiment demonstrate that user-generated comments on online news stories do, in fact, affect judgments of journalistic quality and story liking perceptions. The research also reveals the significance of individuals’ online sense of community in the relationship between comments and journalistic quality perceptions. Further, MAIN model heuristics are also important factors in determining the effects of comments on journalistic quality, either by mediating the relationship between the two or demonstrating the influence of the type of comments on judgments. Future research should investigate why certain heuristics affect judgments of the content of the comments and others affect ratings of journalistic quality.
For more details regarding the study contact
Dr. S. Shyam Sundar by e-mail at sss12@psu.edu or by telephone at (814) 865-2173