On the premise that, if a thing’s worth doing, it’s worth doing well, if we’re anyway about to open a can worms, we’ll go for the biggest one on the shelf. What we wanted to do is some primary research around reader comments on online newspapers so we went for an article by John Crace entitled “Boris Johnson doesn’t let detail stand in way of latest vanity projects”, published on 11^th February 2020 in the online edition of The Guardian.

As a quick summary, the article discusses some large infrastructure projects that the British Prime Minister, Boris Johnson, mentioned in a speech, like the High Speed 2 (HS2) railway or a bridge across the Irish Sea between Scotland and Northern Ireland. To give a little background, Boris Johnson has a history of making inaccurate statements, some very visible as they were written on the side of a bus, and also led a failed bridge project (The Garden Bridge) in London. The article contains videos and photos from reputable sources and The Guardian is not known for promoting deepfakes or satire. The same footage has been shown by other publications so there’s no reason to believe the speech delivered by the PM was fabricated.

Comments on The Guardian

To be completely fair, we need to mention that The Guardian is a left-leaning publication and generally supports the Labour or Liberal Democrat parties, which are currently in opposition. Thus, knowing about the tendency of Internet forums to create echo chambers, we can expect commenters to have similar views and be critical of the Prime Minister.

We’re very interested in comments because they are a great addition to online news, allowing readers to express their opinion shortly after reading the article. However much we like them, it must be said that we were lucky that comments were open on an article about Boris Johnson in the first place, as not all articles have them. The editors explained why they sometimes turn them off, which is mainly for legal reasons – the site owner is responsible for all content including comments and, since commenters online tend to bring up either Hitler or the communists in the second sentence of any heated argument, it seems understandable. Most British online newspapers have taken a similar approach and some have closed down comments completely. While we understand why it makes sense to moderate comments, we’re not really in support of not allowing comments for the most important topics like Brexit, the climate emergency or dubious appointments at Number 10.

Methodology

A bit about the methodology. This is a cross-sectional study, which, as we’ve seen in a previous post, is less reliable than other types. Also, we’re obviously not the first ones to address this issue. We’ll come back to existing research in a future post. Initially we ran the text of the top 250 comments out of 1000 through cognitive services. This performs a standard operation called sentiment analysis which offers a number corresponding to the “positiveness” of the text (similar for negative and neutral). However, we discovered that these numbers don’t go deep enough to tell a story, especially since the automated analysis is not known to work well with irony. Thus, we manually went through half of those comments using human operators and extracted binary indicators that show if the comment has the six qualities below. The choice of characteristics was determined by observation of other comments: many attempt to be funny, use irony to criticise the people talked about in the article, may provide more information not included in the article or may just offer unsupported speculation that attepts to predict future actions. There are some comments that are completely beside the point, a phenomenon which seems more frequent on lower quality sites. We’ve called it à côté de la plaque (ACDLP), meaning way off the mark. Examples include straight ads, comments on a different topic or comments that respond to another comment and are very conversational, which may sound good when spoken or if the readers know the author personally but are out of context.

• Irony
• New Info
• Funny
• Criticising Subjects
• Speculation
• À côté de la plaque (ACDLP)

The hypothesis we wanted to test was that comments improve the experience of reading online in ways including:

• getting more info/context on the topic not included in original article article
• assessing the mood of commenters
• reader interest as the number of comments compared to similar articles
• appreciations (e.g. likes, upvotes) provide a fair way of picking the best comments

Results

The result coming from the automated sentiment analysis shows that most of the comments were negative but also shows the limitation of the AI, which doesn’t work well with ironical statements. As an example, the comment below was scored as mostly neutral although a human operator would notice the irony, intended humour and the critical character:

“These would have to be placed in water up to 1 000 ft deep. Marc Francois said they would do it at low tide.”

Regarding the correlation between votes and time passed since publication, the jury is still out. We can see a greater density of higher votes immediately after the publication, which is normal since a reader would normally just move to another article after rating some comments rather than continue refreshing the page to check out the newest comments. Some people obviously do that as we have comments with many up votes even towards the end of the sampled period.

Finally, in order to trully understand the nature of the comments, we had a human operator score them on the six criteria already mentioned. As expected, only very few comments are à côté de la plaque (ACDLP) since The Guardian uses moderation that would remove obvious ads or suspicious hyperlinks.

It’s also not hard to understand that almost all comments are critical towards the Prime Minister. The two main reasons are the location of this forum (The Guardian being a left-leaning paper) and the nature of the article which itself was critical and highlighted the lack of responsibility on the side of the Prime Minister for coming with such unrealistic proposals.

The remaining results come to confirm the original hypothesis that commenters have a meaningful input to the reading experience. Almost one in three provided information that helped understand the topic better (e.g. comparison to the Millau Viaduct, previous deeds of the PM in the Nazanin Zaghari-Ratcliffe sentence). Also, many of the comments showed irony or a sense humour or both, which again make the article a more pleasurable read.

It’s important to point out that about one in three comments involved some speculation. Commenters would very rarely provide links to back their statements but in some cases everything they’re saying is made of malicious assuptions about what the subjects, in our case Boris Johnson, may or may not do.

The number of comments in 949 at the time of writing but this is hard to compare as many similar articles have the comments turned off.

This has been really fun and we’ll definitely follow up with more results.