Students Research Lab

Some time ago (summer term 2022), I taught the bachelor course "Research Lab". The purpose of the research lab is to familiarize students with research by "getting their hands dirty". I introduced DeLab to the students as one example of a research project. In particular, Julian and I presented our research efforts in finding and analyzing user moderation instances on Twitter and Reddit. We also offered the opportunity to the students setting up an own research project within the context of DeLab. Even though all projects were within the broader context of DeLab, there are three projects that contributed more directly to our project. In this post, I will briefly introduce one of these three projects.

Influence of the opening post

When you read social media posts, you have probably seen two types of conversations: While some conversations are characterized by a (friendly) exchange of ideas and arguments, others -too many- are characterized by users shouting at each other; some users even use hate language. In other words: some conversations are good and some are not. The two students ask the question why this is the case?

While there are many factors that explain the quality and success of conversations (please see, e.g., Sadeque and Bethard 2019 or Arguello et al. 2006), the students argue that one factor is still missing: constructiveness. Think about a first constructive post, offering an argument and an invitation to challenge that argument. Would you expect the follow-up conversation to evolve differently in comparison to an insulting opening post? And would you expect the conversation to be much nicer than in the latter scenario? Well, the two students did.

Based on a brief review of existing literature and theoretical arguments, two hypotheses were derived:

1. A constructive opening post will lead to a more constructive conversation, vice versa.
2. Conversations with polarized topics are more destructive in nature.

A coding scheme was developed to measure constructiveness: A post is constructive given

• it is free of irony, provocation, and insults
• it contains either an argument, a link to an external source, an invitation to join the discussion, or an objective question

In addition, in the follow-up conversation, users should refer to the topic of the opening post rather than simply agreeing or disagreeing with the statement. So there should be at least some kind of engagement with the conversation. The opposite category -destructive- is coded for posts that are offending or full of irony. A third category -neutral- is used for posts that are neither constructive nor destructive.

To analyze the hypotheses, the two students sampled conversations with at least 10 tweets containing one of the two hashtags: #freespeech or #freedomofspeech. In total, their analysis is based on 57 conversations between October 2020 and June 2022. The students arrive at a manually labelled dataset with topics and constructiveness.

Because the topics of two conversations are completely unrelated to the hashtags, 55 topics could be successfully labelled: There are 16 conversations talking about Elon Musk buying Twitter Inc., 7 conversations about the Corona pandemic, 7 topics about censorship, 6 topics about freespeech and freedom, 5 topics about banning Trump from Twitter, and 14 other topics.

In terms of the level of constructiveness, the majority of opening posts are neutral (25). While 11 conversations were destructive, 19 were constructive. Looking at the average level of constructiveness of the subsequent posts, a similar picture emerges: Most of the conversations are neutral (20). 10 conversations can be characterized as destructive, and 25 as constructive.

In the next step, the students analyzed the relationship between the opening post and the follow-up conversation. They decided to use Pearson's correlation as a measure of the relationship. With a positive value of 0.37, there seems to be a moderate positive relationship: Constructive opening posts also associated with more constructive follow-up conversations, and vice versa. Going further, the determination coefficient of R = 0.13 shows that the opening post explains 13% of the variance of the follow-up posts. Therefore, the first hypothesis can be (preliminary) confirmed. However, due to the small sample size, the results must be treated with some caution.

When analyzing the correlation for each of the 5 manually labelled topics, the sample  size decreases even further. Therefore, the results should be taken with (even more) caution: With an average of -0.34, conversations about Corona are the most destructive conversations. Censorship is still destructive (-0.12); only the Elon Musk's purchase of Twitter Inc. (0.03) and  Donald Trump's ban  (0.13) are constructive. This also confirms the second hypothesis - with the aforementioned caveat of the small sample sizes.

All in all, the results are quited promising: Constructive opening posts lead to more constructive follow-up posts. Of course, with only 55 conversations, the sample needs to be expanded to dig deeper into the relationship between opening posts, topics, and constructiveness. If you read (or write) social media posts, you might want to pay attention to this relationship. Hopefully, you'll find a similar distribution of more positive (constructive) instances than negative (destructive) ones.