Skip to content
Home » Shared Task 2024

Shared Task 2024

The Workshop on Computational Linguistics and Clinical Psychology

CLPsych 2024 will be held in conjunction with EACL 2024

Thursday March 21, 2024

CLPsych 2024 Shared Task

Utilising LLMs for finding supporting evidence about an individual’s suicide risk level

Increasingly the clinical community are looking for new and better diagnostic measures and mental health condition monitoring tools. Over the past decade, there has been a surge in methods at the intersection of NLP and mental health, showing that signals for the diagnosis of certain conditions can be found in language. However, most research tasks have been mostly defined on the basis of classifying individuals in a static manner (e.g., on the basis of suicide risk [1, 2]) or over time (e.g., on the basis of capturing changes in their well-being over time [3, 4]). Such tasks lack the explainability component (defined here as finding supporting evidence) which is crucial for shedding light onto the reasons affecting a model’s (or an annotator’s) decisions and for enabling practitioners to quickly extract evidence signalling an individual’s mental health state in natural language. Recent work on large language models (LLMs) suggest that they can perform well on clinical NLP tasks such as information extraction [5] and question answering [6], and have the potential to synthesize diverse medical insights into meaningful, well-formed text. This Shared Task combines the generative abilities of LLMs with the generation of supportive evidence for clinical assessments.

The CLPsych 2024 Shared Task

Our shared task this year provides the combination of the promise brought by the generative abilities of LLMs with the generation of supporting evidence for clinical assessments. Specifically we focus on the problem of using an open source LLM to provide evidence for the assigned suicide risk level of a person on the basis of their linguistic content. For the purpose of the task we focus on posting activity in online social media platforms. In particular, given a user’s posts and a pre-assigned suicide risk level associated with the user, we aim at extracting the evidence within the user’s posts supporting the annotation label in an unsupervised manner. We then want to present this as a coherent explanation.

Data

Social media data annotated for the purposes of the above tasks will be made available to the registered teams upon receiving the signed agreements (see top of this page). The data comes from 2019 CLPsych Shared Task A [1, 2] (University of Maryland Reddit Suicidality Dataset, Version 2) and includes Reddit users and their r/SuicideWatch posts, alongside their suicide risk levels in four classes: No, Low, Moderate and Severe risk.

The data pertains to vulnerable individuals and we endeavour to protect them as much as possible. As stated in the agreement, sharing the data to any entity (person, organisation, company, etc.) outside those listed in the agreement is strictly prohibited. Note that this includes sharing the data via APIs in order to use LLMs (e.g., Bard, ChatGPT, Claude, etc.), which is also strictly prohibited. 

Some teams might have already worked with the dataset. Therefore, to allow for a fair “competition”, the organisers are sharing the data with the registered teams 9 days prior to the release of any further instructions  (see Timeline below), so that all teams can become familiar with it. The organizing team will then be clarifying the task and the evaluation process, and provide the corresponding python scripts.

Timeline
  • Call for Participation: 1 December 2023
  • Team Registration Deadline: 10 19 December 2023
  • Dataset Out: 11 December 2023 (after registration)
  • Task Instructions Out: 20 December 2023
  • System Submission due: 19 January 2024
  • Results out: 23 January 2024
  • Paper Submission: 26 January 2024
  • Reviews Out: 30 January 2024
  • Camera Ready: 4 February 2024
How to participate

Team Registration (required once for each team): online form
Individual Member Registration (required for all participants) : online form
Data Sharing Agreement (required once for each team): PDF form, please fill out, sign (paper or electronic signature) and email to the task organizers.

Organizers

Adam Tsakalidis (Queen Mary University of London & The Alan Turing Institute)
Jenny Chim (Queen Mary University of London)
Dana Atzil Slonim (Bar-Ilan University)
Dimitris Gkoumas (Queen Mary University of London)
Maria Liakata  (Queen Mary University of London & The Alan Turing Institute)

Email organizers at clpsych24-shared-task-organizers@googlegroups.com

Information on previous shared tasks is available for 2021 and 2022, and you can find papers and presentations from the 2022 workshop here.

Follow us on Twitter

Twitter feed is not available at the moment.