Loading
AIcrowd
Challenges
Organize a challenge FAQ Contact Jobs
Log In Sign Up
Back to MEDIQA 2021
Round 1: Completed Post Challenge Round: Completed Weight: 1.0

MEDIQA 2021 - Question Summarization (QS)

ACL-BioNLP Shared Task

By  MEDIQA-NLP 
4053
149
29
314
Participate

🕵️ Introduction

MEDIQA 2021 tackles three summarization tasks in the medical domain:

In this shared task, we will also explore the use of different evaluation metrics for summarization.

MEDIQA 2021 will be organized at the NAACL-BioNLP 2021 workshop.

🤷‍♂️ Question Summarization (QS)

Consumer health questions tend to contain numerous peripheral information that hinders automatic Question Answering (QA). Empirical QA studies based on manual expert summarization of these questions showed a substantial improvement of 58% in performance [1]. Effective automatic summarization methods for consumer health questions could therefore play a key role in enhancing medical question answering.

The goal of this task is to promote the development of new summarization approaches that address specifically the challenges of long and potentially complex consumer health questions.

Relevant approaches should be able to generate a condensed question expressing the minimum information required to find correct answers to the original question [2].

 

💾 Datasets

  • Training Data: The MeQSum Dataset of consumer health questions and their summaries [2] could be used for training.

  • Participants can use available external resources, including, but not limited to question focus and question type recognition datasets. For instance, the CHQs Dataset [3] contains additional annotations (e.g. medical entities, focus, question type, keywords) of the MeQSum questions.

  • Validation and Test Sets: Consist of consumer health questions received by the U.S. National Library of Medicine (NLM) in December 2020 and their associated summaries, manually created by medical experts. 

  • The validation set is available here: https://github.com/abachaa/MEDIQA2021/tree/main/Task1. 

  • The test set will be available for the registered participants in the Resources Section.

Registration

The registration & data usage agreement form is available under the Resources section of the AIcrowd projects. The form covers the three tasks. You can download it from any of the three MEDIQA projects: QS@AIcrowd, MAS@AIcrowd & RRS@AIcrowd.
To register, you need to complete, sign, and upload the form. When approved, you will be able to download the official test sets and to submit your runs on the AIcrowd submission systems.

 📅  Timeline

  • January 29, 2021: Release of the validation sets. 

  • February 26, 2021: Release of the test sets. Run submission opens on AIcrowd.

  • March 5, 2021: Run submission deadline. Participants' ROUGE scores will be available on AIcrowd.

  • March 10, 2021: Release of the official results.

  • March 17, 2021: Papers due date (Submission website and instructions).

  • April 15, 2021: Notification of acceptance.

  • April 26, 2021: Camera-ready papers due (hard deadline).

  • June 11, 2021: BioNLP Workshop @NAACL'21 

 

🖊  Evaluation Metrics

ROUGE will be used as the main metric to rank the participating teams, but we will also use several evaluation metrics more adapted to each task. 

🚀 Submission Format

- Format: ID [tab] Summary

-- Task 1 & Task 2 => question_id [tab] summary 

-- Task 3 => study_id [tab] summary 

- The summary must fit in one line (no line breaks). 

📜 Rules

1) Each team is allowed to submit a maximum of 10 runs.

2) Please choose a username that represents your team, and update your profile with the following information: First name, Last nam, Affiliation, Address, City, Country.

3) For each run submission, it is mandatory to fill in the submission description field of the submission form with a short description of the methods, tools and resources used for that run.

4) The final results will not be considered official until a working notes paper with the full description of the methods is submitted.

📱 Contact us