[Task Submission] slayqa (slayqa)

[bbunzeck]

SlayQA

SlayQA is a version of Social IQa that:

• filters for context+question+answer pairs with at least to coreference chains referring to named entities (humans), and
• systematically replaces gendered pronouns with gender-inclusive neo-pronouns, which are much rarer across common pre-training corpora

Authors

• Bastian Bunzeck bastian.bunzeck@uni-bielefeld.de

Implementation

This task re-implements the format_example() function to combine context and question into a singular prompt.

Usage

The evaluation function should be ran in the default way: (task.evaluate_predictionS(predictions, gold)), see also doc.md for a naive baseline example.

Checklist:

• I and my co-authors agree that, if this PR is merged, the code will be available under the same license as the genbench_cbt repository.
• Prior to submitting, I have ran the GenBench CBT test suite using the genbench-cli test-task tool.
• I have read the description of what should be in the doc.md of my task, and have added the required arguments.
• I have submitted or will submit an accompanying paper to the GenBench workshop.