Submissions

Task 1. LongEval-Retrieval:

Link: https://github.com/clef-longeval/IR-Participants.git

Submissions will be done using git. Each team will have a private repository created by the organizers, which will be used to submit the runs to the task. The information about the repository is for each team provided by the organizers. Please contact the organizers if you did not receive this information.

All the participants can either participate in Sub-task A, short-term persistence, Sub-task B, long-term persistence, or both these sub-tasks. Participants can submit up to 5 systems applied to either one or both tasks. For each system submitted on either sub-task, the participants also need to submit the results acquired by this system on Train heldout queries which represent within a time benchmark estimate. This allows the organizers to acquire the information about the change of the system’s performance. Participants also need to provide a short description of each of the submitted systems.

We further denote the submission of a single system on a single sub-task or on the within a time query set as the run. The individual runs need to be submitted in the TREC format. For each query in each run, it is allowed to return up to 1000 documents.

Each system should be submitted in a single zipped file consisting of a following tuple:

team_system.WT contains a run (a single TREC file) of the system on the Train heldout queries, used to measure within a time (WT) performance.
team_system.ST contains a run (a single TREC file) of the system acquired on the Short Term Test queries.
team_system.LT contains a run (a single TREC file) of the system acquired on the Long Term Test queries.
Please note, that if the participants decide to only participate in the short-term or only in the long-term sub-task, then team_system.LT or team_system.ST respectively might be missing. However, the participants in all cases need to include team_system.WT and team_system.description files.
team_system.meta contains a short description of the approach. This file should contain the information which indexing and ranking methods were applied, what type of training was applied and which training data were used. Please specify if you used statistical or neural approaches and if you used sparse or dense retrieval methods. Also, please include the information if the approach uses a single ranking approach, multiple-stages of rankers or any (and what) other type of fusion. Participants also need to describe if they used French data, provided English translations or their own translations and the resources (memory, GPUs, CPUs) used. Participants should use this provided form for filling all the system details.

All the files in a single zipped document should thus correspond to a single system. The name of this system should contain the name of the team and an unique identifier of this system. The suffix of each file should either be WT, ST, LT or meta. For example, if the file contains the submission of the BM25 system of the UGA team applied on the long-term task, the file name can be UGA_BM25.LT.

Each system might be either run on French or English data (or their combination). The participants might also opt to use their own translations systems or even manual translations. However, if any manual intervention is used, even for the translation, participants need to clearly state this in the system description.

Task 2. LongEval-Classification:

Practice [Pre-Evaluation]

You can access the COMPETITION HERE and submit to Practice to evaluate your model and practice submittion process
You can download the training and practice sets from here: Training data with two temporal practice sets

Submission format
When submitting to Codalab, please submit a single zip file containing a folder called “submission”. This folder must contain THREE files:
1. predicted_eval_within.txt (with within predictions - interim_eval_2016.json)
2. predicted_eval_short.txt (with distant predictions - interim_eval_2018.json)
3. predicted_eval_long.txt (a BLANK file which will be used for interim_eval_2021.json during the evaluation phase)

Evaluation

You can access the COMPETITION HERE and submit to Evaluation to evaluate your model and rank its performance
You can download the evaluation set from here: Three temporal evaluation sets without gold labels

Submission format
When submitting to Codalab, please submit a single zip file containing a folder called “submission”. This folder must contain THREE files:
1. predicted_test_within.txt (with within predictions - interim_test_2016.json)
2. predicted_test_short.txt (with distant/short predictions - interim_test_2018.json)
3. predicted_test_long.txt (with distant/long predictions - interim_test_2021.json)

Notes

Use Format checking script for test your formatting and look into examples provided here: Baseline model results

The submissions for each sub-task will be ranked based on the first metric of macro-averaged F1. We encourage participants to contribute to both sub-tasks in order to be correctly placed on a joint leader board, as well as to enable better analysis of their system performance in both settings.