Submissions

Submissions will be done using through CodaLab compition, you can access COMING SOON . Each team will need to participant in the competition and submit their runs through it.

All the participants need to provide results for each submitted system on both Lag6 and Lag8 test sets. This allows the organizers to acquire the information about the change of the system’s performance. Participants can submit up to 10 systems. Participants also need to provide a short description of each of the submitted systems.

We further denote the submission of a single system as the run. The individual runs need to be submitted in the TREC format. For each query in each run, it is allowed to return up to 1000 documents.

Each system should be submitted in a single zipped file consisting of a following tuple:

team_system.lag6
team_system.lag8
team_system.meta

team_system.lag6 contains a run (a single TREC file) of the system acquired on the Lag6 test collection.
team_system.lag8 contains a run (a single TREC file) of the system acquired on the Lag8 test collection.
team_system.meta contains a short description of the approach. This file should contain the information which indexing and ranking methods were applied, what type of training was applied and which training data were used. Please specify if you used statistical or neural approaches and if you used sparse or dense retrieval methods. Also, please include the information if the approach uses a single ranking approach, multiple-stages of rankers or any (and what) other type of fusion. Participants also need to describe if they used French data, provided English translations, 1-best or n-best translations, or their own translations and the resources (memory, GPUs, CPUs) used. Participants should use this provided form for filling all the system details.

All the files in a single zipped document should thus correspond to a single system. The name of this system should contain the name of the team and an unique identifier of this system. The suffix of each file should either be lag6, lag8 or meta. For example, if the file contains the submission of the BM25 system of the RSA_TU_UGA team applied on the lag6 collection, the file name can be RSA_TU_UGA_BM25.lag6.

Each system might be either run on French or English data (or their combination). The participants might also opt to use their own translations systems or even manual translations. However, if any manual intervention is used, even for the translation, participants need to clearly state this in the system description.

The submitted runs (i.e. team_system.lag6 and team_system.lag8) should follow the TrecRun format:

qid Q0 docno rank score tag

where:

qid is the query number
Q0 is the literal Q0
docno is the id of a document returned for qid
rank (1-999) is the rank of this response for this qid
score is a system-dependent indication of the quality of the response
tag is the identifier for the system

Example:
1 Q0 nhslo3844_12_012186 1 1.73315273652 mySystem
1 Q0 nhslo1393_12_003292 2 1.72581054377 mySystem
1 Q0 nhslo3844_12_002212 3 1.72522727817 mySystem
1 Q0 nhslo3844_12_012182 4 1.72522727817 mySystem
1 Q0 nhslo1393_12_003296 5 1.71374426875 mySystem