LongEval CLEF 2025 Lab

Longitudinal Evaluation of Model Performance

Data

The data for this task is a sequence of web document collections and queries provided by Qwant.

Queries:
The queries are extracted from Qwant’s search logs, based on a set of selected topics. The query set was created in French.

Documents:
The document collection includes all documents that have been displayed in SERPs for the selected queries. Filters were applied to exclude adult content from the collection. A collection.db file (SQLite) is provided to the participants with the mapping of the entire collection (URLs) to the document IDs. This mapping ensures that every document has its own unique ID across the whole collection. The structure of the table 'mapping' is as follows:

    (0, 'id', 'INTEGER', 0, None, 1)
    (1, 'url', 'TEXT', 0, None, 0)
    (2, 'last_updated_at', 'TEXT', 0, None, 0)
    (3, 'date', 'TEXT', 0, None, 0)
    

Record with ID 1:

    {
        "id": 1,
        "url": "https://www.blogduvoyage.fr/roadtrip-usa-conseils/",
        "last_updated_at": [
            1640160479,
            1640160479
        ],
        "date": [
            "2022-06",
            "2022-06"
        ]
    }
    

Participants can check detailed information of the document, such as when it was last updated and where the document is located.

Relevance estimates:
The relevance estimates for LongEval-Retrieval are obtained through automatic collection of user implicit feedback. This implicit feedback is obtained with a click model, based on Dynamic Bayesian Networks trained on Qwant data. The output of the click model represents an attractiveness probability, which is turned to a 3-level scale score (0 = not relevant, 1 = relevant, 2 = highly relevant). This set of relevance estimates will be completed with explicit relevance assessment after the submission deadline.

The overview of the data creation process is displayed in the Figure below:

The full collection is provided and processed to map queries and documents with unique IDs. It is recommended not to use the previous collections from 2023 and 2024 for training.

The purpose of providing the full collection without too much sampling is to give participants full freedom and to provide the IR community with a comprehensive collection for future testing of their methods.

The figure below shows the overlap of documents across the full collection:

More details about the collection can be found in a paper: P. Galuscakova, R. Deveaud, G. Gonzalez-Saez, P. Mulhem, L. Goeuriot, F. Piroi, M. Popel: LongEval-Retrieval: French-English Dynamic Test Collection for Continuous Web Search Evaluation.

Details about the new CORE task will be provided soon.