Challenges / VideoLectures.Net Recommender System Challenge

Status Closed
Type Scientific
Start 2011-04-18 10:00:00 CET
End 2011-07-08 11:59:59 CET
Prize 5,500€

Registration is required.

Overview

 
 
The challenge is over now. Click here to view the Summary.
 
 
Image

Welcome to the web page of ECML/PKDD Discovery Challenge 2011 (European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases).
The tasks of the challenge are focused on making recommendations for video lectures, based on historical data from the VideoLectures.Net website. Prize fund of 5,500€ is ensured from the European Commission through the e-LICO EU project.

ECML/PKDD 2011 - Discovery challenge Workshop, Athens, Greece, 5th Sep 2011
We have prepared a website for the workshop related to the challenge that should be held on. From the beginning of the next week all the changes and news related to the workshop (submission of papers, key dates, etc) will be announced at http://lis.irb.hr/challenge .

Contact: tomislav.smuc AT irb.hr, nino.antulov AT irb.hr

General description of the proposed challenge

VideoLectures.Net is a free and open access multimedia repository of video lectures, mainly of research and educational character. The lectures are given by distinguished scholars and scientists at the most important and prominent events like conferences, summer schools, workshops and science promotional events from many fields of science. The website is aimed at promoting science, exchanging ideas and fostering knowledge sharing by providing high quality didactic contents not only to the scientific community but also to the general public. All lectures, accompanying documents, information and links are systematically selected and classified through the editorial process taking into account also users' comments.
This challenge is organized in order to improve the website’s current recommender system. The challenge consists of two main tasks and a “side-by” contest. The data we provided is for both of the tasks, and it is up to the contestants how it will be used for learning (building up) a recommender. Due to the nature of the problem, each of the tasks has its own merit: task 1 simulates new-user and new-item recommendation (cold-start mode), task 2 simulates clickstream based recommendation (normal mode). Data from VL.Net website does not include any explicit nor implicit user profiles. Due to the privacy-preserving constraints implicit profiles embodied in viewing sequences (clickstreams) have been transformed, so that no individual viewing sequence information can be revealed or reconstructed. There are however other viewing related data included: i) co-viewing frequencies ii) pooled viewing sequences, and iii) content related information available: lecture category taxonomy, names, descriptions and slide titles (where available), authors, institutions, lecture events and timestamps. The dataset (including the leaderboard and the test set) will remain publicly available for experimentation after the end of the challenge.

IMPORTANT: The VideoLectures.Net recommender system engine has been obfuscated in the manner that it is not possible to use it for solving the challenge tasks.

Awards

We have ensured prize-sponsoring from the European Commission through the e-LICO EU project, 2009-2012 whose primary goal is to build a virtual laboratory for interdisciplinary collaborative research in data mining and data-intensive sciences.
The prizes, for each of the tracks are:
  • 1500€ for the first place
  • 700€ for the second place
  • 300€ for the third place
The prizes, for the Workflow contest are:
  • 500€ for best workflow
  • Free admission to RapidMiner Community Meeting and Conference 2012 for the best RapidMiner workflow (sponsor: Rapid-I)

Important Dates

  • April 18, 2011 - start of the challenge (10:00 AM Central European Time)
  • July 8, 2011 - closing of the challenge - participants submit last predictions (11:59 AM Central European Time)
  • July 22, 2011 - paper submission deadline
  • August 2, 2011 - paper notifications
  • August 10, 2011 - camera-ready version due
  • September 5, 2011 - workshop

Organizers


Contact

tomislav.smuc AT irb.hr, nino.antulov AT irb.hr

Evaluation metrics – Mean Average R-precision (MARp)

Taking into account relative scarcity of items available for learning, recommending and evaluation (esp. in case of ‘cold start’ task), we have defined an R-precision variants of standard evaluation measures in information retrieval p@k and MAP. The overall score of the submission is mean value over all queries R (recommended lists r) given in the test sets:

Image

Average R-precision score - AvgRp(r) for a single recommended ranked list r is defined as:

Image

where Rp@z(r) is R-precision at some cut-off length z ∈ Z (e.g. z ∈ {5, 10, 15, 20, 25, 30} for the task 1). Rp@z(r) is defined as the ratio of number of retrieved relevant items and relevant items at the particular cut-off z of the list:

Image

Number of relevant items at cut-off length z is defined as min(m, z), where m is the total number of relevant items. When m < z, number of relevant items at z is m, while for other situations it is limited to top z relevant items from the (real) solution ranked list s. A special situation happens when there are more equally relevant items at the same rank (ties) at the cut-off length of the s list. In that case, any of these items is treated as relevant (true positive) in calculating Rp@z(r).

For the task 1, cut-off lengths z for the calculation of MARp are z ∈ {5, 10, 15, 20, 25, 30}.

For the task 2, cut-off lengths z for the calculation of MARp are z ∈ {5, 10}.

For more information about the evaluation metrics click here.

Partners and sponsors

Image

Copyright © 2008-2013 by TunedIT
Design by luksite