Intrument challenge evaluation

Questions, answers, discussions related to ISMIS 2011 Contest

Intrument challenge evaluation

Postby bosmart » Fri Feb 04, 2011 10:00 am

How is the preliminary performance evaluated exactly? Is it the percentage of correctly classified pairs or does it somehow also take into account the pairs in which only one instrument was classified correctly?

Regards
bosmart
 
Posts: 5
Joined: Thu Jan 13, 2011 2:57 pm

Re: Intrument challenge evaluation

Postby Swietlicka » Fri Feb 04, 2011 2:15 pm

Hi bosmart,

The pairs in which only one instrument was classified correctly are also taken into account, with weight = 0.5. Referring to the task description (solutions and evaluation section):
The evaluation metric is modified accuracy:

* If no recognized instrument matches the actual ones, 0.0 score is assigned
* If only one instrument is correctly recognized, 0.5 is assigned
* If both instruments match the target ones, 1.0 is assigned


Regards,
Joanna, TunedIT
Swietlicka
 

Re: Intrument challenge evaluation

Postby bosmart » Fri Feb 04, 2011 4:27 pm

Hi Joanna,

Thanks, I've somehow missed the description.

I do however have another question: is it possible to find out the individual errors (I mean how many pairs were classified correctly, and for how many only one instrument was recognized)?

Thanks
bosmart
 
Posts: 5
Joined: Thu Jan 13, 2011 2:57 pm

Re: Intrument challenge evaluation

Postby Swietlicka » Sat Feb 05, 2011 2:23 am

Unfortunately, it is not possible to do so directly. The only feedback we provide you with is the evaluation score.

Regards,
Joanna
Swietlicka
 


Return to ISMIS 2011 Contest: Music Information Retrieval

Who is online

Users browsing this forum: No registered users and 1 guest

cron