Final results and availability of data

Q&A related to the challenge: JRS 2012 DM Competition: Topical Classification of Biomedical Research Papers

Final results and availability of data

Postby NosferatoCorp » Fri Apr 06, 2012 6:50 pm

Our JRS'2012 Data Mining Contest has finished! You can find the final results at the leaderboard and a summary of the competition in the Summary section.

Once again, thank you for participation!

To those of you how are waiting for publication of labels for the test data, as well as for PMIDs of documents and names of the MeSH headings/subheadings: please be patient. Although the JRS'2012 competition ended, there is a mini-contest ongoing for our students at University of Warsaw who attend a Machine Learning course. We do not want to disclose any additional data before it finishes, which is going to be on April 30.

Best regards,
Andrzej Janusz
NosferatoCorp
 
Posts: 15
Joined: Tue Jul 13, 2010 5:44 pm

Re: Final results and availability of data

Postby namp » Thu Apr 19, 2012 10:24 pm

Is it possible to make available the script that you used in order to calculate the score appearing on the leaderboard?

Many thanks
namp
 
Posts: 2
Joined: Fri Oct 29, 2010 2:46 pm

Re: Final results and availability of data

Postby NosferatoCorp » Mon Apr 23, 2012 9:03 pm

namp wrote:Is it possible to make available the script that you used in order to calculate the score appearing on the leaderboard?


Hello,

I have placed eval.jar file in the 'Public files' folder. It contains a java script which was used for evaluation of results during the competition. You can access it through the Summary section.

Though, you have to remember that the preliminary and final evaluation scores were computed on disjoint subsets of the test data. We can not reveal this partitioning just yet.

Best regards,

Andrzej Janusz
NosferatoCorp
 
Posts: 15
Joined: Tue Jul 13, 2010 5:44 pm

Re: Final results and availability of data

Postby namp » Thu Apr 26, 2012 9:50 am

That's really great. Thanks for posting!
namp
 
Posts: 2
Joined: Fri Oct 29, 2010 2:46 pm

Re: Final results and availability of data

Postby Ismagilov » Sun May 06, 2012 11:18 am

Can you please reveal all the test data now?
Ismagilov
 
Posts: 2
Joined: Thu Feb 23, 2012 7:31 pm

Re: Final results and availability of data

Postby Ismagilov » Thu May 17, 2012 5:18 pm

Many thanks!
Ismagilov
 
Posts: 2
Joined: Thu Feb 23, 2012 7:31 pm

Re: Final results and availability of data

Postby NosferatoCorp » Thu May 17, 2012 9:12 pm

Ismagilov wrote:Can you please reveal all the test data now?


We have revealed all the data related to our competition, including classification of the test cases, PMIDs of documents, as well as the names of columns and target labels. The files can be accessed from the Summary page.

Best regards,

Andrzej Janusz
NosferatoCorp
 
Posts: 15
Joined: Tue Jul 13, 2010 5:44 pm

Re: Final results and availability of data

Postby j_scheibel » Thu May 24, 2012 5:24 pm

Great! thanks!
j_scheibel
 
Posts: 8
Joined: Fri Jan 08, 2010 11:18 pm

Re: Final results and availability of data

Postby narjes » Wed Jul 18, 2012 8:26 pm

NosferatoCorp wrote:
namp wrote:Is it possible to make available the script that you used in order to calculate the score appearing on the leaderboard?


Hello,

I have placed eval.jar file in the 'Public files' folder. It contains a java script which was used for evaluation of results during the competition. You can access it through the Summary section.

Though, you have to remember that the preliminary and final evaluation scores were computed on disjoint subsets of the test data. We can not reveal this partitioning just yet.

Best regards,

Andrzej Janusz




Hello,

I was wondering if you could guide me in order to use eval1prelim.jar. I try to run it in command line,but it delivers errors. should we set the arguments for it in command line or it will provide a GUI?
narjes
 
Posts: 2
Joined: Fri Feb 25, 2011 12:01 pm

Re: Final results and availability of data

Postby sebov » Fri Aug 03, 2012 9:52 pm

As a basis for understanding, please read the page - http://wiki.tunedit.org/doc:challenges-tutorial

To set up a contest we had to define datasets and an evaluation procedure.
The shared file eval.jar constist of one java class FMeasureEvaluationProcedure
which extends EvaluationProcedure abstract class from TunedIT framework. You need
to inherit from this class if you want to implement an evaluation procedure suitable
for TunedIT based contest.

If you want to use the eval.jar you have a couple of options. If you do not know
the TunedIT research possibilities, you will find the second one easier for start.
1. You can use TunedTester to set up a test and use FMeasureEvaluationProcedure as your
evaluation procedure. Read http://wiki.tunedit.org/ - especially TunedIT Research section.
2. The evaluation procedure was not designed to run by hand, but if you make
a little effort you can achieve what you want.
Of course you may freely extend this example, e.g. to pass paths as args from command line.
Please remember to add eval.jar and core.jar (you can read about and download core.jar
from http://wiki.tunedit.org/doc:research-architecture)
An example may be more or less as follows.

Code: Select all
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.InputStream;
import java.util.List;

import org.tunedit.core.ExternalProcess;
import org.tunedit.core.ResourceLoader;
import org.tunedit.core.ResourceName;
import org.tunedit.core.exception.TunedTesterException;

public class Invoker {

   public static class SimpleResourceLoader implements ResourceLoader {
      @Override
      public InputStream open(ResourceName fileResource)
            throws TunedTesterException {
         String filePart = fileResource.getFilePart();
         try {
            return new FileInputStream(filePart);
         } catch (FileNotFoundException e) {
            e.printStackTrace();
            return null;
         }
      }

      @Override
      public ExternalProcess runProcess(ResourceName fileResource,
            List<String> args) throws TunedTesterException {
         return null;
      }
   }
   
   public static void main(String[] args) throws Exception {
      ResourceName resourceName1 = new ResourceName("path_to_your_user_labels.txt");
      ResourceName resourceName2 = new ResourceName("path_to_yout_true_labels.txt");
      SimpleResourceLoader simpleResourceLoader = new SimpleResourceLoader();
      FMeasureEvaluationProcedure eval = new FMeasureEvaluationProcedure();
      System.out.println(eval.run(resourceName1, resourceName2, simpleResourceLoader)[0]);
   }
}


If it is not exactly what you want or in case of other issues, do not hesitate to ask.

Sebastian
sebov
 
Posts: 5
Joined: Fri Oct 09, 2009 6:45 pm

Re: Final results and availability of data

Postby dupko » Fri Aug 10, 2012 3:10 pm

sebov wrote:As a basis for understanding, please read the page - http://wiki.tunedit.org/doc:challenges-tutorial

To set up a contest we had to define datasets and an evaluation procedure.
The shared file eval.jar constist of one java class FMeasureEvaluationProcedure
which extends EvaluationProcedure abstract class from TunedIT framework. You need
to inherit from this class if you want to implement an evaluation procedure suitable
for TunedIT based contest.

If you want to use the eval.jar you have a couple of options. If you do not know
the TunedIT research possibilities, you will find the second one easier for start.
1. You can use TunedTester to set up a test and use FMeasureEvaluationProcedure as your
evaluation procedure. Read http://wiki.tunedit.org/ - especially TunedIT Research section.
2. The evaluation procedure was not designed to run by hand, but if you make
a little effort you can achieve what you want.
Of course you may freely extend this example, e.g. to pass paths as args from command line.
Please remember to add eval.jar and core.jar (you can read about and download core.jar
from http://wiki.tunedit.org/doc:research-architecture)
An example may be more or less as follows.

Code: Select all
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.InputStream;
import java.util.List;

import org.tunedit.core.ExternalProcess;
import org.tunedit.core.ResourceLoader;
import org.tunedit.core.ResourceName;
import org.tunedit.core.exception.TunedTesterException;

public class Invoker {

   public static class SimpleResourceLoader implements ResourceLoader {
      @Override
      public InputStream open(ResourceName fileResource)
            throws TunedTesterException {
         String filePart = fileResource.getFilePart();
         try {
            return new FileInputStream(filePart);
         } catch (FileNotFoundException e) {
            e.printStackTrace();
            return null;
         }
      }

      @Override
      public ExternalProcess runProcess(ResourceName fileResource,
            List<String> args) throws TunedTesterException {
         return null;
      }
   }
   
   public static void main(String[] args) throws Exception {
      ResourceName resourceName1 = new ResourceName("path_to_your_user_labels.txt");
      ResourceName resourceName2 = new ResourceName("path_to_yout_true_labels.txt");
      SimpleResourceLoader simpleResourceLoader = new SimpleResourceLoader();
      FMeasureEvaluationProcedure eval = new FMeasureEvaluationProcedure();
      System.out.println(eval.run(resourceName1, resourceName2, simpleResourceLoader)[0]);
   }
}


If it is not exactly what you want or in case of other issues, do not hesitate to ask.

Sebastian


Oh thank you,or i should thank wiki? :roll:
dupko
 
Posts: 1
Joined: Fri Aug 10, 2012 3:09 pm

Re: Final results and availability of data

Postby narjes » Mon Oct 08, 2012 9:50 pm

Thank you very much for your comprehensive response.
Finally, I coud use tunedtester to evaluate my algorithm's answer,but there is still one question.

As you know, there are two distinct evaluations for traffic prediction contest,preliminary and final. Preliminary samples comprise 35% of test data and final samples - the remaining 65%.

The question is that; are these subsets selected randomly from the whole answer file OR the first 35% of data is evaluated for preliminary and the remaining for final?
if the subsets are fixed part of answer file, why we get different answers through different runs of TunedTester?

More generally, why there is an option to run Tunedtester for several times?
narjes
 
Posts: 2
Joined: Fri Feb 25, 2011 12:01 pm

Re: Final results and availability of data

Postby purexa » Wed Oct 17, 2012 10:13 pm

From http://tunedit.org/challenge/JRS12Contest?m=task
With a use of our automatic tagging algorithm, which we will describe in details after completion of the contest, we associated all the documents with the most related MeSH terms (headings).


Where can I find a description of the automatic tagging algorithm?
purexa
 
Posts: 1
Joined: Tue Aug 23, 2011 3:28 pm


Return to JRS 2012 Data Mining Competition: Topical Classification of Biomedical Research Papers

Who is online

Users browsing this forum: No registered users and 3 guests

cron