Page 1 of 1

Final results and availability of data

PostPosted: Fri Apr 06, 2012 6:50 pm
by NosferatoCorp
Our JRS'2012 Data Mining Contest has finished! You can find the final results at the leaderboard and a summary of the competition in the Summary section.

Once again, thank you for participation!

To those of you how are waiting for publication of labels for the test data, as well as for PMIDs of documents and names of the MeSH headings/subheadings: please be patient. Although the JRS'2012 competition ended, there is a mini-contest ongoing for our students at University of Warsaw who attend a Machine Learning course. We do not want to disclose any additional data before it finishes, which is going to be on April 30.

Best regards,
Andrzej Janusz

Re: Final results and availability of data

PostPosted: Thu Apr 19, 2012 10:24 pm
by namp
Is it possible to make available the script that you used in order to calculate the score appearing on the leaderboard?

Many thanks

Re: Final results and availability of data

PostPosted: Mon Apr 23, 2012 9:03 pm
by NosferatoCorp
namp wrote:Is it possible to make available the script that you used in order to calculate the score appearing on the leaderboard?


Hello,

I have placed eval.jar file in the 'Public files' folder. It contains a java script which was used for evaluation of results during the competition. You can access it through the Summary section.

Though, you have to remember that the preliminary and final evaluation scores were computed on disjoint subsets of the test data. We can not reveal this partitioning just yet.

Best regards,

Andrzej Janusz

Re: Final results and availability of data

PostPosted: Thu Apr 26, 2012 9:50 am
by namp
That's really great. Thanks for posting!

Re: Final results and availability of data

PostPosted: Sun May 06, 2012 11:18 am
by Ismagilov
Can you please reveal all the test data now?

Re: Final results and availability of data

PostPosted: Thu May 17, 2012 5:18 pm
by Ismagilov
Many thanks!

Re: Final results and availability of data

PostPosted: Thu May 17, 2012 9:12 pm
by NosferatoCorp
Ismagilov wrote:Can you please reveal all the test data now?


We have revealed all the data related to our competition, including classification of the test cases, PMIDs of documents, as well as the names of columns and target labels. The files can be accessed from the Summary page.

Best regards,

Andrzej Janusz

Re: Final results and availability of data

PostPosted: Thu May 24, 2012 5:24 pm
by j_scheibel
Great! thanks!

Re: Final results and availability of data

PostPosted: Wed Jul 18, 2012 8:26 pm
by narjes
NosferatoCorp wrote:
namp wrote:Is it possible to make available the script that you used in order to calculate the score appearing on the leaderboard?


Hello,

I have placed eval.jar file in the 'Public files' folder. It contains a java script which was used for evaluation of results during the competition. You can access it through the Summary section.

Though, you have to remember that the preliminary and final evaluation scores were computed on disjoint subsets of the test data. We can not reveal this partitioning just yet.

Best regards,

Andrzej Janusz




Hello,

I was wondering if you could guide me in order to use eval1prelim.jar. I try to run it in command line,but it delivers errors. should we set the arguments for it in command line or it will provide a GUI?

Re: Final results and availability of data

PostPosted: Fri Aug 03, 2012 9:52 pm
by sebov
As a basis for understanding, please read the page - http://wiki.tunedit.org/doc:challenges-tutorial

To set up a contest we had to define datasets and an evaluation procedure.
The shared file eval.jar constist of one java class FMeasureEvaluationProcedure
which extends EvaluationProcedure abstract class from TunedIT framework. You need
to inherit from this class if you want to implement an evaluation procedure suitable
for TunedIT based contest.

If you want to use the eval.jar you have a couple of options. If you do not know
the TunedIT research possibilities, you will find the second one easier for start.
1. You can use TunedTester to set up a test and use FMeasureEvaluationProcedure as your
evaluation procedure. Read http://wiki.tunedit.org/ - especially TunedIT Research section.
2. The evaluation procedure was not designed to run by hand, but if you make
a little effort you can achieve what you want.
Of course you may freely extend this example, e.g. to pass paths as args from command line.
Please remember to add eval.jar and core.jar (you can read about and download core.jar
from http://wiki.tunedit.org/doc:research-architecture)
An example may be more or less as follows.

Code: Select all
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.InputStream;
import java.util.List;

import org.tunedit.core.ExternalProcess;
import org.tunedit.core.ResourceLoader;
import org.tunedit.core.ResourceName;
import org.tunedit.core.exception.TunedTesterException;

public class Invoker {

   public static class SimpleResourceLoader implements ResourceLoader {
      @Override
      public InputStream open(ResourceName fileResource)
            throws TunedTesterException {
         String filePart = fileResource.getFilePart();
         try {
            return new FileInputStream(filePart);
         } catch (FileNotFoundException e) {
            e.printStackTrace();
            return null;
         }
      }

      @Override
      public ExternalProcess runProcess(ResourceName fileResource,
            List<String> args) throws TunedTesterException {
         return null;
      }
   }
   
   public static void main(String[] args) throws Exception {
      ResourceName resourceName1 = new ResourceName("path_to_your_user_labels.txt");
      ResourceName resourceName2 = new ResourceName("path_to_yout_true_labels.txt");
      SimpleResourceLoader simpleResourceLoader = new SimpleResourceLoader();
      FMeasureEvaluationProcedure eval = new FMeasureEvaluationProcedure();
      System.out.println(eval.run(resourceName1, resourceName2, simpleResourceLoader)[0]);
   }
}


If it is not exactly what you want or in case of other issues, do not hesitate to ask.

Sebastian

Re: Final results and availability of data

PostPosted: Fri Aug 10, 2012 3:10 pm
by dupko
sebov wrote:As a basis for understanding, please read the page - http://wiki.tunedit.org/doc:challenges-tutorial

To set up a contest we had to define datasets and an evaluation procedure.
The shared file eval.jar constist of one java class FMeasureEvaluationProcedure
which extends EvaluationProcedure abstract class from TunedIT framework. You need
to inherit from this class if you want to implement an evaluation procedure suitable
for TunedIT based contest.

If you want to use the eval.jar you have a couple of options. If you do not know
the TunedIT research possibilities, you will find the second one easier for start.
1. You can use TunedTester to set up a test and use FMeasureEvaluationProcedure as your
evaluation procedure. Read http://wiki.tunedit.org/ - especially TunedIT Research section.
2. The evaluation procedure was not designed to run by hand, but if you make
a little effort you can achieve what you want.
Of course you may freely extend this example, e.g. to pass paths as args from command line.
Please remember to add eval.jar and core.jar (you can read about and download core.jar
from http://wiki.tunedit.org/doc:research-architecture)
An example may be more or less as follows.

Code: Select all
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.InputStream;
import java.util.List;

import org.tunedit.core.ExternalProcess;
import org.tunedit.core.ResourceLoader;
import org.tunedit.core.ResourceName;
import org.tunedit.core.exception.TunedTesterException;

public class Invoker {

   public static class SimpleResourceLoader implements ResourceLoader {
      @Override
      public InputStream open(ResourceName fileResource)
            throws TunedTesterException {
         String filePart = fileResource.getFilePart();
         try {
            return new FileInputStream(filePart);
         } catch (FileNotFoundException e) {
            e.printStackTrace();
            return null;
         }
      }

      @Override
      public ExternalProcess runProcess(ResourceName fileResource,
            List<String> args) throws TunedTesterException {
         return null;
      }
   }
   
   public static void main(String[] args) throws Exception {
      ResourceName resourceName1 = new ResourceName("path_to_your_user_labels.txt");
      ResourceName resourceName2 = new ResourceName("path_to_yout_true_labels.txt");
      SimpleResourceLoader simpleResourceLoader = new SimpleResourceLoader();
      FMeasureEvaluationProcedure eval = new FMeasureEvaluationProcedure();
      System.out.println(eval.run(resourceName1, resourceName2, simpleResourceLoader)[0]);
   }
}


If it is not exactly what you want or in case of other issues, do not hesitate to ask.

Sebastian


Oh thank you,or i should thank wiki? :roll:

Re: Final results and availability of data

PostPosted: Mon Oct 08, 2012 9:50 pm
by narjes
Thank you very much for your comprehensive response.
Finally, I coud use tunedtester to evaluate my algorithm's answer,but there is still one question.

As you know, there are two distinct evaluations for traffic prediction contest,preliminary and final. Preliminary samples comprise 35% of test data and final samples - the remaining 65%.

The question is that; are these subsets selected randomly from the whole answer file OR the first 35% of data is evaluated for preliminary and the remaining for final?
if the subsets are fixed part of answer file, why we get different answers through different runs of TunedTester?

More generally, why there is an option to run Tunedtester for several times?

Re: Final results and availability of data

PostPosted: Wed Oct 17, 2012 10:13 pm
by purexa
From http://tunedit.org/challenge/JRS12Contest?m=task
With a use of our automatic tagging algorithm, which we will describe in details after completion of the contest, we associated all the documents with the most related MeSH terms (headings).


Where can I find a description of the automatic tagging algorithm?