namp wrote:Is it possible to make available the script that you used in order to calculate the score appearing on the leaderboard?
Ismagilov wrote:Can you please reveal all the test data now?
NosferatoCorp wrote:namp wrote:Is it possible to make available the script that you used in order to calculate the score appearing on the leaderboard?
Hello,
I have placed eval.jar file in the 'Public files' folder. It contains a java script which was used for evaluation of results during the competition. You can access it through the Summary section.
Though, you have to remember that the preliminary and final evaluation scores were computed on disjoint subsets of the test data. We can not reveal this partitioning just yet.
Best regards,
Andrzej Janusz
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.InputStream;
import java.util.List;
import org.tunedit.core.ExternalProcess;
import org.tunedit.core.ResourceLoader;
import org.tunedit.core.ResourceName;
import org.tunedit.core.exception.TunedTesterException;
public class Invoker {
public static class SimpleResourceLoader implements ResourceLoader {
@Override
public InputStream open(ResourceName fileResource)
throws TunedTesterException {
String filePart = fileResource.getFilePart();
try {
return new FileInputStream(filePart);
} catch (FileNotFoundException e) {
e.printStackTrace();
return null;
}
}
@Override
public ExternalProcess runProcess(ResourceName fileResource,
List<String> args) throws TunedTesterException {
return null;
}
}
public static void main(String[] args) throws Exception {
ResourceName resourceName1 = new ResourceName("path_to_your_user_labels.txt");
ResourceName resourceName2 = new ResourceName("path_to_yout_true_labels.txt");
SimpleResourceLoader simpleResourceLoader = new SimpleResourceLoader();
FMeasureEvaluationProcedure eval = new FMeasureEvaluationProcedure();
System.out.println(eval.run(resourceName1, resourceName2, simpleResourceLoader)[0]);
}
}
sebov wrote:As a basis for understanding, please read the page - http://wiki.tunedit.org/doc:challenges-tutorial
To set up a contest we had to define datasets and an evaluation procedure.
The shared file eval.jar constist of one java class FMeasureEvaluationProcedure
which extends EvaluationProcedure abstract class from TunedIT framework. You need
to inherit from this class if you want to implement an evaluation procedure suitable
for TunedIT based contest.
If you want to use the eval.jar you have a couple of options. If you do not know
the TunedIT research possibilities, you will find the second one easier for start.
1. You can use TunedTester to set up a test and use FMeasureEvaluationProcedure as your
evaluation procedure. Read http://wiki.tunedit.org/ - especially TunedIT Research section.
2. The evaluation procedure was not designed to run by hand, but if you make
a little effort you can achieve what you want.
Of course you may freely extend this example, e.g. to pass paths as args from command line.
Please remember to add eval.jar and core.jar (you can read about and download core.jar
from http://wiki.tunedit.org/doc:research-architecture)
An example may be more or less as follows.
- Code: Select all
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.InputStream;
import java.util.List;
import org.tunedit.core.ExternalProcess;
import org.tunedit.core.ResourceLoader;
import org.tunedit.core.ResourceName;
import org.tunedit.core.exception.TunedTesterException;
public class Invoker {
public static class SimpleResourceLoader implements ResourceLoader {
@Override
public InputStream open(ResourceName fileResource)
throws TunedTesterException {
String filePart = fileResource.getFilePart();
try {
return new FileInputStream(filePart);
} catch (FileNotFoundException e) {
e.printStackTrace();
return null;
}
}
@Override
public ExternalProcess runProcess(ResourceName fileResource,
List<String> args) throws TunedTesterException {
return null;
}
}
public static void main(String[] args) throws Exception {
ResourceName resourceName1 = new ResourceName("path_to_your_user_labels.txt");
ResourceName resourceName2 = new ResourceName("path_to_yout_true_labels.txt");
SimpleResourceLoader simpleResourceLoader = new SimpleResourceLoader();
FMeasureEvaluationProcedure eval = new FMeasureEvaluationProcedure();
System.out.println(eval.run(resourceName1, resourceName2, simpleResourceLoader)[0]);
}
}
If it is not exactly what you want or in case of other issues, do not hesitate to ask.
Sebastian
With a use of our automatic tagging algorithm, which we will describe in details after completion of the contest, we associated all the documents with the most related MeSH terms (headings).
Return to JRS 2012 Data Mining Competition: Topical Classification of Biomedical Research Papers
Users browsing this forum: No registered users and 1 guest