Submitting jar

[wjaskowski]

Hello,

I have another question regarding the type of solutions possible. I have a challenge where I want a solutions in a form of Java programs. I noticed the "With compilation" option and it works fine, but I wonder how the evaluation procedure then should look like. During evaluation I would like to run solutions to test them on my own data set. I thought that in the evaluator I should execute loader.runProcess with userLabelsName as the argument, but I got:

Code: Select all

ERROR. You do not have access rights required to perform the operation
-------------stack-trace-------------
org.tunedit.tester.a.g: You do not have access rights required to perform the operation
        at org.tunedit.tester.e.<init>(Unknown Source)
        at org.tunedit.tester.w.runProcess(Unknown Source)
        at javaTestEval.JavaTestEval.run(JavaTestEval.java:30)
        at org.tunedit.core.EvaluationProcedure.run(EvaluationProcedure.java:70)
        at org.tunedit.tester.C.main(Unknown Source)
Caused by: java.security.PrivilegedActionException: java.io.IOException: Cannot run program "C:\Users\Wojtek\tunedit-cache\wjaskowski\challenges\kck10test\alg\wjaskowski\1286363051_Submission1.jar": CreateProcess error=193, %1 is not a valid Win32 application
        at java.security.AccessController.doPrivileged(Native Method)
        ... 5 more
Caused by: java.io.IOException: Cannot run program "C:\Users\Wojtek\tunedit-cache\wjaskowski\challenges\kck10test\alg\wjaskowski\1286363051_Submission1.jar": CreateProcess error=19
3, %1 is not a valid Win32 application
        at java.lang.ProcessBuilder.start(Unknown Source)
        at org.tunedit.tester.B.run(Unknown Source)
        ... 6 more
Caused by: java.io.IOException: CreateProcess error=193, %1 is not a valid Win32 application
        at java.lang.ProcessImpl.create(Native Method)
        at java.lang.ProcessImpl.<init>(Unknown Source)
        at java.lang.ProcessImpl.start(Unknown Source)
        ... 8 more


I am probably doing something wrong. It would be easier for me if you could provide any example of such a procedure. In particular, I am interested how to deal in a proper way with abnormal program termination (Runtime exception, timeout, wrong exit code, etc.), so a student could have a feedback what went wrong.

Thanks in advance,
Wojtek

[Swietlicka]

Hello,

This functionality wasn't intended to be used in student challenges, yet - that's why it isn't well documented. But of course you're welcome to use it

Indeed, Loader.runProcess can be used to evaluate solutions in the form of programs (i.e. single executable files). However, currently we can only guarantee that it works in Linux. Below is an example evaluation procedure:

Code: Select all

public class AccuracyEval  extends EvaluationProcedure {
   
   private static final Double trainRatio = 0.7; //Number of training examples
   public static final String testStartMarker = "~"; //Contents of the line marking the start of the test part of the data

   @Override
   public Double[] run(ResourceName algorithm, ResourceName dataset,
         ResourceLoader loader, long seed) throws TunedTesterException,
         EvaluationSetupException, AlgorithmErrorException {
      
      StandardLoader sLoader = new StandardLoader(loader);
      BufferedReader data = new BufferedReader(new InputStreamReader(sLoader.open(dataset)));
      ExternalProcess algorithmProcess = sLoader.runProcess(algorithm, null);
      
      List<String> testDecisions = writeAlgorithmInput(data, algorithmProcess, seed);
      
      waitForProcess(algorithmProcess);
      
      Double[] score = new Double[1];
      score[0] =  countAccuracy(algorithmProcess, testDecisions);
      
      return score;
   }

   /**
    * Writes all the data to the input stream of a given process.
    *
    * First, the whole header of the input arff file (attributes etc.) is written.
    * Next, the data is divided to two subsets: training and testing.
    * Training part is written before the testing one.
    *
    * @param data Input data
    * @param algorithmProcess Process to write to
    * @param seed Seed used to divide the data set to training and testing part
    * @return Decisions for the test samples
    * @throws TunedTesterException
    */
   private List<String> writeAlgorithmInput(BufferedReader data, ExternalProcess algorithmProcess, long seed) throws TunedTesterException {

      PrintStream dataStream = new PrintStream(new BufferedOutputStream(algorithmProcess.getDataStream()));
      List<String> testSamples = new ArrayList<String>(); //Temporary list of test samples
      List<String> testDecisions = new ArrayList<String>();
      Random rand = new Random(seed);
      String line;
      
      try {
         // Header of the arff file
         while ((line = data.readLine()) != null){
            if (!line.trim().equals("") && !line.trim().startsWith("%")){ // ignore empty lines and comments
               dataStream.println(line);
               if (line.trim().equals("@data")) // End of the header
                  break;
            }
         }
         
         // Data section
         while ((line = data.readLine()) != null){
            if (!line.trim().equals("") && !line.trim().startsWith("%")){ //ignore empty lines and comments
               if (rand.nextDouble() < trainRatio){ //The sample is added to the training subset
                  dataStream.println(line); // Write training sample
               }else{
                  // Save test sample
                  int lastComma = line.lastIndexOf(",");
                  if (lastComma >= 0){
                     testSamples.add(line.substring(0, lastComma));// Input part
                     testDecisions.add(line.substring(lastComma + 1)); //Decision (kept private)
                  }
               }
            }
         }
         
         dataStream.println(testStartMarker); // Mark the end of training examples
         
         //Write the input parts of the test samples
         for (String testSample: testSamples){
            dataStream.println(testSample);
         }
         
      } catch (IOException e) {
         throw new TunedTesterException("Error while reading data set file. ", e);
      }
      
      dataStream.close();
      
      return testDecisions;
   }

   /**
    * Waits for termination of the process and checks its exit code
    * @param algorithmProcess The process to wait for
    * @throws AlgorithmErrorException
    * @throws TunedTesterException
    */
   private void waitForProcess(ExternalProcess algorithmProcess) throws AlgorithmErrorException, TunedTesterException {
      try {
         int exitCode = algorithmProcess.waitFor();
         if (exitCode != 0){
            throw new AlgorithmErrorException();
         }
      } catch (InterruptedException e) {
         throw new TunedTesterException("Algorithm process interrupted. ", e);
      }
   }

   /**
    * Counts algorithm's score
    * @param algorithmProcess Process of the algorithm
    * @param targetDecisions Correct decisions for given samples
    * @return Accuracy of the classifier (number of correct answers divided by total number of samples)
    * @throws AlgorithmErrorException
    */
   private Double countAccuracy(ExternalProcess algorithmProcess, List<String> targetDecisions) throws AlgorithmErrorException {
      
      List<String> result = algorithmProcess.getResult();
      int totalSamples = 0, correctDecisions = 0;
      
      for (String resultLine: result){
         if (targetDecisions.isEmpty()){
            throw new AlgorithmErrorException("Too many output values.");
         }
         totalSamples++;
         if (resultLine.trim().equals(targetDecisions.remove(0).trim()))
            correctDecisions++;
      }
      if (!targetDecisions.isEmpty()){
         throw new AlgorithmErrorException("Too few output values.");
      }

      return ((double) correctDecisions)/totalSamples;
   }

   @Override
   public Double[] run(ResourceName algorithm, ResourceName dataset,
         ResourceLoader loader) throws TunedTesterException,
         EvaluationSetupException, AlgorithmErrorException {
      // Deprecated method - just link to the one using seed
      return run(algorithm, dataset, loader, 0);
   }

}


If the solutions can only be written in Java, you can also use StandardLoader.loadClass.

Regards,
Joanna Świetlicka