Challenges - Documentation

Go to discussion forum to view and post questions and answers. See tutorial for a brief step-by-step introduction on creating and running the challenge.

Organizer's Guide

New challenge can be created on New Challenge page. After creation, it is empty - New Draft - and requires configuration before publishing and opening.

Configuration

As a challenge owner, you have access to multiple settings, which can be tuned to your needs. Most of them are located just in the place where they are needed or displayed to participants. For example, registration code can be changed on Register subpage and start/end dates - on the Fact sheet (the block of basic facts, located below side menu). Some of the settings are grouped on separate administration pages, like Evaluation Settings. A summary of all the settings and their current values can be found on the Checklist subpage.

Typically, you will configure most of the settings before publishing and opening the challenge, when it is still in New Draft state. However, it is sometimes necessary to correct a setting later on, when the challenge is already published or open. TunedIT allows for such modifications, if only they do not break consistency of the challenge.

Below is a list of available settings, their locations on web pages and conditions when they are editable:

Setting	Description	Editable in state
Title	Human-readable title of the challenge	Draft
Name	Computer-readable name of the challenge. You will pass it to TunedTester to identify the challenge for solution evaluation purposes. Used also to build URL of challenge web pages	New Draft
Folder	Location in Repository where challenge files - datasets, evaluation procedures, submitted solutions - will be stored. You may also upload there any other files manually, through Repository pages	New Draft
Start	The date when registration of participants and submission of solutions will become open. If you wish, you may set a date in the past, but not earlier than the date when the challenge was created	Draft or New Published
End	The date when registration of new participants and submission of solutions will become closed, and final results will be published. The hour is always 23:59:59	Draft, New Published or Open Published
Overview	General description of the challenge: purpose, organizers, schedule, awards. May contain formatting and links to other pages	Anytime
Task	Detailed description of the task: input data, expected output, format of the solution, method of evalution	Anytime
Evaluation procedure	The evaluation procedure(s) you wish to use to assess solutions submitted by participants. Can be distinct for preliminary and final tests	New or Open
Datasets	The dataset(s) you wish to use to evaluate solutions. Can be distinct for preliminary and final tests	New or Open
Timeout	Time limit for solution evaluation. If evaluation of a given solution lasts longer, it is terminated with timeout result	New
Precision	No. of digits after comma displayed on Leaderboard. May influence the ordering of solutions	New or Open
Remember best	Which participant's solution should be the active one (presented on Leaderboard and considered for final evaluation): the best or most recent one. Notice: after becoming inactive, source file of the solution is removed and cannot be tested anymore	New or Open
Limit submissions	Maximum number of solutions the participant may submit for the whole duration of the challenge. If unchecked or zero, there is no limit	New or Open
Registration code	The secret code every user will have to enter when registering to the challenge. Used for security purposes, to limit access to the challenge	New

Dataset files and JARs with evaluation procedures are put into challenge folder in Repository as private resources, accessible only by challenge owner.

Public resources

Apart from resources required for evaluation - evaluation procedure(s) and test dataset(s) - you may upload any other files to the challenge folder and set their access rights as public or private, according to your needs. You may also create subfolders. This can be done directly on Repository page of the challenge folder - just click the folder name shown on the fact sheet and you will be redirected there.

Additionally, if you want to distribute files that should be accessible only to registered participants and no one else, like training datasets, put them into 'public' subfolder of the challenge folder and leave their access rights as 'restricted' - these special rights denote all participants registered to the challenge.

Dry-run tests

Before publication of the challenge, you may wish to check if submission and evaluation of solutions works as expected. To do this, you should:

Open the challenge by setting the Start date in the past
Register yourself as a participant
Create and submit an example solution
Check if the correct result appeared on Leaderboard. Do not forget to start TunedTester for the challenge beforehand

As long as the challenge is not published, you can open and close it multiple times by changing Start and End dates, and change all the settings. It can also be deleted.

Evaluation of solutions

To evaluate solutions submitted by participants, download and un-zip TunedTester application, then open a console and run the following command in TunedTester directory:

./tunedtester.sh -u YourUsername -p YourPassword -c YourChallengeName

That's all!

From now on, Tunedtester will be working in a loop: it will automatically query the server for new submissions, download them, evaluate and send results back to TunedIT, to be published on Leaderboard. It will only stop if you make it so by pressing Ctrl+C. Specification of the test - what evaluation procedure and dataset to use, what is the time limit - will be taken from the server automatically, as well.

The same TT instance can execute both preliminary and final tests, without additional configuration. By default, final tests begin 1 week before the end of the challenge and are only run on the single active solution of each participant. Their results are gathered on the server and published only when the challenge is finished.

Publishing, opening & closing

When you are ready to disclose the challenge publicly, even if it is not yet fully configured, but Overview is filled in and Start and End dates are set, click Publish button located below the side menu. The challenge title will become visible to everybody on Challenges list. Everyone will see the Overview, but only registered participants will have access to Task description and Leaderboard.

Opening and closing the challenge is done automatically by TunedIT, when Start or End dates pass. After the end, final results apear on Leaderboard automatically, as soon as they are fully calculated. If you wish, you may edit Overview to summarize the challenge or explicitly list the winners.


Home - Contact Us - In Media - Challenges List - Create - About Challenges - R&D Outsourcing - Wiki - Blog - Forum Research - Repository - Knowledge Base - TunedTester	Copyright © 2008-2011 by TunedIT Design by luksite