### Data to train empirical performance models

This data contains performance data collected by running algorithm 
configuration procedured on AClib benchmarks. Each folder contains files and
folders explained in the following.

This data was collected in order to train empirical performance models which 
then serve as surrogate benchmarks.

# Files containing scenario specific information
The following files contain information that is necessary to understand and 
process json files. The format should be self-explanatory.

training.txt - A list of instances used during configuration
test.txt - A list in instances used to validate the best found configuration
features.txt - A list of instance features
scenario.txt - The scenario file defining the algorithm configuration experiment
*.pcs - A file defining the parameter configuration space for the target algorithm

# Actual performance data
random_train.json     - Performances of 10000 randomly sampled 
                        configurations/(train)instance pairs
random_test.json      - Performances of 10000 randomly sampled 
                        configurations/(test)instance pairs
random_traintest.json - Merged random_train.json + random_test.json
data_train/*.json     - Performances of all configurations evaluated during 
                        configuration (only train instances)
data_traintest/*.json - Performances of all configurations evaluated during 
                        configuration (on train/test instances)
data_test/*.json      - Performances of the best found configurations over time
                        (only test instances)

# Data format of *.json files
Each line is a valid json string containing the following keys:
    * config:   the configuration which has been run (relates to *.pcs)
    * instance: problem instance (features can be read from features.txt)
    * seed:     seed used to run this configuration (-1 if deterministic)
    * time:     running time spent on this instance
    * misc:     if TIMEOUT/CRASHED: related error message
    * status:   one of SAT/UNSAT, TIMEOUT, CRASHED
   (* quality:) performance for ml-scenarios

# For more information, please contact me: eggenspk @ cs.uni-freiburg.de



