Home
Evaluation summary
Multi-criteria decision aid
An evaluation of all sonifications of the workshop was tried with a method called Multi criteria decision aid.
The simple idea is to find criteria, according to which sonification can be evaluated, that are largely independent of each other. These criteria are then weighted (see below). Then all sonifications are evaluated. Due to time reasons we chose not a consensual method for this evaluation but questionnaires. The mean value for each criterium was then multiplied by the weight for this criterium. Then all weighted evaluated criteria are summed up to one value.
Weighting
We used 2 procedures for the weighting, one was the classic
investigation with questionnaires, which were then averaged, the other
was a silent negotiation method, where the criteria were ranked by the
group but without discussion. The results of both methods were
remarkably similar to each other.
|
Criteria |
Mean values of individual evaluation |
Ranking/Consens: |
|
|
|
Annehmlichkeit (Acceptance of sound) |
11,9% |
3 |
1,88 |
12,5% |
|
Intuitivität (Intuitivity) |
10,1% |
1 |
0,63 |
4,2% |
|
Deutlichkeit (Clearness of gestalts in sound) |
19,1% |
5 |
3,13 |
20,8% |
|
Lernaufwand (Learning afford) |
9,5% |
4 |
2,50 |
16,7% |
|
Potenzial (Potential of the sonification) |
17,2% |
5 |
3,13 |
20,8% |
|
Effizienz (Efficiency of the sonification) |
10,4% |
2 |
1,25 |
8,3% |
|
Kontextfähigkeit (Context ability) |
12,6% |
4 |
2,50 |
16,7% |
|
Technischer Aufwand (Technical effort) |
9,2% |
2 |
1,25 |
8,3% |
|
|
100% |
Summe: |
15 |
100% |
|
|
|
highest rank: |
5 |
|
|
|
|
number of criteria: |
8 |
|
MCDA Results
The ranking of the sonifications can be seen here:
Conclusions
Some conclusions and observations can be drawn from this evaluation attempt.
- the MCDA objectifies the subjective evaluation of individuals, and reproduces (at least) our subjective ranking
- in general, differences between the sonifications were not very high;
it seems that the MCDA has a rather small effective scale (the
theoretical minimum is 1/7th, but this becomes only true when every
person rates every criteria with the smallest value; the same is true
for the theoretical maximum 1)
- the sonification experts rated the sonifications generally higher than the domain scientists
- people rated the results fo their own team higher than the ones of the others, possibly because the understood better what happened there
- there was a slight trend over time, that the evaluation became better
- many criteria were not clearly enough defined
and interpreted differently by the test persons: the set of criteria
was a first test, and will be elaborated due to the results of this
evaluation
A more detailed summary can be found in german.
All data and plots can be found in the excel file.
