Multimediate Challenge

Download

multimediate-challenge@hcics.simtech.uni-stuttgart.de

June 2021

1 participants
2 discussions

Test server for next speaker prediction is up
by Dominike Thomas 24 Jun '21

24 Jun '21

Dear MultiMediate participants, We are happy to announce that the test server for the next speaker prediction task can be used for evaluation now. If your method uses the audio modality for next speaker prediction, please be aware that the volume of the audio can very between recordings. This might be relevant if you rely on features that encode the volume. Furthermore we would like to inform you that baseline results for the eye contact detection task are in the leaderboard. For the eye contact baseline, we extracted head pose and gaze features wiht OpenFace 2.0 and trained a separate RBF-SVM for each seating position that predicts the eye contact classes. This baseline reached 0.52 Accuracy on the test set. If you have any troubles with the evaluation system (building images, submitting them, interpreting the output,...), please do not hesitate to contact us for help! Best regards, the MultiMediate organisers

1 0

Multimediate Challenge: Evaluation Server now up!
by Dominike Thomas 09 Jun '21

09 Jun '21

Dear participants, We would like to announce that the evaluation server for the eye contact detection tasks is available as of now (link: https://eval.ai/web/challenges/challenge-page/1062/overview). The evaluation server for the next speaker prediction challenge will follow soon. In addition there are some important updates and specifications of the rules: (1) For awarding certificates for 1st, 2nd and 3rd place in each subchallenge we will only consider approaches that are described in accepted papers that were submitted to the ACM MM Grand Challenge track. (2) The evaluation servers will be open until the camera ready deadline (August 10, 2021). In case participants' evaluation results in the camera ready version of the paper differ from the results in the initial paper submission, the organisers need to be notified and the reason for the difference needs to be explained. The improved result can only be considered for the challenge ranking if it is obtained with the method described in the accepted paper. (3) Both challenge tasks are formulated as an online prediction scenario at test time, i.e. using only information from a single test sample to perform prediction for that test sample. We are aware that the design of the evaluation server allows for offline prediction (i.e. using information from several test samples jointly). The challenge ranking will only be based on online approaches. However, we also invite submissions using an offline approach. In this case the fact that an offline approach is presented needs to be clearly communicated in the paper and it will be out-of-competition with the online approaches. If you have any questions concerning the challenge or the evaluation procedure, please don't hesitate to get in contact with us! The MultiMediate organisers --------- Dominike Thomas Universität Stuttgart | University of Stuttgart Abteilung Mensch-Computer-Interaktion und Kognitive Systeme | Department Human-Computer Interaction and Cognitive Systems (HCICS) Institut für Visualisierung und Interaktive Systeme (VIS) | Institute for Visualization and Interactive Systems (VIS) Pfaffenwaldring 5a, 01.027 70569 Stuttgart Deutschland | Germany +49 (0) 711 685-60051 dominike.thomas(a)vis.uni-stuttgart.de<mailto:dominike.thomas@vis.uni-stuttgart.de> https://www.perceptualui.org/

1 0