The aim of the workshop is to focus the attention of researchers in Artificial Intelligence and Robotics on the need for evaluation and benchmarking methods that can provide quantitative reliable measurements of the ability of robots to perform complex tasks, that often cannot be achieved with simple reactive behaviors. A substantial basis to tackle this issue is given by robotic competitions and RoboCup in particular. While RoboCup competitions provide a first significant step and experience in benchmarking intelligent robot behaviors, we are still far from having a well-established performance evaluation methodology that can be widely accepted and generalized to other domains and tasks, and that is not restricted to robot subsystems, but rather tackles the full integrated system. Specifically, we believe that a significant contribution to this topic should be brought by the Artificial Intelligence community, both within competitions, such as RoboCup, and in other scientific endeavors. The Workshop programme will include 2 invited talks and presentations of submitted contributions (the targeted number is 12, but this will be adjusted based on the number and quality of the submissions). A final panel discussion will summarize the contributions presented at the workshop. We plan to propose a special issue of ECCAI Communications to publish selected contributions from the workshop.
There is a significant interest in the topic of benchmarking. IEEE RAS TC on "Performance Evaluation and Benchmarking of Robotic and Automation Systems" organizes workshops and events. The topic is also addressed within the workshop series "Performance Metrics for Intelligent Systems" PERMIS, which addresses the issue of performance evaluation in a wider perspective. However, the proposed workshop shall address the evaluation and benchmarking of intelligent robotic systems with a specific focus on the full integrated system that is designed to accomplish complex tasks, that often cannot be achieved with simple reactive behaviours/capabilities and/or involve multiple-robots. The former EURON SIG on Cooperative Robotics pointed out the relevance of the topics in a White Paper but no specific event on benchmarking has been organized. Moreover, the EURON SIG on Good Experimental Methodologies provide a web hub for people interested in benchmarking, objective performance evaluation and methodologies. This group also published a GEM SIG Review Guidelines paper and organized some events and workshops, however, the guidelines do not explicitly take into account multiple robots behavior and the related collective performance evaluation. Finally, a EURON Benchmarking WP worked on a survey of the current efforts in comparative robotics research. Members from the above cited communities are involved in the workshop organization, with the goal of creating an opportunity for the ECAI audience to meet them.
The goal of the workshop is to provide a forum for discussion of proposals and ideas on "Benchmarking Intelligent (Multi)-Robot Systems". In particular, we aim at focusing the attention of the AI research community on the technical aspects performance evaluation of intelligent robotic systems, by presenting the state of the approaches, including (but not limited to) RoboCup competitions. Research contributions are therefore solicited on all aspects concerning the evaluation and benchmarking of intelligent robotics agents. Specifically, the focus will be on those methods and frameworks that aim at the evaluation of systems, possibly involving multiple-robots, with respect to complex tasks such as: soccer, disaster response, surveillance, home companion/helper, domestic and other environment services, etc...
IMPORTANT NOTE: all workshop participants need to register to the ECAI Conference.
ECAI, is the leading Conference on Artificial Intelligence in Europe, and is a biennial organization of the European Coordinating Committee for Artificial Intelligence - ECCAI.