The aim of the workshop is to focus the attention of researchers in Artificial
Intelligence and Robotics on the need for evaluation and benchmarking methods that
can provide quantitative reliable measurements of the ability of robots to perform
complex tasks, that often cannot be achieved with simple reactive behaviors.
A substantial basis to tackle this issue is given by robotic
competitions and RoboCup in particular. While RoboCup competitions provide a
first significant step and experience in benchmarking intelligent robot behaviors,
we are still far from having a well-established performance evaluation methodology
that can be widely accepted and generalized to other domains and tasks,
and that is not restricted to robot subsystems, but rather tackles the full
integrated system.
Specifically, we believe that a significant contribution to this topic
should be brought by the Artificial Intelligence community, both within
competitions, such as RoboCup, and in other scientific endeavors.
The Workshop programme will include 2 invited talks and presentations
of submitted contributions (the targeted number is 12, but this will be adjusted
based on the number and quality of the submissions). A final panel discussion
will summarize the contributions presented at the workshop. We plan to propose
a special issue of ECCAI Communications to publish selected contributions
from the workshop.
There is a significant interest in the topic
of benchmarking.
IEEE RAS TC on "Performance Evaluation and Benchmarking of Robotic and
Automation Systems" organizes workshops and events.
The topic is also addressed within the workshop series
"Performance Metrics for Intelligent Systems"
PERMIS, which addresses the issue of performance evaluation in a wider perspective.
However, the proposed workshop shall address
the evaluation and benchmarking of intelligent robotic
systems with a specific focus on the full integrated system that is designed
to accomplish complex tasks, that often cannot be achieved
with simple reactive behaviours/capabilities and/or involve multiple-robots.
The former EURON SIG on Cooperative Robotics pointed out the relevance of the topics in a White Paper but no specific event on benchmarking has been organized. Moreover, the EURON SIG on Good Experimental Methodologies provide a web hub for people interested in benchmarking, objective performance evaluation and methodologies. This group also published a GEM SIG Review Guidelines paper and organized some events and workshops, however, the guidelines do not explicitly take into account multiple robots behavior and the related collective performance evaluation. Finally, a EURON Benchmarking WP worked on a survey of the current efforts in comparative robotics research.
Members from the above cited communities are involved in the workshop organization, with the goal of creating an opportunity for the ECAI audience to meet them.
The goal of the workshop is to provide a forum for discussion of proposals
and ideas on "Benchmarking Intelligent (Multi)-Robot Systems". In particular,
we aim at focusing the attention of the AI research community on the
technical aspects performance evaluation of intelligent robotic systems,
by presenting the state of the approaches, including (but not limited to)
RoboCup competitions.
Research contributions are therefore solicited on all aspects concerning
the evaluation and benchmarking of intelligent robotics agents.
Specifically, the focus will be on those methods and
frameworks that aim at the evaluation of systems, possibly involving
multiple-robots, with respect to complex tasks such as: soccer,
disaster response, surveillance, home companion/helper,
domestic and other environment services, etc...
IMPORTANT NOTE: all workshop participants need to register to the ECAI Conference.
ECAI, is the leading Conference on Artificial Intelligence in Europe, and is a biennial organization of the European Coordinating Committee for Artificial Intelligence - ECCAI.