In this deliverable we present the results of the first-round evaluations of all our crowdbots (Pepper, the smart wheelchair, cuyBot and Qolo). The evaluations are both quantitative and qualitative. They range from testing the efficacy of core components that are developed within the project and underpin the Crowdbots (e.g. sensing, localisation, simulation, planning), to fully integrated tests of each platform for some of the scenarios identified in D1.1 (Specification of Scenarios Requirements).
The results demonstrate that the prototype crowdbots are currently able to operate in low density crowds. However, throughout the evaluation, some common themes emerged. Our autonomous robots (Pepper and cuyBot), both of which require localisation, were able to operate well in low density crowds over extended periods of evaluation. Complementary approaches (e.g. “visual-inertial ceiling-based localization” and “2D-LiDAR-based localization”) were used to increase robustness. However, as the crowd density increased there were times when both localisation methods failed. In these instances, we would most likely need to rely upon a dead-reckoning approach until a new pose could be determined.
Moreover, people’s perceptions of the autonomous robots (i.e. Pepper and cuyBot) are very different to those of our robots that are co-located with a human user (i.e. smart wheelchair and Qolo). Consequently, we observed different crowd interaction behaviour in a number of scenarios, where the Pepper robot and cuyBot were treated more as mobile obstacles to be avoided, whilst the human crowd was generally more patient with the wheelchair and Qolo. This resulted in behaviours where people would overtake and cut abruptly in front of cuyBot and Pepper, or even deliberately try to impede their motion. Conversely, people within the crowd would often wait to allow the wheelchair to pass, make eye contact with the user and in some cases even move proactively out of the way. The robot speed also appeared to play a substantial role in affecting the crowd behaviour. This has implications on how we model crowd motion and subsequent robotic planning, in response to the type and speed of the crowdbot.
The results of this deliverable are currently being used to inform D1.3 (Specification of Scenarios Requirements Update). In particular we are prioritising and refining the scenarios to be tested (D1.1), as well as developing a common set of metrics, which will allow us to better compare between robots, scenarios and algorithms.