AR Instruction Benchmark

This is the website of the Augmented Reality Instruction Benchmark, which was proposed in our paper [1].

If your results are better then the current benchmark, please email me at albrecht.schmidt@acm.org and we will update the high score table.

We provide printable instructions for our proposed reference tasks.
Download:
Instuctions as PDF: Duplo Tasks (4,8,16,32 steps)
Instuctions as images in a ZIP: Duplo Tasks (4,8,16,32 steps)

Instuctions as PDF: Screws Task (4,8,16,32 steps)
Instuctions as images in a ZIP: Screws Task (4,8,16,32 steps)

Accordingly, we propose using a reference setup for the proposed assembly tasks. The arrangement of the setup and order of the parts can be seen in the pictures below.

duplo5 screwsetup1

We will provide a technical drawing which specifies the assembly areas in detail soon.

Highscore List Duplo Task:

Paper Instruction Method t_locate t_pick t_locate_part t_assemble
Funk et al.[1] Paper Baseline 2.139 0.895 1.258 1.048
Funk et al.[2] In-Situ Projection 1.300 0.78 0.85 0.80
Funk et al.[2] Paper Baseline

1.69

0.77

0.72

0.81

Funk et al.[2] Tablet instructions (Nexus 9)

2.53

0.82

0.78

0.86

Funk et al.[2] HMD instructions (Epson Moverio BT-200)

2.72

0.99

0.97

1.03

Blattgerste et al.[3] Paper Baseline

0.92

0.62

1.12

0.7

Blattgerste et al.[3] HMD instructions (Epson Moverio BT-200)

1.12

0.69

1.14

0.73

Blattgerste et al.[3] Smartphone

1.29

0.73

2.19

0.82

Blattgerste et al.[3] HoloLens

1.11

0.66

1.83

0.7

Highscore List Screws Task:

Paper Instruction Method t_locate t_pick t_locate_part t_assemble
Funk et al.[1] Paper Baseline 2.232 1.483 1.268 6.061

We will update this highscore list with AR instructions that use our proposed reference setup to enable a fast comparison between all AR instructions that are proposed by the community.

References:

[1] Funk, M., Kosch, T., Greenwald, S. W., & Schmidt, A. (2015). “A Benchmark for Interactive Augmented Reality Instructions for Assembly Tasks.” in Proceedings of the 14th international conference on Mobile and Ubiquitous Multimedia, 2015.

[2] Funk, M., Kosch, T., & Schmidt, A. (2016). Interactive worker assistance: comparing the effects of in-situ projection, head-mounted displays, tablet, and paper instructions. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing (pp. 934-939). ACM.

[3] Blattgerste, J., Strenge, B., Renner, P., Pfeiffer, T., & Essig, K. (2017) “Comparing Conventional and Augmented Reality Instructions for Manual Assembly Tasks.” ┬áIn Proceedings of the 10th ACM international conference on pervasive technologies related to assistive environments, 2017