Abstract

The use of flexible and autonomous robotic systems is a possible solution for automation in dynamic and unstructured industrial environments. Pick and place robotic applications are becoming common for the automation of manipulation tasks in an industrial context. This context requires the robot to be aware of its surroundings throughout the whole manipulation task, even after accomplishing the gripping action. This work introduces the deep post gripping perception framework, which includes post gripping perception abilities realized with the help of deep learning techniques, mainly unsupervised learning methods. These abilities help robots to execute a stable and precise placing of the gripped items while respecting the process quality requirements. The framework development is described based on the results of a literature review on post gripping perception functions and frameworks. This results in a modular design using three building components to realize planning, monitoring and verifying modules. Experimental evaluation of the framework shows its advantages in terms of process quality and stability in pick and place applications.

References

References
1.
IFR
,
2018
,
Robots and the Workplace of the Future, Technical Report, International Federation of Robotics Frankfurt
, Germany.
2.
Lammer
,
U. A.
,
2017
, “
Funktionsvereinigung in der Lagertechnik
,”
PhD thesis
,
Technical University of Munich
,
Munich, Germany
.
3.
Winkelhake
,
U.
,
2017
,
Die Digitale Transformation Der Automobilindustrie: Treiber - Roadmap - Praxis
,
Springer Vieweg, Wiesbaden, Germany
.
4.
Salehi
,
V.
, and
Wang
,
S.
,
2017
, “
Using Point Cloud Technology for Process Simulation in the Context of Digital Factory Based on a Systems Engineering Integrated Approach
,”
21st International Conference on Engineering Design (ICED 17) Vol 3: Product Services and Systems Design
,
Vancouver, British Columbia, Canada
.
5.
Salehi
,
V.
,
2019
, “
Development of An Agile Concept for MBSE for Future Digital Products Through the Entire Life Cycle Management Called Munich Agile MBSE Concept (MAGIC)
,”
Comput.-Aided Design Appl.
,
17
(
1
), pp.
147
166
. 10.14733/cadaps.2020.147-166
6.
Salehi
,
V.
, and
Wang
,
S.
,
2019
, “
Munich Agile MBSE Concept (MAGIC)
,”
Proc. Design Soc.: Int. Conf. Eng. Design
,
1
(
1
), pp.
3701
3710
. 10.1017/dsi.2019.377
7.
Kurrek
,
P.
,
Zoghlami
,
F.
,
Jocas
,
M.
,
Stoelen
,
M.
, and
Salehi
,
V.
,
2020
, “
Q-model: An Artificial Intelligence Based Methodology for the Development of Autonomous Robots
,”
ASME J. Comput. Inform. Sci. Eng.
,
20
(
6
), p.
061009
. 10.1115/1.4046992
8.
Kurrek
,
P.
,
Zoghlami
,
F.
,
Jocas
,
M.
,
Stoelen
,
M.
, and
Salehi
,
V.
,
2020
, “
Reinforcement Learning Lifecycle for the Design of Advanced Robotic Systems
,”
3rd IEEE International Conference on Industrial Cyber-Physical Systems
,
Tampere, Finland
.
9.
Saloky
,
T.
, and
Seminsky
,
J.
,
2005
,
Artificial Intelligence and Machine Learning. KEGA research project, Virtual Program Modules of AI Systems
.
10.
Russell
,
S.
, and
Norvig
,
P.
,
2009
,
Artificial Intelligence: A Modern Approach
§, 3rd ed,
Prentice Hall Press
,
Upper Saddle River, NJ
.
11.
Ghosh
,
D.
,
Olewnik
,
A.
, and
Lewis
,
K.
,
2017
, “
Application of Feature-Learning Methods Toward Product Usage Context Identification and Comfort Prediction
,”
ASME J. Comput. Inf. Sci. Eng.
,
18
(
1
), p.
011004
. 10.1115/1.4037435
12.
Shaw
,
S. B.
, and
Singh
,
A. K.
,
2014
, “
A Survey on Cloud Computing
,”
2014 International Conference on Green Computing Communication and Electrical Engineering (ICGCCEE)
,
Coimbatore, India
, pp.
1
6
.
13.
Steder
,
B.
,
2013
, “
Feature-Based 3D Perception for Mobile Robots
,”
PhD thesis
,
Uni Freiburg
,
Freiburg, Germany
.
14.
Zoghlami
,
F.
,
Kurrek
,
P.
,
Jocas
,
M.
,
Masala
,
G.
, and
Salehi
,
V.
,
2019
, “
Usage Identification of Anomaly Detection in An Industrial Context
,”
Proc. Design Soc.: Int. Conf. Eng. Design
,
1
(
1
), pp.
3761
3770
. 10.1017/dsi.2019.383
15.
Kurrek
,
P.
,
Jocas
,
M.
,
Zoghlami
,
F.
,
Stoelen
,
M.
, and
Salehi
,
V.
,
2019
, “
AI Motion Control a Generic Approach to Develop Control Policies for Robotic Manipulation Tasks
,”
Proc. Design Soc.: Int. Conf. Eng. Design
,
1
(
1
), pp.
3561
3570
. 10.1017/dsi.2019.363
16.
Jocas
,
M.
,
Kurrek
,
P.
,
Zoghlami
,
F.
,
Gianni
,
M.
, and
Salehi
,
V.
,
2019
, “
AI-Based Learning Approach with Consideration of Safety Criteria on Example of a Depalletization Robot
,”
Proc. Design Soc.: Int. Conf. Eng. Design
,
1
(
1
), pp.
2041
2050
. 10.1017/dsi.2019.210
17.
Zhang
,
H.
,
Long
,
P.
,
Zhou
,
D.
,
Qian
,
Z.
,
Wang
,
Z.
,
Wan
,
W.
,
Manocha
,
D.
,
Park
,
C.
,
Hu
,
T.
,
Cao
,
C.
,
Chen
,
Y.
,
Chow
,
M.
, and
Pan
,
J.
,
2016
, “
Dorapicker: An Autonomous Picking System for General Objects
,”
IEEE International Conference on Automation Science and Engineering (CASE)
,
Fort Worth, TX
, pp.
721
726
.
18.
Poss
,
C.
,
Irrenhauser
,
T.
,
Prueglmeier
,
M.
,
Goehring
,
D.
,
Zoghlami
,
F.
,
Salehi
,
V.
, and
Ibragimov
,
O.
,
2019
, “
Enabling Robot Selective Trained Deep Neural Networks for Object Detection Through Intelligent Infrastructure
,”
Proceedings of the 2019 4th International Conference on Automation, Control and Robotics Engineering, CACRE2019, Association for Computing Machinery
,
Shenzhen, China
.
19.
Poss
,
C.
,
Irrenhauser
,
T.
,
Prueglmeier
,
M.
,
Goehring
,
D.
,
Zoghlami
,
F.
, and
Salehi
,
V.
,
2019
, “
Perception Based Intelligent Materialhandling in Industrial Logistics Environments
,”
Proceedings of the 2019 11th International Conference on Computer and Automation Engineering, ICCAE 2019
,
Perth, Australia
, ACM, pp.
146
151
.
20.
Poss
,
C.
,
Mlouka
,
O. B.
,
Irrenhauser
,
T.
,
Prueglmeier
,
M.
,
Goehring
,
D.
,
Zoghlami
,
F.
, and
Salehi
,
V.
,
2019
, “
Robust Framework for Intelligent Gripping Point Detection
,”
IECON 2019 – 45th Annual Conference of the IEEE Industrial Electronics Society
,
Lisbon, Portugal
, Vol.
1
, pp.
717
723
.
21.
Schuster
,
M. J.
,
Okerman
,
J.
,
Nguyen
,
H.
,
Rehg
,
J. M.
, and
Kemp
,
C. C.
,
2010
, “
Perceiving Clutter and Surfaces for Object Placement in Indoor Environments
,”
2010 10th IEEE-RAS International Conference on Humanoid Robots
,
Nashville, TN
, pp.
152
159
.
22.
Jiang
,
Y.
,
Lim
,
M.
,
Zheng
,
C.
, and
Saxena
,
A.
,
2012
, “
Learning to Place New Objects in a Scene
,”
Int. J. Robot. Res.
,
31
(
9
), pp.
1021
1043
. 10.1177/0278364912438781
23.
Teodorescu
,
C. S.
,
Vandenplas
,
S.
,
Depraetere
,
B.
,
Anthonis
,
J.
,
Steinhauser
,
A.
, and
Swevers
,
J.
,
2016
, “
A Fast Pick-and-place Prototype Robot: Design and Control
,”
IEEE Conference on Control Applications (CCA2016)
,
Buenos Aires, Argentina
, pp.
1414
1420
.
24.
Rengervé
,
A. D.
,
Hirel
,
J.
,
Andry
,
P.
,
Quoy
,
M.
, and
Gaussier
,
P.
,
2011
, “
On-line Learning and Planning in a Pick-and-place Task Demonstrated Through Body Manipulation
,”
IEEE International Conference on Development and Learning (ICDL)
,
Genova, Italy
, Vol.
2
, pp.
1
6
.
25.
Correll
,
N.
,
Bekris
,
K. E.
,
Berenson
,
D.
,
Brock
,
O.
,
Causo
,
A.
,
Hauser
,
K.
,
Okada
,
K.
,
Rodriguez
,
A.
,
Romano
,
J. M.
, and
Wurman
,
P. R.
,
2016
, “
Analysis and Observations From the First Amazon Picking Challenge
,”
IEEE Transactions on Automation Science and Engineering
,
15
(
1
), pp.
172
188
. 10.1109/TASE.2016.2600527
26.
Patil
,
G. G.
,
2013
, “
Vision Guided Pick and Place Robotic Arm System Based on SIFT
,”
Int. J. Sci. Eng. Res.
,
4
(
12
), pp.
242
248
.
27.
Cosgun
,
A.
,
Hermans
,
T.
,
Emeli
,
V.
, and
Stilman
,
M.
,
2011
, “
Push Planning for Object Placement on Cluttered Table Surfaces
,”
2011 IEEE/RSJ International Conference on Intelligent Robots and Systems
,
San Francisco, CA
, pp.
4627
4632
.
28.
Ongaro
,
F.
,
Yoon
,
C.
,
Abayazid
,
M.
,
Oh
,
S. H.
,
Gracias
,
D. H.
, and
Misra
,
S.
,
2016
, “
Control of Untethered Soft Grippers for Pick-and-place Tasks
,”
6th IEEE International Conference on Biomedical Robotics and Biomechatronics (BioRob)
,
Singapore
, pp.
299
304
.
29.
Kootbally
,
Z.
,
Kramer
,
T.
,
Schlenoff
,
C.
, and
Gupta
,
S.
,
2017
, “
Overview of an Ontology-Based Approach for Kit Building Applications
,” pp.
520
525
.
30.
Altan
,
D.
, and
Sariel
,
S.
,
2020
, “
What Went Wrong?: Identification of Everyday Object Manipulation Anomalies
,”.
ArXiv, abs/2001.09084, preprint submitted to Elsevier
.
31.
Kothari
,
C. R.
,
2004
,
Research Methodology : Methods & Techniques
,
New Age International (P) Ltd, New Delhi, India
.
32.
Blessing
,
L. T.
, and
Chakrabarti
,
A.
,
2009
,
DRM, A Design Research Methodology
,
Springer-Verlag London, London, UK
.
33.
Council
,
D.
,
2007
,
Eleven Lessons: Managing Design in Eleven Global Brands. A Study of the Design Process
. Technical Report, Design Council.
34.
Edsinger
,
A.
, and
Kemp
,
C. C.
,
2006
, “
Manipulation in Human Environments
,”
2006 6th IEEE-RAS International Conference on Humanoid Robots
,
Genova, Italy
, pp.
102
109
.
35.
Yamashita
,
T.
,
Godler
,
I.
,
Takahashi
,
Y.
,
Wada
,
K.
, and
Katoh
,
R.
,
1991
, “
Peg-and-Hole Task by Robot With Force Sensor: Simulation and Experiment
,”
Proceedings IECON’91: 1991 International Conference on Industrial Electronics, Control and Instrumentation
,
Kobe, Japan
, Vol.
2
, pp.
980
985
.
36.
Bo
,
C.
,
Hu
,
H.
,
Xu
,
J.
,
Liu
,
Z.
, and
Chai
,
P.
,
2017
, “
Classification Learning Method for PCB Insertion Holes Based on Shape Context
,”
Third IEEE International Conference on Computer and Communications (ICCC)
,
Chengdu, China
, pp.
1933
1937
.
37.
Wang
,
P.
,
Qin
,
Z.
,
Xiong
,
Z.
,
Lu
,
J.
,
Xu
,
D.
,
Yuan
,
X.
, and
Liu
,
C.
,
2015
, “
Robotic Assembly System Guided by Multiple Vision and Laser Sensors for Large Scale Components
,”
IEEE International Conference on Robotics and Biomimetics (ROBIO)
,
Zhuhai, China
, pp.
1735
1740
.
38.
Karako
,
Y.
,
Moriya
,
T.
,
Abe
,
M.
,
Shimakawa
,
H.
,
Shirahori
,
S.
, and
Saitoh
,
Y.
,
2017
, “
A Practical Simulation Method for Pick-and-place with Vacuum Gripper
,”
56th Annual Conference of the Society of Instrument and Control Engineers of Japan (SICE)
,
Kanazawa, Japan
, pp.
1351
1356
.
39.
Cappelleri
,
D. J.
,
Krishnan
,
G.
,
Kim
,
C.
,
Kumar
,
V.
, and
Kota
,
S.
,
2010
, “
Toward the Design of a Decoupled, Two-Dimensional, Vision-Based μN Force Sensor
,”
ASME J. Mech. Rob.
,
2
(
2
), p.
021010
. 10.1115/1.4001093
40.
Nozu
,
K.
, and
Shimonomura
,
K.
,
2018
, “
Robotic Bolt Insertion and Tightening Based on In-Hand Object Localization and Force Sensing
,”
2018 IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM)
,
Auckland, New Zealand
, pp.
310
315
.
41.
Hayakawa
,
S.
,
Yamada
,
Y.
, and
Tsuchida
,
N.
,
2004
, “
Shaft Insertion for Moving Object by Using Robot Manipulator With One Dimensional PSDs Sensor
,”
Proceedings of the 2004 IEEE International Conference on Control Applications
,
Taipei, Taiwan
, pp.
21118
21123
.
42.
De Gregorio
,
D.
,
Zanella
,
R.
,
Palli
,
G.
,
Pirozzi
,
S.
, and
Melchiorri
,
C.
,
2019
, “
Integration of Robotic Vision and Tactile Sensing for Wire-Terminal Insertion Tasks
,”
IEEE Trans. Auto. Sci. Eng.
,
16
(
2
), pp.
585
598
. 10.1109/TASE.2018.2847222
43.
Erkent
,
O.
,
Shukla
,
D.
, and
Piater
,
J.
,
2017
, “
Visual Task Outcome Verification Using Deep Learning
,”
2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
,
Vancouver, BC, Canada
, pp.
4821
4827
.
44.
Yuan
,
P.
,
Zhong
,
Y.
, and
Yuan
,
Y.
,
2017
, “
Faster R-CNN With Region Proposal Refinement
”.
45.
Krizhevsky
,
A.
,
Sutskever
,
I.
,
Hinton
,
G. E.
,
Pereira
,
F.
,
Burges
,
C. J. C.
,
Bottou
,
L.
, and
Weinberger
,
K. Q.
,
2012
,
Imagenet Classification With Deep Convolutional Neural Networks
,
Curran Associates Inc.
,
New York City, NY
, pp.
1097
1105
.
46.
Zhang
,
Z.
,
Liu
,
J.
,
Wang
,
X.
,
Zhao
,
Q.
,
Zhou
,
C.
,
Tan
,
M.
,
Pu
,
H.
,
Xie
,
S.
, and
Sun
,
Y.
,
2017
, “
Robotic Pick-and-Place of Multiple Embryos for Vitrification
,”
IEEE Rob. Auto. Lett.
,
2
(
2
), pp.
570
576
. 10.1109/LRA.2016.2640364
47.
Komati
,
B.
,
Kudryavtsev
,
A.
,
Clévy
,
C.
,
Laurent
,
G.
,
Tamadazte
,
B.
,
Agnus
,
J.
, and
Lutz
,
P.
,
2016
, “
Automated Robotic Microassembly of Flexible Optical Components
,”
IEEE International Symposium on Assembly and Manufacturing (ISAM)
,
Fort Worth, TX
, pp.
93
98
.
48.
Harada
,
K.
,
Tsuji
,
T.
,
Nagata
,
K.
,
Yamanobe
,
N.
,
Onda
,
H.
,
Yoshimi
,
T.
, and
Kawai
,
Y.
,
2012
, “
Object Placement Planner for Robotic Pick and Place Tasks
,”
IEEE/RSJ International Conference on Intelligent Robots and Systems
,
Algarve, Portugal
, pp.
980
985
.
49.
Toris
,
R.
,
Kent
,
D.
, and
Chernova
,
S.
,
2015
, “
Unsupervised Learning of Multi-hypothesized Pick-and-Place Task Templates Via Crowdsourcing
,”
2015 IEEE International Conference on Robotics and Automation (ICRA)
,
Seattle, WA
, pp.
4504
4510
.
50.
Cortes
,
C.
, and
Vapnik
,
V.
,
1995
, “
Support-Vector Networks
,”
Mach. Learn.
,
20
(
3
), pp.
273
297
.
51.
Baumgartl
,
J.
,
2016
, “
Schnelle, konservative Greif- und Ablageplanung von unbekannten Objekten
”.
PhD thesis
,
Universit ̈ut Bayreuth
,
Bayreuth, Germany
.
52.
Balakirsky
,
S.
, and
Kootbally
,
Z.
,
2014
,
An Ontology Based Approach to Action Verification for Agile Manufacturing
,
Springer International Publishing
,
Cham
, pp.
201
217
.
53.
Kootbally
,
Z.
,
Schlenoff
,
C.
,
Weisman
,
T.
,
Balakirsky
,
S.
,
Kramer
,
T.
, and
Pietromartire
,
A.
,
2014
,
A Simulated Sensor-Based Approach for Kit Building Applications
,
Springer International Publishing
,
Cham
, pp.
241
257
.
54.
Hochreiter
,
S.
, and
Schmidhuber
,
J.
,
1997
, “
Long Short-Term Memory
,”
Neural Comput.
,
9
(
8
), pp.
1735
1780
. 10.1162/neco.1997.9.8.1735
55.
Chandola
,
V.
,
Banerjee
,
A.
, and
Kumar
,
V.
,
2009
, “
Anomaly Detection: A Survey
,”
ACM Comput. Surv.
,
41
(
3
), pp.
1
72
. 10.1145/1541880.1541882
56.
Pimentel
,
M. A. F.
,
Clifton
,
D. A.
,
Clifton
,
L. A.
, and
Tarassenko
,
L.
,
2014
, “
A Review of Novelty Detection
,”
Signal Process.
,
99
(
1
), pp.
215
249
. 10.1016/j.sigpro.2013.12.026
57.
Redmon
,
J.
, and
Farhadi
,
A.
,
2018
, “
Yolov3: An Incremental Improvement
,”
Preprint available online at arXiv:1804.02767
.
58.
Schölkopf
,
B.
,
Williamson
,
R. C.
,
Smola
,
A. J.
,
Shawe-Taylor
,
J.
,
Platt
,
J. C.
,
Solla
,
S. A.
,
Leen
,
T. K.
, and
Müller
,
K.
,
2000
,
Support Vector Method for Novelty Detection
,
MIT Press
,
Cambridge, MA
, pp.
582
588
.
59.
Zoghlami
,
F.
,
Kurrek
,
P.
,
Jocas
,
M.
,
Masala
,
G.
, and
Salehi
,
V.
,
2020
, “
Unsupervised Pose Anomaly Detection for Dynamic Robotic Environments
,”
Third IEEE International Conference on Industrial Cyber-Physical Systems
,
Tampere, Finland
.
60.
Schlegl
,
T.
,
Seeböck
,
P.
,
Waldstein
,
S. M.
,
Schmidt-Erfurth
,
U.
, and
Langs
,
G.
,
2017
, “
Unsupervised Anomaly Detection With Generative Adversarial Networks to Guide Marker Discovery
,”
25th International conference on Information Processing in Medical Imaging (IPMI 2017)
,
Boone, NC
.
You do not currently have access to this content.