As a cutting-edge technology that continues to integrate and develop, machine vision can help all kinds of machines collect visual data, analyze and perform specific tasks, and play an important role in the automation development, transformation and upgrading of various industries. Based on the analysis of machine vision and related concepts, this paper comprehensively introduces the research and application of machine vision technology at home and abroad, and analyzes and looks forward to the development trend of machine vision technology.
Machine vision and related concepts
1. Definition and historical evolution of machine vision
Since the dawn of human civilization, effective technologies have been invented to record, replicate and expand what the human eye sees. From the 15th century, which was used as an auxiliary tool for artists to paint, to today's area scan cameras that can accurately locate and measure objects, and smart cameras that integrate image acquisition, processing, and communication, the technology of using the "machine eye" instead of the human eye to make measurements and judgments is becoming more and more mature. In addition, machine vision is no longer limited to allowing machines to "see" the world like humans, but also allows machines to "see", "understand" and "execute", and realize or even surpass the entire process of receiving, processing and feeding back information with human eyes.
Nowadays, machine vision system has developed into an interdisciplinary comprehensive technology and has become an important branch in the field of artificial intelligence, which is widely used in manufacturing (especially in 3C electronic manufacturing, automobile manufacturing, photovoltaic lithium battery and other subdivisions), and has been applied in logistics, transportation, agriculture, tobacco, medical and other fields. As a cutting-edge technology that continues to integrate and develop, the connotation and extension of machine vision have also expanded. Machine vision technology involves many fields such as computer vision, image processing, pattern recognition, artificial intelligence, signal processing, and optical mechatronics [1].
For example, in the article "The Development of Machine Vision Technology and Its Industrial Application", Zhang Wei argues that machine vision refers to a discipline in which a computer imitates human visual functions, obtains visual information such as color and size from observed objects, and then further digitizes the information for detection and control [2].
The United States Association for Advancing Automation (A3) defines machine vision as encompassing all industrial and non-industrial applications, providing operational guidance for equipment to perform functions based on the capture and processing of images through a combination of hardware and software.
Cognex, a global leader in the machine vision industry, believes that machine vision is an important part of the interaction between digital systems and the real world, allowing automated systems to see parts, products, patterns, codes, or other objects and use that information to make decisions.
KEYENCE, another leading company in the machine vision industry, believes that machine vision systems combine industrial cameras, lenses, and lighting equipment to automatically visually inspect finished products. Vision systems are used in a wide range of applications, such as defect detection, assembly inspection, character and code reading, and industrial robot positioning.
2. Machine vision and computer vision
With the increasing demand for high precision and efficiency in various fields, the synergy between artificial intelligence and vision systems is particularly important. AI augments machine vision and computer vision with its data-driven insights and predictive capabilities. Machine vision and computer vision both belong to the field of artificial intelligence, and they are often used interchangeably, and their identification will help to further develop the discussion in this paper.
Machine vision is biased towards system engineering, which is composed of software and hardware, and its main purpose is to help all kinds of machines collect visual data and analyze and perform specific tasks, which is mainly used in the industrial field and has high requirements for hardware. Machine vision systems help improve production efficiency and product quality, and can perform tasks such as identification, detection, measurement, and robot guidance (assisting intelligent assembly and sorting).
Computer vision, on the other hand, is biased towards computer science, with software as the core, and is dominated by algorithms and models, and its main purpose is to simulate human interpretation and understanding of the visual world, allowing machines to make decisions based on visual data. Computer vision does not need to perform physical tasks, and its application fields are wider than machine vision, such as face recognition, virtual reality, augmented reality, etc., and the environment it faces is more challenging, and the light and environment are uncontrollable.
3. Machine vision system structure
From the above definition and analysis, we can extract the three keys of machine vision system, namely image acquisition, imaging analysis and processing, and decision execution. A typical machine vision system is composed of two parts: hardware equipment and software system, and the software part is the image processing system, which overlaps with computer vision. Its hardware generally includes lighting systems, imaging systems and visual information processing systems, including light source equipment, lenses, industrial cameras (CCD/CMOS), image grabbers, industrial hosts, image processors, PLCs, etc.
(1) Lighting system: The main light sources of machine vision system include halogen lamps, fluorescent lamps, xenon lamps, LEDs, lasers, infrared, X-rays, etc.
(2) Imaging system: The machine vision system uses lenses, industrial cameras, image grabbers and other related equipment to obtain high-quality images of the observed target and transmit them to the processing system.
(3) Visual information processing system: process and analyze the collected images, realize the detection, analysis and recognition of specific targets, and make corresponding decisions.
Research and application of machine vision technology at home and abroad
1. Research and application in the field of machine vision abroad
The study of machine vision originated from the study of "vision". In the 50s of the 20th century, the United States biologist David · Hubel and Sweden biologist Torsten · Wiesel used animal experiments to discover and analyze the conduction of nerve impulses from the retina to the sensory and motor centers of the brain [3], laying the foundation for the study of the visual nervous system and opening up the in-depth exploration of the field of "vision". In 1957, the world's first digital image (PDF) was created (see Figure 1), and Russell · Kirsch, the inventor of the "pixel", mounted his son's photograph on a scanner and digitized the image by transmitting 1 and 0 to the machine through a photocell[4], and it became possible to process digital images. Subsequently, the statistical pattern recognition research of 2D images by all parties started the "sailing journey" of machine vision as an application engineering.
Figure 1: Russell · Kirsch, the father of "pixels", holds the world's first digital image--- a 76x76 pixel photograph
In 1963, Roberts, L.G. published a doctoral dissertation entitled "Machine Perception of Three-Dimensional Entities" [5], in which he proposed the process of deriving three-dimensional information from two-dimensional pictures (see Fig. 2), and his research brought great inspiration to the field of computer vision computing. In the same year, Morrison released the Computable Sensor, a structure that could determine the position of the spot using the light guide effect, which became the beginning of the development of CMOS image sensors. In 1969, the CCD sensor (charge-coupled element) was developed by W.S. Bell Laboratories. Invented by Boyle and G.E. Smith, this critical piece of image acquisition hardware converts photons into electrical pulses for high-quality digital image acquisition tasks in industrial camera sensors.
Fig.2 The process diagram of deriving three-dimensional information from two-dimensional images in the paper "Machine Perception of Three-dimensional Entities".
In the 70s of the 20th century, the Massachusetts Institute of Technology Artificial Intelligence Laboratory officially opened the "Machine Vision" course, during this period, many scholars began to participate in the research of machine vision theory, algorithm, and system design, and the famous Marr vision theory was born in this period. In 1982, Cognex, which had just been formed, manufactured the world's first vision system, DataMan. DataMan is an industrial optical character recognition (OCR) system capable of reading, verifying, and confirming letters, numbers, and symbols directly marked on parts and assemblies [6]. After that, machine vision systems began to be applied in the field of automation, and machine vision has set off a research boom around the world.
In 2006, with the introduction of the concept of deep learning and the popularization and application of algorithms such as convolutional neural networks and recurrent neural networks, machines can independently establish recognition logic through training, and the accuracy of image recognition has been greatly improved, and the development of machine vision has entered a new stage [7]. Currently, the total number of related papers and monographs is more than 150,000 and is increasing by more than 10,000 per year. Its downstream application fields have also been continuously expanded, from the original industry, agriculture to logistics, food and beverage, military, medical and other fields, becoming the preferred "weapon" for automation, transformation and upgrading of various industries.
2. Research and application in the field of machine vision in China
The research in the field of machine vision in mainland China started in the late 80s of the 20th century, when the academic field began to introduce foreign machine vision related technologies on the one hand, and on the other hand, the research team of mainland China began to carry out applied research on machine vision and computer vision in various industries.
During this period, the mainland's research on the use of machine vision to achieve automation mainly focused on the agricultural field, including animal and plant growth detection, agricultural product quality inspection, agricultural product harvesting and processing automation, agricultural robot visual navigation, etc., and there was little research in the industrial field and transportation field.
Before the 21st century, the applied research of machine vision in mainland China has been in a tepid state; In 2004~2014, the number of related studies began to increase, and the number of research papers published per year exceeded 100, entering the initial stage; Since 2014, it has seen explosive growth, with more than 800 articles published annually at its peak (see Figure 3). (Note: In fact, a large number of domestic scholars, especially science and engineering scientists, will choose to publish English papers in international journals with higher influence, and the data here is only the number of Chinese papers collected by CNKI.) From the perspective of the market, the gap between domestic and foreign product technology is not as large as the gap in the number of Chinese and English papers. ）
Fig.3 Statistics on the number of scientific research papers published in the field of machine vision in China (data source: CNKI
In 2004~2014, the domestic machine vision market is also in the early stage of development, although a certain stage of results has been achieved, but there is a large gap between the market development degree and the application of research results and the developed country market. In the field of scientific research, the related research of machine vision mainly focuses on defect detection and identification, and the field of research and design has gradually broadened from agricultural engineering to industrial general equipment, chemical metallurgy, electronics, automobile industry, power industry, transportation, aerospace and other fields, and has produced a number of academic achievements that can be applied to the ground.
For example, Lingyun studied the detection technology of grain appearance quality under static and dynamic conditions, and designed and developed a set of detection devices suitable for the appearance quality analysis of grain in mainland China [8]. Xu Qiaoyou has developed a machine vision system for part identification or defect detection [9]; Bi Xin and Ding Han established an automatic mura defect detection process based on the importance of mura defect detection in the liquid crystal display process and the drawbacks of manual inspection [10]. Wang Lei took the button battery as the object to study the surface appearance defect detection method of the positive and negative pole surfaces[11]; Gui et al. applied machine vision technology to the monitoring of the mineral flotation process [12].
In addition, some innovations have been made in the use of machine vision technology for robot-assisted sorting. For example, Liu Zhenyu et al. completed the construction of an industrial robot sorting system platform based on machine vision [13]; Yan Zugen et al. designed a machine vision hardware system for the grading and positioning of sports foods in response to the needs of the food production industry in mainland China [14].
After 2014, a large number of local suppliers and integrators in the field of machine vision began to emerge, and related research has also penetrated into all aspects of various industries, and scientific research results have ushered in explosive growth, and the number of review studies has increased, such as Tang Bo et al. have done a review study on the detection of surface defects in machine vision in the industrial field [15]; Zhu Yun et al. systematically summarized the key technologies, application fields, challenges, and development trends of machine vision technology [16], and so on. At this time, the popularization and application of deep learning algorithms in the field of computer vision has also led to the innovation of machine vision systems. For example, the State Key Laboratory of Precision Testing Technology and Instruments at Tianjin University systematically summarized the application results of machine vision technology in the field of modern automobile manufacturing in three aspects: visual measurement, visual guidance, and visual inspection [17]; In order to meet the huge demand for machine vision technology in the intelligent manufacturing equipment industry, Wang Yaonan et al. proposed a general machine vision detection and control technology system based on the characteristics of equipment technology and special application environment [18]. Tang Bo et al. conducted a review study on the detection of surface defects in machine vision in the industrial field [19]; Zhu Yun et al. systematically summarized the key technologies, application fields, challenges, and development trends of machine vision technology [20].
The development of the mainland machine vision market is basically synchronized with the research in the field of scientific research. Around 2014, the application of machine vision in the field of scientific research in the field of automation industry and agricultural automation has grown explosively. Around 2013, the machine vision industry in China also became the third largest machine vision market after United States and Japan, and the technology of local manufacturers continued to break through, and the penetration rate of machine vision in industrial fields such as automobiles, electronics, and semiconductors also reached a high level [21].
The development trend and prospect of machine vision technology
Generally speaking, there is a "time difference" between the acceptance and application of technical products in the mainstream market and the theoretical research and technology research and development of scientific research institutions, and machine vision technology is no exception. Machine vision as a science was established in the 70s of the 20th century, the first officially landed machine vision system was born ten years later, the machine vision industry was formally formed in the 90s of the 20th century, and the global machine vision market of 10 billion US dollars [22] was broken through in 2020.
The development of machine vision technology is closely related to the rise of market demand in two aspects: first, traditional industries are seeking transformation due to high costs and labor shortages (especially during the epidemic), and the demand for automation technology has surged; Second, in the development of emerging industries, due to their own needs, they naturally use machine vision technology to improve accuracy and work efficiency and achieve automation. Under the influence of factors such as the upgrading of intelligent manufacturing, the improvement of robot technology, and the innovation of image processing theory, machine vision technology will also develop towards the trend of 3D, integration and intelligence.
1.3D machine vision
From black and white to color, low to high resolution, still images to moving images, these industry innovations have always revolved around 2D machine vision. However, there are always drawbacks to compressing 3D space into a 2D plane. For example, the measurement accuracy is greatly affected by lighting and color changes, and the object recognition and processing ability in complex environments is insufficient, so 2D machine vision cannot fully meet the accurate measurement of object shape, volume, thickness, and position in practical application scenarios.
With the continuous optimization of algorithms and theoretical applications such as binocular vision, point cloud processing, structured light 3D measurement, 3D reconstruction, and machine learning, 3D machine vision that can capture the spatial information of objects will become one of the promising directions in the future. For example, disorderly sorting, bin picking, and dismantling/palletizing operations that are difficult for robots/robotic arms to handle can be successfully completed by identifying the spatial coordinates of the gripping position and rationally planning the path through the 3D machine vision system.
2. Embedded vision applications
At present, traditional PC (or board) machine vision systems and embedded vision systems coexist in the market, and PC-based machine vision is dominant. However, as the field of machine vision applications continues to expand, embedded vision systems will also usher in development. The PC-type machine vision system belongs to the traditional vision system structure, with complex structure, large size and long development cycle, but it is ideal in terms of accuracy and speed, and can provide support in the field of industrial automation and agricultural automation. The embedded vision system is a new type of system that combines "embedded system" and "machine vision", which has the advantages of easy integration, miniaturization and low cost. Embedded vision system is an important part of machine vision "out" of the factory floor, and it has great promise in the security and military fields in the future.
3. Intelligent machine vision products
At present, the trend of adopting AI in machine vision systems has already begun to appear. Humans have spent decades allowing machines to "see" and "interpret" the flat world, and the combination of AI and 3D machine vision has made it possible for machines to "understand" the real world. With the support of AI technology, machine vision will have intelligent capabilities beyond existing solutions, which can be adaptive, can perceive the environment autonomously, can extract key features from the collected visual information, and quickly make judgments based on deep learning algorithms. If machine vision products want to further expand their acceptance in the existing market, software algorithms are adaptive, tailored to needs, and the system is easy to operate and does not require special engineer maintenance. Under these two goals, the introduction of AI is particularly important. AI can provide higher system flexibility and operational flexibility, further meeting the needs of the manufacturing industry for product inspection and quality control, as well as the demand for intelligent collaborative robots in the warehousing and logistics industry.
Bibliography:
[1] Written. Research on the application of automation technology in mechanical manufacturing based on machine vision technology[J].Science and Technology Outlook,2017,27(4): 260.)
[2] Zhang Wei. Development of Machine Vision Technology and Its Industrial Application[J], 2006, (02): 11-17
[3] Britannica, T. Editors of Encyclopaedia (2024, April 16). Torsten Wiesel. Encyclopedia Britannica. https://www.britannica.com/biography/Torsten-Wiesel
[4] Russell Kirsch: Pixel Pioneer And The Father Of Digital Imaging | Hackaday
[5] Roberts, L.G. (1963). Machine Perception of Three-Dimensional Solids. Outstanding Dissertations in the Computer Sciences.
[6]https://www.cognex.cn/zh-cn/company/history
[7] Song Chunhua, Peng Xunzhi. Equipment Manufacturing Technology,2019,(06):213-216.
[8] Ling Yun. Research on grain appearance quality detection technology based on machine vision[D].China Agricultural University,2004.
[9] Xu Qiaoyou. Research on part recognition and detection based on machine vision system[D].Nanjing University of Aeronautics and Astronautics,2006.
[10] Bi Xin, Ding Han. Journal of Mechanical Engineering,2010,46(12):13-19.)
[11] Wang Lei. Research on battery surface defect detection technology based on machine vision[D].University of Science and Technology of China,2011.
[12] Gui Weihua, Yang Chunhua, Xu Degang, et al. Research progress on monitoring technology of mineral flotation process based on machine vision[J].Acta Automatica Sinica,2013,39(11):1879-1888.)
[13] Liu Zhenyu, Li Zhongsheng, Zhao Xue, et al. Research on sorting technology of industrial robot based on machine vision[J].Manufacturing Automation,2013,35(17):25-30.)
[14] Yan Zugen, Li Ming, Xu Kefei, et al. Research on machine vision technology of high-speed robot sorting system[J].Packaging and Food Machinery,2014,32(01):28-31.)
[15] Tang Bo, Kong Jianyi, Wu Shiqian. Chinese Journal of Image and Graphics,2017,22(12):1640-1663.)
[16] Zhu Yun, Ling Zhigang, Zhang Yuqiang. Journal of Graphics,2020,41(06):871-890.)
[17] Yin Shibin, Ren Yongjie, Liu Tao, et al. A review on the application of machine vision technology in modern automobile manufacturing[J].Acta Optica Sinica,2018,38(08):11-22.)
[18] Wang Yaonan, Chen Tiejian, He Zhendong, et al. Control Theory and Application,2015,32(03):273-286.)
[19] Tang Bo, Kong Jianyi, Wu Shiqian. Chinese Journal of Image and Graphics,2017,22(12):1640-1663.)
[20] Zhu Yun, Ling Zhigang, Zhang Yuqiang. Journal of Graphics,2020,41(06):871-890.)
[21] Qianzhan Industry Research Institute, "Panorama of China's Machine Vision Industry in 2023"
[22]市场调研机构markets&markets，Machine Vision Market Size, Share, Industry Trends and Top Companies Growth- [2032] (marketsandmarkets.com)