author
Liu Huiyuan, Gongbi Painting Institute, Chinese Academy of Arts
This article was originally published in Art Studies, No. 2, 2024
Based on interviews with experts in the restoration of Chinese ancient paintings, research and analysis of the characteristics of artificial intelligence virtual restoration, the research group of the National Social Science Foundation of China Art Youth Project "Research on the Restoration and Restoration Method of Ancient Paintings Based on Neural Network Algorithm" presided over by the author proposes a scientific method for the restoration of digital images of Chinese ancient paintings, including the restoration process, evaluation criteria and algorithm selection. At the same time, through algorithm screening, training, and optimization, the research group finally obtained the artificial intelligence deep learning algorithm ACP-LaMa, which is suitable for the restoration of ancient Chinese paintings. The results of this algorithm are suitable for the image restoration of complex ancient paintings with high definition, and can ideally restore the artistic characteristics of ancient Chinese paintings. In 2023, the author will apply for the scientific research project of the Chinese Academy of Arts, "Research on Algorithm Restoration of Chinese Ancient Painting Images", to further optimize the restoration process and algorithm performance, and expand the application of algorithms to calligraphy works and arts and crafts image restoration. At present, the results of the project have received the attention and recognition of many experts in the field of ancient painting restoration, and have been declared for national invention patents and cooperated with restoration institutions.
1. The significance of artificial intelligence restoration of ancient paintings
(1) Assist in the restoration of traditional ancient paintings
The artificial intelligence virtual restoration of ancient painting images can provide a variety of visual virtual restoration results for restoration workers as a reference for restoration in a short time. This method of concretizing and clarifying the original abstract restoration goals can effectively reduce the investment in restoration preparation, and ultimately improve the timeliness, accuracy and reliability of the restoration of physical ancient paintings.
(2) Assist in the protection of ancient paintings and cultural relics
The virtual restoration of digital images does not touch the original work, which can avoid contact and damage to cultural relics. At the same time, the restoration reports and annotations completed in the virtual restoration process can also be used as academic materials for the auxiliary work of cultural relics protection.
(3) Serve the promotion of excellent traditional culture in the mainland
Through digital restoration, the dilapidated and ancient Chinese paintings are transformed into image works that meet the aesthetic preferences of the public, which can be more widely disseminated and displayed, which is a beneficial way to inherit the excellent traditional Chinese culture.
2. Difficulties in the artificial intelligence restoration of ancient paintings
(1) High-definition image restoration
The prior art is mostly aimed at images in the category of low-definition photographs, while the images of ancient paintings scanned or photographed are of higher definition and larger files.
(2) Large-area image restoration capabilities
The repair algorithm can understand the global image information of the picture, instead of only inferring the defect based on a small number of pixels around the defect, and it is easy to accurately fill the large area of defects.
(3) To better express the artistic characteristics of ancient Chinese paintings
The restoration algorithm can better express the texture, texture and color transition of the picture, the line restoration needs to conform to the brush used in Chinese painting, and the restoration area can show the artistic characteristics of ancient Chinese painting.
(4) Objectivity of restoration results
Whether the data used for training is sufficient and whether the data is professional and convergent directly affects whether the repair results can be objectively better than human judgment.
(5) Algorithm process optimization
The computing process requires high efficiency and resource optimization to avoid problems such as long computing time and excessive use of computing resources.
(6) Simple and easy to operate
The restoration process should be convenient for non-computer technicians to learn and use, so as to be conducive to the popularization and use of cultural relics protection and art research workers.
3. Systematic construction of artificial intelligence restoration methods for ancient Chinese paintings
The traditional artificial restoration of ancient paintings is a systematic profession with a complete process and clear standards, and has an independent academic position. Therefore, the algorithm restoration of ancient paintings should refer to the historical experience of manual restoration, and summarize a set of scientific, systematic, reliable technical schemes, restoration processes and evaluation criteria that conform to the logic of digital image virtual restoration.
Based on the current situation and needs of the restoration of ancient Chinese paintings, the research group interviewed more than 10 experts and scholars in the field of cultural relics restoration and protection, and sorted out the implementation plan suitable for the intelligent restoration of digital images of Chinese ancient paintings. Deep learning algorithm is the most promising AI image restoration method at present, so the research group takes it as the core goal of research and experiments.
(1) Data preparation
The traditional restoration of ancient paintings mainly relies on data, experience and learning, while the performance of artificial intelligence algorithms relies on a large amount of relevant data training. In this project, ancient Chinese paintings are selected as the basis for data training.
1. Data collection and establishment of data sets
The research group has sorted out more than 20,000 images of ancient Chinese paintings, which are divided into 300,000 images of ancient paintings through customized software, and continue to update and supplement.
2. Data classification and processing
The data are divided into six sub-databases, as follows (Table 1):
Table 1 List of images of ancient Chinese paintings in a sub-database
The sub-databases listed above all involve data comparison image sets so that the algorithm can converge more easily, as shown in Figure 1.
Figure 1-1 Song "Eighty-Seven Immortals" fairy.
Figure 1-2 Song Dynasty "Dynasty Yuan Immortal Battle Picture" fairy.
Figure 1-3 The fairy of the Qing Dynasty "Chao Yuan Tu".
Fig.1 Comparison of ancient Taoist scroll paintings and fairy paintings.
(2) Repair process design
1. Process design basis
If the algorithm is to play a full role in the actual work of ancient painting restoration, it is necessary to refer to the manual restoration process and principles of traditional ancient paintings, and design a complete algorithm restoration steps or software tools.
(1) "Washing": Through the image "denoising" algorithm, remove the fine mottles and stains on the picture.
(2) "Reveal": Reset the dislocation, settlement and fragmentation in the picture through image editing software.
(3) "Repair": Through the image restoration algorithm of ancient Chinese paintings, the large fracture and large area of the picture are repaired.
(4) "Complete": Through the image restoration algorithm of ancient Chinese paintings, small diseases, discoloration and missing line segments are repaired.
2. Fix the implementation process
In order to achieve the optimal restoration of digital images of ancient paintings, it is necessary to complete multiple restoration steps together, and the evaluation criteria of each step need to be determined. First of all, before restoring ancient paintings, it is necessary to mark the damage, collect colors, and formulate a restoration strategy. Complex diseases should be repaired in layers, and gradually realize the matching and filling from texture, color to line. Secondly, in the repair process, each step needs to compare the repair standards, and the algorithm needs to be repaired multiple times if the standards are not met. Finally, the results are analyzed, the problems are summarized and the report is written, and the unsatisfactory areas can be repaired with the assistance of drawing software. Based on the actual experimental experience, our research group summarized the repair flow chart (Fig. 2):
Fig.2 Workflow diagram of the restoration algorithm of intelligent restoration algorithm for ancient Chinese paintings.
(3) Visual standards and evaluation plans for the restoration of ancient Chinese paintings
By comparing the traditional painting and calligraphy restoration standards, combined with expert guidance and research group discussions, we have identified "washing", "uncovering", "supplementing" and "complete" as the evaluation criteria for the digital restoration of Chinese ancient paintings. The traditional painting and calligraphy restoration process pays attention to achieving "four-sided light", so digital restoration should also be highly matched and integrated. In terms of restoration goals, it is required to "repair the old as the old"; The degree of repair can be divided into "mild", "moderate" and "highly repaired".
(4) The overall planning of the intelligent restoration algorithm system of Chinese ancient paintings
Based on the discipline, systematization and application, the research group made the basic plan of the intelligent restoration algorithm system of Chinese ancient paintings based on data preparation, algorithm screening, process design and evaluation criteria, which are as follows (Table 2 and Figure 3):
Table 2 The basic planning of the intelligent restoration algorithm system of ancient Chinese paintings
(Above) Figure 3-1 The original picture of the Bodhisattva mural in Shanxi. Figure 3-2 "Wash" to remove image noise. Figure 3-3 "Removal" The image is broken and the staggered layer is reset.
Figure 3-4 "Patch" Obvious fracture and detachment. Figure 3-5 "Full" – "Panchromatic" Shedding in a small area. Figure 3-6 "Complete" - "Pick up the pen" Fine-tuning the line segment and picture.
Figure 3-7 Comparison of partial "full" restoration.
Figure 3-8 Partial "full" restoration comparison 2. Fig.3 Demonstration of the intelligent restoration process of the Bodhisattva mural image and the comparison of the local restoration results.
4. Research and experimental analysis of the restoration algorithm of ancient Chinese paintings
(1) Algorithm screening
The research group investigated, analyzed and experimented on nearly 60 algorithm schemes, and the analysis process is as follows:
Firstly, the screening process excludes traditional algorithms that do not have the ability to learn, and the repair accuracy of such algorithms is low
Secondly, in the deep learning algorithm, most of the images that can be processed have low pixel level and insufficient computing power, which does not conform to the data characteristics of ancient painting scanning or photographic images
Finally, algorithms with poor image understanding ability and no large-area defect repair ability are eliminated.
(2) Algorithm optimization and training
Based on a large number of algorithm investigations and experiments, the research group selected and adopted the "Fourier convolution-based" large-scale mask repair method suitable for large-area image defects, and obtained the ideal restoration method, ACP LaMa, by optimizing the LaMa mask strategy in the algorithm and using the "Chinese Ancient Painting Dataset" mentioned above as the algorithm training object. The algorithm has the following advantages:
First, the project uses a fast Fourier convolution (FFC) restoration network, which allows the operation to obtain a receptive field covering the entire image range, so that the network can be generalized to handle high-resolution images. Secondly, the perceptual loss function based on the high receptive field semantic segmentation network can make the algorithm better understand the global image information and effectively deduce the information of large defect areas. Thirdly, the original mask strategy was optimized, and the crack mask strategy and multi-point mask strategy were added that conformed to the characteristics of defects and diseases of ancient Chinese painting images, which improved the understanding of the repaired object by the trained algorithm. Fourthly, the single-stage operation of the artificial intelligence restoration method for ancient Chinese paintings (ACP-LaMa) can achieve more ideal results, save computing resources, and have high operation efficiency. Fifth, the design will be aimed at users to develop a "fool-like" algorithm operating system, with a friendly operation interface, simple functions, high flexibility, and non-technical personnel can quickly grasp it.
After actual case experiments, the ACP-LaMa artificial intelligence image restoration algorithm basically meets the above five requirements. The following is an example (Figure 4):
Figure 4-1 White depiction of the mural of Yongle Palace.
Figure 4-2 Manually adding a damaged area.
Figure 4-3 CoModGAN algorithm fixed.
Figure 4-4 MST-Net repair result.
Figure 4-5 Repair result of this method.
Fig.4 Comparison of the restoration of various algorithms in the white drawing of the murals of Yongle Palace.
It can be seen from the comparison of the algorithm restoration effect that ACP-LaMa can understand the line information and artistic characteristics of Chinese painting well, and can restore the damaged image more accurately. The other two algorithms have excellent restoration performance and close restoration logic to the restoration needs of ancient paintings, but the line restoration ability is insufficient.
5. Examples of algorithm image restoration of ancient Chinese paintings
(1) Image restoration process
1. Disease labeling and data sampling
The research group took the murals of Baofan Temple in Sichuan as the restoration object, and selected the partial digital scanning image of the "Dharma Tribute Map" to carry out artificial intelligence algorithm restoration, as shown in Figure 5-1 before restoration, there are many types of diseases, the disease situation is complex, and it affects the identification of the human image, the main disease types are shown in Figure 5-2, and the small area of diseases is marked by the research group as shown in Figure 5-3. The figure below (Fig. 5) is derived from the "Digital Survey and Mapping Report of the Murals of Baofan Temple", which records in detail the disease labeling, color sampling and analysis results of the surveyed murals.
Figure 5-1 Partial original image of the Bodhidharma Tribute Map.
Figure 5-2 Labeling of major diseases.
Figure 5-3 Labeling of diseases in small areas and line segments.
Figure 5-4 Repair result.
Fig.5 Comparison of local diseases before and after marking and repairing in the Bodhidharma Tribute Map.
2. Algorithm repair process
(1) "Washing": The image has fewer dotted diseases, the picture is relatively clean and bright, and it is judged that there is no need for "denoising" treatment.
(2) "Peeling": The image has no obvious settlement, dislocation and stratification, and it is judged that there is no need to reset the picture.
(3) "Patch": There is a transverse crack running through the face of the Buddha statue in this image, a large area near the right eye, and a large area of sparse and falling off areas on the backlight, chest and white clothes.
(4) "Complete": The small-area diseases and line segment problems marked in Figure 5-3 are many and miscellaneous related diseases, which need to be carefully repaired, most areas can be masked once to achieve the ideal effect, and a few areas need to be repaired 2 to 3 times.
(2) Restoration results and evaluation
Based on the restoration results, the algorithm restoration method of ancient Chinese paintings used in this project basically meets the evaluation criteria formulated by the project. The experiment adopted the standard of "moderate restoration", followed the principle of minimal intervention, and did not carry out fine restoration, but maintained the overall unity and simplicity of the picture. Each damage is repaired by 1 to 3 times algorithm, and the visual standard can be basically met. The restoration of each link can basically achieve natural color transition, reasonable connection of color blocks, smooth lines, exquisite ink color, unified artistic style and accurate texture matching. The research group invited restoration experts to evaluate the "objectivity" of the restoration results, and the restoration experts believed that the restoration was basically up to standard. At the same time, the algorithm's understanding of the information of ancient pictures needs to be improved, and the restoration of eyebrows and corners of the eyes is lacking, which can be assisted by drawing tools. The results of the partial repair are shown in Figure (Figure 6).
Fig.6 Comparison of local leaching and gilding techniques and line segment repair.
(3) Other experimental cases (Fig. 7-Fig. 9)
Fig.7 Partial image restoration comparison of Confucius's disciples in Song Dynasty silk paintings.
Fig.8 Comparison of partial image restoration of Zhang Bi's calligraphy works in the Ming Dynasty.
Fig.9 Comparison of partial image restoration of "lacquerware" works in the Forbidden City.
6. The innovative value of research results
ACP-LaMa, an AI deep learning algorithm for image restoration of ancient Chinese paintings, proposed by the research group, can carry out efficient virtual restoration of digital images of ancient Chinese paintings based on the development of scientific restoration process and evaluation methods, which is a practical case of the development of manual restoration to scientific and technological restoration. The results of the project algorithm can understand the image information of ancient Chinese paintings with high definition and the characteristics of Chinese painting lines in a global manner, and the repair effect of large-area defects is ideal. The restoration area realizes the matching of pen and ink language, color information and texture with the original picture, and the restoration area basically achieves the effect of vivid charm and difficult to distinguish with the naked eye. The algorithm is efficient in application and concise in operation, suitable for the restoration of various types of ancient Chinese paintings, and has good popularity.
The project "Research on Image Restoration and Restoration Methods of Ancient Paintings Based on Neural Network Algorithm" links culture, art and science and technology through an interdisciplinary perspective, and applies them to the fields of auxiliary cultural relics restoration and protection, art research and public service, and comprehensively realizes its academic value, application value, social value and contemporary value.
This paper is the interim result of the National Social Science Foundation of China Art Youth Project "Research on the Restoration Method of Ancient Painting Images Based on Neural Network Algorithm" (Grant No. 17CF198).
This article was written by Liu Huiyuan
Editor in charge: Yang Mengjiao
Courtesy of the author
For ease of reading, the quotation is omitted