Reporting by XinZhiyuan
Edit: Small salted fish good sleepy
【New Zhiyuan Introduction】If you want the "paper man" in the painting to have life, Meta AI will do it! Yesterday, Meta AI announced an ingenious approach that can animate children's hand-drawn characters in just a few minutes.
Look, the "Paper Man" actually jumped out on his own!
Kittens learn hiphops.
Little bees also do wave.
Stickmen can actually play taekwondo.
Candy Man can also fly in the air and kick.
These cartoons are generated using children's stick figures, but the author is an AI!
AI animation tools roll over on children's drawings?
Children's imaginative and creative stick figures can sometimes lead to different thinking.
However, while it is easy for parents or teachers to see what their child's drawings are trying to show, ai-engines have a hard time doing this task.
Why?
This is because children's drawings are often more out of the ordinary, for example, the figure's feet do not support the body, or the figure's arms are on the same side of the body.
As a result, AI systems can sometimes fail to recognize figures in images and paintings.
Researchers already have some eyebrows, though.
Yesterday, Meta AI announced an ingenious approach that will automatically animate children's hand-drawn or humanoid characters (i.e., characters with arms, legs, etc.) in just a few minutes, and the resulting animations will be lifelike.
Simply upload your child's hand-drawn drawings to Meta AI and you can see them transform into dancing characters, which is very realistic.
Hand-drawn change animation in four steps
Many AI tools and techniques are designed to process real human images, but children's drawing increases the diversity and unpredictability of the character's features, which makes identifying hand-drawn figures more complicated.
The "person" in children's paintings has many different forms, colors, sizes and proportions, and there are almost no similarities in terms of body symmetry and form.
Meta AI then addresses this challenge in four steps.
Identify humanoids through object detection
The first step in animating children's characters is to distinguish the characters from other types of characters in the background and pictures.
Meta AI uses MassK R-CNN, an object detection model based on convolutional neural networks, to extract figures from children's drawings.
Although Mask R-CNN has been pre-trained on the largest segmentation datasets, these datasets are made up of photographs of real-world objects and do not contain freehand drawings.
So the researchers fine-tuned the model on about 1,000 hand-drawn drawings.
After a fine-tuning process, the model is able to detect the characters in the test dataset very well.
However, while existing object detection techniques are very effective at distinguishing between figures on children's drawings, the resulting segmentation masks are not precise enough to be used in animation.
To solve this problem, we instead use the bounding box obtained from the object detector and apply a series of morphological operations and image processing steps to obtain the mask.
Use character masks to elevate humanoids from the scene
After identifying and extracting the humanoid from the drawing, the next step is to separate the subject from the background via masks.
Theoretically, after this step is completed, the part of the mask that falls out should only contain the humanoid part, without any background.
However, if the hands, feet, body, and other parts of the character in the hand-drawn drawing are too unconventional in appearance, masks predicted by Mask R-CNN often cannot accurately capture the entire graphic.
For example, the inverted triangle of the little man in the picture below, where the yellow triangle represents the body and a horizontal bar represents the arm.
The mask R-CNN result was obviously very crude, either "cutting off" the arm or thickening the thigh.
To that end, Meta's researchers have developed a classic image-based approach that is more robust to these body parts of different shapes.
The researchers first cropped the image with a predicted humanoid bounding box, and then, applying adaptive threshold processing and morphological manipulation, pan-filled from the edges of the box, assuming that masks were the largest polygons that had not been touched by pan-fill.
While this method can precisely extract and extract masks that are suitable for animation, it is not easy to do when the background is disorganized or there are wrinkles, breakages, or shadows on the paper.
Prepare for animation by "assembling"
The figures drawn by children come in a variety of body shapes, all of which go far beyond the shapes of the heads, arms, legs, and torsos of traditional people that AI has seen.
For example, the child's "stickman" has no torso, and its arms and legs are directly attached to the head.
Therefore, researchers need a completely new way to deal with such morphological "variations".
Meta uses AlphaPose, a model trained specifically for human posture detection, to identify key points on the person that can be used as a recognition of hips, shoulders, elbows, knees, wrists, and ankles.
Since AlphaPose is trained on images of real people, in order for it to detect poses in children's drawings, it must be retrained to handle the strange poses in children's drawings.
To do this, Meta's researchers first collected a wave of data on children's drawings internally.
The researchers then created an internal tool using pose detectors trained on this initial dataset and allowed parents to upload their children's drawings, while the researchers could use the uploaded drawings for additional training.
As more and more data grew, the model was trained repeatedly, eventually reaching a higher level of accuracy.
3D motion capture for 2D character animation
Once you have masks and joint predictions, you have everything you need to make an animation.
Meta first generates a mesh using the extracted mask and textures it with the original graphic.
Using predicted joint positions, the researchers created a skeleton for the characters, generating various poses by rotating the bones and using new joint positions.
Before you animate, create a character from the drawing that needs to be manipulated.
Then, project a frame of motion capture data onto a 2D plane and change the character's pose. Projecting motion capture data from the front (top row), side (middle row), and distorted viewing angle (bottom) is fine.
Finally, by giving the character a series of consecutive poses, you can create an animation.
Meta AI can choose different movements based on the confidence level of the joint prediction: the animation can be produced seamlessly when both the arms and legs are correctly predicted.
But if the torso doesn't appear in the drawing, its joint confidence value will be low, and the AI will have to abandon the task, ask the user to correct the prediction, or declare that the animation "can't be done".
Make more complex animations with AI
AI has become a powerful tool for "creativity", empowering artists and inspiring new forms of expression. Meta's animation tools encourage people to try their hand at drawing and take them in more directions.
Maybe one day, AI can draw a complex picture and then use multiple character interactions as well as elements in the background to instantly produce a detailed animation.
With augmented reality glasses, these animated characters can even come to life in the real world, dancing or talking with kids.
Possibly, ai's imagination, like the human imagination, is infinite.
Resources:
https://ai.facebook.com/blog/using-ai-to-bring-childrens-drawings-to-life/