Recently, a project went viral on social media by a 3D special effects artist from Vienna.
The creation of such a CNN visualization tool was completed in my master's thesis, because I was learning neural networks, and it was difficult to understand how convolutional layers are connected to each other and how to connect with different types of layers.
Therefore, the main features of this project include convolution, maximum pooling, and visual representation of fully connected layers, as well as various simplification mechanisms for clearer visualization, etc.
The purpose of publishing this successful result of my own learning is to let beginners get the key points of CNN in the most intuitive way, and also let people understand more deeply!
How to make a 3D network with Unity
Before getting started with Unity, the author built a visual 3D network prototype in Houdini.
In other words, it is necessary to provide an idea for building a Unity version of the 3D network, and prepare in advance to implement the method of displaying convolution calculation, the shape of the feature map, the effect of edge binding, etc.
In advance, the "shape" of the neural network was preset, and since he had not used Unity before, the author first learned about shaders and procedural geometry.
The author found some limitations in this, using Shaderlab, Unity's language for shaders.
Shading variations are not available in this language, and only predefined variables for semantics can be passed between vertices, geometry, and pixel shaders.
Moreover, it cannot arbitrarily assign vertex attributes, only predefined properties such as position, color, UVs, etc. (Probably this is one of the reasons why the 3D network can't change colors in real time)
After studying some concepts related to instancing, the authors plan to use a geometric shader approach to generate the wires of the neural network. The start and end points are passed to the vertex shader and forwarded directly to the geometry shader.
These lines can consist of up to 120 vertices, as Unity allows the geometry shader to create a variable with a scalar float of 1024.
Subsequently, the corresponding 3D neural network image is generated from the TensorFlow code of the model.
The Tensorflow-native.ckpt file needs to store the data required to reconstruct the model graph, the binary weight read and activation values, and the name of a specific layer.
After that, you need to load these checkpoint files, start a TensorFlow session, enter the training example, and query the activation functions for each layer very well.
Then write a JSON file that stores the shape, name, weights, and activation functions of each layer for easy reading.
The final result is easy to read!
Just last year, a Chinese Ph.D. visualized the convolutional neural network, and the changes in each layer were shown very clearly, as shown in the figure below!
And the 5,002 stars star on GitHub is okay, and this project also uses edge binding, ray tracing and other technologies, combined with feature extraction, fine-tuning and normalization to visualize neural networks.
The project hopes to use these techniques to estimate the importance of different parts of a neural network.
If you are interested in this kind of 3D neural network visualization, you can go to the GitHub project given by the editor to learn more about it!
Development Process:
https://vimeo.com/stefsietz
3D Neural Network Projects that have been open-sourced:
https://github.com/julrog/nn_vis
China's version of the Neural Network Project
https://github.com/poloclub/cnn-explainer
Reference Links:
https://www.reddit.com/r/MachineLearning/comments/leq2kf/d_convolution_neural_network_visualization_made/
https://mp.weixin.qq.com/s/tmx59J75wuRii4RuOT8TTg
https://vimeo.com/stefsietz
http://portfolio.stefansietzen.at/
http://visuality.at/vis2/detail.html