What interests me the most about the iPad Pro 2024 is not the new M4 chip or the new OLED screen, but a small "chemistry" between the Apple Pencil Pro and the iPad Pro 2024: when you use the Apple Pencil Pro to draw and write on the iPad Pro 2024, the shadow of the corresponding brush will appear on the screen, letting the user know the current Apple Pencil Pro What kind of model.
Subsequently, 10 3D models developed by Apple for the feature were found in Apple's PencilKit framework.
What's interesting is that this shadow is not a simple map, it changes according to the position of the Apple Pencil Pro in the real world, the distance from the screen, and the angle of tilt, in short, it has a strong "realism", "sense of space" and "interactivity".
More information about the Apple Pencil Pro has also been revealed: Apple has stuffed it with a new haptic feedback engine, a new pressure sensor, a gyroscope, and a U2 chip. It has the potential to emulate a real-life writing experience on the iPad Pro, as well as the potential to link with the Vision Pro.
Apple's parody game
Apple is the world's favorite company to simulate the real-world sensory experience through hardware.
The Taptic Engine on the 2015 MacBook uses a subtle simulation of vibration and haptics to create a touchpad that is "as real as it is real". Perhaps until now, many people didn't know that the MacBook's Force Touch trackpad itself is a flat surface that "can't be pressed".
Whether it's 3D Touch or Haptic Touch, which now has no pressure sensitivity, the iPhone is on this flat glass, with just the right amount of vibration feedback, simulating a magical feeling of pressing a button and rebounding, with a sound effect that mimics the button press, people can't help but press it a few more times.
The same technology is used on the iPhone 7's Home button and the Apple Watch's rotating crown, all of which have a tactile feel very close to mechanical buttons and gears.
In addition to using the Taptic Engine to "trick" the user's fingers, iOS 14 is paired with AirPods to create "Spatial Audio", which simulates the sound of various directional sensations in 3D space, as well as head tracking technology, to give users a "sound immersive" feeling. On social platforms, many users said that after turning on spatial audio, they put on headphones and thought that their mobile phones were playing outside.
Prior to the release of the Apple Pencil Pro, many sources predicted that Apple would equip the new pen with the new Taptic Engine, allowing the Apple Pencil Pro and iPad to offer a realistic feeling of pen and paper.
This sounds like a bit of a "sci-fi" feature, but it doesn't feel fanciful, because Apple is a manufacturer that likes and is good at playing with vibration motors. Even if Pencil doesn't have the functionality yet, the future is still promising.
And another company that also likes to deceive users' senses with hardware is Nintendo.
The Joy-Con controller on the Nitendo Switch supports "HD Vibration" technology, which allows players to distinguish the subtle difference between "pouring ice into a cup" and "pouring water into a cup" just by the different vibration feels. Combined with the Switch Labo cardboard set, the gyroscope and vibration motor in the handle can also restore the real feel of catching fish out of the water.
Whether it's Apple's vibration magic, or Nintendo's Labo kit and controller, it's all about using "one piece of hardware" to simulate "countless possibilities".
For example, the Taptic Engine on the iPhone 7 is not only designed to simulate the Home button and 3D Touch, but also the vibration of the "shaking head" of the phone with the wrong password, as well as various feedback in the system, all kinds of vibration actions are completed by the huge Taptic Engine inside.
It's a similar story for spatial audio, where AirPods Max are able to simulate complex, spatial-oriented sounds with gyroscopes and accelerometers.
If Nintendo is to bring players a more realistic gaming experience, then Apple's "imitation game" is a full range of simulation, covering touch, hearing, and vision.
Therefore, the projection of Apple Pencil Pro is by no means an isolated little easter egg.
Vision Pro 的补完计划
During the visionOS test last year, some developers found that when the user held the Apple Pencil, the Vision Pro could not properly recognize the pinching operation of the finger, but the official feedback said that "this is a design feature, not a bug", which shows that the recognition of Apple Pencil is likely to exist in the Vision Pro.
The addition of hardware such as a haptic feedback engine, a new pressure sensor, a gyroscope, and a U2 chip makes it more like a smaller VR controller, which is logically quite similar.
Coincidentally, Logitech has introduced MX INK, a stylus for VR devices for the Meta Quest 2 and 3.
Weighing 10 grams more than the Apple Pencil 2, the stylus can create flat content in mixed reality like a traditional stylus, with six-degree-of-freedom tracking for freedom to create in 3D space, with multiple buttons and pressure-sensitive tips.
Logitech MX INK
So, does Vision Pro need a pen? Or does it need "peripherals" other than eyes and fingers?
Before we talk about this open-ended question, let's take a look at a series of popular science videos for developers at WWDC24: "Creating Engaging Spatial Photo and Video Experiences."
There are a lot of conceptual things mentioned in it, and Apple divides the stereoscopic video experience into: 3D video, spatial video, and Apple immersive video:
- 3D video is presented on a flat surface and has a sense of depth, similar to the 3D movies we watch in the cinema
- Spatial video, in addition to having a three-dimensional effect, the content is scaled to the real size of the object, and the boundaries are blurred
- Immersive video, on the other hand, surrounds the user, not watching the video, but being there
Among them, "spatial video" can imitate the sense of presence of people in the real world, and can change the viewing angle within a certain range; "Immersive video" is even more extreme, where users see the video everywhere they look, and the user becomes a part of the video.
Apple also made additional requirements, such as immersive videos that need to have 3D content that can be interacted with.
漫威为 Vision Pro 推出的《What If…? 》沉浸式故事，可以让用户伸手参与其中（图源：YouTube @Adam Savage’s Tested）
The concept of "spatial metadata" is also mentioned in the popular science video, and there is also an explanation of "projection": projection defines the relationship between objects in the world and pixels in an image.
The little easter egg in the Apple Pencil Pro is also explained in the developer video of the Vision Pro: Apple has never treated the iPad Pro 2024 and the Apple Pencil Pro as separate devices, they are linked together through "projection".
In other words, an Apple Pencil may be able to be "projected" into a variety of different forms of pens in the virtual world of Vision Pro in the future.
Since the Vision Pro can explain the doubts about the Apple Pencil Pro, then Apple, as the culmination of this "imitation game", we also have reason to believe that the current Vision Pro is still in the early stage of the product, and it has been able to achieve a considerable degree of "fake and real" in vision and hearing, but it is far from enough in terms of touch, which naturally comes down to the fact that Vision Pro does not have specially adapted peripherals, and it cannot always rely on Vision Pro Pretend to be a horse to shock our brains, right?
Now, we're almost certain that the Apple Pencil Pro with its vibration engine, gyroscope, and U2 chip will work with the Vision Pro, and that the Apple Pencil Pro can also be projected on the screen in "other ways", such as an ink pen. So maybe the pen could be turned into a lightsaber from a Saber Beat game, or a wand from the Harry Potter series, like the controller of other VR devices?
Dexmo Force Feedback Gloves
Even more imaginative, maybe Apple will also release a "force feedback glove", the whole glove is full of vibration motors, and with Apple's tuning, users can really "touch" the virtual world presented by the Vision Pro.
What is almost certain is that Apple is already looking for ways to add more "interactions" to the Vision Pro, because the Vision Pro is not yet complete, and Apple's "imitation game" of replacing reality with the virtual is far from over.
The importance of "realism".
In previous articles, we discussed the importance of keys. When it comes to Vision Pro, which allows users to perform like "no physical performance", interaction and feedback can be said to be more important, as the editor-in-chief of the technology magazine "Wired" once said:
All devices require interaction. If something doesn't have a sense of interaction, it's considered broken.
It was a perfect fit for my first experience with Vision Pro: no matter how or where I tried to use the gestures, I couldn't click the button I wanted to click, and I didn't even know if it was a Vision Pro problem or my own.
As a result, even if Apple had to cut out the 3.5mm headphone jack and reduce the battery capacity, it had to make room for a huge Taptic Engine, which extended the life of the components by discarding the mechanics while ensuring the realism of the interactive feedback.
The reality of interaction is not just a signal to feedback whether the device is broken or not, if you want to go deeper, we have completely shifted from the era of "buttons" to the era of "touch", and we are also entering the era of "air operation". In the future, perhaps we will no longer need to touch any device to achieve various actions, all kinds of holographic UI will surround us, and we can only point in the air at any time if we want to do something, or voice control.
This vague future of the virtual and the real will give us all kinds of uncertainties, such as, how do you know that you are a living person and not a brain in a vat?
Perhaps Apple's "imitation game" will one day come to an end with a change in operation, but until then, we can still expect how Apple will fool our fingers and brains in various ways.
#欢迎关注爱范儿官方微信公众号: Love Fan Er (WeChat ID: ifanr), more exciting content will be presented to you as soon as possible.
Love Faner|Original link· Sina Weibo