The wonderful moments of the Spring Festival Gala in the Year of the Tiger are still vividly remembered.
Perhaps what impresses you the most is the immersion of "Remembering Jiangnan", which seems to be touring in the painting, Zhang Xiaofei's blue coat, or "Only This Green" that has played a thousand miles of rivers and mountains...
In addition to the wonderful song and dance language programs, the Spring Festival Gala of the Year of the Tiger also added a variety of interesting ways to interact with the audience, among which it had to be mentioned that the red envelope grabbing link was mentioned.
During the live broadcast of the CCTV Spring Festival Gala this year, JD.com distributed a total of 1.5 billion yuan worth of red envelopes and physical objects, inviting the whole people to interact. Official data shows that during the Spring Festival Gala, the cumulative interaction volume of global Chinese participating in Jingdong APP red envelopes reached 69.1 billion.
Behind this is the day and night struggle of countless JD cloud team members - when these traffic peaks poured into JD.com on the night of Chinese New Year's Eve, it brought huge challenges to JD.com's teams.
The interaction of the Jingdong Spring Festival Gala red envelope is far more complicated than in previous years.
In addition to the action of "grabbing red packets", it also involves subsequent user orders, distribution and other processes, and the increase of each link link also makes the complexity and difficulty of the Spring Festival Gala scene increase exponentially, refreshing the record of the most complex scenes in the world at the Spring Festival Gala.
How to allocate computing power resources, how to ensure the security of activities and user information, how to ensure the smoothness of payment, accurate preparation of the supply chain, and how to ensure that every question of users is answered, these problems have become the problems that JD's technical teams must face.
This issue of micro-story, focusing on the 19 days before the preparation of the Jingdong Spring Festival Gala red envelope, to see how the Jingdong technical team such as Jingdong Cloud can break through the difficulties of each link link one by one in such a short period of time, so that the Spring Festival Gala red envelope interaction can be carried out smoothly.
Here are the true stories about them:
Text | Little North
Edit | Outstanding
On January 3, before the New Year's Day holiday ended, Zhou Guang of the Jingdong Cloud Fleet team suddenly received news: Jingdong will become the exclusive interactive partner of the 2022 CCTV Spring Festival Gala.
This is a multi-billion level interactive activity, and the matching computing power resources required are very considerable.
In the software, any behavior of users requires computing power resources, and the number of people like the Spring Festival Gala Red Envelope Interaction, the increase in users will inevitably increase to hundreds of millions, and the computing power requirements are extremely high and the challenges are extremely high.
If the computing power is insufficient, it may lead to jd.com's interface or red envelope interface not being refreshed, users can not grab the red packet, in addition to the inability to support the operation of the system, the entire system may collapse. What's more, the business scenario of the Jingdong Spring Festival Gala interaction is more complicated than in previous years.
As a shopping app, Jingdong needs to undertake not only the interaction of red packets, but also the transactions, payments, customer service and other content involved in users after receiving red packets, and because JD.com's self-operated goods are distributed by JD.com, the follow-up will also involve business scenarios such as sorting and distribution, and the entire link is super long.
The difference between the two scenarios of red envelope interaction and shopping transactions is large, and the peak is frequently switched back and forth, which is a great technical challenge. In this case, the lack of computing power resources is undoubtedly fatal to such a hundred million activities.
But there are only 19 days left for JD Cloud.
Figure | The JD Cloud Fleet team is preparing for the battle
The common approach in the industry is to match this demand for computing power and purchase new servers. But the time left for JD Cloud is too small, they can't even purchase, and if they want to meet the complex interactive scene of the Spring Festival Gala, they need to purchase tens of thousands of servers on the basis of the existing CPU core.
This is simply an impossible task. After careful consideration, the JD cloud team decided to use the existing computing power of the JD cloud ship to support the multi-billion level activity of the Spring Festival Gala red envelope interaction.
It's a tough challenge. There are a large number of applications in the Jingdong APP, how are these applications distributed, and to what extent should the resources involved in each application be compressed, so as not to affect its own use, but also to provide space for the red envelope interaction of the Spring Festival Gala?
At the same time, the movement of this resource must also be compressed or released in a second-level time to ensure a smooth user experience. The JD cloud fleet team had to solve this problem through stress testing and system monitoring again and again.
In order to avoid the user's use time, the cloud ship team usually chooses to stress test at night. One night, at 1:00 a.m., I was under pressure testing, and at 3:00 a.m., I suddenly found that some machines had problems and hung up.
This is something that has never happened before. At that time, the colleague on duty called all the relevant personnel to check what the situation was.
In fact, the number of machines hung up is not much, and for the entire large-scale server of JD.com, the proportion of resources in this part is very small. However, the large-scale resource scheduling of the Jingdong Cloud Cloud Ship requires extremely high accuracy of resource scheduling, even if it is only a very small number of server resources, it also needs to be guaranteed in place.
Everyone was very nervous, rushed to the company at the first time, checked what happened to the machine at that time, how the log of the machine was hung up, they kept pulling groups, telephone communication, from 4 o'clock in the morning, until 7 or 8 o'clock in the evening, there was no rest in the middle, and the meal was also takeaway to eat two bites quickly, and they were afraid that it was too late.
Finally, they identified the problem after 14 hours and resolved it quickly. At that time, although everyone was tired, we all had a sense of pride: we solved this problem, and we could more confidently ensure the operation of the Spring Festival Gala.
For a software, computing power resources are the cornerstone of its operation.
This state of the cloud ship team in the face of difficulties and immediately find a way to solve the problem finally enabled JD Cloud to achieve hundreds of millions of Spring Festival Gala red envelope interaction without purchasing a separate server, so that the entire user experience can be smoothly and happily carried out.
Equally silent but indispensable is the security team. The Spring Festival Gala involves a large number of new business launches, and often new business is the most prone to security problems and the easiest to be used externally.
There are some security issues that have very serious consequences, such as command execution vulnerabilities, which can directly operate the server, or even shut down the entire server, making the activity impossible. There are also security vulnerabilities that can target users' login rights or redirect users to other websites, causing strong interference to users.
What the security team has to do is to carry out repeated tests and demining before the Spring Festival Gala activities are launched, so that the activities and users are not affected. In building a security prevention and control network, there is a common difficulty in the industry: how to protect against DDoS attacks?
Figure | The JD security team is working together
It is an attack that attempts to overwhelm and paralyze online services, websites, and web applications with malicious traffic from multiple sources.
It is a normal request, but the volume of requests is particularly large, and when the volume is so large that the server cannot be hosted, it collapses. Under the premise that the Spring Festival Gala red envelope interaction itself has a large number of requests, if there is another DDoS attack, the consequences are unimaginable.
The security team collected a large amount of data, conducted penetration testing and self-examination of the cloud environment that completed the secure deployment from the attacker's point of view, and ensured that the deployment strategy was effective, and that the emergency response and policy disposal met the project requirements.
Wang Taoran of the security team remembers that in order to develop an effective security defense strategy, the team stayed up late until the early hours of the morning. He was touched by the feeling of thinking together and trying to solve problems in order to achieve their goals.
When the Spring Festival Gala event went online, and the user did not have a bad experience due to safety issues, he felt that everything was worth it.
Also to be fully updated in 19 days, there is also jd.com payment technology team and intelligent supply chain Y team.
Under the premise that the computing power can support hundreds of millions of people to complete the red envelope interaction, and the security protection escorts the activities and users, JD.com must also do a good job in the user experience of the red envelope interaction, so that users can get a good experience in the shopping and delivery of goods after grabbing the red packet.
On New Year's Day, Fan Mingxing received a notice about the payment upgrade. As a member of JD.com's payment technology team, he knows that JD.com's payment technology has been continuously improving in countless big promotions.
Figure | JD.com's payment technology team is working intensely
Take the payment page that appears after clicking on the checkout as an example, this page needs to display various payment tools and activity information, and what kind of preferential activities are currently available for various payment methods, which need to confirm the information to the background and display it on this interface.
When the user chooses a certain payment tool, the backend will enter the real deduction process, which needs to verify the password, report the risk, confirm the logic of the coupon, confirm the deduction logic, etc., and after the deduction is successful, it also involves settlement to the merchant. Therefore, although the user only pressed the "Pay" button, there are many links that need to be walked in the background.
In this case, the lack of smoothness on any one node will lead to a slippery slope in the overall user experience. At that time, after the user clicked on the payment, what they saw was often not the payment method page, but a circle displayed, constantly loading.
After many Jingdong 618, 11.11, New Year Festival and other major examinations, Fan Mingxing led the technical team to increase the response speed of payment by 6 times on the original basis, making the user's experience smoother and laying a solid foundation for ensuring the interactive activities of the Spring Festival Gala.
At the Spring Festival Gala, users will pour into Jingdong in large numbers, and there will be a more concentrated order scene, and the improvement of page response speed can help users have a smoother experience.
But for Fan Mingxing, the biggest challenge facing the Jingdong payment technology team this time is not the payment speed, but the red envelope interactive activities of billions of people like the Spring Festival Gala, everything is unknown, and everything has to be built.
"In the past, the payment technology team focused on the user's transaction payment link, but in the process of the Spring Festival Gala red envelope interaction, in addition to the user's payment, there will be a large amount of traffic pouring into the "My Wallet" to check the received red envelope balance, which means that we need to prepare a new link and upgrade it to a better level."
In just 19 days, the JD payment technology team needs to sort out the content involved in the entire new link, estimate the traffic peaks involved in each link, test the risk points, and upgrade and optimize the architecture in a short period of time. In the past, this was a job that needed to be prepared 2-3 months in advance, but this time they only had less than a month to go, and the unknown variables were also large.
But everyone in the team, with the purpose of wanting to do a good job, the plan is not perfect, everyone will make suggestions and modifications together, and some plans even change dozens of versions before they are finally determined.
The results of the subsequent events are also obvious. Under the impact of such a large amount of traffic, JD.com has carried the traffic pressure of users, from red envelope collection, viewing, to using payment, which is very smooth.
In addition to payment, Jingdong also needs to consider how to use digital intelligence supply chain capabilities to ensure the user's shopping experience during the Spring Festival Gala. The supply chain's guarantee of the user's shopping experience mainly includes two aspects, one is the spot rate, and the other is the performance time.
Although it is during the Spring Festival, users also need to have spot when placing orders, and they also need to ensure the performance of the time limit. That is, to let consumers buy what they want to buy, and to deliver goods to users faster. JD.com's self-operated goods are usually delivered on the same day or the next day, and to do this, the advance stockpiling of warehouses in various cities is particularly important.
However, Jingdong only serves tens of millions of goods, there are nearly 1300 warehouse layouts, and each region, each type of commodity sales forecast will be different, need to be by region, according to the warehouse to predict what region, what warehouse should be prepared in advance, the entire amount of calculation reached the level of 100 million, the calculation is very difficult.
Shi Zhengxin of the Intelligent Supply Chain Y Business Department is one of the team members in charge of this work, what users want to buy, what can they buy, how to let users in various regions buy what they want to buy, is a problem that Shi Zhengxin must solve.
Figure | JD Intelligent Supply Chain Y Business Department
At the node of the Spring Festival Gala red envelope interaction, due to the huge number of users watching the Spring Festival Gala, traffic and orders will erupt in a concentrated manner during that period. This outbreak and the promotion have certain similarities, the difference is that the Spring Festival Gala is a brand new scene, in this scene, everything needs to be recalculated, the ratings of the Spring Festival Gala, the different degrees of the epidemic in various places, and the different needs for materials everywhere...
The influence of multiple factors makes the calculation of traffic and commodity inventory very complicated.
In order to better improve the user experience, the small partners of the R&D team of the Y business department of Jingdong Intelligent Supply Chain have reproduced the retention of the previous promotion experience into the scene of the Spring Festival Gala through the algorithm of transfer learning, and made a set of sales estimation models specifically for this event, making more accurate sales forecasts by region and category.
Taking candy as an example, during the New Year Festival, the sales of candy in the southern region tend to increase, such as Guangzhou is about 2 times the usual, while the sales in the northwest region do not change much. Each of JD.com's own tens of millions of goods must be predicted in related regions and quantities.
At the same time, in order to cope with unexpected situations, the R&D team of the Intelligent Supply Chain Y Business Department also optimized the layout of the warehouse and the warehousing network for the Spring Festival Gala event to ensure the fulfillment of orders and the flexible distribution of goods during the event.
Jingdong's warehousing has a large regional warehouse RDC, and some small warehouse FDC, such as the Beijing-Tianjin-Hebei region, Beijing is the regional RDC, covering the entire Beijing surrounding warehousing, while Tianjin, Hebei some cities are FDC, in the logistics transportation, usually the goods of the large warehouse to the small warehouse, and then distributed by the small warehouse.
At this time, the distribution of goods in large warehouses and small warehouses is particularly important. With nearly 1,300 warehouses, the R&D team of the Intelligent Supply Chain Y Business Unit needs to calculate the advance stocking and distribution of each warehouse, which is also a very difficult challenge.
But it is precisely such a large number of and such complex advance preparations that make the Jingdong supply chain truly efficient and agile. The first order of Jingdong's Spring Festival Gala was sent to Guizhou Bijie that night. In this Spring Festival, consumers can buy the goods they want to buy, and the goods can be delivered home in time.
Du Jianjun, 70, is celebrating the New Year alone again this year.
In recent years, because of the repeated epidemics, Du Jianjun is worried about the safety of their return to their hometowns, and every year when he calls them before the New Year, he will preemptively tell them: "You don't come back, this year's epidemic is very fierce, you just have a good life outside." ”
For this, the grandson remotely taught him to make video calls and also taught him to shop on JD.com; his son shared the JD account with him, so that Du Jianjun could buy anything he wanted.
Chinese New Year's Eve evening, Du Jianjun finished video with the children and sat in front of the TV waiting for the Spring Festival Gala. When it came to the interactive session, he heard the host say that he could participate in the distribution of 1.5 billion red envelopes and good things with a shake, so he quickly took out his mobile phone, clicked into JD.com, and shook the mobile phone in his hand.
Hey, I really hit a red packet!
Du Jianjun returned to grab, there are red envelopes with vouchers, happy to close his mouth, he just wanted to buy some New Year goods, but the coupons he snatched ah, red packets, I don't know if they can be used. He hadn't paid attention to these things before.
And there is also a drumming game under the red packet, which is also very interesting to watch, and there are benefits to take, but the rules are a bit complicated. The children weren't around, and he didn't know who to ask for this new thing. He saw on TV that there was a hotline and wanted to call for advice, but he didn't dare – he didn't speak Mandarin, and the child said that he spoke "Tu Sichuan dialect" with a heavy accent, and he was afraid that others would not understand it.
But it was said on TV that dialects can also be played. He hesitated, but still called, "That, you drummer, squirmer?" ”
The opposite AI voice recognized the keyword of "drumming", knew that he actually wanted to say that he was drumming Yingchun, patiently told him how to play, Du Jianjun turned the phone into amplification, according to what was said on the phone to try, and sure enough, he got a wave of benefits. He was elated and asked, "Then what about your red envelope, the bag is used to whisper?" ”
Customer service also patiently told him the method of using the red packet, Du Jianjun browsed the Jingdong page, quickly placed an order for nut New Year goods, did not expect to arrive the next day.
He happily showed off to the children and raved about this year's event.
"I don't know how to play with mobile phones, this year is so comfortable, oh, grabbed the red envelope." Those who answer the phone, I can hear what I say so clearly, oh, very fierce(powerful)! I also bought some east (stuff) and arrived so fast! ”
For these elderly people who are not very good at playing mobile phones, the other party can understand their words, can give them normal answers, and can make them happy for half a day. Because they no longer feel that they don't understand, don't know, and are abandoned by the times.
Behind this good experience that is good for the elderly, it is inseparable from the intelligent voice recognition function of the Jingdong intelligent customer service intelligent voice team that has rapidly iterated out in more than a week.
Figure | Jingdong intelligent customer service intelligent voice team
Fan Lu got the news in early January that Jingdong will participate in the Spring Festival Gala red envelope interaction, and many of the audience watching the Spring Festival Gala are older people such as fathers and grandparents, who do not understand the interaction of the app very well, but can call naturally.
In the scene of the Spring Festival Gala red envelope interaction, there must be many people who have doubts about red envelopes and activities and call for consultation, and the intelligent voice team estimates that this traffic is about tens of times that of the double eleven peak.
How to undertake so many telephone consultations? JD.com's customer service cannot be rapidly expanded by dozens of times in a short period of time, and these older people speak more dialects, which is also a great challenge for customer service.
Based on this, the Jingdong intelligent customer service intelligent voice team decided to answer the questions of users who came to consult through the voice interaction of intelligent AI.
They sorted out the questions that users may ask, identified the user's questions through intelligent AI, and gave accurate replies, so that they could better improve efficiency.
Over the past three years, the intelligent speech team has also built a model of a dialect database by classifying and analyzing dialects by region.
When the intelligent AI answers the phone, it will determine the area from which the user's language comes from by accessing the user's question, and then apply the language expression in this area to distinguish their intentions and give the correct answer.
When applied, the speech recognition accuracy of this intelligent AI (including accented dialects) can basically reach more than 95%.
From computing power transfer to security protection, from payment upgrades to supply chain advance preparations, to the upgrade of intelligent AI dialect recognition rate, behind the new gameplay of this year's Spring Festival Gala, it is the unremitting efforts of a group of people from the Jingdong cloud team that finally supports the smooth progress of the Jingdong Spring Festival Gala red envelopes.
This year's Spring Festival Gala Red Packet has the shortest preparation time in history - 19 days, but it is the world's largest network interactive activity - the peak can reach hundreds of millions of levels, but also the world's most complex scene - the Spring Festival Gala Red Packet Interaction Superimposed Jingdong supports the overall supply chain performance in the national "Spring Festival", refreshing the new record of the world's most complex technical scenes in the Spring Festival Gala.
In the case of such a short time and such a big challenge, JD.com undoubtedly handed over a satisfactory answer.
Figure | After the Spring Festival Gala, everyone was celebrating
After such achievements, it is inseparable from the efforts and synergy of everyone in the JD cloud team.
Fan Lu of the intelligent voice team still remembers that on the Friday before the launch, the team continued to debug the problem, and when they combed the words of the Spring Festival Gala in the specific scene, they found points that had not been considered before and could be optimized, and in only a few days, they re-sorted the work and re-executed. At that time, everyone worked overtime until 3 a.m.
But at that moment, Fan Lu felt very moved. When this goal is in front of us, everyone is goal-oriented and works together to think about doing things well, and this state of "wanting to do a good job" makes everyone willing to pay and work hard, and after the event is completed, a lot of joy will gush out in the heart: this is what I did, and I did it perfectly.
That moment will never be forgotten under any circumstances.
In JD Cloud, there are many such people, who voluntarily give up the opportunity to go home and monitor the dynamics of the system and data at any time in the company; in order to achieve a more satisfactory goal or plan, they modify it many times and forget to eat and sleep.
And it is precisely with them, with such a goal-oriented atmosphere within JD Cloud, JD Cloud has become an efficient and effective team, in order to think of others dare not think, and overcome challenges and difficulties.