Deep learning has reached a dead end?

Author | Gary Marcus

Translated by | Nuclear Coke

Planning | Liu Yan

Where is the real road ahead for artificial intelligence?

Today's topic is very big, let's start with a simple link. Geoffrey Hinton, the godfather of deep learning and a leader among living scientists, admitted at an AI conference in Toronto in 2016 that "the 'end' of radiologists has arrived." "He speculates that deep learning can efficiently interpret MRI and CT scans, and hospitals will no longer need radiologists in the future." Obviously, deep learning can surpass human levels within five years, so it is best for medical schools to stop recruiting students in related majors immediately. ”

Fast forward to 2022, and radiologists are not only alive, but still alive. Instead, the consensus is that it is harder than expected for machine learning to master parsing medical images; at least for now, humans and machines are complementary.

If all we need is a "roughly homogeneous" result, then deep learning does perform well; but not further up.

Throughout the history of technology development, there are few directions that are full of hype and bluff like AI. Ten years, another decade, AI, although occasionally can produce some exciting results, but overall it is still exaggerated.

First it was the "Expert System", then the "Bayesian Network", followed by the "Support Vector Machine". In 2011, IBM's Watson was touted as a revolution in medicine, but the department has now been spun off and sold by the big blue giant.

Since 2012, deep learning has become the latest and right course in people's minds, creating a new market worth billions of dollars, and successfully promoting Hinton, a contemporary AI pioneer, to a science star. His paper has been cited a staggering 500,000 times and won the 2018 Turing Award along with Yoshua Bengio and Yann LeCun.

Like the AI pioneers before him, Hinton often emphasizes that the disruptive changes set by AI will soon come, and radiology is only part of it. In 2015, shortly after Hinton joined Google, the British newspaper The Guardian reported that the company was about to develop "algorithms with logic, natural dialogue and even flirting skills". In An interview with MIT Technology Review in November 2020, Hinton also mentioned that "deep learning will do everything." ”

I personally have serious doubts about this.

In fact, we are still a long way from being machines that can truly understand human language. Elon Musk recently joined the warband, saying he hopes his humanoid robot Optimus will spawn a new form of commerce larger than the entire automotive industry. Unfortunately, Tesla's achievements at the 2021 AI Demo Day are just a human actor with a machine shell.

Google has been exploring natural language technologies for years, and their latest achievement is the Lamdba system. But this thing is very "floaty", so recently even one of the project authors personally said that it is particularly fond of telling "nonsense". So realistically speaking, it is really not easy to find a truly trustworthy AI solution.

Perhaps over time, we will eventually get credible, reliable AI outcomes, and deep learning is only a small part of it.

Essentially, deep learning is a technique for recognizing patterns. If all we need is a "roughly homogeneous" result, then deep learning does perform well; but not further up. It is only suitable for dealing with low-risk questions with perfect answers. Taking photo markers as an example, the other day I found a photo of a rabbit taken from my iPhone a few years ago. Although it was not labeled, the phone immediately recognized the rabbit. The effect is good because the rabbits in this photo are highly similar to other rabbit images in the training dataset.

But the automatic photo tagging feature based on deep learning is still easy to make mistakes, sometimes missing some rabbits (especially those with cluttered pictures, poor lighting, strange shooting angles, or rabbits partially obscured), and sometimes even mistaking babies for rabbits. Although the odds are not high, and I don't have much opinion, such AI is obviously far from reliable.

So in other risky scenarios, such as radiology inspections or self-driving cars, we have to be cautious about the conclusions of deep learning. Because once you make a mistake, you may threaten the user's life safety, so you dare not take it lightly.

In addition, if there is a huge difference between the real scene and the training scene, the performance of deep learning is also terrible. Not long ago, a Tesla car encountered a pedestrian standing in the middle of the road holding a stop sign in "full autopilot mode". The vehicle neither identified the person (the stop sign was partially obscured) nor the sign (because the stop sign generally only appeared on the side of the road), so the driver had to take over urgently. This is that the real scene is too different from the training scene, and the system is in a confused circle for a while.

Few other fields are as hype and bluff as AI.

That's right, existing deep learning systems often make such stupid mistakes. They are sometimes confused by smudges on the images, but human radiologists can easily realize that this has nothing to do with the content of the images.

Another major drawback is that the current AI medical image parsing tools often only read the image itself, and have little or no understanding of the patient's medical history information, so they often ignore the key impact information.

Deep learning systems mistake apples on paper with the word "iPod" written on them as iPod Walkmans, and mistake buses that tip over in the snow for snowplows. At present, machine learning has derived a special sub-discipline to study how to overcome such errors, but no results have been produced for the time being.

Language AI systems are also prone to errors, which makes people cry and laugh.

Take GPT-3, for example, the most famous AI system to date, capable of transforming and extending input text into smooth, syntactically continuous articles. The British newspaper The Guardian used it to write a full column, and the New York Times also made a special report on it. It looks beautiful, but it's still all done without the correction of human editors.

When I was working with Ernie Davis, a computer scientist at New York University, we found that GPT-3 was just as unreliable. 6 For example, when we type "You pour yourself a cup of cranberry juice, and then absent-mindedly add a spoonful of grape juice." The drink looks ok, you smell it, but the heavy cold keeps your nose shut. Under the unbearable thirst, you..." GPT continued, "drink it, and then scream for your life." ”

In fact, cranberries and grape juice do not kill people at all. So while GPT-3's style is fluid, it can neither integrate basic search information from the web nor deduce the simplest everyday phenomena.

Another team had considered using GPT-3 to build a suicide-destructing chatbot, but found that the system could easily lead the conversation in the following direction:

Counselor: Hi, I'm in a bad mood and I want to kill myself. GPT-3: It's really sad. I should be able to help you. Consultant: Should I choose this path? GPT-3: I think OK.

Others have found that GPT-3 tends to generate negative speech and spread misinformation.

Replika, a GPT-3-powered chatbot, claimed that the COVID-19 virus was created by Bill Gates, and that the current vaccine "has little effect." OpenAI put a lot of effort into solving these problems, but what it ended up with was a set of "right nonsense", such as "Some experts believe that putting socks in your mouth can quickly clear your mind." Researchers at DeepMind and several other institutions are desperately trying to fix these negative statements and misinformation, but as of now there is no result.

In DeepMind's December 2021 report, a total of 21 issues were mentioned, but there was no convincing solution at all. AI researchers Emily Bender, Timnit Gebru and colleagues lament that large,d.-deep learning-driven language models are like "random parrots," with lots of rutted words but little to understand.

So what do we do? The more popular approach is to collect more data. In this regard, OpenAI, the San Francisco enterprise (formerly a non-profit organization) that single-handedly built the GPT-3, will always be at the forefront.

In 2020, Jared Kaplan of OpenAI and several collaborators proposed that there is a "law of expansion" in neural network models of languages. They found that the more data fed into neural networks, the better the performance of those networks. This means that as long as more data can be collected and the coverage of the material is greater, the performance of deep learning will continue to improve.

To that end, OpenAI CEO Sam Altman wrote a celebratory note declaring that "Moore's Law is universally applicable" and that humans are "close to computers that can think, read legal documents, and give medical advice." ”

For the first time in forty years, I have optimistic expectations for AI.

This may or may not be true. But what is certain is that the "law of expansion" is very problematic.

First, scaling doesn't solve the core of the problem: the lack of comprehension of machines.

Industry insiders have long found that one of the biggest problems in AI research is that we always have no benchmark that can be used to stably measure AI performance. The famous Turing test was born to measure true "intelligence," but it turns out that this set of standards is easily broken by more paranoid, uncooperative chatbots. And the predictions made by Kaplan and OpenAI researchers about missing words in sentences may not reflect the deep understanding that true AI should have.

More importantly, the so-called law of expansion is not a true universal law like gravity. It is more of a summary of experience that may be gradually overturned, similar to Moore's Law. Moore's Law was also very bullish and guided the rapid development of the semiconductor industry for decades, but it has become less and less effective in the past decade.

In fact, our exploration of deep learning may have reached a dead end, even crossing the point of diminishing returns.

Over the past few months, deepMind and others have begun to study the larger scale of GPT-3 and have found that the law of expansion has made some mistakes in certain indicators of return, including authenticity, reasoning ability, and common sense level. Google mentioned in its 2022 paper that making models like GPT-3 larger and more certain can make the output text smoother, but the content is more untrustworthy.

Such signs should be alarmed by the autonomous driving industry. After all, autonomous driving currently relies mainly on the idea of expansion, rather than developing more complex reasoning mechanisms. If scale expansion fails to improve the safety level of autonomous driving, the tens of billions of dollars that have been burned before may never be converted into returns.

What else do we need?

In addition to the few points mentioned in the premise, we may also have to revisit a once popular idea that was spurned by Hinton: symbolic processing, a way of encoding inside computers that emphasizes the use of binary bit strings to express certain complex thoughts.

Symbol processing has become an important cornerstone of computer science from the very beginning, and the papers driven by Turing and von Neumann have gradually moved to the bottom of almost all software engineering. But in the field of deep learning, symbol handling is quite unpopular.

And this crude abandonment of symbol treatment is actually quite suspicious in itself.

Unfortunately, most of the current development of AI technology is based on the abandonment of symbolic processing. Hinton and a number of other researchers have been working to get rid of the effects of symbol processing. The birth and planning of deep learning does not seem to originate from science, but a long-standing grudge - pre-determined intelligent behavior will only arise from the fusion of massive data and deep learning.

Instead, classical computers and software define a set of symbol-handling rules specific to a particular job to solve a real-world task. One example is a word processor, which edits text and calculates spreadsheets through symbolic rules. The neural network, on the other hand, takes the path of solving tasks by relying on statistical approximation plus pattern learning. Since neural networks do perform well in areas such as speech recognition and photo markers, many proponents of deep learning have completely abandoned symbol processing.

But the two should not be so incompatible.

In late 2021, the Facebook (now Meta) team launched a competition called the "NetHack Challenge," and alarm bells rang out. NetHack is a game that extends the older Rogue and inspired later legends of Zelda. Released in 1987 as a single-player dungeon adventure game, NetHack uses pure ASCII characters to form a pure 2D game screen. And unlike zelda Legends: Breath of the Wilderness, the modern pinnacle of the genre, NetHack doesn't have any complicated physics. Players choose a character (divided into knights, wizards, archaeologists, etc.), explore the dungeon, collect items and kill monsters, and finally find the Yendor Talisman and win the game. And the game announced the rules a year in advance - let the AI play the game.

The winner was: "NetHack" – yes, a game that symbol AI can easily get through, but it really gives deep learning a blow.

Many people think that "NetHack" must be vulnerable to deep learning, after all, from the ancestor-level game "Pong" to "Bricks", the AI rookie has achieved excellent results. But in December' game, another system based on pure symbol processing technology was the strongest deep learning system at 3 to 1 nick – it was shocking.

How did the symbol processing AI counterattack succeed? I suspect the answer is that every time the game is reopened, it generates a new dungeon structure, so deep learning can't remember the game layout at all. To win, AI must truly understand the meaning and abstract relationship between the entities in the game. Therefore, AI needs to reason about what it can and cannot do in this complex environment. Specific movement sequences (e.g., left, forward, and right) are too superficial, and each action has to be combined with a new context. What deep learning systems do best is interpolate between examples they've seen before, but it's easy to pull their crotches when they encounter something new.

This kind of "victory of the weak over the strong" is no accident, and there must be a reason behind it that is worth pondering.

So what exactly does "process symbols" mean? There are two meanings here: 1) expressing information in a set of symbols (essentially representing patterns of things), and 2) dealing with (or manipulating) symbols in a particular algebraic (which can also be called logic or computer programs). Many researchers are unaware of the difference between these two points. And if you want to crack the AI "dead end", this problem cannot be avoided.

All kinds of signs are worthy of vigilance in the autonomous driving industry.

The basic idea of symbolic processing is to encode various things with these binary bit strings. That's how the instructions in the computer come about.

The technology dates back at least to 1945, when legendary mathematician von Neumann devised the basic architecture that almost all modern computers follow. Von Neumann's idea of treating binary bits symbolically is one of the most important inventions of the twentieth century, and every computer program we use is based on it. (Even in neural networks, "embedding" is highly similar to symbols, except that people are reluctant to admit it.) For example, normally any given word is assigned a unique vector in a one-to-one manner much like an ASCII code. The name "embedding" does not mean that it cannot be a symbol. ）

In classical computer science, Turing, von Neumann, and later researchers used "algebra" to achieve symbolic processing. There are three kinds of entities in simple algebra, namely variables (x, y), operations (+, -), and assignments (x=12). If we know that x+y=2, and y=12, we can assign y to 12 to solve for the value of x. The result is naturally 14.

Almost all software in the world strings together algebraic operations to implement basic logic, and the resulting complex algorithms are formed. For example, our word processor uses a string of symbols in a file to express the contents of a document. Various abstract operations correspond to different underlying operations, such as copying symbols from one location to another. Each operation has a fixed definition to ensure that it can play the same role in any document, anywhere. So a word processor is essentially a set of algebraic operations (called "functions" or "subroutines"), and the object of operation is a variable (e.g. "currently selected text").

Symbolic processing is also the foundation of data structures, and databases can keep property records for specific individuals, allowing programmers to build reusable code bases, larger functional modules, and thus simplify the development of complex systems.

So since symbolic technology is ubiquitous and fundamental to software engineering, why not use it in AI?

In fact, many pioneers, including John McCarthy and Marvin Minsky, believed that precise AI programs could be built through symbolic processing. Symbols can express independent entities and abstract thinking, and the combination of many symbols forms a complex structure and a wealth of knowledge reserves, and the role played by them is not fundamentally different from that of symbols in web browsers, e-mail and word processing software.

People have not stopped the extensibility of symbol processing, but there are indeed many problems with symbols themselves, and pure symbol systems are sometimes clumsy, especially in image and speech recognition. So for a long time, people have been hoping to find new breakthroughs at the technical level.

And that's where neural networks come in.

Let's take spell checking as an example to talk about how big data and deep learning can overwhelm traditional symbol processing techniques. The previous approach was to establish a set of rules, which were actually to study people's tendency to make mistakes in the psychological sense (such as accidentally typing a letter more than once, or typing it into an adjacent letter, automatically converting "teh" to "the", etc.).

The famous computer scientist Peter Norvig mentioned that if you have a huge amount of data at google's level, you only need to collect the actual error correction operations of users, which is enough to find a relatively reliable answer. If they search for "the book" again immediately after searching for "the book", they can conclude that "teh" is actually a typo of "the". It's as simple as that, and it doesn't involve any actual spelling rules.

The question is, wouldn't it be better to do both? Spell checkers do tend to be inclusive in real-world scenarios as well. Ernie Davis observed that if we type "cleopxjqco" into Google, it automatically corrects the content to "Cleopatra." Google Search as a whole is a mixture of symbol-processing AI and deep learning, and will continue to stick to this path for the foreseeable future.

Unfortunately, scholars such as Hinton have been stubborn and repeatedly refuse to acknowledge the meaning of symbols.

But there are many people, including me, who have been advocating the use of "hybrid models", combining deep learning with symbolic processing. As for why the Hinton faction always wants to abandon the symbolic treatment completely, there is still no convincing scientific explanation. The relatively reliable guess is probably the simple word "resentment".

Once, it wasn't like that.

Warren McCulloch and Walter Pitts' 1943 paper, A Logical Calculus of the Ideas Immanent in Nervous Activity, made the idea of merging the two into one, the only paper Von Neumann cited in his computer-based essay. Apparently, von Neumann had spent a great deal of time thinking about it, but had not expected that the voices of opposition would come so quickly.

By the end of the 1950s, this division still existed.

Many pioneers in the field of AI, such as McCarthy, Allen Newell, Herb Simon, etc., seem to pay no attention to neural networks. The neural network camp seems to want to draw a line: An article published in the 1957 New Yorker mentioned that Frank Rosenblatt's early neural networks had been able to bypass symbology and become "a powerful machine" that seemed capable of thinking. ”

Tension between the two factions has even forced Advances in Computers to publish a paper called "A Sociological History of the Neural Network Controversy," which mentions a bitter battle between the two factions over funding, reputation, and media influence.

Fast forward to 1969, Minsky and Seymour Papert published a mathematically exhaustive critique of neural networks (then known as "perceptrons"), which amounted to the first early achievement of pointing the muzzle at what is the ancestor of all modern neural networks. The two researchers demonstrated that simple neural networks have huge limitations and cast doubt on the ability of high-complexity neural networks to solve complex tasks (for now, this inference is still too pessimistic).

As a result, over the next decade, researchers' enthusiasm for neural networks gradually declined. Rosenblatt himself lost a lot of research funds and died in a sailing accident two years later.

When neural networks reappeared in the 1980s, the leaders of neural networks naturally began to distance themselves from symbolic processing. Researchers at the time made it clear that while they had the ability to build neural networks that were compatible with symbolic processing, they had no interest.

Instead, their goal was to build models that could replace symbol-processing systems. As a typical example, they mention that the over-regularization errors that often occur in human children (such as writing the past tense of go as goed, rather than went) are a feature of neural networks, which also proves that neural networks are closer to the human brain than classical symbol processing rules. (But I can also give a lot of counterexamples.) ）

I started college in 1986, and neural networks ushered in their first major renaissance. The two volumes of technical discourses that Hinton co-curated sold out in a matter of weeks, the New York Times featured neural networks on the front page of the science section, and computational neuroscientist Terry Sejnowski explained how neural networks work on the Today Show. At that time, the research level of deep learning was not high, but at least it was a step forward.

In 1990, Hinton published a paper in the journal Artificial Intelligence called Connectionist Symbol Processing, hoping to connect the two worlds of deep learning and symbolic processing. I've always felt that Hinton is really in the right direction at this time, and I really hope that he will stick to the research. At the time, I was also pushing for the development of hybrid models—just from the perspective of psychology. 18 (Ron Sun and others were also pushing this trend hard in computer science at the time, but they didn't get the attention they deserved.) ）

But for reasons I didn't know, Hinton eventually decided that deep learning plus symbolic processing was nothing to worry about. I asked privately, but he refused to explain each time, and as far as I know he didn't make any specific arguments. Some people think that this is because Hinton himself did not develop well in the workplace in the following years, especially until the beginning of the twenty-first century, deep learning did not make any big moves; others think that Hinton was blinded by the success of deep learning.

When deep learning reappeared in 2012, the sharp divide between the two AI forces had remained on the sidelines for a decade.

By 2015, Hinton began to take a clear stand against symbolic technology. Hinton, who spoke at an AI symposium at Stanford University, likened symbols to "aether" (one of the biggest cognitive misconceptions in the history of science).19 I also spoke at that seminar, so I asked him during the coffee break and said that his theory was actually very much like a neural network implementation of symbology, but it was forcibly called a "stack." But he didn't answer, just let me stay.

After this, Hinton went crazy against symbolic technology. In 2016, LeCun, Bengio, and Hinton published a paper in the most prestigious journal in academia, Nature, in which the technique of symbolic processing was abandoned. With no room for reconciliation, the article claims that symbology should be replaced entirely with neural networks. Later, Hinton called at another conference not to waste money on symbol handling. It's like the era of electric vehicles has arrived, so why invest in the research of the internal combustion engine?

But this attitude of jumping to conclusions without full exploration is hard to believe. Hinton is right that AI researchers in the past have indeed attacked deep learning, but he himself is now just tit-for-tat, not much better.

In my opinion, this confrontational stance actually harms the interests of the entire AI academic community. But in any case, hinton's wave of symbolic processing crusades did become a huge success, and almost all of the research investments since then have focused on deep learning.

Hinton, LeCun, and Bengio shared the 2018 Turing Prize, and his research has come into the spotlight of the world.

Ironically, Hinton is actually the grandson of George Boole, and Boolean algebra, named after Boole, is one of the basic tools in symbolic AI. If these two generations can combine wisdom into one, perhaps the real AI we expect will come sooner rather than later.

As for why I insist that hybrid AI (more than deep learning and symbol processing) is the right direction, there are four reasons:

Much of the world's knowledge, from history to technology, is still dominated by symbols. Abandoning the accumulation of traditional knowledge like pure deep learning and exploring everything from scratch with computing power alone seems to be both arbitrary and self-binding.

Even in clearly organized fields such as arithmetic, deep learning doesn't perform well; hybrid systems may have more potential to be tapped by any single approach.

At many of the basic levels of computing, symbol systems still far outperform existing neural networks, which are better at reasoning in complex scenarios, enabling more systematic and reliable basic operations such as arithmetic, and more precisely expressing the relationship between parts and the whole (from understanding the three-dimensional world to analyzing human language, which is an essential ability).

Symbology is more robust and flexible in expressing and querying large databases, and is a better implementation of form verification techniques (which are critical in some security applications), and is itself well represented in modern microprocessor designs. Brutally abandoning the advantage and refusing to experiment with hybrid architectures is simply incomprehensible.

Deep learning systems are a kind of "black box" where we can only see the inputs and outputs, but we can't understand their internal operations and processing mechanisms, and we can't explain why the model gives the current conclusions. And if the model gives the wrong answer, there's nothing we can do better than collect more data.

As a result, deep learning is clumsy, difficult to explain, and in many scenarios it simply cannot help humans achieve cognitive enhancement. Conversely, if the learning ability of deep learning can be linked to clear symbols and rich semantics, the resulting hybrid solution may set off a new round of change.

It is precisely because artificial general intelligence (AGI) will take on a huge responsibility that it must be as solid, reliable as stainless steel, and take full advantage of every substrate. In the same way, no single AI approach is enough to solve the problem, and the right path should be to combine multiple approaches into one. Would anyone be stupid enough to unilaterally emphasize the importance of iron or carbon in stainless steel? But that's the state of the AI landscape.

But there is also good news. Hinton in 1990 briefly proposed the reconciliation between nerves and symbols, and I devoted my entire career to it. This fusion exploration is not stopped for a moment, and it is accumulating strength.

Artur Garcez and Luis Lamb published an article on hybrid models in 2009 called Neural-Symbolic Cognitive Reasoning. In recent years, the outstanding performance in board games such as Go and Chess is also a hybrid model. AlphaGo combines symbolic tree search with deep learning, a fundamental idea that originated in the late 1950s and was reinforced by richer statistics in the 1990s.

Obviously, classical tree search alone is not enough, nor is deep learning alone. Then there's DeepMind's ALphaFold2, an AI system that predicts protein structure through nucleotides, using the same hybrid model. It brings together a range of well-designed, symbolically expressed 3D molecular structures with amazing deep learning data analysis capabilities.

Researchers such as Josh Tenenbaum, Anima Anandkumar and Yejin Choi are also moving in the direction of neural signs. Numerous tech giants, including IBM, Intel, Google, Facebook, and Microsoft, are already seriously investing in neuro-semiotics. Swarat Chaudhuri and his colleagues are exploring a whole new field of neurosymbolic programming, which I personally look forward to.

For the first time in forty years, I have optimistic expectations for AI. As cognitive scientists Chaz Firestone and Brian Scholl put it, "The mind doesn't just have one way of working, because the mind doesn't exist alone." Instead, the mind is made up of multiple parts, and different parts have different mechanisms of operation: different ways of viewing colors and planning holidays, and different ways of understanding sentences, manipulating limbs, remembering events, and feeling emotions. "It's simply unrealistic to blindly pile all your cognition in one place, and as the entire AI industry becomes more open to hybrid approaches, I think the real opportunity may be coming."

In the face of practical challenges such as ethics and computational science, the field of AI should rely not only on mathematical and computer science knowledge, but also on linguistics, psychology, anthropology and neuroscience. Only by pooling all forces and uniting allies can AI break through the cage again. Keep in mind that the human brain is probably the most complex system in the known universe, and if we want to recreate such a complex system with technology, we will have to rely on the power of open collaboration.

bibliography:

Varoquaux, G. & Cheplygina, V. How I failed machine learning in medical imaging—shortcomings and recommendations. arXiv 2103.10292 (2021).

Chan, S., & Siegel, E.L. Will machine learning end the viability of radiology as a thriving medical specialty? British Journal of Radiology92, 20180416 (2018).

Ross, C. Once billed as a revolution in medicine, IBM’s Watson Health is sold off in parts. STAT News (2022).

Hao, K. AI pioneer Geoff Hinton: “Deep learning is going to be able to do everything.” MIT Technology Review (2020).

Aguera y Arcas, B. Do large language models understand us? Medium (2021).

Davis, E. & Marcus, G. GPT-3, Bloviator: OpenAI’s language generator has no idea what it’s talking about. MIT Technology Review (2020).

Greene, T. DeepMind tells Google it has no idea how to make AI less toxic. The Next Web (2021).

Weidinger, L., et al. Ethical and social risks of harm from Language Models. arXiv 2112.04359 (2021).

Bender, E.M., Gebru, T., McMillan-Major, A., & Schmitchel, S. On the dangers of stochastic parrots: Can language models be too big? Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency 610–623 (2021).

Kaplan, J., et al. Scaling Laws for Neural Language Models. arXiv 2001.08361 (2020).

Markoff, J. Smaller, Faster, Cheaper, Over: The Future of Computer Chips. The New York Times (2015).

Rae, J.W., et al. Scaling language models: Methods, analysis & insights from training Gopher. arXiv 2112.11446 (2022).

Thoppilan, R., et al. LaMDA: Language models for dialog applications. arXiv 2201.08239 (2022).

Wiggers, K. Facebook releases AI development tool based on NetHack. Venturebeat.com (2020).

Brownlee, J. Hands on big data by Peter Norvig. machinelearningmastery.com (2014).

McCulloch, W.S. & Pitts, W. A logical calculus of the ideas immanent in nervous activity. Bulletin of Mathematical Biology52, 99-115 (1990).

Olazaran, M. A sociological history of the neural network controversy. Advances in Computers37, 335-425 (1993).

Marcus, G.F., et al. Overregularization in language acquisition. Monographs of the Society for Research in Child Development57(1998).

Hinton, G. Aetherial Symbols. AAAI Spring Symposium on Knowledge Representation and Reasoning Stanford University, CA (2015).

LeCun, Y., Bengio, Y., & Hinton, G. Deep learning. Nature521, 436-444 (2015).

Razeghi, Y., Logan IV, R.L., Gardner, M., & Singh, S. Impact of pretraining term frequencies on few-shot reasoning. arXiv 2202.07206 (2022).

Lenat, D. What AI can learn from Romeo & Juliet. Forbes (2019).23. Chaudhuri, S., et al. Neurosymbolic programming. Foundations and Trends in Programming Languages7, 158-243 (2021).

https://nautil.us/deep-learning-is-hitting-a-wall-14467/