[REVIEW] : Google presents PaLM, its new language model capable of programming, explaining jokes and solving mathematical problems.

[REVIEW] : Google presents PaLM, its new language model capable of programming, explaining jokes and solving mathematical problems.

Google submitted PALM (Pathways Language Model), its new language model capable of programming, solving mathematical problems and explaining jokes with a learning efficiency rate that places it above other language models created to date .

Google explains that the PaLM was trained in the Pathways modelwhich allowed him to effectively train a single model with multiple Tensor Processing Units (TPU) Pods, as a statement on his blog indicates. It is based on ‘a few shots’ learning, which reduces the number of examples needed for task-specific training to fit a single application.

To do this, a database was used with 780 billion tokenswhich combines “a set of multilingual data” in addition to Englishincluding web docs, books, Wikipedia, conversations, and GitHub code. There is also a “lossless” vocabulary that “preserves all whitespace”, which is noted as “particularly important” for programmingand dividing Unicode characters that are not in the vocabulary into bytes.

This new AI hosts 540 billion parameterswhich exceeds the 175 billion of OpenAI’s GPT-3, the language model that Google cites as a pioneer in showing that these can be used for learning with “impressive results”.

Google’s new language model combines 6,144 v4 TPU chips at the address Access pathsThe “largest TPU configuration” used in history, the company said. PaLM also achieves a learning efficiency of 57.8% in the use of hardware flops, “the highest ever achieved for language models at this scale”.

This is made possible by a combination of “the parallelism strategy and a reformulation of the transform block” which allows the attention and advance layers to be computed in parallel, thus speeding up “TPU compiler optimizations”.

“PaLM has proven its breakthrough capabilities in many very difficult tasks,” says the company, which presented several examples ranging from language comprehension and generation to reasoning and programming-related tasks.

One of the tests that Google gives as an example is asking PaLM to. to guess a movie from four emojis: a robot, an insect, a plant and the planet Earth. Among all the options (LA Confidential, Wall-E, Lion: The Professional, BIG and Rush), the AI ​​chooses the right one: Wall-E.

In another case, he is asked to choose from a list of words two words associated with “stumble” and he also correctly chooses “fall” and “stumble”.

AI is also able to solve simple math problems and even to explain a joke by contextualizing it and breaking down the elements that appear in it in order to give it meaning.

Finally, Google also notes that PaLM is able to program by translating code “from one language to another”, writing code based on “a natural description of the language” and “fixing compilation errors”.