Microsoft and Nvidia surpass OpenAI by creating "the worldโs largest and most powerful generative language model,โMegatron-Turing Natural Language Generation model (MT-NLG).
Just under a year and a half ago OpenAI announced completion ofย GPT-3, its natural language processing algorithm that was, at the time, the largest and most complex model of its type. This week, Microsoft and Nvidiaย introducedย a new model theyโre calling โthe worldโs largest and most powerful generative language model.โ
The Megatron-Turing Natural Language Generation model (MT-NLG) is more than triple the size of GPT-3 at 530 billion parameters.
GPT-3โs 175 billion parameters was already a lot; its predecessor,ย GPT-2, had a mere 1.5 billion parameters, and Microsoftโsย Turing Natural Language Generationย model, released in February 2020, had 17 billion.
A parameter is an attribute a machine learning model defines based on its training data, and tuning more of them requires upping the amount of data the model is trained on. Itโs essentially learning to predict how likely it is that a given word will be preceded or followed by another word, and how much that likelihood changes based on other words in the sentence.
As you can imagine, getting to 530 billion parameters required quite a lot of input data and just as much computing power. The algorithm was trained using an Nvidia supercomputer made up of 560 servers, each holding eight 80-gigabyte GPUs. Thatโs 4,480 GPUs total, and anย estimated costย of over $85 million.
For training data, Megatron-Turingโs creators usedย The Pile, a dataset put together by open-source language model research group Eleuther AI. Comprised of everything from PubMed to Wikipedia to Github, the dataset totals 825GB, broken down into 22 smaller datasets. Microsoft and Nvidia curated the dataset, selecting subsets they found to be โof the highest relative quality.โ They added data fromย Common Crawl, a non-profit that scans the open web every month and downloads content from billions of HTML pages then makes it available in a special format for large-scale data mining. GPT-3 was also trained using Common Crawl data.
Microsoftโsย blog postย on Megatron-Turing says the algorithm is skilled at tasks like completion prediction, reading comprehension, commonsense reasoning, natural language inferences, and word sense disambiguation. But stay tunedโthere will likely be more skills added to that list once the model starts being widely utilized.
GPT-3 turned out to have capabilities beyond what its creators anticipated, like writing code, doing math, translating between languages, and autocompleting images (oh, and writing aย short filmย with a twist ending). This led some toย speculateย that GPT-3 might be the gateway toย artificial general intelligence. But the algorithmโs variety of talents, while unexpected, still fell within the language domain (including programming languages), so thatโs a bit of a stretch.
However, given the tricks GPT-3 had up its sleeve based on its 175 billion parameters, itโs intriguing to wonder what the Megatron-Turing model may surprise us with at 530 billion. The algorithm likely wonโt be commercially available for some time, so itโll be a while before we find out.
The new modelโs creators, though, are highly optimistic. โWe look forward to how MT-NLG will shape tomorrowโs products and motivate the community to push the boundaries of natural language processing even further,โ they wrote in theย blog post. โThe journey is long and far from complete, but we are excited by what is possible and what lies ahead.โ
ย
ย