The hype and the hope surrounding a powerful new artificial intelligence tool
The latest version of OpenAI’s text processing engine signifies a leap forward in computers’ ability to generate coherent language, but comes with a serious risk of bias.
Depending on where you are sitting, the ability to use artificial intelligence to automatically generate reports, articles and even novels that are indistinguishable from human efforts is either the Holy Grail of AI or your worst nightmare.
As someone who has made a living from writing for my entire life, I lean towards the latter category. But it is hard not to be impressed by some of the output of GPT-3, the potential uses of which go beyond the generation of reports, articles and books to include writing computer code and generating medical reports.
San Francisco-based start-up founder and software developer, Arram Sabeti, has had early access to GPT-3 last month and is impressed with the results.
“I’ve gotten it to write songs, stories, press releases, guitar tabs, interviews, essays, technical manuals,” wrote Sabeti.
“It’s hilarious and frightening. I feel like I’ve seen the future and that full [artificial general intelligence] might not be too far away.”
The examples of poetry, press releases and fil scripts the program generated with just a few words as a prompt from Sabeti, are certainly eye-opening. The examples Sabeti posts are quite entertaining, often well-written, as though a quirky university professor is riffing on an issue they know a lot about, but disappearing down a rabbit hole at the same time.
What exactly is GPT-3?
GPT-3, or Generative Pre-trained Transformer 3, is a text processing computer program and language model that uses deep learning to produce human-like text results. Released to limited testing in May, it is ultimately the creation of OpenAI, the nonprofit organisation co-founded in 2015 by billionaire tech entrepreneur Elon Musk and others, to work on ethical advances in artificial technology.
Musk has since parted ways with OpenAI though remains a funder. The organisation continues in its stated mission of undertaking research to advance the field of AI while trying to ensure it doesn’t do harm. GPT-3, naturally enough, builds on GPT-2 which was released last year and won praise for the convincing passages of text it created with just an opening prompt of a few words.
But GPT-3 is a massive leap forward in terms of the scale of the data drawn on to train the model, but also the parameters or values the neural network works on optimising during training. GPT-3 has 175 billion parameters, at least a 10x increase on GPT-2.
When it comes to data, GPT-3 draws on the “Common Crawl” dataset, which scrapes 60 million internet domains and sites to which they link. That amounts to petabytes of information. Those sites range from reputable media outlets to social media platforms. But the researchers have also fed millions of words from the likes of Wikipedia, textbooks and computer manuals into the model too.
How good is it really?
Computer-generated efforts to create everything from novels to symphonies have been making headlines for years, but more for their novelty value than their actual quality. The early buzz around GPT-3 suggests the massive data set the program has to draw on is leading to it producing much more naturalistic turns of phrase.
Its ease of use, which allows users to start with a prompt and get GPT-3 to predict the resulting language, is seen as a major drawcard. The model doesn’t need to be retrained with numerous examples for every task it is given.
But as Arram Sabeti points out, it is hard to know what exactly you are getting with GPT-3.
“If GPT-3 fails at a task that only proves that your prompt didn’t work, not that the model can’t do the task. Beyond outright failure, you can also get radically different output quality using slightly different prompts.”
There’s clearly a lot of fine-tuning to be done.
Is it dangerous?
One major caveat with GPT-3 is that the swathes of information drawn from the web to train it, also feature the biases of those who originally typed those words. GPT-3 spits out reasonably coherent summaries of information and chat-bot like answers, based on the best and the worst of the web.
Its creators acknowledge this as a potential problem in their research paper released with the software. They initially chose to delay the release of its previous version over those concerns that the Generative Pre-trained Transformer could be used to reinforce existing biases and even auto-generate racist or discriminatory texts that could be used by spammers or for online abuse.
“Mitigating negative effects such as harmful bias is a hard, industry-wide issue that is extremely important,” write OpenAI’s creators.
“Ultimately, our [application programming interface] models do exhibit biases that will appear on occasion in generated text. Our API models could also cause harm in ways that we haven’t thought of yet.”
OpenAI is taking a cautious approach working with early customers to develop user guidelines and develop “tools to label and intervene on manifestations of harmful bias”.
In the meantime, it is unlikely that any major company is going to trust GPT-3 to generate text for public consumption without some serious vetting for quality.
Why is OpenAI selling its text generator?
While it comes under the control of the non-profit OpenAI Inc. and was developed by the OpenAI research lab, GPT-3 is being commercialised through OpenAI LP the for-profit company.
Currently, in a limited beta with a small number of customers, it will soon be made available as a commercial product in the form of a cloud computing service and API that customers can adapt to their needs. The tight control over GPT-3, says OpenAI, is to try and avoid the abuse and misuse it knows could derail progress with its AI text generator.
But as a powerful AI service, GPT-3’s commercial debut sees the organisation beginning to compete in the cloud computing space, dominated by the likes of Amazon Web Services, Microsoft, IBM and Google.
More likely however is a scenario where some of those players draw on OpenAI’s toolsets in conjunction with their own AI platforms. Microsoft last year invested US$1 billion in OpenAI LP, with that aim in mind. OpenAI now runs on Microsoft’s Azure platform.
Where is GPT-3 taking us?
OpenAI’s co-founder and CEO, Sam Altman, who along with Musk and other Silicon Valley heavyweights pledged US$1 billion to launch the organisation’s quest for powerful but ‘friendly’ AI, has tried to tone down some of the hype around the release of GPT-3.
On July 20, he tweeted:
Still, Altman is clearly impressed and excited at its potential. In an earlier tweet he wrote: “For non-programmers, it’s like experiencing the magic of programming for the first time.”
That is likely the biggest leap forward for OpenAI’s efforts with natural language processing, making it accessible to a much wider audience who will come up with ways to explore ita potential and also to identify its limitations.
We are still a long way from getting beyond the relatively ‘narrow’ uses of artificial intelligence that are nevertheless delivering major improvements in everything from business process automation to diagnosing cancers.
There’s another aspect to the fast-paced advances in GPT – the computer processing power required to support it. As more of these systems come online, the task of churning through massive amounts of data will require increasing amounts of computing capacity, which in turn will require additional power.
As with the Bitcoin mining craze of the last decade, that has a potentially serious environmental impact. Along with the quest for more sophisticated AI will be a need to find new ways to more efficiently power the computer processing that underpins it.
Microsoft is a technology company whose mission is to empower every person and every organisation on the planet to achieve more. We strive to create local opportunity, growth, and impact in every country around the world.