What Is ChatGPT Doing: ... and Why Does It Work?
()
About this ebook
Nobody expected this—not even its creators: ChatGPT has burst onto the scene as an AI capable of writing at a convincingly human level. But how does it really work? What's going on inside its "AI mind"? In this short book, prominent scientist and computation pioneer Stephen Wolfram provides a readable and engaging explanation that draws on his decades-long unique experience at the frontiers of science and technology. Find out how the success of ChatGPT brings together the latest neural net technology with foundational questions about language and human thought posed by Aristotle more than two thousand years ago.
Read more from Stephen Wolfram
Computation and the Future of the Human Condition Rating: 0 out of 5 stars0 ratingsHow to Teach Computational Thinking Rating: 0 out of 5 stars0 ratingsOn the Quest for Computable Knowledge Rating: 0 out of 5 stars0 ratingsPredicting the Eclipse: A Multimillenium Tale of Computation Rating: 0 out of 5 stars0 ratings
Related to What Is ChatGPT Doing
Related ebooks
Hexagonal Architecture Explained Rating: 0 out of 5 stars0 ratingsRead Write Code: A Friendly Introduction to the World of Coding, and Why It’s the New Litera Rating: 0 out of 5 stars0 ratingsHow to Teach Computational Thinking Rating: 0 out of 5 stars0 ratingsSQL for Data Scientists: A Beginner's Guide for Building Datasets for Analysis Rating: 0 out of 5 stars0 ratingsVisualize This: The FlowingData Guide to Design, Visualization, and Statistics Rating: 0 out of 5 stars0 ratingsThe Chief Data Officer Management Handbook: Set Up and Run an Organization’s Data Supply Chain Rating: 0 out of 5 stars0 ratingsGetting Data Science Done: Managing Projects From Ideas to Products Rating: 0 out of 5 stars0 ratingsData Smart: Using Data Science to Transform Information into Insight Rating: 4 out of 5 stars4/5Deep Learning with Python: Learn Best Practices of Deep Learning Models with PyTorch Rating: 0 out of 5 stars0 ratingsDocs for Developers: An Engineer’s Field Guide to Technical Writing Rating: 0 out of 5 stars0 ratingsSoftware Mistakes and Tradeoffs: How to make good programming decisions Rating: 0 out of 5 stars0 ratingsJulia for Data Analysis Rating: 0 out of 5 stars0 ratingsMove Fast: How Facebook Builds Software Rating: 5 out of 5 stars5/5GROKKING ALGORITHMS: Tips and Tricks of Grokking Functional Programming Rating: 0 out of 5 stars0 ratingsLeading Projects with Data: Overcome Behavioral and Cultural Barriers to Unlock the Hidden Value of Data in Projects Rating: 0 out of 5 stars0 ratingsManaging Machine Learning Projects: From design to deployment Rating: 0 out of 5 stars0 ratingsEffective Data Science Infrastructure: How to make data scientists productive Rating: 0 out of 5 stars0 ratingsDigital Asset Ecosystems: Rethinking crowds and clouds Rating: 0 out of 5 stars0 ratingsBig Data Engineer A Complete Guide - 2021 Edition Rating: 0 out of 5 stars0 ratingsAdvanced Analytics with Transact-SQL: Exploring Hidden Patterns and Rules in Your Data Rating: 0 out of 5 stars0 ratingsThe Infinite Machine: How an Army of Crypto-Hackers Is Building the Next Internet with Ethereum Rating: 3 out of 5 stars3/5Build a Career in Data Science Rating: 5 out of 5 stars5/5Enterprise Architecture at Work: Modelling, Communication and Analysis Rating: 2 out of 5 stars2/5Beginners' Artificial Intelligence and Python Programming Rating: 0 out of 5 stars0 ratingsGROKKING ALGORITHMS: Advanced Methods to Learn and Use Grokking Algorithms and Data Structures for Programming Rating: 0 out of 5 stars0 ratingsDesigning Deep Learning Systems: A software engineer's guide Rating: 0 out of 5 stars0 ratingsText as Data: A New Framework for Machine Learning and the Social Sciences Rating: 0 out of 5 stars0 ratingsWriting for Computer Science Rating: 4 out of 5 stars4/5How to Speak Tech: The Non-Techie’s Guide to Key Technology Concepts Rating: 4 out of 5 stars4/5Data Insights: New Ways to Visualize and Make Sense of Data Rating: 2 out of 5 stars2/5
Intelligence (AI) & Semantics For You
Summary of Super-Intelligence From Nick Bostrom Rating: 5 out of 5 stars5/5Artificial Intelligence: A Guide for Thinking Humans Rating: 4 out of 5 stars4/5ChatGPT For Dummies Rating: 0 out of 5 stars0 ratingsDark Aeon: Transhumanism and the War Against Humanity Rating: 5 out of 5 stars5/52084: Artificial Intelligence and the Future of Humanity Rating: 4 out of 5 stars4/5Creating Online Courses with ChatGPT | A Step-by-Step Guide with Prompt Templates Rating: 4 out of 5 stars4/5Mastering ChatGPT: 21 Prompts Templates for Effortless Writing Rating: 5 out of 5 stars5/5ChatGPT For Fiction Writing: AI for Authors Rating: 5 out of 5 stars5/5Midjourney Mastery - The Ultimate Handbook of Prompts Rating: 5 out of 5 stars5/5Dancing with Qubits: How quantum computing works and how it can change the world Rating: 5 out of 5 stars5/5Chat-GPT Income Ideas: Pioneering Monetization Concepts Utilizing Conversational AI for Profitable Ventures Rating: 4 out of 5 stars4/5The Secrets of ChatGPT Prompt Engineering for Non-Developers Rating: 5 out of 5 stars5/5Impromptu: Amplifying Our Humanity Through AI Rating: 5 out of 5 stars5/5Our Final Invention: Artificial Intelligence and the End of the Human Era Rating: 4 out of 5 stars4/5101 Midjourney Prompt Secrets Rating: 3 out of 5 stars3/5A Quickstart Guide To Becoming A ChatGPT Millionaire: The ChatGPT Book For Beginners (Lazy Money Series®) Rating: 4 out of 5 stars4/5Large Language Models Rating: 2 out of 5 stars2/5Enterprise AI For Dummies Rating: 3 out of 5 stars3/5AI for Educators: AI for Educators Rating: 5 out of 5 stars5/5ChatGPT Ultimate User Guide - How to Make Money Online Faster and More Precise Using AI Technology Rating: 0 out of 5 stars0 ratingsChatGPT: The Future of Intelligent Conversation Rating: 4 out of 5 stars4/5The Algorithm of the Universe (A New Perspective to Cognitive AI) Rating: 5 out of 5 stars5/5
Reviews for What Is ChatGPT Doing
0 ratings0 reviews
Book preview
What Is ChatGPT Doing - Stephen Wolfram
What Is ChatGPT Doing ... and Why Does It Work?
Copyright © 2023 Stephen Wolfram, LLC
Wolfram Media, Inc. | wolfram-media.com
ISBN-978-1-57955-081-3 (paperback)
ISBN-978-1-57955-082-0 (ebook)
Technology/Computers
Library of Congress Cataloging-in-Publication Data:
Names: Wolfram, Stephen, 1959- author.
Title: What is ChatGPT doing ... and why does it work? / Stephen Wolfram.
Other titles: ChatGPT
Description: First edition. | [Champaign, Illinois] : Wolfram Media, Inc., [2023] | Includes bibliographical references.
Identifiers: LCCN 2023009927 (print) | LCCN 2023009928 (ebook) | ISBN 9781579550813 (paperback) | ISBN 9781579550820 (ebook)
Subjects: LCSH: Natural language generation (Computer science)—Computer programs. | Neural networks (Computer science) | ChatGPT. | Wolfram language (Computer program language)
Classification: LCC QA76.9.N38 W65 2023 (print) | LCC QA76.9.N38 (ebook) | DDC 006.3/5—dc23/eng/20230310
LC record available at https://lccn.loc.gov/2023009927
LC ebook record available at https://lccn.loc.gov/2023009928
For permission to reproduce images, contact permissions@wolfram.com.
Visit the online version of this text at wolfr.am/SW-ChatGPT and wolfr.am/ChatGPT-WA. Click any picture to copy the code behind it.
ChatGPT screenshots were generated with GPT-3, OpenAI’s AI system that produces natural language.
First edition.
Contents
Preface
What Is ChatGPT Doing ... and Why Does It Work?
It’s Just Adding One Word at a Time · Where Do the Probabilities Come From? · What Is a Model? · Models for Human-Like Tasks · Neural Nets · Machine Learning, and the Training of Neural Nets · The Practice and Lore of Neural Net Training · Surely a Network That’s Big Enough Can Do Anything!
· The Concept of Embeddings · Inside ChatGPT · The Training of ChatGPT · Beyond Basic Training · What Really Lets ChatGPT Work? · Meaning Space and Semantic Laws of Motion · Semantic Grammar and the Power of Computational Language · So ... What Is ChatGPT Doing, and Why Does It Work? · Thanks
Wolfram|Alpha as the Way to Bring Computational Knowledge Superpowers to ChatGPT
ChatGPT and Wolfram|Alpha · A Basic Example · A Few More Examples · The Path Forward
Additional Resources
Preface
This short book is an attempt to explain from first principles how and why ChatGPT works. In some ways it’s a story about technology. But it’s also a story about science. As well as about philosophy. And to tell the story, we’ll have to bring together a remarkable range of ideas and discoveries made across many centuries.
For me it’s exciting to see so many things I’ve so long been interested in come together in a burst of sudden progress. From the complex behavior of simple programs to the core character of language and meaning, and the practicalities of large computer systems—all of these are part of the ChatGPT story.
ChatGPT is based on the concept of neural nets—originally invented in the 1940s as an idealization of the operation of brains. I myself first programmed a neural net in 1983—and it didn’t do anything interesting. But 40 years later, with computers that are effectively a million times faster, with billions of pages of text on the web, and after a whole series of engineering innovations, the situation is quite different. And—to everyone’s surprise—a neural net that is a billion times larger than the one I had in 1983 is capable of doing what was thought to be that uniquely human thing of generating meaningful human language.
This book consists of two pieces that I wrote soon after ChatGPT debuted. The first is an explanation of ChatGPT and its ability to do the very human thing of generating language. The second looks forward to ChatGPT being able to use computational tools to go beyond what humans can do, and in particular being able to leverage the computational knowledge superpowers
of our Wolfram|Alpha system.
It’s only been three months since ChatGPT launched, and we are just beginning to understand its implications, both practical and intellectual. But for now its arrival is a reminder that even after everything that has been invented and discovered, surprises are still possible.
Stephen Wolfram
February 28, 2023
What Is ChatGPT Doing ... and Why Does It Work?
(February 14, 2023)
It’s Just Adding One Word at a Time
That ChatGPT can automatically generate something that reads even superficially like human-written text is remarkable, and unexpected. But how does it do it? And why does it work? My purpose here is to give a rough outline of what’s going on inside ChatGPT—and then to explore why it is that it can do so well in producing what we might consider to be meaningful text. I should say at the outset that I’m going to focus on the big picture of what’s going on—and while I’ll mention some engineering details, I won’t get deeply into them. (And the essence of what I’ll say applies just as well to other current large language models
[LLMs] as to ChatGPT.)
The first thing to explain is that what ChatGPT is always fundamentally trying to do is to produce a reasonable continuation
of whatever text it’s got so far, where by reasonable
we mean what one might expect someone to write after seeing what people have written on billions of webpages, etc.
So let’s say we’ve got the text "The best thing about AI is its ability to. Imagine scanning billions of pages of human-written text (say on the web and in digitized books) and finding all instances of this text—then seeing what word comes next what fraction of the time. ChatGPT effectively does something like this, except that (as I’ll explain) it doesn’t look at literal text; it looks for things that in a certain sense
match in meaning. But the end result is that it produces a ranked list of words that might follow, together with
probabilities":
And the remarkable thing is that when ChatGPT does something like write an essay what it’s essentially doing is just asking over and over again given the text so far, what should the next word be?
—and each time adding a word. (More precisely, as I’ll explain, it’s adding a token
, which could be just a part of a word, which is why it can sometimes make up new words
.)
But, OK, at each step it gets a list of words with probabilities. But which one should it actually pick to add to the essay (or whatever) that it’s writing? One might think it should be the highest-ranked
word (i.e. the one to which the highest probability
was assigned). But this is where a bit of voodoo begins to creep in. Because for some reason—that maybe one day we’ll have a scientific-style understanding of—if we always pick the highest-ranked word, we’ll typically get a very flat
essay, that never seems to show any creativity
(and even sometimes repeats word for word). But if sometimes (at random) we pick lower-ranked words, we get a more interesting
essay.
The fact that there’s randomness here means that if we use the same prompt