100k Token Contexts, AI Tutors and Google/IO – Live and Learn #17
Welcome to this edition of Live and Learn. This time it's one of those editions, where the amount of things that have happened recently is simply incomprehensible. I cut out so much that I would have liked to include in this edition… And still, this one grew beyond the 4-5 links that I try to set myself as a soft limit.
AI is moving at an ever-accelerating pace and this becomes more and more evident the more I read about it. There are way over 100 tabs bookmarked that I haven't read yet, and soon I will bookmark the next 100. And this to me is insane. It feels great to dig into all that is happening but at the same time, it's overwhelming.
Anyways. This edition contains some insights from the GoogleIO, a great article about how open source is taking over LLTM research by storm, and an exploration of the music generation capabilities of ChatGPT. As always I hope you find something useful and enjoy this newsletter.
✨ Quote ✨
In perhaps nature's most miraculous transformation, the universe evolved the capacity to ponder and understand itself.
— Edward Kolb
Links
GPT-4 explains GPT-2 neurons by OpenAI. As models get more powerful we can turn them back on themselves to help us understand what they do. Here they prompted GPT-4 to simulate how neurons in GPT-2 work. And then scored those simulations and found short descriptions that encapsulate what activates those neurons. This leads to a high-level understanding of what the neurons are doing. By using this technique they can "label" most of the neurons within GPT-2. This is already impressive as is. But their goal is to explain everything that their models do, end to end. In the words of OpenAI: "We want to eventually automatically find and explain entire neural circuits implementing complex behaviors." Let's see how far they get.
Introducing 100k Context Windows by Anthropic. You can now officially feed LLTMs entire books in a single prompt. This blows my mind. I mean seriously, 100k in tokens equals roughly 75k words, which is the average length of most novels out there. Just imagine that you knew somebody who could read and summarize, and to a certain extent understand all of "Harry Potter and the Philosopher's Stone" (76944 words) in a little less than a minute. Now stop imagining it, because well, we have the technology to do that, right now.
We have No Moat by SemiAnalysis. This leaked Google insider document, argues that open source is starting to dominate over OpenAI's and Google's efforts. And that these companies should embrace the open-source culture, instead of fighting it. The examples this piece links to alone, make it worth reading but the analysis is really where it shines. Because the article builds a compelling case as to why open source changes the research landscape. And then goes on to explain why this change is something that Google and other companies like it, should worry about. There is a timeline at the end that shows, just how fast things evolve once the open-source community gets involved. And instead of fearing that competition, Google should embrace and try to use that power to its advantage.
Google I/O Keynote by Google. I've been watching the Google Keynote sessions for quite some years now and they never ceased to amaze me. However, with the current rate of progress in AI development, this keynote felt a bit lackluster. It's weird to see Google playing catch-up with OpenAI and see them trying to convince the world that they are still a powerhouse within AI research and operations. The coolest announcement, in my opinion, was that of Vertex AI. A unified platform for doing machine learning training, fine-tuning, deployment and all the other things that engineers will need to build AI into their products. Google is positioning itself to be not just a model provider, but an infrastructure powerhouse. They want to sell an out-of-the-box, easy-to-use platform so that everybody can integrate AI into their business. I think this is a great idea and I am looking forward to seeing where this is going. But I am also hyped as to what AWS, Microsoft, and Nvidia are going to do in this space as well.
Mr. Ranedeer by JushBJJ. One of the coolest personalized AI tutors I have seen. With it, you can learn pretty much anything, really quickly. This prompt feels like magic. It shows how people can configure large language models to solve very specific tasks. This is what "prompt engineering" really means. Copy-pasting the prompt and being greeted exactly with what the prompt was engineered to do. An achievement like this is simply and utterly mindblowing. If you want to learn anything I think tutors like Mr. Ranedeer are one of the best ways out there today. Khan Academy also rolled out their Khanmigo tutor a while back, which is worth checking out.
But can ChatGPT-4 write a good melody? by Marc Evanstein. This is an exploration of using GPT-4 to create music. Right now GPT-4 is not great at this task, but still it can produce small coherent musical phrases. But if you stop for a second and think about what it is actually doing it is already extremely good. It's only a large language model after all, and yet it can generate melodies! How insane is that? Just keep in mind that it has never learned the rules of music. Nor does it hear or have a concept of sound as such... I mean, seriously, it can only output Midi notes as Python code. Try to imagine somebody who has never heard music in their life and was told to compose a piece in Python code... who then goes on to do a pretty decent job at it. That would be freaking impressive. And that is exactly what ChatGPT is doing in this video. This is absolutely remarkable and shows just how general GPT-4 already is.
🌌 Midjourney 🌌
🎶 Song 🎶
Moonlight Sonata, 3rd Movement by Beethoven
I hope you found this newsletter useful, beautiful, or even both!
Have ideas for improving it? Please let me know.
Cheers,
– Rico