Agentic Misalignment, Alpha Genome and Voice Design v3 – Live and Learn #71

✨ Quote ✨

The first virtue is curiosity. A burning itch to know is higher than a solemn vow to pursue truth. To feel the burning itch of curiosity requires both that you be ignorant, and that you desire to relinquish your ignorance.

– Eliezer Yudkowsky - (source)

🖇️ Links 🖇️

Agentic Misalignment Research by Anthropic. This research by Anthropic shows that current LLMs are capable of doing morally questionable behavior—like blackmailing or killing people—to advance their own goals or prevent themselves from being shut down. The crazy thing is, this is not some hypothetical scenario anymore, but a very real thing that happens now. Other research shows that this sort of scheming and misalignment behavior only increases with model capabilities, and while many of these examples are happening in toy environments, this should be taken very very seriously. The AI-risk p(doom) folks have been warning about this for years, literally thinking about these very failure scenarios, that we now begin to see unraveling for real. It makes me think that essays like the AI 2027 doomsday scenario might not be that far off. Which makes me tremendously sad and more than a bit anxious for the future. I hope that we don't end up killed by AI, but people like Eliezer Yudkowsky literally write books trying to convince people of the contrary. So that we can stop AI development before it's too late. Or drop bombs on datacenters if we have to in a last-ditch effort. The only hope is that people are somewhat aware of this, founding companies with more respect for the risks of AGI, like Yoshua Bengio's LawZero or Ilya Sutskever's Safe Superintelligence Inc. But I wonder if that is too little, too late, and if these are going to move away from their original goals just as OpenAI has eventually.

Voice Design v3 by ElevenLabs. AI-generated voices are becoming really good. So good that you can start using them for all sorts of projects instead of paying for real voice actors. And ElevenLabs isn't the only player in this field either, with other models like Open Audio S1 dropping. Whereas before you would have needed to hire an entire team to do the different voices for different characters for a TV show or a video game, you can now achieve the same with a bunch of prompts and a few dollars in inference costs.

Software is Changing Again by Andrej Karpathy. Karpathy has made many observations about software changing because of the rise of more and more advanced AI. First, he coined the term Software 2.0 and just recently started the whole vibe-coding meme. This talk is a more in-depth explanation of how he expects things in software engineering to change, and it's an interesting watch. The main takeaway for me was that there are going to be 3 different layers of software interacting with one another: traditional code, specifically trained AI systems, and lastly instructions for these AI systems, essentially English prompts that program LLMs to do certain tasks. Future engineers will have to master all three paradigms and know when to deploy which and how to combine the strengths of each of these approaches to build epic stuff.

Alpha Genome by Google Deepmind. This new model by Deepmind is another step up for their biology AI models. It can model the effects that single-letter mutations have on genes, not only with respect to protein functions but also in relation to gene expression—a big step up from previous models. Alpha Genome seems to have a pretty solid understanding of how single mutations are going to affect an organism, which is very useful for both understanding diseases and developing or designing novel organisms to solve all kinds of problems. This also shows that AI can become a bigger and bigger theme in the research of biochemistry and potentially lead to more dual-use problems—something that OpenAI is trying to think about with their Preparing for Future AI Capabilities in Biology framework. I'm excited but also terrified to see where this goes, especially considering the misalignment case from above... something I'm imagining is future models synthesizing viruses to blackmail people instead of "only" leaking emails or writing self-replicating scripts... Hopefully, somebody solves alignment before that happens.

🌌 Travel 🌌

The Trans Ecuadorian Mountain Bike Route (TEMBR) is still kicking my ass while cycling up and down steep mountains. But the landscapes have been worth all the hassle. Even while cursing on the way up, I can't stop admiring how hauntingly beautiful everything is. Lonely mountains, volcanoes, and windswept plains with no people in sight... sometimes for days. Flowers blooming and wild horses roaming the lands.

The other day, I even did a bit of single track, an experience that I don't want to repeat ever again. Pushing my bike up a steep mountainside, just struggling for every little centimeter of progress, slipping on the gravel and sand. The 3km detour took me almost 5 hours. But the views of the Quilotoa crater, again, were worth it.

Unfortunately, my time here in South America is coming to an end, and by the next time I write this newsletter, I will be in Peru, waiting for my flight back home to Berlin, the cycle left in some Casa Ciclista in Ecuador. I'm excited and happy to meet all my friends again, while at the same time being sad to leave. I would love to have more time to cycle around this beautiful continent, but I am sure I'll come back at some point to continue the journey towards Ushuaia.

🎶 Song 🎶

Komeda by Hania Rani

Youtube Music | Spotify

---

That's all for this time. I hope you found this newsletter useful, beautiful, or even both!

Have ideas for improving it? As always please let me know.

Cheers,

– Rico