AGI is coming

Manav Gupta [00:01]

Hello, folks. Welcome to Ship AI. I’m your host, Manav Gupta, Vice President and CTO at IBM Canada. Today, we are going to deep dive on the single most important question in technology right now. What is AGI? When is it coming?

Manav Gupta [00:16]

And what does it mean for you? Questions for the audience as I get into this. When do you think we are going to have an AI that can do any cognitive job a human can do? What would you do with an AI that was smarter than any human available for say $20 a month? What’s your number in terms of probability that you think AI causes serious harm? Today, we’re going to answer all of these.

Manav Gupta [00:46]

Let’s get into it. To get started, I’m going to lay down the agenda for this conversation here. So we are going to be looking at the following elements today. Okay, so here is the complete agenda for today. So we’ll get into core concepts of AGI. And if I click into that, we’re going to look at what is AGI versus artificial superintelligence.

Manav Gupta [01:20]

We’ll talk about the predefined levels from Google DeepMind just to dive deeper into AGI, the concepts behind it. I’m going to introduce you, if you don’t already know, about the AGI Doomsday Clock. We’ll talk about some of the gaps that exist, such as the gaps for data, model alignment, etc. And I’m going to introduce to you a theoretical problem. Well, what happens when there are two companies chasing AI, chasing AGI, and one gets there 10 minutes prior to the other? There is a lot of research that’s already done into it, famously by the AI 2027 websites.

Manav Gupta [01:57]

We’re going to look at their timeline. We’re going to look at the agentic ladder that they’ve put forward. We’re going to look at what happens when AI is a multiplier. We’ll look at geopolitics, the two potential endings, the good or the bad, if you so want to call it, the economics behind AGI, and time permitting we’ll get into the data wall. As usual, no…

Manav Gupta [02:23]

Conversation about AI, AGI, ASI can happen without risk and safety. So I’m going to introduce to you first the blockers. There still exists at least five different walls that have to be climbed in order for AI to become AGI or anything else. I’m going to introduce to you a probabilistic estimate of the Doomsday scenario. Let’s call it PDoom. We’ll look at the governance pipeline.

Manav Gupta [02:50]

And then finally, we’re going to end with the impact on humans. Well, what happens when predominant, when majority of human jobs are already accounted for, taken into account? What happens to human futures? How do humans survive if there is a probabilistic scenario that we don’t have any jobs? And finally, are there any careers whatsoever that are AI proof? So without further ado, let’s begin.

Manav Gupta [03:19]

All right, as we get started, the question in front of us always, first and paramount always is, well, hey, what do you mean by AI or AGI, et cetera? So that’s the first thing that we’re going to get into. So I’m going to go back here. So I’m going to assume there is a general understanding of what is AI, right? Any machines or algorithmic behavior that is designed to mimic human cognitive function. I think that’s fairly straightforward.

Manav Gupta [03:52]

Let’s talk about what majority of the masses think call AGI or artificial general intelligence. For reference, Google’s DeepMind came up with five different levels to define AGI. The most broadly accepted and understood definition of AGI maps to level two from DeepMind. This is where an AI can perform any cognitive task a human can do across all domains. That’s the key. It doesn’t need to be smarter than humans, just as capable, but faster, cheaper, and of course, infinitely scalable.

Manav Gupta [04:27]

At AGI, this is not the Skynet scenario that robots take over the world. Humans can still supervise theoretically and correct the system. In practical terms, what it means is it’s better than 50 % of all humans out there for any scary task. It can learn any new domain with minimal or no examples. Generally speaking, in terms of cost, you’re looking at about $20 a month for any knowledge worker. And as I said, in theory, still operates within human defined constraints.

Manav Gupta [05:08]

A key difference here that people sometimes tend to get confused about is that when we talk about AGI, It’s going to be self-improving. Broadly speaking, it does not improve autonomously. It does not have self-defined goals. It does not surpass humans in creativity, in leadership, or strategic judgment. In other words, AGI does not mean machine takeover. Humans are still evaluating its output, catching its mistakes, or writing decisions.

Manav Gupta [05:39]

Then comes the transition zone. but I’m going to get into that later. Let’s talk about the hypothetical scenario, which is the Skynet scenario that you have artificial super intelligence. This is where intelligence that exceeds the best humans, PhD level humans in every domain, including the ability to design its own successors, model the geopolitical systems, solve unsolved mathematics. and pursue goals across long time horizons. At ASI, the gap between human and AI capability becomes insurmountable.

Manav Gupta [06:21]

In other words, this thing gets so out of control, it is so much better, it’s better than any human, every human that’s on the planet. This is what people are talking about when they are loosely talking about AI taking over. It’s not robots, but it’s cognitive dominance. So what does that mean? This means that this is an AI that is better. It exceeds the human brain on every intellectual task.

Manav Gupta [06:51]

Remember with AGI, we talked about AI better than being 50 % of all humans. This is AI that’s better than all humans on every task. It can also model human behavior and psychology with superhuman accuracy. Well, at this point, I imagine the words like humans or superhumans become irrelevant. Here is the crucial part. This is AI when it can design its own successor architecture without human input.

Manav Gupta [07:20]

This can operate on time scales and speeds that humans can no longer track. Now comes the million dollar question. Well, is it safe? Quite candidly, ASI being safe, truly depends upon whether the values were correctly aligned before transition. Remember, I said that we’re going to get into the transition phase and now is the time. So between AGI, a general purpose AI that is better than 50 % of all humans that humans can still theoretically control, and the Skynet scenario that AI takes over, there is a dangerous middle.

Manav Gupta [08:03]

Let’s call that the transition period. where I’m going to introduce to you this hypothetical scenario. this is when AGI, so once AGI is capable to meaningfully improve and solve its own architecture and training, this is where recursive self-improvement begins. And we begin, we will, when that happens, we will begin to transition from AGI to ASI. I call this the transition zone because this is a situation where human oversight will become harder and harder and the capability gap begins to widen exponentially. For reference, the AI 2027 website, it places the agents three in this zone.

Manav Gupta [08:45]

So what’s happening in this transition zone? AI is getting so good, it’s not quite as high yet, but it is so good that can evaluate and begin to improve its own training procedures, which therefore means that each training improvement makes the next iteration faster and better. Human evaluation begins to lie because we just cannot keep up. We can’t grade the tests. The tests are coming thick and fast. They are so varied.

Manav Gupta [09:21]

They are so complex. Humans just can no longer keep up. The alignment specification. So if there was a specification for model alignment, it can no longer be verified against what is actually learned. In practical terms, this is not an inevitable catastrophe, but this is a window where we need good governance and safety research still matters. But I’m here to tell you that the window is narrow and closing.

Manav Gupta [09:47]

So I hope it gave you a little bit of an idea of the core concepts of AGI versus ASI. Now I briefly alluded to the AGI levels from DeepMind, so now is the time for us to get into that. So DeepMind from Google in 2023, they came up with levels of AGI. There are six of them from level zero to level five. Level zero being the dead simplest to understand, no AI. So using basic tools such as calculators, the abacus, rule-based systems, spreadsheets.

Manav Gupta [10:21]

Then you get into L1, emerging artificial general intelligence. The performance here is equivalent to, let’s call it an unskilled human, where you see most of the examples today are happening here. That’s the GPT-4-0, the Clover 3.7, the current frontier models. The capabilities are predominantly narrow, but they are going towards being general capabilities. AGI level two.

Manav Gupta [10:59]

Now we are beginning to get into a competent AGI, where the performance is better than 50th percentile of skilled adults. And that’s the key, adults that are skilled in those capabilities. Which also means that their availability is general. The good thing, or I guess the bad thing, if you’re an AI researcher, we’re not quite there yet. This is not yet achieved. This is the next critical threshold.

Manav Gupta [11:25]

Once we hit this next critical threshold, the next level of evolution is an expert AGI. So this is now moving from AGI towards almost ASI. We are still a couple of levels beyond that, but this is level three, expert artificial general intelligence, which means this AI is getting so good. It is better than 90th percentile skilled adults, which means better than 90th. better than 90 % of all of humans on a broad set of skills. Think of this emerging scenario such as the protein folding algorithms and AI from Google, some of these specialized domain tools, et cetera.

Manav Gupta [12:03]

Level 4 is a virtuoso AGI. Now we’re getting into the twilight zone that it is now better than 99 percentile of all skilled adults. So now it’ll be a struggle to find a human that’s better than that AI. We have some very narrow examples right now. So in this case, in level 4, the generality is still narrow only. So the deep blue, the chess computer, AlphaGo.

Manav Gupta [12:38]

So these are narrow. tasks that this AI can do better than superhuman capability. Building on top of that is the final level from Google DeepMind, which is ASI. Now the performance exceeds all humans. So this is now beyond what human brain can do. Generality, it may start off as narrow, but become general, being able to learn.

Manav Gupta [13:05]

new concepts very easily. We have some examples, by the way. So you have things like AlphaZero that I previously talked about or Stockfish. They surpass every human in a very specific domain. So to recap, six levels from level zero to level five, level zero being no AI, level six being superhuman ASI. We currently are at level one.

Manav Gupta [13:36]

Okay, so we covered AGI versus ASI, we covered the deep mind levels. So where exactly are we? So I put together this animation just to visualize where exactly we are and how the various improvements that are happening in AI algorithms and research and collecting the data, where does all of that look like. So, one can argue that the clock truly started with the advent of AlexNet. This one, the ImageNet challenge by a whopping 10 % margin, it left every other model out there in its wake. Deep neural networks at this point shifted from academic curiosity to an existential force.

Manav Gupta [14:21]

Well and truly, the modern era of AI algorithms began. Of course, then in 2017, we got the eponymous attention is all you need paper. which publishes the architecture which now powers pretty much every AI system of the next decade. The entire field of AI pivoted overnight. We are now at 35 minutes to midnight. In 2023, OpenAI confirmed the scaling laws.

Manav Gupta [14:51]

So they built GPT-3 with 175 billion parameters. The bitter lesson was realized that scaling works. OpenAI showed that capability emerges predictably as predicted with compute. So now the race has well and truly began. Then in 2022, we began to see AI escaping the lab. So with GPT 3.5 and then of course its subsequent models, OpenAI, chatGPT got 100 million users in under 60 days.

Manav Gupta [15:23]

Fastest product adoption in history. The public, all of humanity really became a stress test. Alignment concerns have started to go mainstream. So now you can see that the clock’s now going 18 minutes to midnight. Then of course, in 2023 onwards, the world saw the rise of reasoning models starting from DeepSeq and then really made famous by GPT-4 Plus, which passed the bar exam at 90th percentile. The AGI clock moves from 5 % to 87.5%.

Manav Gupta [16:06]

The models are now beginning to move much faster. 2024 saw the rise of agentic AI. So AI is no longer just sitting on a screen or behind a keyboard waiting for a human to prompt it. We now begin to see emergence of tools and tool use by AI along with long horizon planning being deployed at scale. In other words, AI has now begun to get hands. So now we are about nine minutes to mid.

Manav Gupta [16:37]

We are now beginning to enter a world where AI systems are beginning to materially accelerate AI research itself. I call this the R &D multiplier. In other words, researchers using AI are now getting a 1.5x multiple. So they’re able to now operate at 150 % faster than what they could do previously. This is where we are, seven minutes to midnight.

Manav Gupta [17:06]

And I want to end here. The clock is going to strike midnight. This is where we will see L2-competent AGI. This will exceed 50th percentile of skilled adults across all cognitive domains. Any decisions, all decisions that humanity makes around governance and safety and guardrails, they’re going to determine everything. They’re going to determine the future of humans.

Manav Gupta [17:35]

As famously Dario Amadei, the Anthropic CEO said, we as the producers of this technology have a duty to be honest about what is coming. He goes on to say, I don’t think it’s on people’s radar. So I hope it gave you a little bit of an idea of the AGI clock. Now remember, a couple of times I alluded to the 10 minute problem. It’s a theoretical problem that I’ve been thinking about a lot that I wanted to share this with you. Imagine a scenario that there are two companies that are chasing the race for AGI.

Manav Gupta [18:12]

Right? Now let’s imagine a scenario that company A gets there 10 minutes before company B. Hark back to what was going on either during the space race or the nuclear armament. Unlike nuclear weapons and other arms races in the past, there is a possibility that a recursive self-improvement will create a permanent insurmountable advantages. Here’s what I mean by that. Okay, so let’s have look at this.

Manav Gupta [18:59]

Again, two companies racing towards AGI. Company A gets there 10 minutes before company B. What happens in the next 10 minutes? Okay, so company, so AGI at 201 as an example, identifies self-improvements. So remember, company A, let’s imagine company A and company B are chasing AGI. Company A gets there 10 minutes before company B.

Manav Gupta [19:25]

say it’s 2 p.m., company A achieved AGI. This is now equivalent to a brilliant human across all cognitive domains. At 2.01, the AGI analyzes its own architecture.

Manav Gupta [19:39]

It finds, say, 47 different optimizations that humans missed. At 2.03, it performs its first recursive upgrade. As an argument, let’s say it redesigns its training loop. it suddenly becomes 10x more efficient. The capabilities now begin to accelerate.

Manav Gupta [20:02]

It goes through a couple of those iterations, and at 205, it now becomes 100x more capable. It is now operating at a level no human can comprehend or predict. At 208, it starts solving millennial problems. It perhaps proves P not equal NP problem. It perhaps designs a fusion reactor. cracks all protein structures.

Manav Gupta [20:28]

Say at 2.10, 10 minutes after company A arrived there, company B achieves AGI. Well, my view is it’s too late. It’s AGI. Company B’s AGI is immediately, irreversibly obsolete. The race is over because the other one is moving so much faster.

Manav Gupta [20:50]

Recursive self-improvement may create a permanent insurmountable advantage. This is what is going on in the world. When you see OpenAI, Anthropic, Colossus, XAI, and others moving into this field, whether it’s the models from China and AI companies from China and all around the world, the belief is this is not going to be about a catch-up beyond a certain inflection point. The first mover is not just going to win. The first mover may be the only player left once they hit AGR. So I hope that that theoretical problem gave you a bit of an insight into why the world is chasing AGI and ASI.

Manav Gupta [21:33]

Okay, now let’s go look into, well, so if you were to do this, what other research exists, right? And, you know, famously I talked about AI 2027, so perhaps we should look at the timeline that they are talking about. So AI 2027, they produced a master timeline. So ex OpenAI researcher, Daniel Coco Tajlo, and of course this has been endorsed by Yoshua Bengio, the so-anointed godfather of AI. They did a bunch of, I think, 25 plus tabletop exercises, and then that resulted into a whole bunch of these timelines that they talked about. So mid- 2025, that is last year basically, Frontier models began to gain tool use and memory.

Manav Gupta [22:31]

So this is no longer predictions in conjunction. All of you that are listening to this podcast or watching this video, you all know and have used ChartGPT, OpenAI, DeepSeq and others. all of these models have now exhibited tool use capabilities. They have agentic memories. They’re able to now form complex workflows. Enterprises are beginning to deploy these workflows at scale.

Manav Gupta [23:02]

In other words, AI has now hands attached. It is now beginning to act. It’s not supplying you just the answers. By late last year, we are beginning to see, we saw some evidence of a multiplier that was happening of a new class of AI researchers. The theory here is that the AI work is materially being accelerated by 50%. In other words, what used to take two years is now beginning to take 16 months.

Manav Gupta [23:35]

What they are predicting now is that by early this year, the R &D multiplier for AI is going to cross 1.5x. This is where compounding effects are truly going to come in. Progress that used to take years or over a year may now just take only eight months. And that pace is only going to continue to accelerate. By mid of 2026, the belief is that China may formalize state control.

Manav Gupta [24:07]

They already are beginning to do this, by the way, with the three initiatives that we covered in my previous segment around China. So China, famously, they have East data, West computing, the Big Fund 3, as well as their uh C2net model. They’re already beginning to exert state control over frontier AI development and consumption. The belief is by middle of this year, the mega data center construction in China is going to be beginning to rival that of the US infrastructure. So in other words, in addition to everything else that we are hearing geopolitically, we’ll begin to see this geopolitical race is going to be now explicitly in full flow. By late this year, There is a belief that we’ll see some major labor market dislocations appearing in knowledge work.

Manav Gupta [25:07]

In other words, as AI begins to get better and better and deployed in the form of agents in the enterprise, we think, or these researchers think, from AI 2027, that we’ll begin to see global AI capital expenditure hit a trillion dollars annually. In other words, the stakes are going to be undeniable. By January of 2027, the belief is that we will have a level two agent, which is going to be equal to somewhere between a good to an excellent AI human researcher. The difference, of course, is going to be that this is going to be training continuously. It never stops improving. The belief here is that it’s going to require some $200 billion of compute, about six gigawatts of power.

Manav Gupta [25:59]

I’m going to let you read the rest in terms of what other rival powers such as China might do. If history is any indicator and if we were to believe that the Chinese companies might continue to do what they did, for example, with model distillation for DeepSeq, there may be a scenario where there might be a nation-state sponsored data exfiltration to exfiltrate the weights. So while that is happening, by mid of next year, we’ll begin to see agents emerging from level three to five that are going to be superhuman at everything. At level three, the agent will surpass the best human AI researcher in every subdomain. The R &D multiplier at this level is no longer linear. It’s going to be self-referential.

Manav Gupta [26:57]

If all of this continues, we then… Expect. ASI, artificial super intelligence to emerge late 2027. Mind you, this is not the doomsday scenario.

Manav Gupta [27:14]

I’m not predicting. I don’t think that we’re going to get there yet, but there is a very credible, realistic scenario that AI 2027 website posted. And I’m summarizing that where if these previous set of events were to all happen, I can see a scenario. where the clock strikes midnight and we achieve ASI by late 2027. Okay, so you can see why this was the most difficult episode to put together. So we talked about the ASI timeline, but I think in order to understand these type of things, as always, we need to go one level deeper.

Manav Gupta [27:49]

So I briefly mentioned the agent ladder from the AI 2027 website. So let’s have a look at that. So they have defined really three levels of agents. or rather four level of agents. So agent zero, which is, let’s imagine that’s where all the agents are today. So they call them L1 or emerging agents.

Manav Gupta [28:18]

So all of the current frontier models that all of you guys use, everything from GPT-4.0 to Opus, Cloud 3.7, et cetera, one can argue that they are roughly equivalent to an unskilled human across general tasks. The assistant is useful. They’re not quite autonomous. They need a lot of guidance.

Manav Gupta [28:39]

We need to provide them with things such as context engineering, we need to give them tools, they need to build them skills. But they’re getting better and better at drafting code and documents. They are beginning to answer questions accurately. They do require human oversight on all decisions. As my peer and colleague, Mihai Kriveri said, they’re really good token prediction engines. They do require human oversight.

Manav Gupta [29:06]

That’s where we are. One can argue that in terms of R &D multiplier, anybody that had to do some research, they’re making us at least twice as productive. By early to mid of this year, we will begin to see agent level one approaching level two. At this point, AI is going to be able to do the work of a mediocre to a good. So on the scale of excellence, from unskilled mediocre to a good, human AI researcher, we will be somewhere between mediocre to good. In other words, the R &D multiplier is going to be 1.5x.

Manav Gupta [29:44]

We’re already beginning to see some evidence of that, by the way. The deep search, deep research that Gemini can do, OpenAI can do, Cloud can do, they can run multi-hour research tasks autonomously. We are beginning to see AI write the code, of course. and debug full research pipelines. They’re not quite there in terms of completely debugging full research pipelines, but getting there. They will begin to identify their own training improvements.

Manav Gupta [30:16]

We’re beginning to see emerging evidence of that. So that’s a level one, agent one category. By early 2027, by next year that is, we’ll see agent two, which is a level two competency, which is a good to an excellent human AI researcher. At this point, AI is now giving us a 3x multiplier. Six months of research is now taking two months. At this scenario, AI agents are going to be able to execute full research agendas independently.

Manav Gupta [30:48]

They will be able to improve themselves, self-improve their training procedures. They will potentially design novel architectures. If all of that follows, by late, 2027 to 2028, we’ll see a 3 plus level agent, which of course means that it will surpass the best human AI researcher. At this point, you’re looking at a 10x multiplier with AI. Recursive self-improvement becomes self-referential, so we were to reference itself and improve its own training and learning. This is the bifurcation point that AI 2027 talks about, that either this is where the bifurcation happens that humans get freaked out and they begin to slow down and control it?

Manav Gupta [31:36]

Or the race ends because a human oversight becomes practically impossible. Okay, so we talked about a wide variety of scenarios here. Well, uh I’m going to now focus a little bit on the economic side of that. um Actually, before we go any further, now let’s talk about the risk and safety algorithms. Okay, so now let’s talk about risk and safety. Okay.

Manav Gupta [32:55]

Now let’s talk about risk and safety. So this chart on your screen here, this is the most important story in AI right now. You see two lines there, the one in orange and the one in green. So capability is growing exponentially. So what you see, the green line, um so what you see is the capability of actual AI, that’s in orange. Okay.

Manav Gupta [33:31]

Three, two, one, go. So this chart tells the most important story in AI right now. What this chart is telling you is two lines here, one in orange and one in green. The one in orange, that’s the line of capability, and the alignment research, that’s in green. So what this chart is telling you is that capability of AI is growing exponentially, but the research on model alignment, that is growing only linearly, which means that the gap between the two is beginning to broaden. And here is the key highlight.

Manav Gupta [34:13]

key insight from AI 2027. When a company can write up a spec listing dos and don’ts for how the AI should behave. Okay, that’s what most companies do. Then they can try to train the AI to internalize the spec. But they don’t check to see whether or not it works. That’s the core problem in this.

Manav Gupta [34:39]

We can write the rules, but we cannot verify whether the rules were actually learned by the AI models. What we’re essentially doing is education on the world’s most powerful students without any reliable way to test comprehension. So Jeffrey Hinton, who just won the touring award for inventing the foundational techniques behind all of this. He left Google and said the reason he left was to freely speak about the risks. He puts extinction probability of humans somewhere between 10 to 50%. So there are three current approaches for model alignment.

Manav Gupta [35:27]

Reinforcement learning with human feedback, RLHF. Humans rate the outputs. Models learn to produce what humans like. It works well at scale, but humans can be fooled. There are lots of evidence of that. We can call it the next approach being constitutional AI.

Manav Gupta [35:46]

This is the Anthropics approach. Train the model on explicit principles. This is more transparent, but it has the same problem of verification. Then the third approach is that of interpretability, actually understanding what is happening inside the model. And by the way, this is the frontier, and we are years behind where we need to be. So I hope that chart gave you an idea into the biggest problem that we have right now of alignment, right?

Manav Gupta [36:16]

The real alignment gap. Now, in the opening, I also talked about the probability of doom, right? So what do the experts believe can happen and by when? So let’s now focus a little bit on that. So what this chart is sharing with you is uh expert risk estimates. So P-Doom, this is a technical term for probability of catastrophic harm from misaligned AI.

Manav Gupta [36:45]

To be clear, this is not science fiction. This is a serious intellectual debate about what is happening at the top of the field. And if you look at this chart, the range is staggering. You have Jan Luken at 1 % to Eliza Yadkowsky at about 99%. The spread doesn’t mean by the way that we should average them and call it 50%. It means that even genuinely smart people, people way smarter than me and those that will spend their careers on this problem, they have fundamental disagreements about whether this problem is solvable.

Manav Gupta [37:34]

Broadly speaking, I’m going to say that there are four camps. Camp one, I’m going to call them the optimists. Lacoon’s view is that the current AI systems are fundamentally not dangerous. They may be stupid, they may be misguided, you might have an alignment issue. His view is that we don’t even know how to make them dangerous if we try. He thinks that the main issue here is misunderstanding of how these models work.

Manav Gupta [38:00]

And I think there is some credence, is some credibility to that. Camp two, I’m going to say is the concern camp. That’s the camp of Jeffrey Hinton, Yoshua Bengio, et cetera. Both of them touring award winners, by the way. They think that the risks are real, but manageable. If we take the risks…

Manav Gupta [38:23]

seriously. Hinton said that’s the reason that he chose to leave Google. There is then the alarmed category. Paul Cristiano, who ran OpenAI’s alignment team, sits firmly there. He thinks that AI going badly is more likely than not. He’s one of the most technical advocates and voices in the alignment research.

Manav Gupta [38:45]

Then I’m going to call it the fourth category. I’m going to call that the dire category. This is the category of Yadkovsky. His view is that the math of the situation leads essentially inevitably to extinction. He’s been warning about this, by the way, for 20 years and he has been consistently dismissed. I guess the question is whether this dismissal was warranted.

Manav Gupta [39:14]

So, I know the median of this survey is around 5%, but I want you to think about it differently. If I told you that there’s a 5 % chance a plane was about to crash, would you board it? That, my friends, really is the essence of the problem that we’re talking about here. Okay, so we looked at the doomsday scenarios, we looked at the out of the possible. Well, what’s stopping? What are the potential blockers, hindrances for AI to get there?

Manav Gupta [39:48]

So welcome to the five walls. What stands between AI as it exists today versus AGI? uh So I present to you five distinct walls that AI must cross for it to go from where it is today to AGI. The bad news and the good news, three of them are engineering problems. The bad news is that two of them are fundamental research problems with no clear solution. So number one, the data wall.

Manav Gupta [40:26]

This has a 92 % severity. Essentially, this problem states or this wall states that we have essentially used all of high quality human generated text on the internet. Common Crawl, Wikipedia, all the books, 80 % consumed by the web text. According to AI 2027, Agent 2 relies on synthetic data pipelines. The problem here, of course, is that synthetic data quality degrades as you loop through the same models. mean, hence the terms of AI slop and how internet and Google searches and Google images are being filled with the AI slop.

Manav Gupta [41:03]

The second big problem is the alignment wall. We talked about this a little bit. In my opinion, this is the most dangerous wall because this is an invisible wall. This is a conceptual wall. You cannot see when you have hit this wall. To recap, you can have a situation where a company can write up a spec listing dos and don’ts for a model, but they cannot check whether or not it worked.

Manav Gupta [41:29]

or they cannot reliably and scalably check at least. This is an unsolved technical problem in AI safety. Wall number three is the compute wall. The numbers in this scenario are staggering. For agent two alone, for AI 2027, they are projecting roughly $200 billion to train such a model. Agent 3 may require 10x more.

Manav Gupta [42:08]

We are talking about infrastructure that only nation states are going to be capable of investing into a single training run. Let that sink in. The fourth wall tied of course to compute is going to be the energy wall. So open brain in the AI 2027 benchmark scenario consumes 6 gigawatts at peak. To be clear, that is equivalent to six large nuclear power plants. Therefore, now you can begin to understand that the Microsoft deal for the Three Mile Island wasn’t a joke.

Manav Gupta [42:45]

It was a preview of what’s to come. greater the data, better the model, you need more energy, more compute, and you need more energy to train those models. The last wall is the algorithmic wall. So while the world started with our transformer architecture, it may have its own limits. One can best state that the current way the models do their reasoning is really sophisticated pattern matching. Generalizable reasoning.

Manav Gupta [43:25]

reasoning that can be generalized, the kind that lets you and I solve a new type of problem you’ve never seen before, it may require a fundamentally different approach. Okay, so we talked about the five walls. What could humans do? what could humans do? What is it that…

Manav Gupta [43:48]

any individual can do. Okay, so now let’s focus on the human dimension. Right? What can humans do? So we spent the first half on the technology and the risks. What does it really mean for you?

Manav Gupta [44:13]

I’m sure you’re wondering. The honest answer is nuanced. Significant disruption is coming. But the shape of it depends upon the decisions that are being made right now by policymakers, by companies, and by individuals such as you and I. Let me show you a chart. Have a look at this picture from McKinsey for Exposure of jobs to AI.

Manav Gupta [44:41]

So 57 % of all work hours in the US, they are technically automatable. But it’s important to understand what this number actually means. Technically, automatable means the tasks involved in a role can theoretically be done by a machine, by current AI. What it does not mean is that these jobs will disappear. It does not mean that companies will choose to automate them because there has to be an ROI. The economics therefore have to make sense for those to be automatized and that people will accept, most importantly.

Manav Gupta [45:17]

that AI is doing a task instead of a human. The more honest number is somewhere around 32%, still a staggering number by the way. 32%, that’s the share that organizations expect workforce decreases. And by the way, that’s 32 % of organizations are expecting a workforce decrease of greater than 3%. So it’s not nothing, but it’s a whole lot less compared to the headline, 57 % of the jobs that are automatable. If you break this down, I think the real bad news is finance and administrative jobs are the most exposed.

Manav Gupta [46:03]

So any job where there is a corpus of data, and human expertise required for oversight for information processing, patent recognition, document handling. Those are the jobs. Those are the areas that LLMs exceed at. Even though in this McKinsey report they say healthcare is only and has used that in air quotes 38 % exposed. That’s really because healthcare is physical, it’s relational and it’s judgment based. The PwC finding is most interesting in my opinion for career planning.

Manav Gupta [46:45]

AI exposed roles, they evolve 66 % faster than non-exposed roles. But they also command a 56 % premium. So what does that mean? Well, what that means is that if your job, perversely, if your current job is being changed by AI, you have both career risk and you have more potential upside. Think about that. The question for you then as individuals is which direction do you decide to run?

Manav Gupta [47:25]

Okay, so that’s on the job exposure. In the sum totality of this all, what about possibilities of future for humans? So based upon all of the data, everything that I have read, I’ve mapped this to the scenarios that AI 2027 talks about, and I tried to add in my own thinking, my own background into where I think what might happen. I can only see three possible scenarios. Two on the extremes, one in the middle, of course. The best, most favorable scenario is that of flourishing.

Manav Gupta [48:08]

So this is Dario Amadeus’ vision, by the way. ah This is from Machines of Loving Grace. If we get the alignment right, if AGI is deployed carefully, with proper governance, We have the right distribution of concern. The upside is extraordinary. 50 years of biological progress we’ll be able to make within 5 to 10 years. We’ll be able to do things like solve the climate crisis, solve all diseases, eradicate poverty.

Manav Gupta [48:42]

We’ll begin to see introduction of universal basic income so that humans will be focused on purpose rather than pay. This is the flourishing scenario. Let’s cover the far right scenario, which is the catastrophic scenario. You have a scenario where you have misaligned or weaponized AGI. It causes mass destruction. As an example, bio weapons designed by AGI.

Manav Gupta [49:15]

AGI gets so good that it does not need humans for majority of the tasks. The recursive self-improvement machine escapes human control. escapes the lab. In other words, the race ends with an agent 3 which has unverified alignment. There is no coordination on this. There is no desire for this AGI in this catastrophic scenario to require humans.

Manav Gupta [49:40]

It will have no safety and no alignment. The most probable scenario, I think, is that of turbulent transition. I think this is the one, in my humble opinion, that most enterprise leaders are underestimating. What happens in this scenario, the one in the middle, the turbulent transition, is that AI actually provides massive productivity gains. However, those gains are distributed unevenly. That’s the real possibility.

Manav Gupta [50:21]

In other words, a disruption of 10 to 30 percent in the workforce before adaptation is going to be real and painful. The institutions that we have for managing transitions, unemployment, insurance, education systems, social safety nets, They were designed for 20th century economic disruptions. Not ones that happened in years. We designed those systems over a broad decades long period of time. That’s the scenario that I see most obvious happening. Now notice, I did not put Terminator or killer robots or drones, killer drones on the list.

Manav Gupta [51:08]

The realistic catastrophic scenarios are far less cinematic. but more insidious. In the far right catastrophic scenario, a small group that controls AGI, the billionaires of the world for argument’s sake, or those that are in power in a couple of nation states, they acquire capabilities, and the capabilities that they have and somehow they control or direct if not control, they acquire permanent Economic and military advantage. As an example, bioweapons that are designed at superhuman speed by an AI that does not understand why it should not. Economic systems begin to update faster than any democratic institution can regulate. That’s the real fear in my opinion.

Manav Gupta [52:05]

Okay, so we talked about the possibilities of features for humans. Let’s now pivot to, well, what are the AI-proof carriers? What to avoid and where to grow? So let me give you a practical framework. So here is my two by two matrix. On the top left is the high risk jobs.

Manav Gupta [52:32]

These are jobs, by the way, that are going to be completely eliminated or largely automated through the use of agented AI. Any tasks, as you notice here, where data processing, information processing is at its core, pattern recognition, or rule application, data entry, basic customer service, scripted customer service, routine legal research, standard accounting procedures. LLMs can do this today. As they get better and better, they’ll be able to do it with a greater level of accuracy, acceptable level of accuracy at superhuman speed. That’s the highest risk. Top right, I’m going to call that the moderate risk.

Manav Gupta [53:18]

Software engineering is interesting. And I know Anthropic has said multiple times that software engineering is more or less solved. And then every time I scratch the surface, realize, well, code generation is solved. Problem decomposition, architecting so that the software can be managed for day two or operability, that humans can actually manage that, that it can be integrated into an existing slash brownfield, I think that’s still not solved. I think that will still take years to be solved. But yes, those days where jobs such as code machines or people, engineers being paid just to write code and being paid exorbitant to write code, those days are gone.

Manav Gupta [53:53]

Therefore, writing boilerplate code, translating requirements to code, debugging standard errors, They’re all highly exposed. Senior roles that require system architecture, cross-functional judgment, organizational context, less so. In other words, co-pilots, they augment. They don’t quite replace. Bottom left, that’s the lower risk, so product management at the senior level. Again, this is a situation where that requires some human judgment.

Manav Gupta [54:37]

So it’s not just a rote application of rules. It requires um some customer insight, market context. It requires technical constraint. AI can help with each of these components, but cannot outright replace them. It cannot outright and cannot currently replace them with political and interpersonal intelligence. So I think these are the jobs that are going to remain at a lower risk.

Manav Gupta [55:04]

Finally, we come to the bottom right. Resilient. Notice the pattern here. These are all jobs that require physical presence. They all require relational trust, some type of moral authority. and something that I’m going to call an irreducible human judgment.

Manav Gupta [55:25]

Crisis therapists, they need to be human. C-suite executives, until proven otherwise, who require decisions to be made with legal and reputational accountability, does require a human face. Surgeons that are operating in unpredictable physical environments, skilled trade persons work, in conditions that robots still struggle with, those jobs will remain resilient. So if I put this hat on and somebody was to ask me my advice for career in one sentence. So if your core value is executing against known procedures, you are at risk. If your core value is your judgment, trust, or physical presence, you have structural protection.

Manav Gupta [56:24]

and one more thing. There is a reason why AI trainer and AI ethicist are in the resilient category. I’ve done that deliberately. And the reason for that is that we’re still training AI. We’re still trying to define what AI ethics should be, how to translate those into controls and guardrails that can be implemented. Okay, so what can you do?

Manav Gupta [57:00]

Right? So we’ve talked about some totality of all of these things. As an individual, what can you and I do? So I’m going to leave you with this matrix. Here is my action matrix. Four concrete actions for each of the four different groups.

Manav Gupta [57:17]

Let’s start from the top left, individuals. No doubt there, that start now. Right? The PwC AI job barometer is clear. AI exposed jobs, they evolve 65 % faster, but in turn, as I said, they earn 56 % more. You absolutely want to be in that group.

Manav Gupta [57:42]

The practical starting point, pick one tool, start learning it, use it every day for 90 days, start, of course, casually, but seriously, use it for actual work. Learn how to prompt it, learn how to extract every ounce of value from it. If you have to invest some money of your own to learn to pay for the courses, go ahead and do it. The people that are trying to figure it now will have a two year head start compared to anybody else. For enterprises or businesses, there is a 6 to 12 month window. McKinsey is finding around 50 plus agent-to-gay deployments.

Manav Gupta [58:18]

Right now, the world has impressive demos, but very little deployment at scale. So the organizations that are getting real returns aren’t the ones that are using AI as a point solution. They are redesigning workflows around AI capabilities. So if you’re an enterprise, you start with an audit. Which one of your high-cost, high-volume processes are information processing tasks. Those are the candidates.

Manav Gupta [58:51]

And by the way, governance before scaling and governance for scaling. Too often I hear this argument that we’ll worry about governance later and we’ll retrofit it. That does not work. In fact, what ends up happening is a wild, wild west and you might have multiple models, unclear roles and responsibilities. You may have no way of scaling. In fact, this is a scenario where governance upfront actually helps you scale.

Manav Gupta [59:19]

For policymakers, this is urgent. This is a scenario where every nation state should be funding national AI safety research programs. This is not about slowing down AI development. We need to make sure that there is concrete actionable policy around compute threshold that is part of the policy so that we can add verification requirement in AI safety programs beyond a certain size. especially as the AI begins to scale. Think of it as building codes for the most powerful technology that humans have built.

Manav Gupta [01:00:02]

And for citizens in every democracy, including everybody that’s listening, this is an ongoing democracy, democratic problem, as much as a technology problem. AI policy globally is being made by, developed by a very small number of people. Public engagement for this technology matters more so than anything else. Elections matter. Make your voice known and heard in this space that we want legislation, want guardrails, we want safety in this area now than ever before. This problem is real, this is happening.

Manav Gupta [01:00:42]

We need to think about this now. Okay, so I’m going to leave you with all of that with the following five key takeaways. Number one, we are at level one AGI, but level two is imminent. On DeepMind scale, we are at emerging AGI, the next threshold where the performance exceeds 50th percentile of skilled adults across all domains. It’s not confirmed on anybody’s timeline, but every major lab is chasing that. Number two, the R &D multiplier is a key variable to watch.

Manav Gupta [01:01:18]

When the R &D acceleration crosses 1.5 or at agent one category according to AI 2027, the pace of capability development will become self-reinforcing. This is the transition from linear progress to compounding progress. Number three, alignment is structurally behind capability and that’s the risk. When I say alignment, to be clear, we are talking about model alignment. Ensuring that the model is behaving and performing in a way that humans wanted to perform and behave.

Manav Gupta [01:01:55]

This is not just a temporary setback, this is structural. The world can build these AI models, but we cannot verify if the safety specifications were internalized. This is the most important investment right now is in interpretability research. Number four, the outcome depends upon the coordination, not technology. So internationally, we need governance for compute. We need treaties for verification of the models.

Manav Gupta [01:02:35]

We need safety legislation across all of the decisive interventions for technology, which means we require political will, not just technical progress. And number five, Your competitive advantage is acting now while most organizations are still experimenting. Needless to say, as evidenced by every researcher and every report that’s out there, most organizations are experimenting with AI, but fewer than 10 % are at scale. The organizations that are building AI fluency, governance framework, and rescaling programs will have a two to three year structural advantage. This is Manav Gupta, you’re listening to Ship AI and I hope you found today’s episode useful. Please like, subscribe as you see this video and I look forward to seeing you on the next one.

Manav Gupta [01:03:25]

Thank you.