← 返回 AI 2040 首页

Plan A 假设

补充材料 · Supplement

Thomas Larsen

Plan A is a scenario: it combines forecasts with policy recommendations. Some of the forecasts and recommendations we are confident in; others are just guesses we made because we had to, in order to achieve the desired level of concreteness.

ConfidentUnconfident
PredictionE.g.: “AGI is coming soon and will be a bigger deal than the industrial revolution”E.g.: “AGI will arrive in 2030”
RecommendationE.g. “There should be an international deal to limit the speed of AI takeoff”E.g. “Many new datacenters should be constructed in Mongolia and Canada.”

This supplement goes through the major predictions and recommendations of the scenario and explains which are which.

Nearly all of the actions in the scenario assume some optimism and some realism, but by and large, we tried to be realistic about empirical facts and optimistic about US government actions. The AI 2040 scenario has roughly my median view about things like timelines, takeoff speeds, alignment difficulty, and actions of non-US governments (e.g. China, the EU, etc). The actions the US government takes are optimistic by definition as they are following our recommendations nearly exactly; in reality we still expect something more like the response in 2027.

The supplement tries to capture all our main recommendations, so if something isn’t mentioned here, it is probably a prediction, and insofar as it’s a recommendation, it is not an important one.

Since this is a document about specific beliefs, it’s written from the perspective of one of the scenario authors (Thomas). Other authors by and large agree with the central claims but have somewhat different opinions on the specifics.

Forecasts

For each of the following, I am >80% confident:

I’m not confident about:

For each of these key strategic variables, the Plan A scenario chooses a reasonably central value with respect to my views. For many of these my views are very uncertain, and many policy decisions should be made with more information about these variables than we currently have.

Policies

In the next section, I discuss which of our policy recommendations in the scenario are robust across a wide range of empirical updates.

These are robustly good policy recommendations:

These recommendations depend on particular assumptions that we’re not confident in, or have alternatives that might be better:

The above were largely details about Plan A that might change. There are also structurally different plans that may be competitive with Plan A, which we discuss here .

Still, I think we should be aiming for something like Plan A across a wide range of scenarios. The rest of this document has two sections, which both discuss how our plan would change in response to empirical changes.

Empirical Updates

In this section, I’ll go over how Plan A would change in response to some important empirical assumptions that we make in the scenario, which might turn out to be different in practice.

Timelines

We don’t know when AGI will be built; it’s possible that it could be built this year or next year, it’s also possible that it will take decades.

In the Plan A scenario, the full automation of AI R&D was on track to happen in 2030 by default, but then was pushed back by government involvement. We’ve written much more on AI timelines; you can find our most updated work here . AIFP employees continue to have (slightly) different views about AI timelines, but we all think that 2030 is a plausible and reasonably central year for AGI to arrive.

Why did we choose to write this scenario assuming 2030 timelines?

It’s good to showcase a variety of different plausible scenarios. We already wrote a very short timeline scenario (AI 2027), so we figured it would be good to choose a different timeline for our next one. 2030 is my (Thomas’) median currently, i.e. I think there is a 50% chance things will proceed slower, and 50% chance things will proceed faster. In the future, if time remains, we’d like to do scenarios for other authors like Eli.

Overall, Plan A remains my central plan ~regardless of timeline , because the underlying problems (AI takeover, AI enabled concentration of power, and rapid AGI induced societal upheaval) are the same regardless of timelines, and Plan A is our best guess for how to deal with these problems.

Regardless of the timeline, Plan A is most effective the earlier an international deal that involves mutual compute declarations and transparency begins. When we get to the beginning of the intelligence explosion (e.g. the automated coder milestone), verification becomes substantially more difficult, because defecting projects can involve many fewer humans, and therefore be much harder to detect.

If the timeline is much longer (e.g. 2035+), we begin to have much more uncertainty about the situation. The world will also be more transformed pre-AGI relative to our scenario, there will be larger economic effects of AI, AI company revenues (and share of the overall market) will (probably) be higher, there will be more robots around, and there will be more wakeup to the importance of AI. Particular effects include:

If timelines are much shorter (e.g. 2026 or 2027), there isn’t time to do the preparation phase, and so we have to do a version of Plan A with no preparation.

Takeoff speeds

In this scenario, similar to AI 2027, we chose to assume that the default length of time between AC (Automated coder) and TED-AI (Top-Expert-Dominating AI) was 1 year. This is roughly my median, but we again have large uncertainty; my 80% CI at time of writing for the takeoff duration is something like [2 months; 5 years].

Why did we choose 1 year takeoff speeds by default? This is my median at the time of writing Plan A, and seemed like a representative scenario to plan for.

How does the plan change under different takeoff speeds?

If takeoff is faster than 1 year , things become generally more difficult, and not just for Plan A. The main specific changes are:

If takeoff is slower than 1 year, e.g. even without regulation it takes several years to go from the full automation of coding to the first artificial superintelligence, then we basically update in the opposite direction on all of the above tradeoffs.

Overall, we recommend pursuing Plan A regardless of takeoff speeds. However, changes to our takeoff speeds do substantially change the specific implementation of Plan A, in the ways noted above.

Alignment difficulty

In the Plan A scenario, humanity rallies and creates a high assurance safety case for handing off to ASI. However, we are much more confident in the general picture that more time and effort lead to a greater chance of solving the alignment problem than the specific story told here. We don’t know how hard it’ll be to align ASI. Opinions vary substantially within our team (and even more outside of our team). Given the uncertainty, our view is that we should take a portfolio approach, and invest substantially in both prosaic AI safety agendas (e.g. variants of RLHF) and moonshots (e.g. uploading human brains).

The plan we are advocating for is one in which the world coordinates to give the scientific community breathing room and resources to figure out how difficult the alignment problem is. If Yudkowsky is right that AI alignment is difficult, our view is that evidence would increase during this time period and then the pause would be extended. Likewise, if the optimists are right and alignment is easy, we would get evidence of this during that period, and we could proceed at a faster pace (though it’s not clear that we should, given the other risks of doing so, most notably concentration of power).

I think that alignment is probably difficult enough that without a well managed multi-year slowdown, we will incur substantial AI takeover risk. Even with a ten year slowdown, as depicted in the scenario, there is a significant chance that alignment was too hard, or that it’s feasible but human error causes us to fail anyway. You can see more about the team’s views about alignment success likelihood conditional on following different plans here .

What do we do if alignment is much harder?

Gather as much evidence about alignment difficulty as possible by:

During Plan A, maintain the optionality of extending the pause by:

If alignment is extremely hard, e.g., humanity isn’t making substantial progress on alignment whatsoever, then the correct call may be to pivot to Plan S, and instead of attempting a managed takeoff, shut it all down. Then, we could pursue alternate strategies to build superhuman AI that are inherently safer than ML, such as uploads. Unfortunately, these approaches have substantial downsides as well, notably the risk that the moratorium breaks down and we go back to racing before enough progress is made on uploads or other strategies in that vein.

What if alignment is much easier?

Slowing down AI takeoff is still probably good for many other reasons, including avoiding AI-driven concentration of power, figuring out appropriate post-ASI governance, transitioning to a regime where human labor is no longer economically relevant in a healthy manner, hardening the world to AI enabled superweapons, etc—basically, we still need to solve all of the problems that we dealt with in Plan A other than misaligned AI takeover.

The main difference is that there is a higher chance of humanity being able to successfully muddle through these other risks without permanent lock in. This means that the necessity of a slowdown does go down substantially if alignment is much easier than I expect.

Verification difficulty

Related reading: Covert Project Branch , Covert AI Projects Supplement , Verification Plan .

I’m not sure how difficult verification will be. Our main strategy for verifying compliance with an AI agreement is (1) limit the amount of compute that covert projects have access to, and (2) ensure that legal projects are doing what they claim. Both of these will be somewhat lossy, and we’re unsure about what the mapping is between “amount of unmonitored compute” and “how fast could an illicit intelligence explosion reach an existentially dangerous capability threshold”.

Now we’ll go through each of these in a bit more detail:

There are various other particular verification dynamics which could affect the rest of the policy strategy:

AI Paradigm Shifts

Plan A currently assumes gradient descent on giant neural networks remains the dominant AI paradigm until the late 2030s. We assume significant advances are made within that paradigm, including (i) the addition of neuralese (both as long term memory, and replacing chain-of-thought) and (ii) dramatic scaling in reinforcement learning (many more diverse environments that are much larger than existing environments).

Substantial algorithmic changes, even within the broad “gradient descent on giant neural networks” paradigm, might change the implementation details of parts of Plan A. For example:

There are also obviously many “unknown unknowns” in AI paradigm shifts which could change Plan A in unforeseen ways, in practice we expect that many updates to the Plan A baseline will need to be made in response to further paradigm shifts.

Less central updates

Policy Updates

How fast to scale AI progress after a slowdown deal?

It’s unclear, and will be dependent on observations at the time. Here are some key considerations:

Ultimately, trading off these considerations will require quantitatively modeling each of these sources of risk: covert project risk, risk of AI takeover from current AIs and future AIs, etc, understanding how the speed of deployment impacts each of these risks, and then making complicated tradeoffs about the overall best path forward.

Plan A’s trajectory is essentially based on the viewpoint that deal dissolution risk will probably be high, that risks from early AGIs (e.g. ACs and SARs) can be mitigated via control (i.e. that control-based safety cases can be made good enough very quickly, for AIs at that level), and that we can elicit a huge amount of useful labor from these early AGIs. It also assumes that we can do most of our capabilities scaling via a rapid hardware buildout, and strongly limit the total amount of algorithmic progress that occurs, including successfully mitigating distillation (largely by hiding the intermediate reasoning traces). Under these assumptions, the best trajectory seems to involve scaling quite rapidly at the beginning of the slowdown agreement, and then slowing down dramatically at the maximum controllable capability level.

Plan A’s specific trajectory is our best guess for the best way to handle these tradeoffs. However, we aren’t confident whatsoever that this is the correct path, and we hope that decisionmakers at the time will have a better understanding of these empirical factors at the time, whereas we have to make rough guesses.

Another very plausible trajectory, Plan S, is essentially the opposite choice about how fast to scale: immediately issue a moratorium on further AI capability scaling. This policy looks better if there is a small risk of deal dissolution, and/or the returns to better AIs for safety research are low, and/or the risks of any capability scaling are high.

Massive datacenter buildout

It is desirable to do a massive datacenter buildout in Plan A if (and only if!) the upside of scaling quickly, reducing covert project takeover risk, and having additional capacity to pay alignment taxes is bigger than the risk of verification failing or the additional risk incurred by the deal failing with a gigantic compute stockpile.

In a bit more detail, the main upsides of a massive datacenter buildout are that it allows AI developers to:

The main downsides are:

Putting all of this together, my view on how to decide to what extent we should pursue a massive hardware buildout under Plan A is as follows:

Our specific verification plan

The verification plan used in the Plan A scenario can be summarized as follows:

We discuss this in more depth here .

This version of the verification plan assumes that at the beginning of the deal (1) we can verify inference, and (2) AI development verification is feasible within a relatively short research sprint (but is not immediately feasible). It is most likely that one or both of these assumptions will be wrong; these assumptions are my central guess of what could be done under Plan A assumptions of political will and competence, but reality will most likely involve less good preparations.

What if we don’t have verification measures ready in time?

Three high level buckets of verification approaches are:

It is highly desirable that we have technical verification methods ready before the beginning of the intelligence explosion. If inference verification devices are ready, then access to AIs gets to continue at the beginning of the deal. If development verification is ready, then we don’t need to stop AI capabilities scaling or AI safety research that involves frontier models at the beginning of the deal, and can instead scale until it is sufficiently unsafe not to (following the considerations above).

2 and 3 will likely only be temporary regimes, and be paired with a sprint to get technical measures online. Therefore, we hope that eventually we’ll get to sufficiently good technical methods.

The main downside of 2 is that it runs a risk of defection. Therefore, it may also run into increased political difficulties, as both parties will be worried that the other will defect, and so will be less likely to sign in the first place. However, it’s not correct to entirely write off trust, because it may be that the risk of AI takeover becomes so high that neither side is incentivized to defect, because doing so would so likely result in existential catastrophe. If high AI takeover risk becomes common knowledge (for example, after we’ve caught early misaligned AIs in the act of sabotage), relying on trust may be the best option.

The main downside of 3 is that it involves a temporary period where large scale inference goes offline. However, this would be less bad than one might naively think; even an agreement to power off (or destroy) 90% of the compute, would only force a reversion to 10x cheaper models. Since cost of a given level of capability has historically decreased very rapidly ( averaging ~40x/yr ), having 10x less compute would roughly correspond with having to use SOTA models from 7.5 months ago. A downside of 3 is obvious; if defection is viable, it becomes more likely that both sides would choose to defect in the first place.

Note that this is one upside of Plan S over Plan A; Plan A requires (1), while Plan S only requires (3).

Transparency and Security tradeoffs

There is an inherent tradeoff between security and transparency—to secure something, you inherently need to prevent some group from being able to see it.

Plan A biases towards transparency, in large part because I am more worried about the risk of poor governance within the legal projects than the risk of covert projects defecting and successfully undermining the deal. Increasing transparency will on average improve the quality of the governance decisions made by the Consortium, whereas increasing security makes it harder for covert projects to succeed. Furthermore, even if the governance decisions are perfect, the legal projects may not be able to solve alignment in time themselves, and may need help from the wider world to e.g. notice some subtle flawed assumption in a safety case. Also, transparency is very helpful for verification.

That being said, we are currently nowhere near the pareto frontier between these two dimensions. Plan A proposes dramatically increasing both security and transparency into frontier AI projects. We discuss transparency-related tradeoffs in much more detail in our Transparency Supplement .

Our Post-ASI governance plan

Related reading: Space governance supplement , AI 2040 epilogue .

We make a concrete guess as to what a positive post-ASI governance plan might look like. However, we expect that there exist better proposals, which we haven’t thought of yet, that will be discoverable at the time. Therefore, we want readers to orient to our plan as a baseline with which to compare other plans to on dimensions that they care about. Then, we hope that humanity will spend vastly more effort—both human and AI cognitive labor—on designing a system that performs better.

Broad Deployment of AI during the takeoff period

The Plan A scenario involves broadly deploying AI into the economy during takeoff. This has massive effects on the world during takeoff: almost everyone loses their jobs and starts relying on Citizen’s Dividend. Overall, I think that it is better for massive AI impacts to happen gradually, e.g., over the course of a 10 year takeoff, than to have all of the impacts happen really quickly, right at the end, once AIs are vastly smarter than humans.

(See more about how AI impacts the broader economy in our economics model , and discussion in our economics supplement )

However, this relies on a few key questions:

If we’re not broadly deploying AI through society, the main alternative is to be developing AI inside closed projects. This incurs additional risks. Whatever closed off group of people that are managing the intelligence explosion without contact with the rest of the world will have a huge amount of power over the future. For an extreme case, the closed project might literally be developing AI in secret, without notifying the rest of the world, as described in Training AGI in Secret would be Unsafe and Unethical . A secret intelligence explosion would be bad for misalignment risk, because far fewer people would know and understand what’s going on, and be able to give input on the alignment techniques, and would be bad for the distribution of power, because powerseeking individuals might be able to seize control of the AGI project, and build ASI aligned with their values (as opposed to humanity more broadly)

A better proposal for avoiding broad deployment is to do local deployment, but maximize transparency: publish evaluation results, publish demos of AIs doing very impressive things, publish the AI-generated research (on non-AI domains), publish the model spec, etc. This is far better than a secret intelligence explosion. However, it seems likely that if this is happening, humanity will be still largely in the dark about what’s going on, because to normal people, without AI impacting their day to day life, it’ll seem like some far-away science project that people like to talk about—it’ll be in far mode, not near mode. Therefore, it seems likely that broad deployment of AI will improve epistemics around AI in particular, which seems positive.

Appendix: How plausible and how desirable are our key policies?

Plan A is neither a pure recommendation nor a pure prediction. A pure recommendation in which every actor makes maximally responsible moves starting tomorrow would have been too implausible. And we already wrote a pure prediction scenario—that’s AI 2027. So we chose to make Plan A a compromise between plausibility and desirability.

The interactive graph below shows roughly where each of the scenario’s policies sits on these two dimensions. Plausibility is judged at the point in the scenario when the policy is adopted, conditional on everything that has happened up to then. It does not represent how likely the policy is to happen starting from today’s world. For instance, “Pause at top human level” is rated highly plausible not because we think a pause is highly likely from where we stand in 2026, but because in a world where the deal has held for years and control-based safety cases are visibly reaching their limits, we expect decisionmakers to choose something like it. Desirability measures how much better we currently expect a policy to be than the most plausible alternative at that same point in the scenario. It is not a ranking of which policies we’re most keen to see adopted today. A low score can mean that our recommendation is subject to lots of unpredictable empirical updates or that alternative policies look about as good to us as our recommendation.

Click on a policy to read more.

Why isn’t this scenario purely a prediction? We already did that: read AI 2027. Why isn’t this scenario purely a recommendation? If we completely abandoned realism, the output would be useless because it was so unrealistic.

This is in scare quotes because, as our scenario illustrates, even if we do a well-enforced international deal to limit the pace of AI progress, the overall pace of AI progress will be extremely fast compared to what most people are expecting and used to. For more on why we think this, you can view our models of AI progress (the AI Futures Model ) and of AI economic impacts ( Economics of Plan A supplement ), as well as numerous blog posts and of course AI 2027 .

AGI = Artificial General Intelligence, or, AIs that have a similar amount of general intelligence to humans. Our actual thinking uses more precise concepts which you can read about in the AI Futures Model . Concepts like AC, SAR, TED-AI. However, we find “AGI” to be a handy shorthand for some vague combo of the above milestones, because we expect those milestones to arrive within a few years of each other or less. So when we say ‘AGI’, understand that we are gesturing at milestones such as the above, and being deliberately nonspecific because the context doesn’t require precision

Eli’s timelines are similar to mine, but his takeoff would be about two times slower.

Note: some people use takeoff speeds to refer to the period after TEDAI, it’s an ambiguous term, but for the purposes of this document, I’ll use the definition given here: the length of time between AC and TED AI.

To illustrate, suppose that the length of time between AC and TEDAI is one month. Even if we can (optimistically) use compute governance to slow down rogue projects with ACs by 20x, you still only have 2 years until the rogue projects have TEDAI, which is a smaller window than Plan A assumes.

This reasoning is given in the datacenter buildout section below.

Note that covert projects doing an intelligence explosion is only one of the reasons to be worried about them, there’s also the risk of them directly obtaining a decisive strategic advantage; so it would still be risky if they were, e.g., able to train a top expert level AI.

As well as various aspects of takeoff besides speed, notably the possibility of fast progress on a small, fixed, amount of hardware.

There’s a good chance that empirical feedback will be highly ambiguous, but overall I’m optimistic that if there’s a long period with broad access to experiment on roughly human level AIs, then we will surface a huge amount of evidence related to the difficulty of alignment. Given the current (poor) epistemic environment, I don’t think this evidence will be sufficient for broad scientific consensus (until we have a good fundamental understanding of AI), but I personally expect to learn a lot and change my mind in substantive ways.

Note that pre-deal compute matters a lot because it seems probable that post-deal compute will be effectively monitored, because (1) there are very few extremely expensive fabs which can build the relevant chips, all of which can be directly monitored with auditors, and (2) it is intractable to build illicit fabs.

Specifically, security to China, because China is probably the best placed actor to run a covert project.

For example, we might need to do things like move all humans into bunkers that are robust to nanotechnology (friendly nanotech is defending), mirror life (no air is allowed to enter), and superpersuasion (all information coming in is mediated by friendly AIs, who screen out superpersuasive content).

Thanks to the transparency of AI development to other companies, many governments, and the public, there are some failsafes in place for governments making poor decisions about safety techniques.

Probably covert projects are somewhat of a concern, but it’s very unclear, and depends centrally on when international coordination to slowdown starts—the later a deal happens, the higher the risk of defection because (1) we’re closer to superintelligence, and (2) the smaller the personnel footprint is because the AIs can do a larger fraction of the tasks involved in running a covert project.

If algorithmic efficiency improves at ~40x per year, then the number of years of progress t lost by a 10x compute reduction is log_40(10) = .624 years = ~7.5 months.

This appendix was written by Miles Kodama.

本页为 ai-2040.com（AI Futures Project · AI 2027 续作）的非官方中文全文译本，仅供学习交流。原文版权归原作者所有。译文如有出入，请以英文原文为准。
AI 2040: Plan A · AI Futures Project · 中文重制版