Existing AI Governance Proposals
- 9 minsI compiled an introductory list of AI governance proposals while writing my master’s thesis, and one of my supervisors found it a helpful introduction to the area, so I’ve decided to reproduce it here. Please note that this was originally written in February 2025, so any important proposals from after that date will be missing. Please also note that as this was part of a thesis, it’s written in academese. If that’s not what you’re looking to read, take a look at any of my other posts instead.
Existing AI Governance Proposals
The following briefly reviews existing, but mostly unrealized, AI governance proposals. It splits these proposals into four main buckets: apolitical organizations for establishing scientific consensus on AI; testing and auditing measures to catch dangerous model capabilities before they are released; technical safety research to better understand models and avoid training dangerous ones; and export control proposals to influence which actors can train powerful models. Testing and auditing proposals are the most numerous, and they are subdivided into suggestions concerning the organizations responsible for regulating AI, and suggestions concerning the content of the regulations themselves.
Proposals for building scientific consensus have roughly been modelled on the United Nations’ Intergovernmental Panel on Climate Change (IPCC). For example, one paper has suggested an intergovernmental Commission on Frontier AI,[1] although it notes that “the scientific challenges and potential of politicization imply that a Commission—especially one that aims at broad political representation—may not be able to build scientific consensus effectively.”[2] Its proposal has arguably now been realized in the form of the Expert Advisory Panel behind the 2025 International AI Safety Report.[3]
Regarding organizations for regulating AI, Demis Hassabis, CEO of Google DeepMind, has called for an entity like the International Atomic Energy Agency (IAEA) to monitor unsafe projects.[4] Analogously, an intergovernmental or multi-stakeholder “Advanced AI Governance Organization” to promote norms and standards, and to support implementation, monitoring and compliance with safety protocols has been proposed.[5] Papers have also suggested that states create “an International AI Organization (IAIO) to certify state jurisdictions for compliance with international oversight standards”[6] (this would be comparable to the approaches of the International Civil Aviation Organization and the International Maritime Organization), and have more broadly called for mechanisms both to give “regulators visibility into frontier AI development,” and to ensure compliance with safety standards.”[7] Alternatively, others have suggested private regulatory markets, arguing that AI is “simply too complex to be understood by politicians, bureaucrats, and judges and represented in the text-based statutes, regulations, and judicial decisions produced and enforced by them,”[8] and that markets would decentralize enforcement of government policy objectives and enable faster regulatory responses to technological developments. An international AI consortium has also been proposed to “coordinate AI risk evaluations amongst the three core groups of stakeholders—developers of frontier AI systems, independent evaluators, and governments and regulatory agencies”[9] around the shared goal of “robust verification of the safe training and deployment of frontier AI systems.”[10] The paper in question argues that this would avoid potential conflicts of interest present in regulatory markets.[11]
On the content of regulations and standards, the UK AISI’s “Emerging Processes for Frontier AI Safety” includes assessment and mitigation of model risks, model “red-teaming” and evaluation, sharing model information to facilitate governance, securing model weights (the trained components of models that define their behaviour), establishing a reporting structure for model vulnerabilities, and monitoring deployments for model misuse.[12] Other researchers have concurrently proposed the following: “Conduct thorough risk assessments informed by evaluations of dangerous capabilities and controllability. Engage external experts to apply independent scrutiny to models. Follow standardized protocols for how frontier AI models can be deployed based on their assessed risk. Monitor and respond to new information on model capabilities.”[13] It has also been suggested that model audits should include structured access via interfaces that provide researchers with a greater ability to inspect system internals.[14] More ambitiously, audits could include complete “white-box access” to models, with the argument that the potential intellectual property and security risks of such evaluations could be mitigated by secure on-site research environments,[15] and that there is legal precedent for this form of confidential auditing in the finance industry.[16] And moving beyond predominantly technical evaluations, a broader, three-layered approach to auditing AI developers has also been proposed, including “governance audits that assess their organisational procedures, accountability structures and quality management systems … model audits, assessing their capabilities and limitations after initial training but before adaptation and deployment in specific applications … [and] continuous application audits that assess the ethical alignment and legal compliance of their intended functions and their impact over time.”[17]
Regarding organizations for technical AI safety research, Demis Hassabis has proposed a “CERN (European Organization for Nuclear Research) for AI” science institution,[18] and others have suggested an “AI Safety Project” to “promote AI safety R&D by increasing its scale, resourcing and coordination.”[19] As a first set of technical R&D goals, Turing Award winner Yoshua Bengio proposes the following areas for model safety improvements: oversight and honesty, robustness to new situations, interpretability and transparency, inclusivity, and addressing as-yet-unseen model dangers such as refusal to shut down.[20] The development of a means for monitoring AI training runs via specialized hardware has also been suggested.[21]
Less has been written on export controls for AI, but Dario Amodei has recommended the US “secure the AI supply chain in order to maintain its lead while keeping these technologies out of the hands of bad actors.”[22] Others have proposed a multilateral regime similar to the Wassenaar Arrangement for conventional arms and sensitive dual-use technologies,[23] although it has been argued that this approach would be more workable for hardware than for “models, algorithms, and other digital inputs,” and that it would require “unprecedented government oversight of major AI industry players.”[24]
Finally, it should be noted that proposals to address AI which are more ambitious or crisis-oriented than those above also exist; these are generally pessimistic about the ability of regulations to adequately protect humanity from AI risks. A former OpenAI board member supported “the development of crisis management plans for AI accidents,”[25] the Carnegie Endowment published a paper suggesting planetary asteroid defence measures as a potential blueprint for AI crisis response,[26] and a founding engineer of Skype has stated that “humanity should collectively maintain the ability to gracefully shut down AI technology at a global scale, in case of emergencies caused by AI.”[27] More pessimistically still, one researcher has called for an indefinite and global moratorium on LLMs, stating that “the most likely result of building a superhumanly smart AI, under anything remotely like the current circumstances, is that literally everyone on Earth will die,” and that nuclear powers should be “willing to run some risk of nuclear exchange if that’s what it takes to reduce the risk of large AI training runs.”[28]
Footnotes
[1]: Lewis Ho et al., ‘International Institutions for Advanced AI’ (arXiv, 11 July 2023), https://doi.org/10.48550/arXiv.2307.04699.
[2]: Ibid., 9.
[3]: Yoshua Bengio et al., ‘International AI Safety Report’ (arXiv, 29 January 2025), https://doi.org/10.48550/arXiv.2501.17805.
[4]: John Werner, ‘AI Superpowers & Global Treaties’, Forbes, accessed 27 February 2025, https://www.forbes.com/sites/johnwerner/2025/02/19/international-collaboration-right-now-treaty-verifications-and-more/.
[5]: Lewis Ho et al., ‘International Institutions for Advanced AI’ (arXiv, 11 July 2023), https://doi.org/10.48550/arXiv.2307.04699.
[6]: Robert Trager et al., ‘International Governance of Civilian AI: A Jurisdictional Certification Approach’ (arXiv, 11 September 2023), 3, https://doi.org/10.48550/arXiv.2308.15514.
[7]: Markus Anderljung et al., ‘Frontier AI Regulation: Managing Emerging Risks to Public Safety’ (arXiv, 7 November 2023), 16, https://doi.org/10.48550/arXiv.2307.03718.
[8]: Gillian K. Hadfield and Jack Clark, ‘Regulatory Markets: The Future of AI Governance’ (arXiv, 25 April 2023), 2, https://doi.org/10.48550/arXiv.2304.04914.
[9]: Ross Gruetzemacher et al., ‘An International Consortium for Evaluations of Societal-Scale Risks from Advanced AI’ (arXiv, 6 November 2023), 18, https://doi.org/10.48550/arXiv.2310.14455.
[10]: Ibid., 35.
[11]: Ibid., 36.
[12]: ‘Emerging Processes for Frontier AI Safety’ (AI Safety Summit, October 2023), https://www.gov.uk/government/publications/emerging-processes-for-frontier-ai-safety.
[13]: Markus Anderljung et al., ‘Frontier AI Regulation: Managing Emerging Risks to Public Safety’ (arXiv, 7 November 2023), 23, https://doi.org/10.48550/arXiv.2307.03718.
[14]: Benjamin S. Bucknall and Robert F. Trager, ‘Structured Access for Third-Party Research on Frontier AI Models: Investigating Researchers’ Model Access Requirements’, Oxford Martin School, accessed 14 March 2024, https://www.oxfordmartin.ox.ac.uk/publications/structured-access-for-third-party-research-on-frontier-ai-models-investigating-researchers-model-access-requirements/; Esme Harrington and Mathias Vermeulen, ‘External Researcher Access to Closed Foundation Models’ (The Mozilla Foundation, 21 August 2024), https://blog.mozilla.org/wp-content/blogs.dir/278/files/2024/10/External-researcher-access-to-closed-foundation-models.pdf.
[15]: Stephen Casper et al., ‘Black-Box Access Is Insufficient for Rigorous AI Audits’, in The 2024 ACM Conference on Fairness, Accountability, and Transparency, 2024, 8, https://doi.org/10.1145/3630106.3659037.
[16]: Ibid., 8–9.
[17]: Jakob Mökander et al., ‘Auditing Large Language Models: A Three-Layered Approach’, AI and Ethics, 30 May 2023, 10, https://doi.org/10.1007/s43681-023-00289-2.
[18]: John Werner, ‘AI Superpowers & Global Treaties’, Forbes, accessed 27 February 2025, https://www.forbes.com/sites/johnwerner/2025/02/19/international-collaboration-right-now-treaty-verifications-and-more/.
[19]: Lewis Ho et al., ‘International Institutions for Advanced AI’ (arXiv, 11 July 2023), 2, https://doi.org/10.48550/arXiv.2307.04699.
[20]: Yoshua Bengio et al., ‘Managing Extreme AI Risks amid Rapid Progress’, Science 384, no. 6698 (24 May 2024): 844, https://doi.org/10.1126/science.adn0117.
[21]: Yonadav Shavit, ‘What Does It Take to Catch a Chinchilla? Verifying Rules on Large-Scale Neural Network Training via Compute Monitoring’ (arXiv, 30 May 2023), https://doi.org/10.48550/arXiv.2303.11341.
[22]: ‘Oversight of A.I.: Principles for Regulation’ (Washington, D.C., 25 July 2023), https://www.judiciary.senate.gov/committee-activity/hearings/oversight-of-ai-principles-for-regulation.
[23]: Sujai Shivakumar, Charles Wessner, and Hideki Tomoshige, ‘Toward a New Multilateral Export Control Regime’ (Center for Strategic & International Studies, 10 January 2023), https://www.csis.org/analysis/toward-new-multilateral-export-control-regime.
[24]: Emma Klein and Stewart Patrick, ‘Envisioning a Global Regime Complex to Govern Artificial Intelligence’, Carnegie Endowment for International Peace, 21 March 2024, https://carnegieendowment.org/research/2024/03/envisioning-a-global-regime-complex-to-govern-artificial-intelligence.
[25]: Helen Toner et al., ‘Skating to Where the Puck Is Going’ (Center for Security and Emerging Technology), 19, accessed 10 April 2024, https://cset.georgetown.edu/publication/skating-to-where-the-puck-is-going/.
[26]: Emma Klein and Stewart Patrick, ‘Envisioning a Global Regime Complex to Govern Artificial Intelligence’, Carnegie Endowment for International Peace, 21 March 2024, https://carnegieendowment.org/research/2024/03/envisioning-a-global-regime-complex-to-govern-artificial-intelligence.
[27]: Jann Tallinn, ‘Priorities [for Reducing Extinction Risk from AI]’, accessed 12 April 2024, https://jaan.info/priorities/.
[28]: Eliezer Yudkowsky, ‘The Open Letter on AI Doesn’t Go Far Enough’, Time, 29 March 2023, https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-not-enough/.
Alex Chalk
AI Research | Ontario Graduate Scholar | Software Developer