Is open-sourcing AI a safety win or a risk? I'll weigh both sides with the latest evidence.
The answer is both - neither is inherently safer, it depends on what type of risk you mean. The debate is framed as a trade-off between transparency and control:
1. Why Open-Source AI is argued to be Safer
Transparency and Auditability (Security through transparency, not obscurity):
•Open models accelerate innovation by allowing researchers to examine, modify and improve systems, and facilitate greater scrutiny for bias, safety and ethical considerations.
•The open-source community argues "security through obscurity" is a fallacy. From this perspective, open-source AI is actually safer because it eliminates the single point of failure and prevents a small group of engineers from having unilateral control.
•Generally speaking, open-source is argued to be strictly safer than closed-source because closed source relies on security through obscurity, open source benefits from security through transparency.
•The belief that hiding source code makes a system safer is security through obscurity, and it usually fails. Hiding the blueprint is not a security posture.
The US National Telecommunications and Information Administration (NTIA), directed by President Biden's Executive Order on Safe, Secure and Trustworthy AI to review widely available weights, took a cautious yet optimistic view that the benefits of open-source AI will outweigh the risks.
Specific safety benefits cited by NTIA:
•Open foundation models allow a broader range of actors to examine and scrutinise models to identify potential vulnerabilities and implement safety measures and patches.
•Just as open source models can be used to build attack vectors, they can also be used to build at lower cost and more quickly cyber-defence toolkits and create open-source channels for collaboration on cyber-defence.
•Widely available model weights more readily allow neutral third-party entities to assess systems, perform audits and validate internal developer safety checks.
•A downstream developer building from an open model for specific purposes can narrow and concretize the task and add on more targeted and effective safety training, testing, and guardrails.
Regulatory argument: Open-source models, with inspectable weights and training data, are compliant by default — unlike black-box proprietary APIs.
2. Why Open-Source AI is argued to be Riskier
Irreversible loss of control and safeguards removal:
•Once model weights have been widely released, it is difficult to un-release them.
•Three things follow if weights are widely available: a third party can customise outside the developer's initial scope, the developer gives up control over and visibility into end users' actions, and users can run inference locally without sharing data.
•Dual-use foundation models with widely available model weights could plausibly exacerbate risks by allowing a wider range of actors, including malicious users, to leverage capabilities and augment them to create more dangerous systems. Even if the original model has built-in safeguards like content filters, blocklists, and prompt shields, direct weight access can allow individuals to strip these safety features.
•The accessibility of open source AI models also introduces substantial risks. Malicious actors can fine-tune these models.
Anthropic's head of international Guillaume Princen argued that open alternatives to its AI models carry "more risk" of cybersecurity incidents or other dangers: "We need to be really careful with open-source on safety... by definition, there's more risk with them".
Survey data shows many experts perceive open models as riskier:
•Amid benefits there are a number of risks — primarily related to security and privacy. Survey respondents consider open source AI tools to be riskier than proprietary AI for most types of AI risks. The most relevant AI risks cited include cybersecurity (62 percent), regulatory compliance (54 percent), and intellectual property (50 percent).
•Nevertheless, many experts still perceive open models as riskier than closed models. The time-tested open-source security logic doesn't necessarily hold up when dealing with tools as powerful as today's models.
Specific risks where marginal risk is debated:
•Biological and chemical weapons: Effective biological design tools require only hundreds of millions of parameters, but NTIA concluded "the risk delta between jailbreaking future closed models for [chemical and biological weapons] content and augmenting open models is unclear".
•Cyberattacks: Open source AI could lower barriers to automatically generating malware, such as viruses, ransomware, and Trojans.
•Political deep fakes/misinformation: Most open models capable of producing political deepfakes today have fewer than 10 billion parameters, meaning they are relatively small and more easily built and proliferated. Similarly, research shows increase in AI-generated non-consensual intimate imagery (NCII) and CSAM more readily enabled by downstream apps developed from open source models, such as 'nudifying' apps.
•Closed models are also vulnerable: While studies show closed models typically demonstrate fewer safety and security risks compared to open source, closed models still demonstrate weaknesses and are also vulnerable to jailbreaking techniques. While users can circumvent guardrails in closed models (jail-breaking), service providers can better mitigate these risks because closed models tend to be accessed through APIs, such as by monitoring data sent to a model.
3. Why Closed/Proprietary AI is argued to be Safer (and its risks)
Arguments for closed:
•Proprietary models benefit from more extensive RLHF and safety testing.
•Closed-source models address many risks by controlling who can access systems: minimizes misuse by limiting access to vetted users, ensures compliance with content moderation filters and bias detection mechanisms.
•Conversely, closed or proprietary AI models maintain that restricted access ensures better security, quality control, and responsible development, preventing malicious actors from exploiting capabilities for disinformation, cyberattacks, or autonomous weapons. They contend that controlled environments allow for more rigorous safety testing before deployment.
•Main advantage is provider handles performance, safety, and reliability, with mature safety systems.
Arguments against closed:
•Creates risks of black boxing, methodological uniformity, and systemic failures, and societal concerns including centralized cultural power and algorithmic monoculture where ecosystem relies on one or few foundation models.
•Introduces opaque "black-box" architectures, external data privacy risks.
4. Consensus View
Most recent analyses conclude the framing "open is dangerous, closed is safe" is too simplistic. Closed models are also misused, and open benefits in deconcentrating power are genuine but not sufficient in themselves.
NTIA concluded the marginal risk of open models compared to closed models is as yet unclear for many harms, and recommended an evidence-based approach rather than preemptive bans on open weights.
The pragmatic emerging view is hybrid:
> Neither approach offers a perfect solution — open models democratize access but raise legitimate safety concerns, while closed models promise control but risk creating AI monopolies. The reality is that the AI ecosystem likely needs both approaches serving different purposes and risk profiles. Hybrid models that combine transparency in research with controlled deployment may offer the most pragmatic path forward