Is open-source AI safer or riskier than closed/proprietary AI?

Both sidesSays it is contestedSources conflict

The Stanford study finds limited evidence of greater marginal risk from open models, while Anthropic, RUSI and CBS-cited sources stress that open weights pose unique irreversible risks for capable models.

It depends on the dimensions of "safety," the capability level of the models, the threat models considered, and the time horizon—there is no simple consensus that one is inherently safer or riskier. Open-source/open-weight AI and closed/proprietary AI involve different trade-offs. "Open-source" often loosely refers to open-weight models (downloadable parameters that can be run/modified locally), though true open-source typically includes code, training data, and more. Evidence of marginal extra risk from openness is often limited for current models, but unique challenges exist for highly capable ones.[1]

Key Risks of Open/Open-Weight AI

Open models make certain misuses easier and harder to reverse:

•Irreversibility and lack of control: Once weights are released, they cannot be recalled, patched centrally, or have access revoked. Safeguards (e.g., RLHF refusals) can be stripped cheaply and quickly via fine-tuning or techniques like "abliteration" (sometimes in minutes on modest hardware for under a dollar).[2]

•Easier misuse by bad actors: Anyone can download, modify, and run them privately without monitoring. This enables "uncensored" variants for cybercrime (e.g., FraudGPT/WormGPT based on open models like GPT-J), disinformation, scams, non-consensual intimate imagery (NCII), CSAM generation, or potentially higher-stakes risks like cyberattacks or bioweapons assistance as capabilities grow.[3]

•Weaker ongoing oversight: No central logging, usage bans, or iterative safety updates post-release. Closed providers can monitor, ban accounts, and strengthen guardrails based on observed misuse.[4]

•Expert concerns (e.g., from Anthropic’s Dario Amodei and others) highlight that for models with dangerous capabilities, openness precludes many safety measures available to closed systems, creating persistent risks. Some view advanced open AI as a "very dangerous path."[5]

Closed models are not immune: they can be jailbroken via clever prompting, fine-tuned via APIs into unsafe versions, leaked, reverse-engineered, or compromised. Massive distribution of popular closed models also creates scale risks.[6]

Key Benefits and Safety Advantages of Open AI

Openness can enhance certain aspects of safety and resilience:

•Transparency, auditability, and collective scrutiny: Weights enable independent researchers to inspect internals, find vulnerabilities, study biases/alignment, and develop defenses. Closed models are black boxes controlled by a few companies. This supports better safety research and reduces reliance on corporate self-reporting.[7]

•Democratization and reduced concentration of power: Avoids monoculture/single points of failure where a few labs control critical infrastructure. Distributes decision-making on "acceptable" behavior, fosters competition/innovation, and enables local/sovereign deployment (data stays private). Concentration itself is a risk (e.g., corporate or state misuse, regulatory capture). Yann LeCun strongly advocates openness for democracy, cultural diversity, and to prevent a handful of companies mediating knowledge/information.[8]

•Defensive and scientific upsides: Cybersecurity defenders often need controllable open models (closed ones may refuse forensic work due to guardrails). Openness accelerates safety techniques, innovation in applications, and science (reproducibility, interpretability). Lower costs and customizability benefit legitimate users.[9]

•Marginal risk often limited empirically: A major 2024 Stanford HAI/CRFM position paper (with many co-authors) analyzed open foundation models’ distinctive properties (broader access, customizability, local inference, irreversibility, poor monitoring). It found significant benefits for innovation, competition, power distribution, and transparency (with caveats). For risks (disinformation, bio, cyber, phishing, NCII, CSAM, etc.), it concluded there is currently limited evidence of substantial marginal risk relative to closed models or pre-existing technologies (e.g., web search). Some proposed restrictions could disproportionately harm the open ecosystem. Further research is needed as capabilities evolve.[1]

Other analyses note overlapping risks between open and closed, and that open models can strengthen overall safety/cybersecurity through diffusion and defender access. Geopolitically, Chinese open-weight models are proliferating rapidly; U.S. abandonment of open weights could cede the ecosystem and worsen global safety.[10]

Expert and Stakeholder Divide

•More cautious on open (especially frontier): Anthropic (Amodei) sees higher misuse/alignment risks for powerful open-weights due to lost control, though they do not advocate bans and support testing all capable models (open or closed). Some safety-focused voices and earlier OpenAI decisions emphasized risks of releasing advanced weights. Geoffrey Hinton has expressed reservations about open weights of large models enabling cheap fine-tuning for harm.[5]

•Pro-open: Yann LeCun (and many in open-source communities, Meta historically with Llama, Mistral, etc.) argue openness is essential and safer long-term; doom narratives can enable regulatory capture favoring incumbents. Industry letters (including from Google, Microsoft, NVIDIA, etc.) defend open weights for accessibility and innovation while acknowledging distinct risks.[4]

•Broader views (e.g., panels, arXiv analyses): Risks exist in both; closed models have distribution advantages that can amplify harm, and jailbreaks/fine-tuning affect both. Hybrid approaches, mandatory safety testing for high-capability models (regardless of openness), better defenses, and focus on deployment/use (not just release) are common recommendations.[11]

Bottom Line and Context

•Near-to-mid term / current capabilities: Open models do not appear dramatically riskier on net for many documented harms, per analyses like Stanford’s, and offer clear benefits in innovation, access, transparency, and resilience. Closed models’ guardrails are imperfect and can hinder legitimate defensive/research work. Existing tech already enables much of the harm.

•High-capability / catastrophic risks (advanced cyber, bio, autonomous agents, loss-of-control): Openness introduces unique irreversible challenges. Many experts prefer structured access, testing, and controls for the most powerful systems. Offense-defense balances (e.g., in biology) matter greatly and are uncertain.

•Other factors: True safety depends more on overall capability progress, alignment techniques, governance, international coordination (chips, testing), defenses, and incentives than openness alone. Open models lag frontier closed ones but the gap has narrowed. Poisoning/backdoors and supply-chain issues affect both (though transparency helps detection in open cases).

•Policy sweet spots often involve: mandatory rigorous safety evaluations for sufficiently capable models (open and closed); targeted measures (e.g., on distillation or high-risk uses); supporting safer open development (especially U.S./allied leadership); and avoiding blanket bans that could backfire by ceding ground or stifling benefits.[5]

In short, open AI is riskier in controllability and certain misuse scenarios (especially irreversible high-stakes ones) but can be safer or net-positive via transparency, distributed power, innovation in defenses, and reduced concentration. Closed AI offers better centralized monitoring and iteration but risks opacity, monopoly power, and imperfect guardrails. The optimal path involves evidence-based assessment of marginal risks, continued research, and balanced governance rather than treating openness as binary good or bad. Capabilities and societal defenses are evolving quickly, so positions should update with evidence.

Citations (11)

1the-decoder.comNews and media
https://the-decoder.com/stanford-study-open-source-ai-models-pose-no-greater-risks-than-closed-models/
Backs: “Evidence of marginal extra risk from openness is often limited for current models, but unique challenges exist for highly capable ones.”
evidence, article, points to yes. Claim: Stanford study found limited evidence of substantial marginal risk from open foundation models relative to closed models or existing technologies.
2rusi.orgNonprofits and think tanks
https://www.rusi.org/explore-our-research/publications/commentary/responding-risks-open-weight-ai-models
Backs: “Safeguards (e.g., RLHF refusals) can be stripped cheaply and quickly via fine-tuning or techniques like "abliteration" (sometimes in minutes on modest hardware for under a dollar).”
evidence, article, points to no. Claim: Safeguards on open-weight models can be stripped cheaply and quickly via fine-tuning or abliteration, and weights cannot be recalled.
3ibm.comCompanies
https://www.ibm.com/think/insights/unregulated-generative-ai-dangers-open-source
Backs: “This enables "uncensored" variants for cybercrime (e.g., FraudGPT/WormGPT based on open models like GPT-J), disinformation, scams, non-consensual intimate imagery (NCII), CSAM generation, or potentially higher-stakes risks like cyberattacks or bioweapons assistance as capabilities grow.”
evidence, article, points to no. Claim: Uncensored open-model variants enable cybercrime, disinformation, scams, NCII, CSAM, and potentially higher-stakes harms.
4cbsnews.comNews and mediaquality 0.88
https://www.cbsnews.com/news/open-weight-ai-models-safety-risks/
Backs: “Closed providers can monitor, ban accounts, and strengthen guardrails based on observed misuse.”
evidence, article, points to no. Claim: Closed providers can monitor, ban accounts and strengthen guardrails; industry letters defend open weights while acknowledging distinct risks.
5anthropic.comCompanies
https://www.anthropic.com/news/position-open-weights-models
Backs: “Expert concerns (e.g., from Anthropic’s Dario Amodei and others) highlight that for models with dangerous capabilities, openness precludes many safety measures available to closed systems, creating persistent risks. Some view advanced open AI as a "very dangerous path."”
attribution, official document, points to no. Claim: Anthropic sees higher misuse and alignment risks for powerful open-weights models, though it does not advocate bans and supports testing all capable models.
6broadbandbreakfast.comNews and media
https://broadbandbreakfast.com/risks-of-open-source-ai-also-exist-in-closed-systems-panelists/
Backs: “Massive distribution of popular closed models also creates scale risks.”
evidence, article, points to yes. Claim: Closed models can also be jailbroken, fine-tuned via APIs, leaked or compromised, and their mass distribution creates scale risks.
7arxiv.orgAcademic
https://arxiv.org/abs/2505.10603
Backs: “This supports better safety research and reduces reliance on corporate self-reporting.”
evidence, article, points to yes. Claim: Open weights let independent researchers inspect internals, find vulnerabilities and develop defenses, unlike closed black boxes.
8frontierminds.aiLow provenance
https://frontierminds.ai/compare/
Backs: “Yann LeCun strongly advocates openness for democracy, cultural diversity, and to prevent a handful of companies mediating knowledge/information.”
attribution, other, points to yes. Claim: Yann LeCun advocates openness for democracy, diversity and preventing a few companies from mediating information.
9provos.orgNonprofits and think tanks
https://www.provos.org/p/defenders-need-models-they-control/
Backs: “Openness accelerates safety techniques, innovation in applications, and science (reproducibility, interpretability). Lower costs and customizability benefit legitimate users.”
evidence, article, points to yes. Claim: Cybersecurity defenders need controllable open models because closed ones may refuse forensic work.
10techpolicy.pressNews and media
https://techpolicy.press/us-leadership-on-openweight-models-is-key-to-global-ai-safety-
Backs: “U.S. abandonment of open weights could cede the ecosystem and worsen global safety.”
evidence, article, points to yes. Claim: US abandonment of open weights could cede the ecosystem to Chinese open models and worsen global safety.
11arxiv.orgAcademic
https://arxiv.org/html/2404.17047v2
Backs: “Hybrid approaches, mandatory safety testing for high-capability models (regardless of openness), better defenses, and focus on deployment/use (not just release) are common recommendations.”
evidence, article, neutral. Claim: Risks exist in both open and closed models; recommendations include hybrid approaches and mandatory safety testing regardless of openness.