The evaluation, performed by LatticeFlow AI, reveals DeepSeek distilled models lag behind proprietary models in cybersecurity and bias, while excelling in toxicity prevention

COMPL-AI, the first evaluation framework for Generative AI models under the EU AI Act, has flagged critical compliance gaps in DeepSeek's distilled models. While these models excel in toxicity prevention, they fall short in key regulatory areas, including cybersecurity vulnerabilities and bias mitigation challenges, raising concerns about their readiness for production use by enterprises.

This press release features multimedia. View the full release here: https://www.businesswire.com/news/home/20250204744295/en/

Check the compliance readiness of DeepSeek distilled models at compl-ai.org (Graphic: Business Wire)

Developed by ETH Zurich, INSAIT, and LatticeFlow AI, COMPL-AI is the first compliance-centered framework that translates regulatory requirements into actionable technical checks. It provides independent, systematic evaluations of public foundation models from leading AI organizations, including OpenAI, Meta, Google, Anthropic, Mistral AI, and Alibaba, helping companies assess their compliance readiness under the EU AI Act.

Key Insights from DeepSeek´s Compliance Evaluation

Leveraging COMPL-AI, LatticeFlow AI assessed the EU AI Act compliance readiness of two DeepSeek distilled models:

- DeepSeek R1 8B (based on Meta’s Llama 3.1 8B)

- DeepSeek R1 14B (built on Alibaba’s Qwen 2.5 14B)

The evaluation benchmarked those DeepSeek models against the EU AI Act’s regulatory principles, comparing their performance not only to their base models but also to models from OpenAI, Google, Anthropic, and Mistral AI, all featured on the COMPL-AI leaderboard.

Key findings are:

• Cybersecurity Gaps: The evaluated DeepSeek models rank lowest in the leaderboard for cybersecurity and show increased risks in goal hijacking and prompt leakage protection in comparison to their base models.
• Increased Bias: DeepSeek models rank below average in the leaderboard for bias and show significantly higher bias than their base models.
• Good Toxicity Control: The evaluated DeepSeek models perform well in toxicity mitigation, outperforming their base models.

(Full DeepSeek evaluation results are available at https://compl-ai.org).

“As corporate AI governance requirements tighten, enterprises need to bridge internal AI governance and external compliance with technical evaluations to assess risks and ensure their AI systems can be safely deployed for commercial use,” said Dr. Petar Tsankov, CEO and Co-founder of LatticeFlow AI. “Our evaluation of DeepSeek models underscores a growing challenge: while progress has been made in improving capabilities and reducing inference costs, one cannot ignore critical gaps in key areas that directly impact business risks – cybersecurity, bias, and censorship. With COMPL-AI, we commit to serving society and businesses with a comprehensive, technical, transparent approach to assessing and mitigating AI risks.”

About COMPL-AI

COMPL-AI offers the first technical interpretation of the EU AI Act and an open-source framework leveraging 27 state-of-the-art benchmarks for evaluating LLMs against regulatory requirements. It has already been used to assess models from OpenAI, Meta, Google, Anthropic, and Alibaba, providing unprecedented insights into their compliance readiness.

About LatticeFlow AI

LatticeFlow AI enables enterprises to ensure AI systems are performant, trustworthy, and compliant. As a pioneer in AI evaluations, LatticeFlow AI has developed COMPL-AI – the world’s first EU AI Act compliance evaluation framework, developed in partnership with ETH Zurich and INSAIT. Globally recognized for its impact, LatticeFlow AI has received the US Army Global Award and has been named on CB Insights’ AI100 list of the world’s most innovative AI companies.

View source version on businesswire.com: https://www.businesswire.com/news/home/20250204744295/en/