Hugging Face Chief Policy Officer Irene Solaiman Says Embedded AI Evaluators Are Not Simple Solutions

Artificial intelligence labs rely on model benchmarking to measure whether one AI product outperforms another, but these evaluation methods remain complicated and immature. Speaking at TechCrunch Disrupt, Hugging Face chief policy officer Irene Solaiman stated that AI testing must develop into a more consistent science for the industry to depend on future results.

Reproducing evaluation results is difficult, Solaiman noted, adding that open-weights access is vital for checking these outcomes. Testing methods also lose effectiveness quickly, and researchers still lack a reliable way to compare different evaluations against one another. To address safety concerns, the industry has popularized embedded evaluators, where neutral third-party observers are placed inside AI organizations.

However, doubts persist regarding their qualifications and neutrality. For instance, Anthropic faced heavy criticism after hiring consulting firm Accenture as its first evaluator. Furthermore, modern testing now examines entire AI systems rather than standalone models. Solaiman explained that testing must cover the access harness and holistic agents, which remain expensive and difficult to evaluate properly.