OpenAI Privacy Filter Underperforms on Non-Latin Scripts in 42-Benchmark Study
An independent study has assessed OpenAI's Privacy Filter (OPF), a bidirectional PII detector with 1.5 billion parameters, using 42 synthetic benchmarks across five domains and 22 languages. The findings, published on arXiv (2608.02616), indicate that OPF excels compared to Presidio and XLM-RoBERTa on PII-annotated benchmarks, achieving F1 scores of 0.855 on AI4Privacy and 0.464 on SPY medical, while Presidio scores 0.431 and 0.273, and XLM-RoBERTa scores 0.269 and 0.111. However, OPF falters with non-Latin scripts and narrative prose, scoring between 0.04 and 0.57 on NER benchmarks. The study represents the first comprehensive evaluation of OPF, underscoring its challenges in multilingual and contextually rich environments.
Key facts
- OpenAI Privacy Filter (OPF) is a 1.5B-parameter bidirectional PII detector.
- Evaluation covered 42 synthetic benchmarks across 22 languages and 5 domains.
- OPF achieved F1=0.855 on AI4Privacy and 0.464 on SPY medical.
- OPF outperformed Presidio (0.431, 0.273) and XLM-RoBERTa (0.269, 0.111) on PII-annotated benchmarks.
- XLM-RoBERTa led OPF on all 13 Indic and non-Latin languages in multilingual NER.
- GPT-4o led on medical, legal, and financial PII (SPY: 0.643 avg, Gretel: 0.527).
- OPF led on structured synthetic PII (0.71 avg) and customer support (0.60).
- OPF's F1 dropped to 0.04-0.57 on NER benchmarks with narrative prose, and collapsed for Arabic (0.04) and Cyrillic (0.03).
- OPF was strongest on email (0.78) and phone (0.76) PII, weakest on person (0.40) and address (0.30).
Entities
Institutions
- OpenAI
- arXiv