Strikingness-Aware Evaluation for Temporal Knowledge Graph Reasoning
A novel evaluation framework for Temporal Knowledge Graph Reasoning (TKGR) tackles the problem of current metrics inflating reasoning capabilities by giving equal importance to all events, including insignificant repetitions. The innovative strikingness-aware evaluation introduces a rule-based strikingness measuring framework (RSMF), which assesses event strikingness by contrasting expected occurrences with similar events derived from temporal rules. This strikingness is incorporated as a weighting factor in metrics such as weighted MRR and Hits@k. Experiments conducted on four TKG benchmarks reveal that all representative models exhibit diminished performance as event strikingness rises, with path-based approaches performing best on low-strikingness events, while representation-based methods excel on those with high strikingness.
Key facts
- Temporal Knowledge Graph Reasoning (TKGR) infers missing or future events from historical data.
- Current evaluation uniformly weights all events, overestimating true reasoning ability.
- Rare outstanding events require deeper reasoning and should be emphasized.
- Rule-based strikingness measuring framework (RSMF) quantifies event strikingness.
- Strikingness is integrated as a weighting factor into metrics like weighted MRR and Hits@k.
- Experiments conducted on four TKG benchmarks.
- All representative models perform worse as event strikingness increases.
- Path-based methods excel on low-strikingness events; representation-based on high-strikingness.
Entities
—