GPASV: Shapley Value for Cyclic Priority Graphs in ML Valuation
Researchers have introduced a new idea known as the generalized priority-aware Shapley value (GPASV). This innovative random order value can be used with any directed weighted priority graphs, tackling the limitations of existing methods that depend on binary and acyclic pairwise priorities. These current methods often fail to capture real-world situations, like combined human preferences and evaluations with multiple criteria. GPASV includes classical models as specific cases and is built on an axiomatic framework that features key computational techniques and a priority sweeping diagnostic. Its use in valuing LLM ensembles on the cyclic Chatbot Arena preference graph shows the need for a careful balance in prioritizing pairwise graph priorities. The research is accessible on arXiv under the identifier 2605.15018.
Key facts
- GPASV is a random order value defined on arbitrary directed weighted priority graphs.
- Existing priority-aware Shapley value extensions require binary and acyclic pairwise priorities.
- Real-data examples like aggregated human preferences and multi-criterion comparisons violate the acyclic restriction.
- GPASV covers a range of classical models as boundary cases.
- GPASV is established through an axiomatic characterization.
- Computational methods and a priority sweeping diagnostic are developed.
- GPASV is applied to LLM ensemble valuation on the cyclic Chatbot Arena preference graph.
- The work is published on arXiv with identifier 2605.15018.
Entities
Institutions
- arXiv