EntropyMoE: Entropy-Aware Sparse Expert Routing for Tokenizer-Free LLMs
A recent submission on arXiv (2608.06398) presents EntropyMoE, an innovative Mixture-of-Experts (MoE) framework tailored for tokenizer-free large language models (LLMs) that function with dynamic byte patches. This study tackles a challenge found in current byte-patch models, which uniformly apply dense feed-forward computations across all patches, neglecting to adjust model capacity according to variations in patch semantics and granularity. EntropyMoE substitutes the dense feed-forward components in the global patch Transformer with Top-K expert layers, utilizing each dynamic patch for expert routing. The routing process is informed by patch entropy, the same signal employed in dynamic patch creation, with both patch entropy and length jointly defining the routing feature space. This method enhances the efficiency of sparse computation, potentially boosting performance in tokenizer-free LLMs.
Key facts
- EntropyMoE is a Mixture-of-Experts (MoE) architecture for tokenizer-free LLMs.
- It operates on dynamic byte patches, grouping bytes into dynamically sized patches.
- It replaces dense feed-forward modules with Top-K expert layers.
- Expert routing is based on patch entropy and length.
- The router uses the same granularity signal as dynamic patch construction.
- The paper is available on arXiv with ID 2608.06398.
- The announcement type is 'new'.
- The approach aims to adapt model capacity to patch semantics and granularity.
Entities
Institutions
- arXiv