FPMoE: Sparse MoE Model Boosts Functional Code Generation

A new lightweight open-source code generation model named FPMoE has been developed by researchers, utilizing a sparse Mixture-of-Experts (MoE) architecture to enhance the performance of large language models (LLMs) on functional programming languages (FPLs) such as Haskell, OCaml, and Scala. Current LLMs, including leading models, struggle significantly with FPLs due to their training primarily on imperative languages. The research team discovered that fine-tuning for individual languages does not effectively capture shared functional abstractions, and combined multi-language fine-tuning leads to cross-language interference. FPMoE addresses these challenges by employing three language-specific routed experts (one for each of Haskell, OCaml, and Scala) alongside a shared expert that identifies cross-language functional patterns, including monadic reasoning and type-directed programming. The findings are published in arXiv:2605.27849.