Input-Anchored Logic Gate Networks Overcome Depth Scalability Limits
A recent study published on arXiv (2607.21633) reveals two primary reasons for the shortcomings of current Logic Gate Networks (LGNs) when it comes to depth enhancement: the phenomenon of optimization collapse in deep relaxed LGNs and a topology-related constraint that remains even with skip-biased initialization and straight-through estimation. The researchers propose Input-Anchored Logic Gate Networks (IALGNs), which integrate a developing hidden feature with a direct input anchor at each gate, maintaining a computational backbone and conditioning all layers on the initial input. They demonstrate that a depth-D path can rely on as many as D+1 input bits, establishing a definitive path-wise depth hierarchy. Additionally, a random-k anchor relaxation optimizes anchor selection, tackling a key limitation in LGN depth scalability.
Key facts
- arXiv paper 2607.21633 identifies two causes for LGN depth scalability failure
- Optimization collapse occurs in deep relaxed LGNs
- Topology-induced limitation persists despite skip-biased initialization and straight-through estimation
- Trainability alone is insufficient for deeper layers to receive useful information
- Input-Anchored Logic Gate Networks (IALGNs) combine evolving hidden feature with direct input anchor
- IALGNs preserve a computational spine while conditioning every layer on original input
- Depth-D path can depend on up to D+1 input bits
- Strict path-wise depth hierarchy is established
- Random-k anchor relaxation improves anchor selection
Entities
Institutions
- arXiv