pico-type: A 1.5M-Parameter Byte-Level Multi-Head Content Classifier
A groundbreaking content classifier named pico-type has been developed, functioning at the byte level with approximately 1.5 million parameters. This innovative model can assess seven distinct content characteristics directly from raw UTF-8 bytes, eliminating the need for tokenizers or pre-existing embeddings. Pico-type categorizes data into 12 coarse type classes, eight modalities, 24 subtypes, 62 programming languages, 30 text languages, and 90 MIME types, alongside six risk flags. Its structure includes byte embeddings, three convolutional blocks, two bidirectional attention layers, and a pooling layer. Four versions of the model are available, detailed in a study accessible on arXiv with ID 2608.14658.
Key facts
- pico-type is a byte-level multi-head content classifier with approximately 1.5 million parameters.
- It predicts seven content properties simultaneously from raw UTF-8 bytes in a single forward pass.
- It operates without a tokenizer, subword vocabulary, or pretrained embeddings.
- It classifies coarse type (12 classes), modality (8), subtype (24), code language (62), text language (30), file MIME type (90), and risk flags (6-label multi-label).
- The architecture includes a learned byte embedding, three convolutional blocks, two bidirectional attention layers with rotary position encodings, and a statistical pooling layer.
- It uses seven Matryoshka-style classification heads.
- Four tiered variants (tiny/small/base/pro) share the same trunk with sliced representations from 16 to 576 dimensions.
- The paper is available on arXiv under the identifier 2608.14658.
Entities
Institutions
- arXiv