Open-Source RL Training Teaches Coding Model to Paint Watercolours
On August 23, a video by Surya Narreddi showcasing watercolors generated by a language model went viral, amassing over 1.5 million views. The model employs JavaScript through p5.brush. Although Narreddi's blog previously mentioned training on close-up flowers, it did not provide any artifacts, with a comprehensive technical report expected soon. An author from Hugging Face replicated this idea using TRL and OpenEnv, offering resources such as a dataset and training scripts. The pipeline functions on Hugging Face, leveraging Jobs, Spaces, Inference Providers, and a Hub collection. The reward function integrates HPSv3 and a pairwise judge, resulting in varied outcomes from three training runs. The initiative highlights a transition towards model training and editable code, tackling infrastructure issues.
Key facts
- Surya Narreddi posted a video of watercolours painted by a language model on 23 August, which received over 1.5 million views.
- The model writes JavaScript through p5.brush, a library that adds natural drawing tools to p5.js.
- The author reproduced Narreddi's idea using TRL and OpenEnv, publishing all resources openly.
- The pipeline runs on Hugging Face, using Jobs, Spaces, Inference Providers, and the Hub.
- The reward function uses HPSv3 and a pairwise judge (Qwen3-VL-30B-A3B-Instruct).
- The reference pool consists of 178 paintings generated by four open-weight models, rated by the author into 'love' and 'okay' tiers.
- Three runs were trained with different reward mixes: hps-only, hps-led, and judge-led.
- The judge-led run produced the most diverse and artistically interesting paintings.
- The project demonstrates RL over aesthetic preference, with no correct answer.
- The author fixed a bug in OpenEnv and submitted the fix upstream.
Entities
Artists
- Surya Narreddi
- Alejandro Campos Uribe
- Mario Klingemann
- Anna Ridler
- Alex Yango
- Brendan Hogan
- Jason Liu
Institutions
- Hugging Face
- TRL
- OpenEnv
- p5.brush
- HPSv3
- Qwen3-VL-30B-A3B-Instruct
- iNaturalist
- DeepDream
- Edmond de Belamy
- GAN
- Simon Willison's pelican benchmark