Muscle-Driven AI Controls Piano-Playing Hands

A novel data-centric method facilitates dexterous control of musculoskeletal hands, allowing for accurate piano performances on new compositions beyond the existing reference dataset. This hierarchical framework integrates high-frequency muscle-level management with low-frequency coordination in latent space. Reinforcement learning is employed to train low-level policies, which produce dynamic muscle-tendon activations while adhering to trajectories from an extensive reference motion dataset. These policies are transformed into variational autoencoder (VAE) models, resulting in smooth latent spaces that simplify low-level muscle dynamics. High-level, piece-specific policies function within this latent space to synchronize bimanual movements according to note events derived from musical scores.