Self-PreTraining Shows Limited Benefits for Medical Time-Series Transformers
A new preprint on arXiv (2608.06122) examines how Self-PreTraining (SPT) influences transformer models in medical time-series applications, despite its proven benefits in handling long contexts. The study evaluates transformer designs across three significant medical contexts: rehabilitation robotics using the Camargo dataset, stress detection without EEG, and identifying Parkinson's disease through gait analysis. Researchers developed models from scratch or applied SPT, experimenting with four masking techniques to improve learning across time and different modalities. Findings indicate that SPT doesn’t consistently boost diagnostic accuracy in these datasets, particularly with sparse data, suggesting that the benefits seen in long-context tasks may not translate to medical time-series, highlighting a need for specific pre-training strategies. This research is written by a group of researchers and is awaiting peer review on arXiv.
Key facts
- The study is published as arXiv preprint 2608.06122.
- It assesses Self-PreTraining (SPT) for transformer models on medical time-series.
- Three datasets are used: Camargo (rehabilitation robotics), Non-EEG Stress (stress detection), and Gait Parkinson's Disease.
- Four masking-based pre-training objectives are tested.
- Model depth is varied to study capacity interactions.
- SPT did not consistently improve performance across datasets.
- The paper is a cross-type announcement (likely cross-posted).
- The study is relevant for AI in healthcare, particularly under limited data conditions.
Entities
Institutions
- arXiv