Paper challenges text-only pretraining, proposes visual pretraining for language models
A new arXiv paper argues that current language model pretraining discards rich visual information from documents and web pages. The authors propose scalable visual pretraining to incorporate figures, equations, and layouts, aiming to improve language intelligence beyond text-only approaches.
Entities
Related
Paper proposes video generation as general-purpose vision pretraining
A new arXiv paper argues that large-scale text-to-video generation can serve as a general-purpose pretraining paradigm for computer vision, analogous to next-token prediction in NLP. The authors introduce GenCeption, a method that uses a pretrained video generative diffusion backbone for feed-forward visual perception tasks.
New paper proposes LLM-GCN hybrid for semi-supervised image classification
A new arXiv paper introduces a method that integrates Large Language Models with Graph Convolutional Networks to improve semi-supervised image classification. The approach addresses the challenge of graph construction for visual data by leveraging LLMs to generate better graph representations, potentially reducing the need for labeled datasets.
Research paper proposes reusing spectral patterns from pretrained GPT-2 checkpoints as initialization for language model pretraining
A new arXiv paper analyzes eleven pretrained GPT-2-style checkpoints and finds that they share structured weight spectra, with consistent depth trends in Frobenius norm and effective-rank entropy. The authors propose reusing these recurring spectral patterns as initialization signals for GPT-2-style language model pretraining, potentially reducing training time or improving convergence.
CLAP paper proposes direct VLM-to-VLA adaptation to isolate semantic transfer
A new arXiv paper introduces CLAP, a method that converts pretrained vision-language models (VLMs) into vision-language-action models (VLAs) with minimal architectural changes, avoiding the distribution mismatch caused by predicting actions as bare numeric tokens. The approach aims to clarify how VLM capabilities transfer across model scales for robot control.
Self-Guided Test-Time Training Improves Long-Context LLM Accuracy
A new arxiv paper proposes Self-Guided Test-Time Training (SG-TTT) to improve long-context utilization in LLMs without requiring labeled data. The method uses the model's own predictions to generate pseudo-labels for fine-tuning on the test context, addressing accuracy degradation in long inputs. This approach is more efficient than full TTT and shows promise for practical deployment.