Foundation models,
from the ground up.

Abstract

We pursue foundational research on novel pretraining paradigms — architectures and training objectives that depart from standard autoregressive next-token modeling. Our work focuses on representations that transfer cleanly across tasks and modalities.