Post-transformer architectures for foundation models
The dominant approach to foundation models is a Transformer trained on next-token prediction, then scaled in data and parameters. It works well, but it carries structural costs. Attention grows quadratically with sequence length, context is bounded, and inference gets expensive in exactly the regimes where deployment needs it cheap.
We work on post-transformer architectures. The hypothesis has two parts: sub-quadratic sequence models can hold Transformer-level quality at scale, and a shared trunk can learn representations that are not bound to next-token prediction. Both parts are unproven. If they hold, three properties follow.
What follows if the hypothesis holds
Longer context
Longer effective context becomes affordable, because compute no longer grows quadratically with sequence length.
Transfer
Representations transfer across tasks and modalities, because the trunk is not shaped by next-token prediction alone.
Cheaper adaptation
A new task is a head on the shared trunk. Serving runs at sub-quadratic cost.
Where this sits
The precedent is real. State-space models (S4, Mamba), linear and recurrent attention, and Transformer–SSM hybrids have all shown that sub-quadratic sequence models can be competitive at scale. The open question is whether those results carry into the pretraining of general foundation models rather than isolated benchmark tasks. That is the question we work on.
What we are building
A post-transformer pretraining stack in which the architecture itself is the experiment. We treat the sequence-mixing layer, the memory mechanism, and the interface between trunk and heads as first-class and measurable, because those choices decide quality and transfer.
Neognathae and Kestrel
Research that clears that bar ships as domain-specialized models under Neognathae. The neognathae.com site is in development. Kestrel, a text classifier, is available as a service today — write to contact@auxerta.com to get started.
Recent notes
All notes- Project Pigeon: a small model that holds long context
- Adversarial robustness in domain-specific models: red-teaming beyond the generic benchmark
- Interlocking specialized models: routing and merging domain experts for compound AI systems
- Improving synthetic data generation bounds via constrained decoding
Cite · Auxerta Labs. Post-transformer architectures for foundation models. Research note v1.0, 2026. auxerta.com/research