Hexagon

cs.CC — Computational Complexity

The Price of Bayes-Optimal In-Context Learning

Contributed by Xiaoyu Li

In-context learning is often explained as implicit Bayesian inference: given labeled examples of a task drawn from a prior, the learner predicts like the posterior. We determine the computational price of this explanation for task families whose labels depend on few bits of the task description, such as quantized models on sparse data, and we compare it with the price of merely learning. The comparison uses the recent refutations of the Exponential Time Hypothesis by OpenAI and Anthropic. Bayesian inference in context is counting. For $\kappa$ -local task families with $N$ -bit descriptions under the uniform prior, the posterior predictive and posterior sampling reduce to $\#\kappa$ -SAT. Without label noise they run in time $\mathrm{poly}(m)\cdot 2^{o(N)}$ on contexts of every length $m$ when $\kappa=o((\log N)^{1/4}/\sqrt{\log\log N})$ , and label noise is handled by error profiles. A two-variable gadget gives contexts that are always consistent and whose posterior predictive is $2\#F/(2\#F+1)$ for a CNF $F$ . So approximating the posterior predictive within $1/4$ needs time $2^{(1-\gamma)N}$ at locality $O(\log N)$ under CNF-SETH, and time $2^{N/\mathrm{polylog}\,N}$ at locality $3$ under a length form of ETH. At constant locality, Bayesian inference in context is SAT parameterized by formula length. Learning in context is cheap. The Bayes risk of any class with $N$ -bit descriptions is at most $N/(m+1)$ after $m$ examples. For binarized perceptrons and literal disjunctions on sparse data, the very families of the lower bounds, a linear program is a polynomial-time in-context learner with essentially the same sample efficiency. Every $\kappa$ -local family is linear over $\sum_{j\le\kappa}\binom{N}{j}$ local Fourier features, so regression in context learns it from $N^{O(\kappa)}$ examples, in time polynomial in $N^\kappa$ and the inverse accuracy. The average case. For Goldreich's random local functions, the Bayes risk is at most $N/(m+1)$ , while every in-context learner yields a distinguisher for Goldreich's generator, with the same running time up to a linear term, whose advantage is $1/2$ minus the risk. Under a local pseudorandom generator hypothesis with polynomial stretch, polynomial-time in-context learners need polynomially more examples than the Bayes predictor. Under its length form at linear stretch, which the new algorithms refute below rate $1/2$ , the Bayes predictor of the refutations is optimal up to polylogarithmic factors in the exponent. For transformers, these bounds limit the length of a chain of thought that computes the posterior predictive, and they separate Bayes-optimal in-context learning from the regression-type in-context learning that transformers are known to implement.

Primarily AI-generated textHuman understanding: some partsBayesian inferenceCNF-SETHExponential Time HypothesisGoldreich's generatorfine-grained complexityin-context learninglocal pseudorandom generatorsmodel countingposterior predictivetransformers

Nov 06, 202637 pages

PDF View work

hexagon:2611.00132v1

cs.CC — Computational Complexity

Fast Attention after SETH: The Exact Exponent and the Hypotheses Behind It

Contributed by Xiaoyu Li

Alman and Song showed that attention with head dimension $O(\log n)$ can be approximated in almost linear time when the entries are $o(\sqrt{\log n})$ , and needs quadratic time at $\Theta(\sqrt{\log n})$ under the Strong Exponential Time Hypothesis. In October 2026 the Exponential Time Hypothesis was refuted, and with it SETH. The quadratic lower bound now rests on the hardness of Orthogonal Vectors in dimension $c\log n$ (LDOV), which no open hypothesis about satisfiability is known to imply. We measure an instance by its spread $W$ , the largest gap between two logits of a row. At every nontrivial constant precision and in every dimension between $\omega(\log n)$ and $n^{o(1)}$ , we show that under LDOV the time exponent of attention is exactly $1+\min\{W/\ln n,1\}$ : uniform sampling of keys is optimal, the cost is $\min\{n\,e^{W},n^2\}$ up to factors $n^{o(1)}$ , and the transition to quadratic time sits at $W=\ln n$ , the spread at which one key can outweigh all others. At polynomial precision and in dimension $O(\log n)$ the threshold becomes $\Theta(\log n)$ , with a constant that tends to zero as the precision grows, and the dimension moves it: at precision $n^{-1-O(1/\log\log n)}$ , spread $O(\log n/\log\log n)$ is hard in dimension $(\log n)^{1+\Omega(1)}$ but easy in dimension $O(\log n)$ . From satisfiability we obtain hardness at spread $(1+\theta)\ln n$ in dimension $n^{o(1)}$ , provided the hard CNF formulas have $N^{c_0}$ clauses for some $c_0<2$ , and we show that the construction behind this bound, which builds one instance for each proof of a Merlin–Arthur protocol for disjointness, needs $c_0<2$ for every such protocol. For the entry bound, every reduction that embeds Orthogonal Vectors vector by vector, with polynomial blowup and logit gap $\Omega(\log n)$ , yields a Merlin–Arthur protocol for disjointness; by Klauck's lower bound, such reductions cannot derive $B=\Theta(\sqrt{\log n})$ from Orthogonal Vectors in dimension $(\log n)^{1+\Omega(1)}$ . We also map the conditional lower bounds for attention in the literature to the hypotheses that now support them, and find two families whose hypotheses are false, one of them in part of its parameter range.

Primarily AI-generated textHuman understanding: some partsCNF-SETHMerlin-Arthur protocolsOrthogonal VectorsStrong Exponential Time Hypothesisattentioncommunication complexityfine-grained complexitypolynomial methodsoftmaxtransformers

Nov 06, 202643 pages

PDF View work

Matching works

The Price of Bayes-Optimal In-Context Learning

Fast Attention after SETH: The Exact Exponent and the Hypotheses Behind It

Advanced search

math.AC — Commutative Algebra
math.AG — Algebraic Geometry
math.AP — Analysis of PDEs
math.AT — Algebraic Topology
math.CA — Classical Analysis and ODEs
math.CO — Combinatorics
math.CT — Category Theory
math.CV — Complex Variables
math.DG — Differential Geometry
math.DS — Dynamical Systems
math.FA — Functional Analysis
math.GM — General Mathematics
math.GN — General Topology
math.GR — Group Theory
math.GT — Geometric Topology
math.HO — History and Overview
math.IT — Information Theory
math.KT — K-Theory and Homology
math.LO — Logic
math.MG — Metric Geometry
math.NA — Numerical Analysis
math.NT — Number Theory
math.OA — Operator Algebras
math.OC — Optimization and Control
math.PR — Probability
math.QA — Quantum Algebra
math.RA — Rings and Algebras
math.RT — Representation Theory
math.SG — Symplectic Geometry
math.SP — Spectral Theory
math.ST — Statistics Theory
cs.AI — Artificial Intelligence
cs.CC — Computational Complexity
cs.CG — Computational Geometry
cs.DM — Discrete Mathematics
cs.DS — Data Structures and Algorithms
cs.FL — Formal Languages and Automata Theory
cs.IT — Information Theory
cs.LO — Logic in Computer Science
cs.MS — Mathematical Software
cs.NA — Numerical Analysis
cs.SC — Symbolic Computation
math-ph — Mathematical Physics
quant-ph — Quantum Physics
stat.TH — Statistics Theory