Faster fuzzy minds

I have been thinking about looped transformers and latent reasoning ever since working the PhySense paper, where the model might exhibit fuzzy behaviors all expressed in certain float point numbers instead of readable text. It is hard to say if such reasoning mechanism / test time scaling is more human or less human.

Despite the recent emergence of multiple models like LOTUS and Loopie, together with the classical paper in latent CoT COCONUT and CODI, it becomes somehow clear that we do not have out-of-the-box support from vLLM or SGLang for them. In this post I want to share some of thoughts in putting some of the models in this family into mini-sglang and maybe it will help the community in some ways.

On mini-sglang, and why vLLM/SGLang is kinda hard

Things in this section are mostly based on 9a91cfa commit of mini-sglang.

My starting point is mini-sglang (and the reason why I did not choose nano-vllm is simply picking a newer implementation). As a lightweight proxy of the full SGLang, it well explains the structure of the modern inference systems.

Let’s go through the lifecycle of a request, high-levely.

• The API server hands the request a uid, wraps the raw prompt string into a TokenizeMsg, and pushes it onto the outbound queue.
• A tokenizer process (you can run several; by default it shares with the detokenizer) pulls it, turns the string into input_ids, re-wraps it as a UserMsg, and pushes it onto the backend queue, which is the one the scheduler listens on.
• The scheduler, again a separate process, one per TP rank, and the only one holding the model and the KV cache on the GPU — picks up the UserMsg and drops it into its waiting pool. This is where the real work happens.
• Every time the scheduler produces a token, it fires a DetokenizeMsg back out.
• A single detokenizer process turns those ids back into text incrementally, wraps it as a UserReply, and pushes it onto the inbound (frontend) queue.
• The API server is listening on that queue; it matches the reply by uid, wakes the coroutine that has been waiting on this request, and streams the text back to your curl.

The request lifecycle (left) and one scheduler's internals (right) in mini-sglang.

When scheduler does the work

Everything interesting lives in the scheduler’s loop. Every iteration it does, roughly:

• look at who is waiting and who is mid-generation, and decide the next batch — new prompts first (a prefill), otherwise one step for everyone already going (a decode);
• prepare it: allocate KV pages, build the position ids, set up the attention metadata;
• run the model forward once and sample a token per request;
• ship those tokens to the detokenizer, and hand the still-running requests back to itself for the next step.

Three of the core mechamism matter, and they are the same three as full SGLang:

• Continuous batching. The scheduler does not wait to gather a fixed batch. It keeps a running set of decoding requests and, every step, forms a decode batch from whoever is ready — admitting new prompts and retiring finished ones on the fly. A request that finishes early does not stall the rest, so the GPU stays busy. This is the biggest throughput win.
• Paged KV cache. Every layer caches the keys and values of every token so attention can look back without recomputing. Instead of one contiguous buffer per sequence (which fragments horribly as requests come and go), the KV is a flat pool of fixed-size slots addressed through a page table: page_table[request, position] maps a logical position to a physical slot. So a sequence’s KV need not be contiguous, and — this matters later — two sequences can point at the same physical slots.
• CUDA graphs. Because a decode step has a fixed shape, the scheduler captures the whole step once as a CUDA graph and just replays it. More a CUDA side feature.

There is a nice payoff from the paged layout on top: since prefixes can be shared, mini-sglang keeps a radix tree over token sequences (RadixAttention), and a new request that shares a prompt prefix — a system prompt, a few-shot preamble — just inherits the already-computed KV instead of recomputing it. Shared preamble, computed once.

So why is this hard for latent / looped models?

Here is the thing. Every optimization above relies on two properties of a “request”:

1. each step appends exactly one token, so the KV cache only ever grows by one slot per step — that append-only shape is precisely what lets it be paged;
2. the decode step has a fixed shape — one query per request — which is precisely what lets it be frozen into a CUDA graph.

But if we look at the latent CoT models or the looped transformers or the recurrent depth models, they break the properties:

• Latent CoT (COCONUT, CODI) feeds a hidden vector back in as the next input, not a token id. There is nothing to look up in the embedding table — the decode step is no longer a pure function of token ids.
• Looped transformers (LOTUS) lay down a block of latent positions and then re-forward that same block in place, several times, before reading it out. The KV is overwritten, not appended, and the cursor does not move — which is exactly the thing the append-only scheduler assumes can never happen.
• Recurrent-depth models loop the whole layer stack a number of times per token, and some of them want that number chosen per token — which changes the shape and breaks the graph.

When batch size is 1, it does not really matter, but inside an engine like SGLang, it becomes bad:

• continuous batching means the running batch is a mix of requests at different latent phases — one still injecting thoughts, one mid-loop, one already decoding its answer — while the reference loop assumes the whole batch is in lockstep;
• paged KV means an extra KV entry is a page allocation at a physical address, not a list.append;
• CUDA graphs mean you cannot just write if halted: drop it in the middle of a step.

That is really why neither vLLM nor SGLang hands you these out of the box. The support would have to live in the scariest, most heavily optimized part of the whole stack. And that is the exact reason why I start with the mini versions of these inference engines.

Fast-Subconcious - built on top of mini-sglang

Github repo: https://github.com/dibbla/fast-subconscious

Making them work, one contract at a time

The thing that finally made this tractable is that you should not try to teach the scheduler about “latent reasoning” as a general concept. There isn’t a clean general concept there to teach. Instead, for each family you find the single smallest place it violates the append-only / fixed-shape contract, and you put your one special case exactly there. Everything else stays on the boring fast path, untouched.

Before the three families, one piece of shared plumbing. I gave every model one declaration and every request one knob:

class LatentSpec:
    kind: Literal["feedback", "looped"]   # WHAT mechanism the model uses
    default_budget: int                   # its trained default
    c_thought: int = 1                    # looped only: positions per loop

# ...and on the request side, a single integer:
SamplingParams(latent_budget=6)           # HOW MUCH, or None for the default

The request owns the budget. A plain model returns None for its spec and nothing about it changes. This is the one line that lets a whole test-time-scaling sweep be identical code for any latent model, and because the budget is per-request, different budgets can ride in the same batch. However, it is the only thing all three families share, and it is deliberately a description (declarative) rather than the actual driver.

Before we dive into the details of different implementation, here is a taxonomy of model families along those two axes.

Feedback latent CoT — just an inject hook

This family (COCONUT, and CODI/SIM-CoT) is the easy one, because it never actually broke contract #1: it still appends one KV slot per step. The only thing it does differently is what goes in at the latent positions — a hidden vector instead of a looked-up token embedding.

So the entire mechanism is two fields on the batch and one line in the model’s forward. The scheduler, when a request is in its “thinking” phase, fills in an inputs_embeds tensor (the vectors to inject) and a boolean latent_mask (which rows they replace). The model does the substitution right after the embedding lookup:

x = self.embed_tokens(input_ids)
if batch.inputs_embeds is not None:
    x = torch.where(batch.latent_mask.unsqueeze(-1), batch.inputs_embeds, x)

Where does the injected vector come from? The previous step’s last hidden state, passed through a per-model hook:

def latent_feedback(self, hidden):   # base class: raw passthrough (COCONUT)
    return hidden
# CODI overrides it with its trained projection head:
def latent_feedback(self, hidden):
    return self.prj(hidden)

That is essentially all of it. COCONUT feeds the raw hidden back; CODI feeds it through a small MLP it trained for exactly this. When the budget runs out, one transition token (COCONUT’s <|end-latent|>, CODI’s <|eot|>) is force-fed to close the block, and normal decoding resumes.

The part I’m quietly proud of is that this stays inside the CUDA graph. My first version fell back to eager execution for any latent step, and profiling said that eager fallback was more than half of single-request latency. The fix was to make every captured decode graph latent-capable — the inject buffers are always present, and a plain (non-latent) decode just passes an all-False mask, which makes the torch.where an identity at about ten microseconds of tax. Zero scheduler surgery, and the fast path keeps its graph.

LOTUS — the one place I opened the scheduler

Looped transformers sit in a special corner of the taxonamu, so they get the one genuine special case. LOTUS lays down a fixed block of latent positions and re-forwards that same block several times in place, each pass refining it, before it ever emits an answer. The KV of the block is overwritten, and the write cursor does not advance — the exact thing the append-only scheduler assumes cannot happen.

The pleasant surprise was that the two hard primitives this needs already existed in the engine, courtesy of paging:

• overwrite-in-place is free — store_kv already scatters each token’s K/V to whatever physical slots you hand it, so “overwrite the block” is just pass the same block slots again as the write target (out_loc);
• re-forwarding a span is just an extend-prefill, which the attention backend already supports.

So the whole feature is one small, isolated driver living beside the prefill and decode managers, not inside them. A looped request finishes its ordinary prompt prefill, and instead of going to the decode pool it goes into a LotusLoopManager. The scheduler’s pick-a-batch step gains exactly one branch in its priority order:

prefill  →  LOTUS loop (if any looped reqs are ready)  →  decode

When the LOTUS branch fires, the driver:

1. allocates the block’s pages once and remembers those physical slots;
2. runs loop_iters + 1 passes over the block — each pass is one parallel extend-prefill over all block positions at once (not a per-position decode, that would throw away LOTUS’s whole speed advantage), writing K/V back to the same slots, and capturing every position’s hidden state to feed into the next pass;
3. appends the <end-latent> token, runs it as one ordinary decode step to get the first answer token, and hands the request off to the normal decode pool.

The core prefill/decode loop that serves every other model never learns that LOTUS exists: one ~40-line manager plus one marker type, sitting next to the built-in ones, rather than a concept smeared through the scheduler.

The passes stay eager, by the way — they are prefill-shaped and compute-bound, so a CUDA graph buys nothing. Graphs are for the skinny fixed-shape decode step, which LOTUS reaches only after the loop.

Recurrent depth — a deep model in a trench coat

The last family — Ouro and Huginn — loops the entire layer stack several times per token. This one is my favorite, because it needs zero scheduler changes once you see the reframing:

A fixed-depth recurrent model is just a deeper model whose weight modules happen to be reused across iterations.

Concretely: if the stack has $L$ weight layers and you loop it $T$ times, you tell the engine the model has $T \cdot L$ layers (so the KV pool sizes for $T \cdot L$ banks), but you only build $L$ weight modules and reuse them. Iteration $t$, layer $\ell$ simply writes into KV bank $t\cdot L + \ell$:

# one stateless attention op per iteration, pinning a distinct KV bank id
AttentionLayer(layer_id = t * num_weight_layers + layer_idx)

The loop lives entirely inside model.forward(). Paged KV, FlashInfer, and the decode CUDA graph all treat it as a perfectly ordinary (if unusually deep) transformer. The only subtlety is that a token keeps one RoPE position across all $T$ of its banks — depth is not sequence position — but that is a detail in the model file, not the engine.

The two models differ in a way that turns out to matter later:

• Ouro loops the full stack with no injection, and its per-iteration state orbits — it never settles down.
• Huginn has a prelude / recurrent-core / coda structure, re-injects the token embedding into every core iteration, and its state converges — after enough iterations the trajectory stops moving.

For recurrent depth the “budget” (the number of iterations) is not the per-request latent_budget, because per-request variable depth would mean a variable number of KV banks, which fights the fixed pool. This would be an interesting idea for future works.

Can we actually stop early?

The tempting last move: if a recurrent model has a learned halting gate (Ouro, in the ACT / PonderNet lineage) or a convergence test, can we skip the remaining iterations and reclaim the compute? Here the Ouro-vs-Huginn split pays off, because the answer is gated by one property: does the state converge?

I probed both. Huginn’s per-iteration output distribution settles — the step-to-step KL drops below $5\times10^{-4}$ by around iteration 23, and its depth-32 and depth-64 trajectories sit on top of each other. So a token that halts early has a deep KV that is essentially its converged KV, and letting later tokens attend to that “latest” bank is near-lossless: real FLOPs saved, and it’s the mode Huginn was trained for. Ouro, by contrast, keeps orbiting; its shallow KV is not a stand-in for its deep KV, so reusing it degrades the answer. Its learned gate is therefore best served as an exact readout selector — run all the iterations, just pick which one to read out — which saves nothing but stays exact. This is another future work direction to adapt the engine and save the real compute.

Why not one grand abstraction?

It might feel like three families that all “loop and inject,” the obvious temptation is a single RecurrenceDriver(kv_policy, inject_policy, stop_policy) that parameterizes everything. But at the execution level the three families share almost nothing: feedback latent is a normal decode with a vector swapped in, LOTUS is an in-place extend, recurrent depth is a deep forward. What they actually share is a couple of primitives — the scatter-write (store_kv), the inject mask, a loop over a span — and the declarative knob (LatentSpec + latent_budget).

Closing thoughts

This is still early, and mini-sglang is a teaching-scale engine, not a production one. The point of fast-subconscious isn’t for production but a minimal PoC like my favorite CleanRL project. If LOTUS, COCONUT/CODI, Huginn, or Ouro are new to you, hopefully this saved you a few days of reading four papers and one very optimized codebase to find the one place each of them actually needs special-casing.

There’s plenty left undone, and I’d genuinely like help with it:

• More families. PCCoT, KaVa, SIM-CoT, and whatever ships next likely slot into one of the three buckets above — but I’ve only implemented one representative per bucket. PRs adding a model are the easiest way to stress-test whether the taxonomy actually holds.
• Adaptive recurrent depth. Per-request variable iteration count fights the fixed KV pool (see above); if you have ideas for a KV layout that tolerates it, I want to hear them.
• Exact early exit for non-converging models. Ouro’s readout-selector approach is correct but saves nothing — a real speedup here needs someone who knows the ACT/PonderNet literature better than I do.
• Benchmarks against a real engine. I only have mini-sglang numbers; if someone ports this to vLLM or full SGLang, I’d love to see how the story changes at production scale.

Citation

If you found this useful, please cite it as:

@article{xu2026fastsubconscious,
  title   = "Faster fuzzy minds",
  author  = "Xu, Yinggan",
  journal = "dibbla.space",
  year    = "2026",
  url     = "https://dibbla.space/posts/matrices/fast_subconscious/"
}

References

• PhySense — arXiv:2505.24823
• LOTUS — arXiv:2606.31779
• Loopie — arXiv:2607.16051
• COCONUT — arXiv:2412.06769
• CODI — arXiv:2502.21074
• Ouro — arXiv:2510.25741
• Huginn — arXiv:2502.05171
• PCCoT — arXiv:2506.18582
• KaVa — arXiv:2510.02312v2
• SIM-CoT — arXiv:2509.20317