Chunk code by symbols, not arbitrary token windows

Somebody wires up a local code index, points it at a repository, and slices every file into 800-token blocks because that's what the tutorial did. It works, sort of, for a while. Then the retrieval step hands the model half a function: the body, no signature, no imports, no idea what the thing even returns. The model writes something plausible and wrong, and nobody notices until it ships. That's not a hypothetical. It's the default outcome of treating a codebase like a document.

Text splitters exist because they work fine on prose, where meaning sits in the nearby sentences. Code doesn't behave that way. A function's meaning is scattered across its signature, its callers, the tests that pin its behavior, and the imports it depends on, and none of that respects a fixed character count. This is the same discipline problem every retrieval setup runs into: nobody measures it like a system until it breaks like one.

What the index should actually hold

The fix is mechanical, not clever: parse symbols instead of characters, keep the file path and signature attached to every chunk, link each function to its tests and callers, and pull in surrounding imports only when the symbol genuinely uses them, not by default. How you embed a signature is a separate problem from how you embed prose, one covered separately in embedding code for retrieval; this piece is about what you hand the embedder, not what it does with it.

Run the same inputs every time you change something, and save the exact launch command next to the result, or you'll spend an afternoon wondering what you actually tested. Include a cold start if a human will ever hit one, run long enough for heat or queuing to show up, and write down whether the answers stayed correct, not just how fast they arrived. A setup that answers faster by answering differently isn't a faster version of the old one, it's a new, unverified one wearing its clothes.

Before any of that, decide what job you're actually indexing for: interactive chat, inline completion, document extraction, an overnight batch run. Pick one, decide what "good" means for it before you touch a setting, or you'll spend a week tuning a number nobody asked for.

The handful of numbers worth writing down

Not every measurement earns a place in the record. Keep the ones that could actually change what you decide next:

• time to first token
• prompt-processing speed
• generation speed
• peak memory
• wall power, when that matters for where this runs
• task success, meaning the retrieved chunk actually answered the question

Medians tell you what a normal run looks like. A slow percentile, p95 if you can get it, tells you about the pause that makes someone tab away. Neither substitutes for the other. Whether it feels responsive, whether it's noisy, whether it was miserable to set up: none of that shows up in a clean number, so it goes in a note next to the numbers, because it often decides whether anyone keeps using the setup.

Where the easy wins quietly rot

"It loaded" is not a performance result. "The answer looked fine" is not an evaluation, it's a shrug dressed up as a conclusion. Local inference tolerates a huge number of configurations that technically run without telling you whether they're any good, so check the runtime logs and the OS's own metrics instead of trusting whatever flag you asked for. When two runs come out different, change exactly one variable and be able to say why you expect the difference, or the whole exercise turns into a folder of anecdotes.

There's a second trap, quieter than the first: the fragile five-percent win. It looks real in the moment, then evaporates the next time a model, driver, or runtime changes underneath it, and you won't notice until it stops working and you can't remember what you tuned. You don't need an observability platform for a workstation, just a small script, a handful of prompts that represent the workload, and a plain-text file with the results, so retesting after an upgrade doesn't mean rebuilding your mental model.

Sizing chunks to what a person could hold in their head

Here's the decision rule I actually use: chunk boundaries should match units a developer could reason about unassisted, a function, a class, a test paired with what it tests, a call and its definition. That's a decision about meaning before it's one about token counts. Test the smallest plausible change first, leave yourself headroom instead of tuning to the edge of what fits, and stop once the workflow clears its latency and quality bar. Spare capacity absorbs a longer prompt, a background process you forgot was running, or the next model you'll want to try.

None of this is really about hardware, though hardware is what people reach for first. A local model rewards drawing the workload boundary correctly and measuring it the same way twice more than it rewards a bigger card. The setup worth keeping isn't the one with the most impressive isolated number, it's the one you understand well enough to trust when it's wrong.

What I still don't have is a clean rule for that word "selectively" in the import step. Some functions need a line or two of context to make sense; others need a type defined pages away, and I end up deciding by eye, per language, sometimes per file, sometimes per mood. It works, mostly. I haven't found a version of this that isn't still a judgment call.