Verify Flash Attention is actually active

People flip the fused-attention flag on, watch the throughput number climb, and ship the config without ever confirming the kernel ran. Months later someone traces a strange latency regression back to that same box and finds the flag was set the whole time while the runtime quietly fell back to the unfused path, because a dtype didn't match or the context length tripped an unsupported branch. Nothing crashed. The output stayed correct the entire time. The only thing wrong was the number everyone had trusted since the day they set the flag and moved on.

Kernel selection depends on hardware, dtype, context length, and the exact build of the runtime you happen to be running. A flag in a launch command is a request, not an instruction the hardware is obligated to honor, and a runtime can decline it silently with no warning you'd notice in normal use. I test this on an ordinary CUDA-capable card running llama.cpp or vLLM, nothing exotic, because the point is answering the question on hardware you operate, not hardware a vendor benchmark chose.

name the job before you touch the flag

Before any of this is worth measuring, write down what job you're actually running: interactive chat, code completion, document extraction, or an overnight batch queue. Each has a different definition of good. Chat cares about time to first token above almost everything else. Batch work cares about total throughput and barely notices the first token. Extraction cares about correctness ahead of both. Decide what good looks like before you touch a setting, or you'll optimize whatever number is on screen and call it progress.

run it twice, change nothing else

The only test that answers the question is a controlled comparison: same inputs, same model, same everything except the one flag, with the launch command saved next to whatever numbers come out. One warm run tells you almost nothing on its own. Include a cold start if real users will ever hit one, and let the workload run long enough for thermal throttling or request queueing to show up, because a card that looks fast at the start can behave like different hardware once it's been under sustained load for a while. Watch the output quality too, not just the speed. If the faster run answers differently than the slower one, you haven't sped up the same system. You've swapped it for another one, and that's a decision worth making on purpose rather than a side effect to shrug off later.

what earns a place in the log

Record only the numbers that could change your mind about keeping the flag on.

Medians describe the ordinary run; a slow percentile catches the pause that makes a demo look bad at the worst moment. The memory row deserves more attention than people give it, because attention kernels change how the KV cache gets allocated, and if you've worked through the KV cache math before you already know how fast that headroom disappears once context grows.

the flag isn't proof, the log is

The common trap is treating a clean install as proof that every request takes the fast path. Local inference is full of configurations that technically run without ever touching the optimized kernel. "It loaded" is not a performance result, and "the answer looked fine" is not an evaluation, however confident either sounds in the moment. Check the runtime logs and the operating-system metrics instead of trusting the flag you asked for. Change one variable at a time and state the mechanism you expect before you look at the result, because two numbers that differ with no explanation aren't a benchmark. They're an anecdote wearing a chart.

There's a maintenance cost hiding under all of this. A fragile five-percent win evaporates the next time a model, a driver, or the runtime changes underneath you, usually without telling you. Keep a small script, a handful of representative prompts, and a plain-text result file you can diff against the last one. That's enough to retest after every upgrade without building a full observability setup for what is, in the end, one workstation. Keep the optimization only once runtime evidence shows it applies to the shapes you deploy, and stop tuning the moment the workflow clears its latency and quality bar. Spare memory is what absorbs a longer prompt or a background process later, which matters more once you're doing any juggling of multiple models on one card instead of running a single pinned workload all day.

The tradeoff I accept on purpose is time. I spend real minutes re-running the same handful of prompts after every driver or runtime bump instead of assuming last month's flag still does what it says, and that habit costs something on every upgrade cycle for as long as the machine stays in service. What I get for it is never having to wonder, mid-incident, whether the number on the dashboard was ever real. I'd rather pay the minutes than owe the doubt.