Model list is a bit limited, and whole harness affects the speed, so the libs used in harness and their versions might be useful to show for each individual model setup.
I’ll look into adding harnesses. The benchmark sources usually don’t contain harness so the problem is limited data. What models would you like to see?
He's exagerating, I hope. The only bottleneck a harness attachs is pre-context fill and whatever tooling is installed. A proper harness reduces these to negligable startup costs.
Unless of course he's driving some weird lambo that dumps 200k of context into their chats. Who knows, I feel like we're on a yellow submarine.
This is a great idea but I'm far less concerned about output token speed and way more about the ability to parallelize prefill - for agentic coding that's pretty much way more important than output speed.
Unless of course he's driving some weird lambo that dumps 200k of context into their chats. Who knows, I feel like we're on a yellow submarine.
I wished there was such a tool before picking my hardware, it's useful. But the estimates needs to be somewhat correct.