What is LLM latency?
Latency is how long you wait for a reply; time to first token is how long until the first word appears.
Definition
Latency is how long you wait for a reply; time to first token is how long until the first word appears.
How it works on TokenRex
Fast variants give roughly 2x faster responses at 2x the cost.
Streaming improves perceived latency.