Prefill/Decode Disaggregation Can Make Tail Latency Worse
Prefill/decode disaggregation can worsen tail latency through queue imbalance and KV transfer. Measure the full TTFT path and use load-aware routing.
This is a summary aggregated from HackerNoon. Read the complete article on the original site:
Read full article at HackerNoon
