Stop Asking the LLM to Review Everything