What are you using to test your LLM responses, benchmark them, maybe compare different versions?

I've seen a few YC startups focusing on this but I haven't decided yet if we should build this internally or use an external tool.