Answers
How do you evaluate an LLM application before launch?
Build a test set of real questions with known-correct answers before writing any feature code, run the application against it, and score both accuracy and faithfulness to source material against a defined threshold before shipping — the same discipline as automated testing for any other piece of software.
"Good enough" isn't a feeling, it's a number agreed before launch: what accuracy rate on the test set counts as safe to ship, what faithfulness score to the source material is required so the application isn't quietly making things up, and what error rate is acceptable given the actual consequence of a wrong answer — a wrong product recommendation and a wrong medical-adjacent answer don't get the same bar.
Cost and latency need the same treatment, decided at the same time as accuracy rather than discovered after launch when a bill arrives. What's the maximum acceptable response time for the real use case, and what's the cost ceiling per interaction at the volume you expect once it's live, not the volume in a demo?
Run this evaluation the same way you'd run automated tests for any other software feature — as a required gate before deployment, not a one-time check before the first launch. The same test set, expanded over time with real production failures, should keep running against every change going forward, catching regressions before a user does.