Let’s talk about your bug reports. Specifically, let’s talk about the word “Hallucination.” If you are testing AI features today, I bet your Jira is full of tickets that say, “The model hallucinated.” How can I put it? Not really… Continue Reading…

Let’s talk about the dirty secret of our industry. We’ve all been there. You come in on a Monday morning, check the nightly run, and see red. Again. Not “we broke the build” red, but that annoying, flickering, “it worked… Continue Reading…

The recording of Webinar III in the AI Quality series is up! In our previous sessions, we focused on the basics of AI testing. We learned how to paddle the “AI Kayak” in the safe harbor of development. But eventually,… Continue Reading…

Who doesn’t like asserts? We have a habit of confusing “simple” with “easy.” In traditional automation, defining quality was simple. It was binary. Assert.AreEqual(expected, actual). It either matched, or it didn’t. Green or Red. But with AI, “Good” isn’t binary.… Continue Reading…

So, you did it. You built a fantastic “AI Testing Kayak.” You followed the AI Quality Funnel. You have developer tests, sanity checks, and a “Golden Dataset” that defines “good” responses. You even have an “Automated Scorecard” that runs in… Continue Reading…

Most people think of testing an AI feature like testing a chatbot – you assess the quality of a single response. But in our real-world systems, it’s almost never a single response. We build chains of calls. The response from… Continue Reading…

Last week, I published our new strategic model: Are You Building an “AI Testing Kayak” or an “AI Testing Ship”? That post explained the “why.” It defines the “Kayak” as the collection of fast, practical tactics for the practitioner in… Continue Reading…

If AI is so smart, why can’t it tell me how much I can trust its answers? Well, maybe it can… We talked about creating “Golden Datasets” – benchmarks that help us evaluate an answer from an AI model. (If… Continue Reading…

Ahoy, mateys! In our last webinar (recording is coming soon), “AI Testing: Beyond the Basics” we did something amazing: we built a functional “AI Testing Kayak.” We created a frameworkf for testing AI products, that’s fast, practical, and, most importantly,… Continue Reading…

Let’s talk about AI responses. You ask your genie a question, you get an answer. But is it a good answer? How do we evaluate a response that isn’t deterministic, isn’t repeatable, and might have some nuance? LLM answers are… Continue Reading…