Skip to main content

Auditing Chats

Trust but verify. That's our motto (one of them, anyway). We take great pride in the quality of answers that users receive from Copilot. Of course, most of the magic happens in the training data; we're just the messengers (really really eloquent messengers).

Sometimes, it can be helpful to check Copilot's work. What this means is to peer into Copilot's mind and ask "why did you respond this way". Common situations in which you might want to do this are:

  • You identify an incorrect or suboptimal answer and want to understand why it was produced (for example, maybe Copilot was referencing incorrect source data)
  • You are especially impressed with Copilot's answer and want to understand how it produced such brilliance
  • You're curious how Copilot works under the hood

How to audit individual chats

To audit individual Copilot chats, head over the Copilot analytics dashboard. From here, select a chat. This will make it appear from the right. At the top of the chat, you'll see a button called View sources. Click that, and you'll see a panel with one row for each response Copilot gave during the chat.

Adding APIs to Copilot

Clicking a card will reveal the source material that Copilot had available when generating its answers. This inclues

  • Any API endpoints that were used (including the request and response payload)
  • System prompt information (with interpolated values) that were set at the time of the chat
  • Relevant source content. This means passages that were provided to Copilot as part of our answer generation process (which picks out relevant passages from which Copilot can form its answer). These come from source doc passages (we break down every doc into smaller passages) as well as answers. Any source content marked in green was cited by Copilot directly (implying it was the source content that informed the answer the most)