Analytics

Outcome metrics explained

Comxbot tracks real business outcomes, not just conversation counts. The Analytics dashboard shows:

Leads Captured: Conversations where the assistant successfully collected contact information or qualification data. Tracked via form submissions or detected structured data in conversations.

Meetings Booked: Calendar events created through the actions framework. Tracked via the built-in calendar action or custom webhook confirmations.

Issues Resolved: Conversations that ended with the user's question answered satisfactorily, without needing human handoff. Measured via user feedback, sentiment at close, and absence of follow-up.

Handoffs Avoided: Conversations that would have been escalated (based on complexity/sentiment) but were successfully resolved by the assistant. Shows the direct value of good knowledge sources and prompts.

Cost per Outcome: Total AI spend (token costs) divided by total outcomes achieved. Lower is better. Helps you optimise model selection and prompt efficiency.

All metrics can be filtered by assistant, time range, channel, and outcome type. Export to CSV for further analysis.

Cost tracking

Every AI API call is tracked with its token usage and cost. The billing section shows a breakdown by:

Per-assistant costs: See which assistants are most expensive to run. Helps identify opportunities to switch to cheaper models or optimise prompts.

Per-conversation costs: Average cost per conversation. Useful for pricing decisions if you charge per interaction.

Token breakdown: Input tokens (your prompts + knowledge context) vs output tokens (model responses). High input costs may indicate too much context being retrieved.

Daily/weekly/monthly trends: Track cost over time. Sudden spikes may indicate retrieval issues (pulling too many chunks) or prompt inefficiency.

Budget alerts: Set monthly budget caps and receive alerts at 50%, 80%, and 100% of budget. Optional hard cap stops conversations when budget is exhausted.

BYOK savings: If using your own API keys, costs are paid directly to the provider. Comxbot tracks usage for analytics but does not mark up provider costs.

Usage monitoring

The Usage section (Settings > Usage) provides detailed operational metrics:

Conversations: Total conversations started, completed, and abandoned. Breakdown by channel (widget, email, Telegram, etc.).

Messages: Total messages sent and received. Average messages per conversation. Helps identify if conversations are too long (may indicate retrieval issues).

Knowledge retrieval: Total retrieval queries, average chunks returned, cache hit rate. Low cache hits may indicate diverse query patterns.

Actions executed: Total actions proposed, approved, rejected, and failed. Action success rate by type.

Handoff rate: Percentage of conversations that result in human handoff. Track over time to measure improvement.

Response latency: P50, P95, and P99 response times. Includes retrieval time and model inference time separately.

Rate limit hits: How often users are hitting rate limits. If frequent, consider adjusting limits or upgrading plan.