# Changelog
Source: https://docs.monocle.sh/changelog
See what's new in Monocle
# Access & Permissions
Source: https://docs.monocle.sh/features/access-and-permissions
Understand how roles, teams, and application access work in Monocle
Monocle uses a two-layer permission system: **organization** roles for managing the workspace, and **team-based application access** for deciding who can use each app.
The key idea: being part of an organization doesn't automatically grant access to every application. Access is distributed through **teams**.
## Organization roles
Organization roles control who can manage the workspace itself.
| Role | What they can do |
| - | - |
| **Admin** | Manage members, invitations, teams, billing, integrations. Create, edit, and delete applications. Assign team access to apps. Has access to all applications. |
| **Member** | Belongs to the organization but only sees applications their teams have access to. No organization admin capabilities. |
Admins automatically have access to every application in the organization. No team assignment needed.
## Application access
Applications are private to the teams that are assigned to them.
There is no separate read-only role inside an application:
* If a team has access to an app, its members can use that app fully.
* If a team does not have access, its members do not see that app at all.
## Teams
Teams are the glue between members and applications. Instead of granting access user by user, you assign it at the team level.
Here's how it works:
1. A user joins the organization
2. They're added to one or more teams
3. Teams are assigned to specific applications
4. The user inherits access through their teams
## Default team
Every new organization gets a default team called **General**.
* The organization creator is added to `General`
* The first application created in the organization is shared with `General`
* Applications created after that are not shared automatically
This keeps onboarding simple without making every new app visible by default.
### Example
Say you have three teams and three apps:
| Team | Application |
| - | - |
| Backend | api-prod |
| DevOps | api-prod |
| DevOps | internal-tools |
| Finance | billing-service |
This means:
* Backend engineers can operate `api-prod`
* DevOps can operate both `api-prod` and `internal-tools`
* Finance stakeholders can access `billing-service`
## Invitations
When inviting someone to your organization, you pick their organization role first: `Admin` or `Member`.
If you invite them as a **Member**, you can assign them to teams right away. They'll join with the correct application access from day one. No temporary over-permissioning.
If you invite them as an **Admin**, they automatically have access to the whole organization and every application.
# AI Agents
Source: https://docs.monocle.sh/features/ai-agents
Monitor LLM calls, token usage, costs, and conversation flows
Monocle provides a dedicated AI Agents dashboard for monitoring your application's AI activity. Track LLM calls, tool usage, token consumption, estimated costs, and full conversation flows.
## What Monocle tracks
The dashboard has four tabs:
* **Overview**: total calls, token usage, estimated costs, and error rates at a glance
* **Models**: per-model breakdown with latency, token consumption, and cost
* **Tools**: tool call stats with success/error rates and duration
* **Scorers**: evaluation and scorer results with score distributions and explanations
* **Conversations**: multi-turn conversation flows with drill-down into individual runs
## Token tracking
Monocle captures detailed token usage per LLM call, including provider-specific breakdowns when available:
* Input and output tokens
* Cached input tokens (cache hits)
* Cache write tokens
* Reasoning tokens (for models like o1 and o3)
These metrics are used to estimate costs based on published model pricing.
## Conversation tracking
You can group multiple LLM calls into a single conversation by setting a conversation ID. This enables the Conversations tab, where you can follow the full flow of an agentic loop: which models were called, which tools were invoked, and how many tokens were consumed at each step.
## Trace viewer integration
When viewing a trace that contains AI spans, an **AI tab** appears in the span details sidebar. It shows token breakdowns, input/output previews, and model information for that specific call.
## Span attributes
All AI instrumentations must emit these attributes for the AI dashboard to work correctly.
### Standard GenAI semconv attributes
These follow the official [OpenTelemetry GenAI Semantic Conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/):
| Attribute | Description |
| - | - |
| `gen_ai.operation.name` | Operation type (`chat`, `generate_content`, `text_completion`, `invoke_agent`, `execute_tool`, etc.) |
| `gen_ai.system` | LLM provider name |
| `gen_ai.request.model` | Requested model ID |
| `gen_ai.response.model` | Model ID used in response |
| `gen_ai.response.id` | Provider response ID |
| `gen_ai.response.finish_reasons` | Finish reasons array |
| `gen_ai.request.temperature` | Temperature setting |
| `gen_ai.request.max_tokens` | Max tokens setting |
| `gen_ai.request.top_p` | Top-p setting |
### Token usage attributes
| Attribute | Description |
| - | - |
| `gen_ai.usage.input_tokens` | Input tokens consumed |
| `gen_ai.usage.output_tokens` | Output tokens generated |
| `gen_ai.usage.total_tokens` | Computed total (input + output) |
| `gen_ai.usage.input_tokens.cached` | Cached/cache-hit input tokens |
| `gen_ai.usage.input_tokens.cache_write` | Cache write tokens |
| `gen_ai.usage.output_tokens.reasoning` | Reasoning tokens |
### Tool call attributes
| Attribute | Description |
| - | - |
| `gen_ai.tool.name` | Tool name |
| `gen_ai.tool.call.id` | Tool call ID |
| `gen_ai.tool.type` | Always `"function"` |
### Evaluation result attributes
AI evaluations and scorer results should be emitted as span events named `gen_ai.evaluation.result`. The event name is a Monocle convention; the attributes below reuse the OpenTelemetry GenAI namespace.
The core attributes follow the official [OpenTelemetry GenAI attribute registry](https://opentelemetry.io/docs/specs/semconv/registry/attributes/gen-ai/):
| Attribute | Description |
| - | - |
| `gen_ai.evaluation.name` | Evaluation or scorer name |
| `gen_ai.evaluation.score.value` | Numeric score value |
| `gen_ai.evaluation.score.label` | Optional score label such as `pass`, `warning`, or `fail` |
| `gen_ai.evaluation.explanation` | Human-readable explanation for the score |
| `gen_ai.response.id` | Provider response ID being evaluated, when available |
Monocle also recognizes these `gen_ai.evaluation.*` extensions for richer scorer dashboards:
| Attribute | Description |
| - | - |
| `gen_ai.evaluation.id` | Unique evaluation result ID |
| `gen_ai.evaluation.scorer.id` | Stable scorer identifier |
| `gen_ai.evaluation.scorer.name` | Display name for the scorer. Falls back to `gen_ai.evaluation.name` |
| `gen_ai.evaluation.scorer.version` | Scorer version |
| `gen_ai.evaluation.source` | Source of the score, for example `live`, `offline`, or `manual` |
| `gen_ai.evaluation.target.trace_id` | Trace ID of the evaluated AI run |
| `gen_ai.evaluation.target.span_id` | Span ID of the evaluated AI run |
| `gen_ai.evaluation.target.type` | Type of target being evaluated, for example `agent`, `llm`, `tool`, or `workflow` |
| `gen_ai.evaluation.target.name` | Name of the evaluated target |
| `gen_ai.evaluation.trace_id` | Trace ID of the scoring/evaluation run, when scoring is traced separately |
| `gen_ai.evaluation.metadata` | JSON-encoded scorer metadata |
### Monocle attributes
These custom attributes power specific dashboard features:
| Attribute | Dashboard Feature |
| - | - |
| `gen_ai.conversation.id` | Conversations tab, grouping multi-turn interactions |
| `gen_ai.response.model` | Models tab, per-model cost and usage breakdown |
| `gen_ai.tool.name` | Tools tab, per-tool call stats and error rates |
| `gen_ai.usage.*_tokens` | Token consumption charts and cost estimation |
| `gen_ai.usage.input_tokens.cached` | Cache hit rate display and cost savings calculation |
| `gen_ai.function_id` | Pipeline name display |
| `ai.streaming` | Distinguishing streaming vs non-streaming calls |
## Supported SDKs
Auto-instruments generateText, streamText, generateObject, streamObject, embed, embedMany, and rerank
## Getting started
See the instrumentation page above for setup instructions, or the [AdonisJS guide](/frameworks/adonisjs#ai-sdk) if you are using AdonisJS.
# Cache
Source: https://docs.monocle.sh/features/cache
Monitor cache hit rates, latency, and per-key performance
Monocle provides a dedicated Cache dashboard for monitoring your cache layer. See how your cache is performing at a glance and identify which keys need attention.
## What Monocle tracks
* **Hit rate** with color-coded percentage (green for healthy, red for low)
* **Total operations** count and average/P95 latency
* **Hit/miss chart** showing cache efficiency over time
* **Latency chart** tracking response time trends
* **Per-key breakdown** with individual hit/miss ratio and latency metrics
## Key sanitization
Cache keys often contain dynamic values (user IDs, UUIDs, session tokens) that would create high-cardinality attributes. Monocle automatically sanitizes keys by replacing dynamic segments with `*`:
* `users:550e8400-e29b-41d4-a716-446655440000` becomes `users:*`
* `post:12345` becomes `post:*`
* `hash:a1b2c3d4e5f6` becomes `hash:*`
This keeps the per-key table useful instead of showing thousands of unique entries. You can provide a custom sanitizer if the defaults do not match your key format.
## Span attributes
All cache instrumentations must emit these attributes for the Cache dashboard to work correctly.
| Attribute | Dashboard Feature |
| - | - |
| `cache.operation` | Operation type grouping (`get`, `set`, `getOrSet`, `delete`, etc.) |
| `cache.key` | Per-key breakdown table (sanitized) |
| `cache.hit` | Hit rate calculation and hit/miss charts |
| `cache.store` | Cache store identification |
## Supported libraries
Multi-tier caching library for Node.js. Enable it with `@monocle.sh/instrumentation-bentocache`.
## Getting started
See the [AdonisJS guide](/frameworks/adonisjs#cache) for setup instructions if you are using AdonisJS.
# Custom Dashboards
Source: https://docs.monocle.sh/features/custom-dashboards
Build personalized dashboards to visualize any OpenTelemetry metric
Custom Dashboards let you create your own monitoring views by combining widgets that visualize any metric collected by your application. Pick the metrics that matter to you, choose how to display them, and arrange everything on a drag-and-drop grid.
## Widget types
Six visualization types are available:
| Type | Best for | Default size |
| - | - | - |
| **Chart** | Time-series trends (line, area, or bar) | 6×5 |
| **Stat** | Single important number at a glance | 3×3 |
| **Sparkline** | Latest value with a mini trend chart | 3×3 |
| **Table** | Tabular breakdown of metric data | 6×5 |
| **Pie** | Proportional breakdown by attribute | 4×4 |
| **Bar gauge** | Ranked comparison across groups | 4×5 |
**Pie** and **Bar gauge** widgets require a `groupBy` attribute to split the data.
## Configuring widgets
Each widget is backed by an OpenTelemetry metric. You pick:
* **Metric name** — any metric collected by your application (use the metric explorer to discover them)
* **Aggregation** — avg, sum, count, min, max, rate, or percentiles (p50, p90, p95, p99)
* **Group by** — split the data by any attribute (e.g. `service.name`, `host.name`)
* **Filters** — narrow down by attribute values
### Display options
* **Chart type** — line, area, or bar (for chart widgets)
* **Stacking** — stack grouped series on top of each other
* **Format** — number, duration, percentage, or bytes
* **Unit** — custom label for axes and tooltips (e.g. "ms", "req/s", "MB")
* **Color** — pick from a preset palette or use the default auto-coloring
## Grid layout
Dashboards use a **12-column grid**. In edit mode, you can drag widgets to reposition them and resize them from the bottom-right corner. The layout is saved per dashboard.
A few tips for clean layouts:
* Widgets on the same row should have the **same height** to avoid gaps
* Plan each row as a unit — decide the row height first, then size all widgets to match
* Stat and sparkline widgets work well as a row of 4 (3 columns each)
* Chart and table widgets pair nicely as two per row (6 columns each)
## MCP Server
Custom dashboards are fully manageable through the [Monocle MCP Server](/features/mcp). Your AI assistant can create dashboards, add widgets, and update layouts programmatically.
| Tool | Description |
| - | - |
| `list_dashboards` | List all custom dashboards |
| `get_dashboard` | Get a dashboard with all its widgets |
| `create_dashboard` | Create a new empty dashboard |
| `update_dashboard` | Update name, description, or icon |
| `delete_dashboard` | Delete a dashboard and all its widgets |
| `add_widget` | Add a widget with metric, visualization, and layout config |
| `update_widget` | Update an existing widget |
| `remove_widget` | Remove a widget |
Example: *"Create a dashboard called API Health with charts for request latency, error rate, and throughput"*
# Discord
Source: https://docs.monocle.sh/features/integrations/discord
Get uptime alerts delivered to your Discord channels
Connect Discord to get notified in real time when something goes down, and when it comes back up.
## What Gets Sent
Monocle sends alerts to your Discord channel when an uptime monitor detects an incident:
* **Monitor Down**: your service is unreachable or returning errors. Includes the monitor name, URL, and environment.
* **Monitor Recovered**: your service is back online, with the total downtime duration.
## Setup
1. In your Discord server, go to **Server Settings > Integrations** and create a webhook.
2. In Monocle, go to **Organization Settings > Integrations**.
3. Paste the webhook URL under **Discord**.
4. Hit **Test** to make sure it works, then **Save**.
# GitHub
Source: https://docs.monocle.sh/features/integrations/github
Link repositories and create issues from Monocle exceptions
Connect your GitHub repositories to Monocle so you can turn exceptions into GitHub issues without leaving your dashboard.
## Why Connect GitHub?
When an exception pops up in Monocle, you can create a GitHub issue in one click. The issue is pre-filled with the exception details, the affected application, and a link back to the Monocle dashboard. No copy-pasting between tools.
You can also link existing GitHub issues to Monocle exceptions if someone already opened one.
When a linked issue is closed on GitHub, Monocle automatically marks it as resolved. Everything stays in sync.
## Setup
1. Go to **Organization Settings > Integrations** and click **Connect** next to GitHub.
2. You'll be redirected to GitHub to install the Monocle app and select which repositories it can access.
3. Then, in your **Application Settings**, link a specific repository to each Monocle application.
That's it. Your team can now create and link GitHub issues directly from exceptions.
# Linear
Source: https://docs.monocle.sh/features/integrations/linear
Create and sync issues between Monocle and Linear
Connect your Linear workspace to Monocle so your team can create properly categorized issues directly from exceptions.
## Why Connect Linear?
When you spot an exception in Monocle, you can create a Linear issue with the full context: team, project, assignee, priority, labels, and status. Everything is pre-filled with the exception details and a link back to the Monocle dashboard.
You can also search and link existing Linear issues if the bug is already tracked.
When a linked issue is completed in Linear, Monocle automatically marks it as resolved. No manual status updates needed.
## Setup
1. Go to **Organization Settings > Integrations** and click **Connect** next to Linear.
2. Complete the OAuth flow to authorize Monocle.
Once connected, click **Configure** to map each Monocle application to a Linear team. This way, issues created from exceptions automatically land in the right team.
# Slack
Source: https://docs.monocle.sh/features/integrations/slack
Get uptime alerts delivered to your Slack channels
Connect Slack to get notified in real time when something goes down, and when it comes back up.
## What Gets Sent
Monocle sends alerts to your Slack channel when an uptime monitor detects an incident:
* **Monitor Down**: your service is unreachable or returning errors. Includes the monitor name, URL, and environment.
* **Monitor Recovered**: your service is back online, with the total downtime duration.
## Setup
1. [Create an Incoming Webhook](https://api.slack.com/messaging/webhooks) in your Slack workspace and choose the channel where alerts should go.
2. In Monocle, go to **Organization Settings > Integrations**.
3. Paste the webhook URL under **Slack**.
4. Hit **Test** to make sure it works, then **Save**.
# Job Sampling
Source: https://docs.monocle.sh/features/job-sampling
Reduce trace volume for high-throughput jobs while keeping failed executions
Job sampling lets you reduce trace volume for noisy background jobs without disabling job monitoring entirely. It is useful when a small number of jobs run very often or create very large traces.
Job sampling is currently available only through the `@monocle.sh/adonisjs-agent`.
With the Monocle AdonisJS agent, sampling is configured in `config/monocle.ts` with the `Sample` helper
```typescript title="config/monocle.ts" theme={"theme":"vesper"}
import { defineConfig, Sample } from "@monocle.sh/adonisjs-agent";
export default defineConfig({
// ...
sampling(ctx) {
if (ctx.kind !== "job") return Sample.default();
if (ctx.name === "SendNewsletterEmail") {
return Sample.rate(0.01).keepErrors();
}
return Sample.default();
},
});
```
This keeps roughly 1% of successful `SendNewsletterEmail` executions, but still exports failed executions.
## Sampling rules
### Keep default behavior
Use `Sample.default()` when you do not want to override sampling for an execution.
```typescript theme={"theme":"vesper"}
sampling(ctx) {
return Sample.default();
}
```
### Always keep
Use `Sample.always()` when a specific job should always be exported.
```typescript theme={"theme":"vesper"}
sampling(ctx) {
if (ctx.kind === "job" && ctx.name === "BillingReport") {
return Sample.always();
}
return Sample.default();
}
```
### Never keep
Use `Sample.never()` for jobs you do not want to export.
```typescript theme={"theme":"vesper"}
sampling(ctx) {
if (ctx.kind === "job" && ctx.name === "HeartbeatJob") {
return Sample.never();
}
return Sample.default();
}
```
### Sample by rate
Use `Sample.rate(rate)` to keep a percentage of executions. The rate is a number between `0` and `1`.
```typescript theme={"theme":"vesper"}
sampling(ctx) {
if (ctx.kind === "job" && ctx.name === "SyncContacts") {
return Sample.rate(0.1); // keep 10%
}
return Sample.default();
}
```
### Keep failed executions
Use `.keepErrors()` when successful executions should be sampled, but failed executions should still be exported.
```typescript theme={"theme":"vesper"}
sampling(ctx) {
if (ctx.kind === "job" && ctx.name === "SendNewsletterEmail") {
return Sample.rate(0).keepErrors();
}
return Sample.default();
}
```
`Sample.rate(0).keepErrors()` drops successful executions and keeps failed executions. This is the most aggressive setting for high-volume jobs where you mainly care about failures.
## Limit buffered spans
When using `keepErrors()`, the agent buffers spans locally until the job finishes so it can decide whether to export the trace. You can cap the number of buffered spans with `maxSpans`.
```typescript theme={"theme":"vesper"}
sampling(ctx) {
if (ctx.kind === "job" && ctx.name === "SendNewsletterEmail") {
return Sample.rate(0.01).keepErrors({ maxSpans: 500 });
}
return Sample.default();
}
```
If a trace exceeds the limit, Monocle keeps the root span and marks it with truncation attributes:
| Attribute | Description |
| - | - |
| `monocle.sampling.truncated` | `true` when spans were dropped from the local buffer |
| `monocle.sampling.dropped_spans` | Number of spans dropped from the local buffer |
## Sampling context
For job executions, the `sampling(ctx)` callback receives:
| Field | Description |
| - | - |
| `ctx.kind` | Always `"job"` for job executions |
| `ctx.system` | Queue system, for example `"bullmq"` or `"boringqueue"` |
| `ctx.name` | Job name |
| `ctx.queue` | Queue name, when available |
| `ctx.jobId` | Job ID, when available |
| `ctx.attempt` | Retry attempt, when available |
Example:
```typescript theme={"theme":"vesper"}
sampling(ctx) {
if (ctx.kind !== "job") return Sample.default();
if (ctx.system === "bullmq" && ctx.queue === "critical") {
return Sample.always();
}
if (ctx.name.includes("Import")) {
return Sample.rate(0.01).keepErrors();
}
return Sample.default();
}
```
## Supported job libraries
### BullMQ
Install `@monocle.sh/instrumentation-bullmq` and add `bullmq()` to the agent's `integrations` array to enable native BullMQ sampling. The integration inherits the agent's `sampling` callback unless you provide an override.
BullMQ sampling is applied before the worker body runs. When a job is dropped with `Sample.never()` or an unsampled rate rule without `keepErrors()`, Monocle suppresses tracing for the worker execution, so child spans are not created.
See the [BullMQ instrumentation guide](/instrumentations/bullmq) for setup and span details.
### @adonisjs/queue
`@adonisjs/queue` job executions are detected from their root job spans. Sampling works with `Sample.rate()`, `Sample.never()`, and `keepErrors()`.
See the [@boringnode/queue instrumentation guide](/instrumentations/boringnode-queue) for span details.
## How keepErrors works
`keepErrors()` is application-side tail sampling. The agent records spans locally for the job execution, buffers them until the root job span ends, and then exports the trace if the execution was sampled or ended with an error.
Because the decision happens after spans are recorded, Monocle configures the default OpenTelemetry head sampling ratio to `1` when `sampling` is enabled. If you provide a custom OpenTelemetry `sampler`, traces dropped by that sampler cannot be recovered by `keepErrors()`.
# Jobs
Source: https://docs.monocle.sh/features/jobs
Monitor background job execution, failures, and performance
Monocle tracks your background job activity with a dedicated Jobs dashboard. You get visibility into job throughput, error rates, processing times, and queue delays across all your queues.
## What Monocle tracks
* **Throughput** over time with success/error breakdown
* **Error rate** trends to spot problematic jobs
* **Processing time** (average and P95) per job
* **Queue time** between dispatch and processing start
* **Retry attempts** and failure reasons
* **Per-job detail pages** with recent executions and error breakdown
## How it works
Monocle creates two types of spans for each job:
* A **producer span** when a job is dispatched (enqueued)
* A **consumer span** when a job is processed by a worker
These spans carry standardized [OpenTelemetry Messaging Semantic Conventions](https://opentelemetry.io/docs/specs/semconv/messaging/) attributes, so the Jobs dashboard can aggregate metrics regardless of which queue library you use.
## Span attributes
All job instrumentations must emit these attributes for the Jobs dashboard to work correctly.
### Standard semconv attributes
These follow the official [OpenTelemetry Messaging Semantic Conventions](https://opentelemetry.io/docs/specs/semconv/messaging/):
| Attribute | Description |
| - | - |
| `messaging.system` | Queue system identifier (e.g., `"bullmq"`) |
| `messaging.destination.name` | Queue name |
| `messaging.operation.type` | `"send"` for producers, `"process"` for consumers |
| `messaging.operation.name` | Operation name (e.g., `Queue.add`, `Worker.run`) |
| `messaging.message.id` | Job ID |
| `messaging.client.id` | Worker name (consumer spans) |
| `messaging.batch.message_count` | Number of jobs in bulk operations |
| `messaging.message.retry.count` | Number of retry attempts |
### Monocle attributes
These custom attributes power specific dashboard features:
| Attribute | Dashboard Feature |
| - | - |
| `messaging.job.name` | Job name grouping, per-job detail page |
| `messaging.operation.type` | Distinguishing published vs processed jobs in throughput charts |
| `messaging.job.queue_time_ms` | Queue time metrics and charts |
| `entry_point.type` = `"job"` | Identifies the span as a job entry point, used for dashboard routing |
## Supported libraries
Redis-backed job queue with priorities, retries, rate limiting, and flows
Lightweight job queue for Node.js. Instrumentation is built into the Monocle agent when `@boringnode/queue` is installed.
## Getting started
See the individual instrumentation pages above for setup instructions, or the [AdonisJS guide](/frameworks/adonisjs#background-jobs) if you are using AdonisJS.
## Sampling high-volume jobs
If one job runs very often or creates very large traces, use [job sampling](/features/job-sampling) to reduce trace volume for successful executions while keeping failed executions.
```typescript title="config/monocle.ts" theme={"theme":"vesper"}
import { defineConfig, Sample } from "@monocle.sh/adonisjs-agent";
export default defineConfig({
// ...
sampling(ctx) {
if (ctx.kind === "job" && ctx.name === "SendNewsletterEmail") {
return Sample.rate(0.01).keepErrors();
}
return Sample.default();
},
});
```
# Monocle MCP Server
Source: https://docs.monocle.sh/features/mcp
Connect AI assistants to your observability data
Monocle supports the [Model Context Protocol (MCP)](https://modelcontextprotocol.io/), enabling AI assistants like Claude Code, Cursor, and others to interact directly with your observability data. Let your AI tools help you debug issues, analyze traces, and explore your application's telemetry.
## Authentication
Monocle uses [OAuth 2.1](https://datatracker.ietf.org/doc/html/draft-ietf-oauth-v2-1) to authenticate MCP connections. Your MCP client handles the full authorization flow — you just approve access in your browser.
OAuth users are not tied to a single application. You can navigate between organizations, applications, and environments using the `list_apps` and `list_environments` tools.
## Getting Started
```bash theme={"theme":"vesper"}
npx add-mcp https://api.monocle.sh/mcp
```
This works with any MCP client that supports OAuth (Claude Code, Claude Desktop, Cursor, etc.).
### Start Asking Questions
Once connected, you can ask your AI assistant questions about your application:
* "What exceptions occurred in the last hour?"
* "Show me the slowest API endpoints"
* "Find traces with errors from the checkout service"
* "What's the P95 latency for the /api/users endpoint?"
## Available Tools
Your AI assistant has access to the following tools:
### Navigation
| Tool | Description |
| - | - |
| `whoami` | Get current authentication context (org, app, environment) |
| `list_apps` | List applications in your organization |
| `list_environments` | List available environments for your application |
### Traces
| Tool | Description |
| - | - |
| `list_traces` | List distributed traces with filters (time range, status, service, search) |
| `get_trace` | Get all spans for a specific trace by ID |
### Logs
| Tool | Description |
| - | - |
| `list_logs` | List application logs with filtering (severity, service, time range, trace ID) |
| `search_logs` | Full-text search across log messages |
### Exceptions
| Tool | Description |
| - | - |
| `list_exceptions` | List exception groups with occurrence counts and affected users |
| `get_exception` | Get detailed info about an exception group (stacktrace, timeline, affected routes) |
| `archive_exception` | Archive an exception group to hide it from the default list (duration: 1h, 1d, 1w, 30d, or forever) |
| `create_issue` | Create an issue to track one or more exceptions with priority and assignees |
| `update_issue` | Update an issue's status (open, resolved, ignored), priority, or assignees |
### Metrics
| Tool | Description |
| - | - |
| `list_instances` | List service instances (pods/replicas) reporting metrics |
| `list_metrics` | Discover all available metrics (gauges, sums, histograms, summaries) with their types and units |
| `get_metric` | Query time-series data for any metric with aggregation (avg, sum, min, max, p50–p99), filtering, and grouping |
| `get_metric_attributes` | Discover attribute keys and values for a metric to use as filters or groupBy in `get_metric` |
### Endpoints
| Tool | Description |
| - | - |
| `list_endpoints` | List HTTP endpoint patterns with request count, avg/p95 duration, and error rate. Supports sorting, search, and method filtering |
| `get_endpoint_stats` | Get detailed stats for a specific endpoint: latency percentiles, error rates, throughput, and time-series data |
### Database Queries
| Tool | Description |
| - | - |
| `list_db_queries` | List database query patterns with execution count, avg/p95/max duration, error count, and slow query detection |
| `get_db_query_detail` | Get details for a specific query pattern: recent executions with trace IDs and related HTTP routes |
### Background Jobs
| Tool | Description |
| - | - |
| `list_jobs` | List background job executions with published/processed counts, error rate, and avg/p95 processing time |
| `get_job_detail` | Get details for a specific job: recent executions with duration, queue time, status, and error messages |
### Cache
| Tool | Description |
| - | - |
| `list_cache_stats` | List cache keys with hit/miss rates, operation counts, and avg/p95 latency |
| `get_cache_key_detail` | Get detailed metrics for a specific cache key: hit/miss time series, duration, and operation breakdown |
### External Calls
| Tool | Description |
| - | - |
| `list_external_calls` | List third-party API domains with request count, avg/p95 latency, error rate, and status code breakdown |
| `get_external_call_detail` | Get details for a specific domain: URL patterns, recent requests with trace IDs, and performance metrics |
### Custom Dashboards
Build and manage custom dashboards with metric widgets directly from your AI assistant. Dashboards use a 12-column grid layout.
| Tool | Description |
| - | - |
| `list_dashboards` | List all custom dashboards with widget counts |
| `get_dashboard` | Get a dashboard with all its widgets (data sources, display config, layout) |
| `create_dashboard` | Create a new empty dashboard |
| `update_dashboard` | Update a dashboard's name, description, or icon |
| `delete_dashboard` | Permanently delete a dashboard and all its widgets |
| `add_widget` | Add a widget to a dashboard (chart, stat, table, sparkline, pie, or bar-gauge) |
| `update_widget` | Update a widget's metric, aggregation, display, or layout |
| `remove_widget` | Remove a widget from a dashboard |
Dashboard tools that create or modify data (`create_dashboard`, `update_dashboard`, `delete_dashboard`, `add_widget`, `update_widget`, `remove_widget`) require the `mcp:write` scope.
## Example Queries
* *What exceptions occurred in the last hour?*
* *What's the P95 latency for POST /api/orders?*
* *Search logs for "payment failed"*
* *Which database queries are the slowest?*
* *Create an issue for the recurring TypeError in checkout*
* *Create a dashboard with charts for request latency and error rate*
# MCP Monitoring
Source: https://docs.monocle.sh/features/mcp-monitoring
Monitor how your MCP server is being used by AI tools
If your application exposes an MCP server, Monocle provides a dedicated dashboard for tracking how AI tools interact with it. See which tools are called, by whom, and how they perform.
This page is about monitoring **your** MCP server. If you are looking for how
to connect AI tools (Claude Code, Cursor, etc.) to **Monocle's** MCP server,
see the [Monocle MCP Server](/features/mcp) page.
## What Monocle tracks
* **Overview stats**: total tool calls, average and P95 latency, error rate, unique users and clients
* **Calls and duration charts**: time-series showing success/error distribution and latency trends
* **Clients table**: which MCP clients are making calls, with volume and timing data
* **Users table**: which users are using the MCP server
* **Tools table**: breakdown by tool with call count, latency percentiles, and error rate. Click any tool to see its detailed metrics and recent calls with links to full traces.
## Span attributes
Since there is no official OpenTelemetry semconv for MCP yet, attributes use the `mcp.*` namespace. All MCP instrumentations must emit these attributes for the dashboard to work correctly.
### Core attributes
| Attribute | Description |
| - | - |
| `mcp.method.name` | JSON-RPC method (e.g., `tools/call`) |
| `mcp.request.id` | JSON-RPC request ID |
| `mcp.session.id` | Transport session ID |
| `mcp.transport` | Transport class name |
### Identity attributes
| Attribute | Description |
| - | - |
| `mcp.server.name` | Server name |
| `mcp.server.version` | Server version |
| `mcp.client.name` | Client name (from `initialize` request) |
| `mcp.client.version` | Client version |
| `mcp.protocol.version` | Negotiated protocol version |
### Target attributes
| Attribute | When |
| - | - |
| `mcp.tool.name` | `tools/call` requests |
| `mcp.resource.uri` | `resources/*` requests |
| `mcp.prompt.name` | `prompts/get` requests |
### Monocle attributes
| Attribute | Dashboard Feature |
| - | - |
| `mcp.tool.name` | Tool breakdown table, per-tool detail page |
| `mcp.client.name` | Clients table |
| `mcp.tool.result.is_error` | Error rate calculations |
### Error attributes
These follow the [RPC Semantic Conventions](https://opentelemetry.io/docs/specs/semconv/rpc/json-rpc/):
| Attribute | Description |
| - | - |
| `rpc.jsonrpc.error_code` | JSON-RPC error code |
| `rpc.jsonrpc.error_message` | JSON-RPC error message |
## Setup
Install and configure @monocle.sh/instrumentation-mcp
# Uptime Monitoring
Source: https://docs.monocle.sh/features/uptime-monitoring
Monitor your services availability with health checks
Monocle can monitor your services availability by periodically checking your health endpoints. Get notified when something goes wrong and track your uptime over time.
## How It Works
Monocle pings your configured URL **once per minute**. By default, it simply checks if your endpoint responds with a `2xx` status code. But with **Health Mode** enabled, Monocle goes further by parsing the response body to understand the detailed health status of your services.
## Health Mode
Health Mode allows Monocle to understand what's happening inside your application. Instead of just checking "is the server responding?", it parses a structured health response to detect degraded services—even when the endpoint technically returns a `200 OK`.
For example, your API might respond successfully while your database connection is failing. With Health Mode, Monocle detects this and creates an incident.
### Response Format
Your health endpoint must return a JSON response matching this structure:
```json theme={"theme":"vesper"}
{
"isHealthy": true,
"status": "ok",
"finishedAt": "2026-01-01T20:50:11.945Z",
"checks": [
{
"name": "Database health check (postgres)",
"isCached": false,
"message": "Successfully connected to the database server",
"status": "ok",
"finishedAt": "2026-01-01T20:50:11.945Z",
"meta": {
"connection": {
"name": "postgres",
"dialect": "postgres"
}
}
},
{
"name": "Redis health check (main)",
"isCached": false,
"message": "Successfully connected to the redis server",
"status": "ok",
"finishedAt": "2026-01-01T20:50:11.944Z",
"meta": {
"connection": {
"name": "main",
"status": "ready"
}
}
}
]
}
```
| Field | Type | Description |
| - | - | - |
| `isHealthy` | boolean | Overall health status |
| `status` | string | `"ok"`, `"warning"`, or `"error"` |
| `finishedAt` | string | ISO 8601 timestamp |
| `checks` | array | Individual health check results |
| `checks[].name` | string | Name of the health check |
| `checks[].status` | string | `"ok"`, `"warning"`, `"failed"`, `"crashed"`, or `"skipped"` |
| `checks[].message` | string | Human-readable status message |
| `checks[].meta` | object | Optional metadata about the check |
### AdonisJS Integration
If you're using AdonisJS, this format is **natively supported** by the health checks module. Follow the [AdonisJS Health Checks guide](https://docs.adonisjs.com/guides/digging-deeper/health-checks) to set up your health endpoint.
Once configured, your `/health` endpoint will automatically return the expected format and work seamlessly with Monocle's Health Mode.
## Configuring a Monitor
Navigate to the **Uptime** section in your dashboard and click **Configure** to set up your monitor.
### Basic Settings
| Setting | Description |
| - | - |
| **URL** | The health endpoint URL to monitor |
| **Health Mode** | Enable to parse the structured health response |
### Advanced Settings
| Setting | Default | Description |
| - | - | - |
| **Timeout** | 10s | How long to wait before considering a check failed (1-30 seconds) |
| **Custom Headers** | - | Add custom HTTP headers (useful for authentication) |
| **Failure Tolerance** | 3 | Number of consecutive failures before creating an incident |
| **Recovery Tolerance** | 1 | Number of consecutive successes before resolving an incident |
### Custom Headers
If your health endpoint requires authentication, you can add custom headers:
```
Authorization: Bearer your-api-token
X-Custom-Header: some-value
```
## Check Details
Click on any check in the uptime timeline to see detailed results. When Health Mode is enabled, you'll see the status of each individual health check (database, Redis, disk space, etc.).
This makes it easy to understand exactly what failed and why, without digging through logs.
## Incident Thresholds
To avoid false alarms from temporary network issues, Monocle uses tolerance thresholds:
* **Failure Tolerance**: An incident is only created after this many consecutive failures. Set it higher if you experience occasional network hiccups.
* **Recovery Tolerance**: An incident is resolved after this many consecutive successes. Keep it at 1 for fast recovery notifications, or increase it to ensure the service is truly stable.
# AdonisJS
Source: https://docs.monocle.sh/frameworks/adonisjs
Install and configure the Monocle agent in your AdonisJS application
This guide covers everything you need to instrument your AdonisJS application with Monocle. You will learn how to:
* Install and configure the agent
* Track authenticated users across traces
* Capture exceptions and custom messages
* Use tracing helpers and decorators
* Configure instrumentations (mail, cache, jobs, AI, MCP)
* Fan out telemetry to multiple destinations
## Overview
The agent includes Adonis instrumentation and the Node auto-instrumentation bundle. Add Mastra, Vercel AI, BullMQ, Bentocache, and MCP support explicitly so your application only installs the integrations it uses.
This guide uses `@monocle.sh/adonisjs-agent` 2.x. Upgrading from 1.x? Follow the [migration guide](/migrations/adonisjs-2.0.0).
Mail, Adonis queue, and Ace CLI tracing are built in. They activate automatically when their packages are available and keep their top-level `mail`, `queue`, and `cli` options. Optional mail and queue packages are resolved from your application; the agent does not install them.
## Prerequisites
* AdonisJS 6.2.0+ or AdonisJS 7.0.0+
* A Monocle account with an API key
## Installation
Install and configure the package using the Ace CLI.
```bash theme={"theme":"vesper"}
node ace add @monocle.sh/adonisjs-agent
```
1. Creates `config/monocle.ts` configuration file
2. Creates `otel.ts` initialization file at project root
3. Adds the `otel.ts` import as the first import in `bin/server.ts` and `bin/console.ts`
4. Registers the Monocle provider in `adonisrc.ts`
5. Registers the Monocle middleware as the first router middleware
6. Adds required environment variables to `.env` and `start/env.ts`
After running the command, add your API key to `.env`:
```bash title=".env" theme={"theme":"vesper"}
MONOCLE_API_KEY=mk_live_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
```
You can find your API key in your application settings on the Monocle dashboard.
That's it. Your application now has automatic tracing for HTTP requests, database queries, Redis operations, and more.
**The import order is critical.**
OpenTelemetry must initialize before any other code loads. The SDK needs to patch libraries like `http`, `pg`, and `redis` before they are imported. That is why `otel.ts` is imported as the very first line in `bin/server.ts`.
If you move or remove the `import '../otel.js'` line, auto-instrumentation will not work.
## Environment Variables
The agent requires these environment variables:
| Variable | Description | Required |
| - | - | - |
| `MONOCLE_API_KEY` | Your Monocle API key | Yes |
| `APP_NAME` | Service name for identification in Monocle | Yes |
| `APP_VERSION` | Service version (e.g., git sha, semver) | Yes |
| `APP_ENV` | Environment: `development`, `staging`, or `production` | Yes |
The `APP_ENV` variable determines which environment your telemetry data is
associated with in Monocle. You can filter traces, logs, and metrics by
environment using the environment selector in the dashboard.
## Configuration
The configuration file is located at `config/monocle.ts`. Here is a minimal setup:
```typescript title="config/monocle.ts" theme={"theme":"vesper"}
import { defineConfig } from "@monocle.sh/adonisjs-agent";
import env from "#start/env";
export default defineConfig({
apiKey: env.get("MONOCLE_API_KEY"),
serviceName: env.get("APP_NAME"),
serviceVersion: env.get("APP_VERSION"),
environment: env.get("APP_ENV"),
});
```
### Full configuration reference
All options are optional except `apiKey`, `serviceName`, `serviceVersion`, and `environment`.
```typescript title="config/monocle.ts" theme={"theme":"vesper"}
import { defineConfig } from "@monocle.sh/adonisjs-agent";
import env from "#start/env";
export default defineConfig({
apiKey: env.get("MONOCLE_API_KEY"),
serviceName: env.get("APP_NAME"),
serviceVersion: env.get("APP_VERSION"),
environment: env.get("APP_ENV"),
// Custom ingestion endpoint (default: https://ingest.monocle.sh)
endpoint: "https://ingest.monocle.sh",
// Skip OPTIONS/CORS preflight requests (default: true)
ignoreOptionsRequests: true,
// Enable gzip compression (default: true)
compression: true,
// Trace batching configuration
batch: {
maxExportBatchSize: 512,
scheduledDelayMillis: 5000,
exportTimeoutMillis: 30000,
maxQueueSize: 2048,
},
// Host metrics: CPU, memory, network (default: enabled)
hostMetrics: {
enabled: true,
},
// Sampling ratio (0.0 to 1.0). Use 0.1 for high-traffic production.
// Default: 1.0 (100% of traces sampled)
samplingRatio: 1.0,
// Adonis features (enabled by default when the corresponding libraries are available)
mail: { enabled: true },
queue: { enabled: true },
cli: { enabled: true },
// Install and register optional integrations below as needed
integrations: [],
});
```
### Customizing instrumentations
You can configure individual OpenTelemetry instrumentations by passing options keyed by their package name.
```typescript title="config/monocle.ts" theme={"theme":"vesper"}
export default defineConfig({
// ...
instrumentations: {
"@opentelemetry/instrumentation-http": {
// Add URLs to ignore (merged with defaults by default)
ignoredUrls: ["/internal/*", "/api/ping"],
// Set to false to replace default ignored URLs instead of merging
mergeIgnoredUrls: true,
// Ignore static files like css, js, images (default: true)
ignoreStaticFiles: true,
},
// Disable a specific instrumentation entirely
"@opentelemetry/instrumentation-pg": { enabled: false },
},
});
```
Default ignored URLs include `/health`, `/healthz`, `/ready`, `/readiness`, `/metrics`, `/favicon.ico` and other common health check paths.
## User Context
To associate telemetry data with authenticated users, call `Monocle.setUser()` in your authentication middleware. This lets you filter traces and exceptions by user in the dashboard.
```typescript title="app/middleware/silent_auth_middleware.ts" theme={"theme":"vesper"}
import type { NextFn } from "@adonisjs/core/types/http";
import type { HttpContext } from "@adonisjs/core/http";
import { Monocle } from "@monocle.sh/adonisjs-agent";
export default class SilentAuthMiddleware {
async handle(ctx: HttpContext, next: NextFn) {
await ctx.auth.check();
if (ctx.auth.user) {
Monocle.setUser({
id: ctx.auth.user.id,
email: ctx.auth.user.email,
name: ctx.auth.user.fullName,
});
}
return next();
}
}
```
The `name` field takes priority in the Monocle UI, falling back to `email`, then `id`.
Do not include sensitive data (passwords, tokens, API keys) in user
attributes. These values are sent to Monocle and will be visible in trace
viewers.
## Exception Tracking
Exceptions thrown during a request are automatically captured by the agent. It hooks into AdonisJS's `ExceptionHandler.report()` method and respects your ignore rules (`ignoreExceptions`, `ignoreStatuses`, `ignoreCodes`).
### Manual exception capture
You can manually capture exceptions anywhere in your code.
```typescript title="app/services/payment_service.ts" theme={"theme":"vesper"}
import { Monocle } from "@monocle.sh/adonisjs-agent";
try {
await processPayment(order);
} catch (error) {
Monocle.captureException(error, {
user: { id: "123", email: "user@example.com" },
tags: { component: "payment" },
extra: { orderId: order.id, amount: order.total },
});
throw error;
}
```
### Capture messages
Use `Monocle.captureMessage()` to capture custom messages when you want to track something that is not an exception. Messages appear in the Exceptions dashboard alongside errors.
```typescript title="app/services/order_service.ts" theme={"theme":"vesper"}
import { Monocle } from "@monocle.sh/adonisjs-agent";
// Simple message
Monocle.captureMessage("User failed to complete checkout");
// With severity level and context
Monocle.captureMessage("Order processing took too long", {
level: "warning",
user: { id: "123" },
tags: { feature: "checkout" },
extra: { orderId: "order-456", durationMs: 15000 },
});
```
Supported levels: `debug`, `info`, `log`, `warning`, `error`, `fatal`.
## Tracing Helpers
### Creating custom spans
Use `record()` to wrap a section of code in a span. This is useful when you want to measure the duration of a specific operation that is not automatically instrumented.
```typescript title="app/services/report_service.ts" theme={"theme":"vesper"}
import { record } from "@monocle.sh/adonisjs-agent/helpers";
const report = await record("report.generate", async (span) => {
span.setAttributes({ "report.type": "monthly", "report.userId": userId });
const data = await fetchReportData(userId);
return generatePdf(data);
});
```
`record()` handles both sync and async callbacks, automatically records exceptions, and closes the span when done.
### Decorators
The `@span()` decorator wraps a single method in a span. The `@spanAll()` decorator wraps every public method in a class.
```typescript title="app/services/order_service.ts" theme={"theme":"vesper"}
import { inject } from "@adonisjs/core";
import { span, spanAll } from "@monocle.sh/adonisjs-agent/decorators";
// Wrap a single method
export class OrderService {
@span()
async findById(id: string) {
return Order.findOrFail(id);
}
@span({ name: "order.create", attributes: { operation: "create" } })
async create(data: CreateOrderData) {
return Order.create(data);
}
}
// Or wrap all methods at once
@spanAll({ prefix: "user" })
export class UserService {
async findById(id: string) { /* span: user.findById */ }
async create(data: UserData) { /* span: user.create */ }
}
```
### Other helpers
```typescript theme={"theme":"vesper"}
import {
getCurrentSpan,
setAttributes,
recordEvent,
injectTraceContext,
extractTraceContext,
} from "@monocle.sh/adonisjs-agent/helpers";
// Get the current active span
const span = getCurrentSpan();
// Add attributes to the current span
setAttributes({ "order.total": 99.99, "order.currency": "EUR" });
// Record an event on the current span
recordEvent("order.confirmed", { orderId: "123" });
// Distributed tracing: inject context into outgoing headers
const headers = {};
injectTraceContext(headers);
// Distributed tracing: extract context from incoming headers
const context = extractTraceContext(request.headers);
```
## Instrumentations
### HTTP
HTTP tracing is configured with the top-level `http` option.
Alongside the standard HTTP instrumentation options, Monocle adds few specific features:
* `sanitizeUrls` to redact sensitive query parameters from URL span attributes
* `captureBodies` to record request and response payloads on spans
Both features are disabled by default. Configure `sanitizeUrls` to enable URL sanitization. For body capture, set `captureBodies.enabled: true`, or opt in per side with `incoming.request.enabled`, `incoming.response.enabled`, and so on.
```typescript title="config/monocle.ts" theme={"theme":"vesper"}
export default defineConfig({
// ...
// High-level HTTP tracing options.
http: {
// Ignore noisy routes before spans are exported.
ignoredUrls: ["/internal/*", "/api/ping"],
// Redact sensitive query params in URL span attributes.
sanitizeUrls: {
queryParams: ["code", "state", "token", "id_token", "session_state"],
censor: "REDACTED",
},
// Capture request and response payloads on HTTP spans.
captureBodies: {
enabled: true,
// Server-side traffic handled by your Adonis app.
incoming: {
request: {
// Limit the number of bytes stored on the span.
maxBodySize: 16_384,
},
response: {},
},
// Client-side traffic instrumented by node:http / node:https.
outgoing: {
request: {},
response: {},
},
// Declarative redaction merged with Monocle's built-in safe defaults.
redact: {
bodyKeys: ["password", "token", "secret"],
headers: ["authorization", "cookie"],
queryParams: ["token"],
censor: "REDACTED",
},
// Final hook to mutate or replace the captured payload.
transform: (record) => {
if (!record.http.url?.includes("/oauth/callback")) return;
record.http.body = { redacted: true };
},
},
},
});
```
#### URL sanitization
`sanitizeUrls` rewrites URL span attributes such as `http.url`, `url.full`, `http.target`, and `url.query`.
```typescript title="config/monocle.ts" theme={"theme":"vesper"}
export default defineConfig({
// ...
http: {
sanitizeUrls: {
queryParams: ["code", "state", "id_token", "session_state"],
},
},
});
```
For example, an OAuth callback like:
```text theme={"theme":"vesper"}
/oauth/callback?code=oauth-code&state=oauth-state&id_token=jwt-token
```
becomes:
```text theme={"theme":"vesper"}
/oauth/callback?code=REDACTED&state=REDACTED&id_token=REDACTED
```
For more control, you can provide a custom `sanitize` function that takes the full URL object and returns a sanitized string.
```typescript title="config/monocle.ts" theme={"theme":"vesper"}
export default defineConfig({
// ...
http: {
sanitizeUrls: {
sanitize: (url) => {
// Only keep the pathname for OAuth callbacks
if (url.pathname === "/oauth/callback") return url.pathname;
// Otherwise, keep the full URL
return url.href;
},
},
},
});
```
#### Body capture scope
`incoming` captures traffic handled by your Adonis app:
* `incoming.request` for the HTTP request body sent to your server
* `incoming.response` for the HTTP response body returned by your server
`outgoing` captures traffic created by Node HTTP clients inside your app:
* `outgoing.request` for the request body your app sends to another service
* `outgoing.response` for the response body returned by that service
`captureBodies.outgoing` only applies to traffic instrumented by
`@opentelemetry/instrumentation-http`, which means `node:http` and
`node:https`. It does not cover `fetch` or `undici`.
#### Body capture defaults
`captureBodies` is opt-in because payload capture can expose sensitive data.
When enabled, Monocle:
* captures `application/json` and `application/x-www-form-urlencoded` by default
* stores at most `10_000` bytes per request or response unless you increase `maxBodySize`
* shows captured payloads in the **HTTP Body** tab of the trace details view
Body capture should be enabled deliberately and reviewed carefully. Payloads can
contain credentials or sensitive information. Keep capture scoped to the
endpoints you need, prefer the smallest possible `maxBodySize`, and use
`redact` or `transform` to remove anything sensitive before it is exported.
#### Redaction order
`captureBodies.redact` applies declarative redaction rules first.
`captureBodies.transform` runs after that and lets you mutate or replace the captured payload programmatically.
The declarative `redact` rules extend Monocle's built-in safe defaults. Common secrets such as `password`, `token`, `code`, `access_token`, `refresh_token`, `authorization`, and many others are redacted automatically even if you do not list them yourself.
#### Advanced overrides
If you need low-level OpenTelemetry overrides, you can still configure `instrumentations["@opentelemetry/instrumentation-http"]`.
When both are present, the low-level instrumentation config takes priority over the top-level `http` option.
### Mail
If `@adonisjs/mail` is installed, the agent automatically instruments all emails sent through the mail service. Every email appears as a span in your traces with subject, recipients, and transport info. You can monitor email activity in the dedicated Emails dashboard.
To disable:
```typescript title="config/monocle.ts" theme={"theme":"vesper"}
export default defineConfig({
// ...
mail: false,
});
```
### Cache
Install the Bentocache integration explicitly to trace cache operations, including when using `@adonisjs/cache`. Keep `bentocache` as a direct application dependency so the integration uses the same instance as your cache service.
```bash title="Terminal" theme={"theme":"vesper"}
npm install @monocle.sh/instrumentation-bentocache bentocache
```
```typescript title="config/monocle.ts" theme={"theme":"vesper"}
import { defineConfig } from "@monocle.sh/adonisjs-agent";
import { bentocache } from "@monocle.sh/instrumentation-bentocache/integration";
import env from "#start/env";
export default defineConfig({
apiKey: env.get("MONOCLE_API_KEY"),
integrations: [bentocache()],
});
```
The defaults preserve the former `cache` configuration: keys are captured and sanitized, parent spans are not required, and internal Redis/L2 spans are suppressed. UUIDs, numeric IDs, and hex hashes in keys become `*`.
Pass your previous cache options to `bentocache()`. Omit the integration or use `bentocache({ enabled: false })` to disable it. See the [Bentocache guide](/instrumentations/bentocache) for customization.
### Background Jobs
#### @adonisjs/queue
If `@adonisjs/queue` (boringqueue) is installed, the agent instruments job dispatch and execution. Jobs appear in the Jobs dashboard with execution count, error rate, and latency metrics.
```typescript title="config/monocle.ts" theme={"theme":"vesper"}
export default defineConfig({
// ...
queue: {
// How consumer spans relate to producer spans
// 'link': consumer span links to producer (default)
// 'parent': consumer span is a child of the producer span
executionSpanLinkMode: "link",
},
});
```
#### BullMQ
Install `@monocle.sh/instrumentation-bullmq` in your BullMQ application, then register `bullmq()` to trace job dispatch and processing.
```bash title="Terminal" theme={"theme":"vesper"}
npm install @monocle.sh/instrumentation-bullmq
```
```typescript title="config/monocle.ts" theme={"theme":"vesper"}
import { defineConfig } from "@monocle.sh/adonisjs-agent";
import { bullmq } from "@monocle.sh/instrumentation-bullmq/integration";
import env from "#start/env";
export default defineConfig({
apiKey: env.get("MONOCLE_API_KEY"),
integrations: [
bullmq({ useProducerSpanAsConsumerParent: false }),
],
});
```
Consumer spans link to producer spans by default. Set `useProducerSpanAsConsumerParent: true` to keep both in the same trace. See the [BullMQ guide](/instrumentations/bullmq) for all options.
Omit `bullmq()` to disable the Monocle BullMQ integration. Use `queue: false` to disable the separate Adonis queue integration. The Node auto-instrumentation bundle can still trace underlying Redis operations independently.
#### Sampling noisy jobs
Use the `sampling(ctx)` hook to reduce trace volume for high-throughput jobs while keeping failed executions.
```typescript title="config/monocle.ts" theme={"theme":"vesper"}
import { defineConfig, Sample } from "@monocle.sh/adonisjs-agent";
export default defineConfig({
// ...
sampling(ctx) {
if (ctx.kind === "job" && ctx.name === "SendNewsletterEmail") {
return Sample.rate(0.01).keepErrors();
}
return Sample.default();
},
});
```
See [Job Sampling](/features/job-sampling) for all sampling rules and caveats.
### AI SDK
Install the Vercel AI integration and register `vercelAi()` to trace AI SDK calls, token usage, and costs.
```bash title="Terminal" theme={"theme":"vesper"}
npm install @monocle.sh/instrumentation-vercel-ai
```
```typescript title="config/monocle.ts" theme={"theme":"vesper"}
import { defineConfig } from "@monocle.sh/adonisjs-agent";
import { vercelAi } from "@monocle.sh/instrumentation-vercel-ai/integration";
import env from "#start/env";
export default defineConfig({
apiKey: env.get("MONOCLE_API_KEY"),
integrations: [
vercelAi({ recordInputs: false, recordOutputs: false }),
],
});
```
Omit `vercelAi()` to disable this integration. See the [Vercel AI guide](/instrumentations/vercel-ai) for supported operations and recording options.
#### Conversation tracking
Import `withConversationId()` directly from the integration package to group AI calls in the Conversations tab.
```typescript title="app/services/chat_service.ts" theme={"theme":"vesper"}
import { generateText } from "ai";
import { openai } from "@ai-sdk/openai";
import { withConversationId } from "@monocle.sh/instrumentation-vercel-ai";
export async function reply(conversationId: string, prompt: string) {
return withConversationId(conversationId, () =>
generateText({ model: openai("gpt-4o"), prompt }),
);
}
```
### Mastra
Install the Mastra integration and register `mastra()` to normalize spans from your Mastra OpenTelemetry bridge. Configure the bridge using the [Mastra guide](/instrumentations/mastra).
```bash title="Terminal" theme={"theme":"vesper"}
npm install @monocle.sh/instrumentation-mastra
```
```typescript title="config/monocle.ts" theme={"theme":"vesper"}
import { defineConfig } from "@monocle.sh/adonisjs-agent";
import { mastra } from "@monocle.sh/instrumentation-mastra/integration";
import env from "#start/env";
export default defineConfig({
apiKey: env.get("MONOCLE_API_KEY"),
integrations: [
mastra({ recordInputs: false, recordOutputs: false }),
],
});
```
Mastra recording options are independent of `vercelAi()` options. Set them on both integrations if you use both. Omit `mastra()` to disable normalization.
### CLI Commands
Ace CLI command execution is traced by default. Scaffolding commands (`make:*`, `generate:*`) and long-running queue workers are excluded automatically.
Command arguments and flags are captured as span attributes (`cli.args.*`, `cli.flags.*`).
```typescript title="config/monocle.ts" theme={"theme":"vesper"}
export default defineConfig({
// ...
cli: {
// Glob patterns to include (all commands by default)
include: ["migration:*", "db:*"],
// Glob patterns to exclude (defaults below)
exclude: ["make:*", "generate:*", "queue:work", "queue:listen"],
},
});
```
To disable entirely:
```typescript title="config/monocle.ts" theme={"theme":"vesper"}
export default defineConfig({
// ...
cli: false,
});
```
**CLI tracing requires `otel.ts` in `bin/console.ts`.**
Just like `bin/server.ts`, your `bin/console.ts` file must import `otel.ts` as its very first import for CLI instrumentation to work. The `node ace configure` command adds this automatically, but if you set up manually or upgraded from an older version, make sure it is there:
```typescript title="bin/console.ts" theme={"theme":"vesper"}
import '../otel.js' // Must be the first import
import { Kernel } from '@adonisjs/core/ace'
// ...
```
### MCP Server
Install the MCP integration explicitly, then attach it to your server instance. MCP does not need an `integrations` entry.
```bash title="Terminal" theme={"theme":"vesper"}
npm install @monocle.sh/instrumentation-mcp
```
```typescript title="app/mcp/server.ts" theme={"theme":"vesper"}
import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { instrumentMcpServer } from "@monocle.sh/instrumentation-mcp";
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
const server = new McpServer({
name: "my-mcp-server",
version: "1.0.0",
});
// Instrument before registering tools
instrumentMcpServer(server, {
// Capture tool arguments in spans (default: false)
recordInputs: true,
// Capture tool results in spans (default: false)
recordOutputs: true,
});
// Register tools, then connect transport as usual
server.tool("ping", async () => ({ content: [{ type: "text", text: "pong" }] }));
await server.connect(new StdioServerTransport());
```
## Multi-Destination OTLP
You can send telemetry to additional OTLP backends alongside Monocle. The Monocle destination is always injected automatically.
```typescript title="config/monocle.ts" theme={"theme":"vesper"}
import { defineConfig, destinations } from "@monocle.sh/adonisjs-agent";
export default defineConfig({
// ...
destinations: [
destinations.otlp({
name: "grafana",
url: "https://otlp-gateway.grafana.net/otlp",
headers: {
Authorization: `Basic ${btoa("instance_id:api_key")}`,
},
}),
],
});
```
Each destination receives the full telemetry data (traces, metrics, logs).
## Disabling Telemetry
If no API key is provided, telemetry is automatically disabled. This is useful for local development.
```bash title=".env.development" theme={"theme":"vesper"}
MONOCLE_API_KEY=
```
## Migrating from @adonisjs/otel
If you are already using `@adonisjs/otel`, switching to the Monocle agent is straightforward. The agent wraps `@adonisjs/otel` and provides the same API with additional capabilities.
### 1. Swap the package
```bash theme={"theme":"vesper"}
node ace remove @adonisjs/otel
node ace add @monocle.sh/adonisjs-agent
```
### 2. Rename config file
Rename `config/otel.ts` to `config/monocle.ts` and replace `defineConfig` import:
```typescript title="config/monocle.ts" theme={"theme":"vesper"}
import { defineConfig } from "@adonisjs/otel"; // [!code --]
import { defineConfig } from "@monocle.sh/adonisjs-agent"; // [!code ++]
```
### 3. Update all imports
Replace all `@adonisjs/otel` imports with their Monocle equivalents:
```typescript theme={"theme":"vesper"}
import { record, getCurrentSpan } from "@adonisjs/otel/helpers"; // [!code --]
import { record, getCurrentSpan } from "@monocle.sh/adonisjs-agent/helpers"; // [!code ++]
import { span, spanAll } from "@adonisjs/otel/decorators"; // [!code --]
import { span, spanAll } from "@monocle.sh/adonisjs-agent/decorators"; // [!code ++]
```
### Why use Monocle's helpers
The Monocle agent re-exports most helpers from `@adonisjs/otel` unchanged (`getCurrentSpan`, `setAttributes`, `recordEvent`, etc.). However, `record`, `handleError`, `@span`, and `@spanAll` are enhanced versions that add **source code context extraction**.
When an exception is caught by these helpers, the agent reads the actual source files from disk and attaches the surrounding lines of code to the span. This is what powers the source code preview in the Monocle exception viewer. You get the exact line that errored with surrounding context, without needing source maps.
The agent also fully serializes the **error cause chain** (`Error.cause`), including `AggregateError` children, so nested errors are never lost.
These enhancements only work when importing from `@monocle.sh/adonisjs-agent`. If you keep using `@adonisjs/otel` imports directly, you will get standard OpenTelemetry error recording without the extra context.
# Effect
Source: https://docs.monocle.sh/frameworks/effect
Install and configure the Monocle agent in your Effect application
The Effect agent is experimental and under active development. APIs may change between releases.
This guide covers how to add Monocle observability to your [Effect](https://effect.website) application.
## Prerequisites
* Effect 3.21+
* `@effect/opentelemetry` 0.63+
* A Monocle account with an API key (not required for [Monocle Studio](/studio/overview))
## Installation
```bash theme={"theme":"vesper"}
npm install @monocle.sh/effect-agent
```
**Peer dependencies:** `effect >= 3.21.0`, `@effect/opentelemetry >= 0.63.0`
## Setup
`MonocleLayer` returns an Effect Layer that you compose with the rest of your application layers. It configures OTLP exporters for traces, metrics, and logs.
```typescript title="src/main.ts" theme={"theme":"vesper"}
import { Effect, Layer } from "effect";
import { NodeRuntime } from "@effect/platform-node";
import { MonocleLayer } from "@monocle.sh/effect-agent";
const TracingLive = MonocleLayer({
serviceName: "my-app",
apiKey: process.env.MONOCLE_API_KEY,
environment: "production",
});
const MainLive = Layer.mergeAll(DbLive, TracingLive);
program.pipe(Effect.provide(MainLive), NodeRuntime.runMain);
```
That's it. All spans (`Effect.withSpan`), logs (`Effect.log`), and metrics are automatically exported to Monocle.
## Configuration
| Option | Description | Required |
| - | - | - |
| `serviceName` | Service name for identification in Monocle | Yes |
| `apiKey` | Monocle API key. Falls back to `MONOCLE_API_KEY` env var | No |
| `environment` | Environment name. Falls back to `MONOCLE_ENVIRONMENT`, `NODE_ENV`, then `development` | No |
| `serviceVersion` | Service version (e.g., git sha, semver). Defaults to `0.0.0` | No |
| `endpoint` | OTLP endpoint URL. Defaults to `https://ingest.monocle.sh` | No |
| `dev` | Enable dev mode for Monocle Studio (see below) | No |
When no `apiKey` is provided (neither in config nor in `MONOCLE_API_KEY`) and `dev` mode is off, `MonocleLayer` returns an empty layer. No telemetry is sent.
## Auto-Instrumentations
`MonocleLayer` handles the OTLP export and exception enrichment, but does not register OpenTelemetry auto-instrumentations (HTTP, PostgreSQL, Redis, etc.). These need to be set up separately.
The simplest approach is the `@opentelemetry/auto-instrumentations-node` meta-package, which patches all supported libraries automatically:
```bash theme={"theme":"vesper"}
npm install @opentelemetry/auto-instrumentations-node
```
```typescript title="src/register.ts" theme={"theme":"vesper"}
import { getNodeAutoInstrumentations } from "@opentelemetry/auto-instrumentations-node";
import { registerInstrumentations } from "@opentelemetry/instrumentation";
registerInstrumentations({
instrumentations: [getNodeAutoInstrumentations()],
});
```
**The import order is critical.**
This file must be imported before any library it instruments (http, pg, redis, etc.). Import it as the very first line in your entrypoint:
```typescript title="src/main.ts" theme={"theme":"vesper"}
import "./register.js";
import { Effect, Layer } from "effect";
// ...
```
You can also install individual instrumentations if you prefer to keep dependencies minimal:
```bash theme={"theme":"vesper"}
npm install @opentelemetry/instrumentation-http @opentelemetry/instrumentation-pg
```
```typescript title="src/register.ts" theme={"theme":"vesper"}
import { HttpInstrumentation } from "@opentelemetry/instrumentation-http";
import { PgInstrumentation } from "@opentelemetry/instrumentation-pg";
import { registerInstrumentations } from "@opentelemetry/instrumentation";
registerInstrumentations({
instrumentations: [new HttpInstrumentation(), new PgInstrumentation()],
});
```
## User Context
To associate telemetry data with authenticated users, use `Monocle.setUser()`. This lets you filter traces and exceptions by user in the dashboard.
```typescript theme={"theme":"vesper"}
import { Effect } from "effect";
import { Monocle } from "@monocle.sh/effect-agent";
const program = Effect.gen(function* () {
yield* Monocle.setUser({
id: "123",
email: "john@example.com",
name: "John Doe",
});
// ... rest of the effect
}).pipe(Effect.withSpan("handle-request"));
```
The `name` field takes priority in the Monocle UI, falling back to `email`, then `id`.
You can also pass custom attributes:
```typescript theme={"theme":"vesper"}
yield* Monocle.setUser({
id: user.id,
email: user.email,
name: user.fullName,
plan: "enterprise",
role: "admin",
});
```
Custom attributes are stored as `user.` on the span.
`Monocle.setUser` is a no-op when called outside of a span. It will never throw.
Do not include sensitive data (passwords, tokens, API keys) in user
attributes. These values are sent to Monocle and will be visible in trace
viewers.
## Dev Mode (Monocle Studio)
Enable `dev: true` to send telemetry to [Monocle Studio](/studio/overview) running locally. Dev mode uses faster batch delays (100ms) and points to `localhost:4200` by default. No API key required.
```typescript theme={"theme":"vesper"}
const TracingLive = MonocleLayer({
dev: true,
serviceName: "my-app",
});
```
See the [Monocle Studio docs](/studio/overview) to get started with local development.
# Bentocache
Source: https://docs.monocle.sh/instrumentations/bentocache
Instrument Bentocache cache operations for observability
This guide explains how to:
* Install and enable the cache integration
* Customize key sanitization
* Suppress internal cache operations
## Overview
The explicit integration traces Bentocache operations, including caches used through `@adonisjs/cache`, using your agent's OpenTelemetry provider. See the [Cache dashboard](/features/cache) for the attributes it uses.
This integration requires AdonisJS agent 2.x. To replace the 1.x top-level `cache` option, follow the [migration guide](/migrations/adonisjs-2.0.0).
## Installation
Install the integration and declare Bentocache in your application. The integration supports Bentocache `^1.6.0`.
```bash title="Terminal" theme={"theme":"vesper"}
npm install @monocle.sh/instrumentation-bentocache bentocache
```
## With the AdonisJS agent
Register `bentocache()` in `config/monocle.ts`. Keep `otel.ts` first in your application entry points so hooks initialize before application modules load.
```typescript title="config/monocle.ts" theme={"theme":"vesper"}
import { defineConfig } from "@monocle.sh/adonisjs-agent";
import { bentocache } from "@monocle.sh/instrumentation-bentocache/integration";
import env from "#start/env";
export default defineConfig({
apiKey: env.get("MONOCLE_API_KEY"),
integrations: [bentocache()],
});
```
Omit the integration or use `bentocache({ enabled: false })` to disable it. Installing the package alone does not activate tracing.
## Configuration
The default options preserve the agent's previous cache behavior: keys are captured and sanitized, parent spans are not required, and internal Redis/L2 operations are suppressed. Override key handling or require a parent span through the factory.
```typescript title="config/monocle.ts" theme={"theme":"vesper"}
import { defineConfig } from "@monocle.sh/adonisjs-agent";
import { bentocache } from "@monocle.sh/instrumentation-bentocache/integration";
import env from "#start/env";
export default defineConfig({
apiKey: env.get("MONOCLE_API_KEY"),
integrations: [bentocache({
requireParentSpan: true,
includeKeys: true,
keySanitizer: (key) => key?.replace(/session:.+/, "session:*"),
})],
});
```
### Key sanitization
The default sanitizer replaces UUIDs, numeric IDs, and hexadecimal identifiers with `*`. For example, `users:123` becomes `users:*`. A custom `keySanitizer` replaces this default. Set `includeKeys: false` to omit keys.
### Internal operation suppression
The Monocle integration suppresses internal Redis, L2, and bus spans so traces show high-level cache operations. This retains the previous Adonis agent behavior.
## Without the AdonisJS agent
For a standalone OpenTelemetry setup, use `BentoCacheInstrumentation` from `@bentocache/otel` directly. Its defaults differ from the Monocle wrapper. Initialize your SDK and module hooks before loading your application, and include the instrumentation in your existing SDK setup.
```typescript title="instrumentation.ts" theme={"theme":"vesper"}
import { BentoCacheInstrumentation } from "@bentocache/otel";
export const cacheInstrumentation = new BentoCacheInstrumentation({
requireParentSpan: false,
includeKeys: true,
suppressInternalOperations: true,
});
```
# @boringnode/queue
Source: https://docs.monocle.sh/instrumentations/boringnode-queue
Instrument @boringnode/queue job queues for observability
This guide covers how to instrument `@boringnode/queue` for monitoring background jobs. You will learn how to:
* Install and configure the instrumentation
* Track job dispatch and processing
* Understand producer and consumer spans
For the full list of required span attributes and what powers the Jobs dashboard, see the [Jobs feature page](/features/jobs).
## Installation
```bash theme={"theme":"vesper"}
npm install @boringnode/queue
```
The OTel instrumentation is built into `@boringnode/queue` itself. No additional package is needed.
## Usage
Create an instance of `QueueInstrumentation` and call `enable()`. This must happen before you call `QueueManager.init()`.
```typescript title="instrumentation.ts" theme={"theme":"vesper"}
import { QueueInstrumentation } from "@boringnode/queue/otel";
const instrumentation = new QueueInstrumentation();
instrumentation.enable();
```
Once enabled, the instrumentation automatically patches `QueueManager.init()` to inject tracing wrappers. All job dispatches and worker executions will create spans.
If the automatic patching does not work (e.g., ESM hook timing issues), you can register manually:
```typescript title="instrumentation.ts" theme={"theme":"vesper"}
import * as queue from "@boringnode/queue";
instrumentation.manuallyRegister(queue);
```
If you are using AdonisJS with the Monocle agent, this instrumentation is configured automatically when `@boringnode/queue` is installed. No manual setup is needed.
## Configuration
```typescript theme={"theme":"vesper"}
const instrumentation = new QueueInstrumentation({
// Messaging system name used in span attributes (default: "boringqueue")
messagingSystem: "boringqueue",
// How consumer spans relate to producer spans (default: "link")
executionSpanLinkMode: "link",
});
```
### Span linking vs parenting
By default, consumer spans are **linked** to the producer span. This means they appear as separate traces connected by a link. Set `executionSpanLinkMode: "parent"` to make the consumer span a **child** of the producer, keeping them in the same trace.
## Spans
### Producer spans
When you dispatch a job, the instrumentation creates a **PRODUCER** span named `publish {queueName}`. Trace context is injected into the job payload so the consumer can link back to the producer.
For batch dispatches, a single PRODUCER span is created with a `messaging.batch.message_count` attribute.
### Consumer spans
When a worker processes a job, the instrumentation creates a **CONSUMER** span named `process {queueName}`. The span captures job metadata, queue wait time, and any errors thrown during processing.
If the job is retried, a `messaging.retry` event is recorded on the span with the next retry timestamp.
### Internal operation suppression
Internal queue operations (Redis calls, etc.) are automatically suppressed to keep traces clean. Only the high-level dispatch and execution spans are visible.
## Span attributes
In addition to the [standard job attributes](/features/jobs#span-attributes), the `@boringnode/queue` instrumentation emits these attributes:
| Attribute | Description |
| - | - |
| `messaging.job.name` | Job class name |
| `messaging.job.queue_time_ms` | Time between dispatch and execution start (ms) |
| `messaging.job.group_id` | Job group ID (if set) |
| `messaging.job.priority` | Job priority (if set) |
| `messaging.job.delay_ms` | Job delay in ms (if set) |
| `messaging.message.retry.count` | Number of retry attempts |
# BullMQ
Source: https://docs.monocle.sh/instrumentations/bullmq
Instrument BullMQ job queues for observability
This guide covers how to instrument BullMQ queues and workers. You will learn how to:
* Install and configure the instrumentation
* Track job dispatch and processing
* Understand producer and consumer spans
## Overview
For the full list of required span attributes and what powers the Jobs dashboard, see the [Jobs feature page](/features/jobs).
## Installation
```bash title="Terminal" theme={"theme":"vesper"}
npm install @monocle.sh/instrumentation-bullmq
```
**Peer dependency:** `bullmq >= 5.0.0`
## With the AdonisJS agent
Registration through `integrations` requires AdonisJS agent 2.x. See the [migration guide](/migrations/adonisjs-2.0.0) when upgrading from 1.x.
Install this integration separately and add `bullmq()` to `config/monocle.ts`. Installing the package alone does not activate it.
```typescript title="config/monocle.ts" theme={"theme":"vesper"}
import { defineConfig } from "@monocle.sh/adonisjs-agent";
import { bullmq } from "@monocle.sh/instrumentation-bullmq/integration";
import env from "#start/env";
export default defineConfig({
apiKey: env.get("MONOCLE_API_KEY"),
integrations: [bullmq()],
});
```
The integration inherits the agent's `sampling` callback. Pass `sampling` to `bullmq()` to override it for BullMQ only. Omit the integration or use `bullmq({ enabled: false })` to disable it. See [Job Sampling](/features/job-sampling) for sampling rules.
## Without the AdonisJS agent
Create an instance of `BullMQInstrumentation` and call `enable()`. This must happen before you import or create any BullMQ `Queue`, `Worker`, or `FlowProducer` instances.
```typescript title="instrumentation.ts" theme={"theme":"vesper"}
import { BullMQInstrumentation } from "@monocle.sh/instrumentation-bullmq";
const instrumentation = new BullMQInstrumentation();
instrumentation.enable();
```
Once enabled, all `Queue.add`, `Queue.addBulk`, `FlowProducer.add`, `FlowProducer.addBulk`, and `Worker` processing calls are automatically instrumented.
```typescript title="example.ts" theme={"theme":"vesper"}
import { Queue, Worker } from "bullmq";
const queue = new Queue("emails", { connection });
// This creates a PRODUCER span
await queue.add("welcome-email", { userId: "123" });
// This creates a CONSUMER span for each processed job
const worker = new Worker("emails", async (job) => {
await sendEmail(job.data.userId);
}, { connection });
```
## Configuration
```typescript theme={"theme":"vesper"}
const instrumentation = new BullMQInstrumentation({
// Create individual spans for each job in addBulk (default: true)
emitCreateSpansForBulk: true,
// Create individual spans for each job in FlowProducer.add (default: true)
emitCreateSpansForFlow: true,
// Only create producer spans when a parent span exists (default: false)
requireParentSpanForPublish: false,
// Use producer span as consumer parent instead of linking (default: false)
useProducerSpanAsConsumerParent: false,
});
```
### Span linking vs parenting
By default, consumer spans are **linked** to the producer span. This means they appear as separate traces connected by a link. Set `useProducerSpanAsConsumerParent: true` to make the consumer span a **child** of the producer, keeping them in the same trace.
## Spans
### Producer spans
**`Queue.add`** creates a **PRODUCER** span named `Queue.add {queueName}`.
**`Queue.addBulk`** creates an **INTERNAL** parent span named `Queue.addBulk {queueName}`, with individual **PRODUCER** child spans for each job (named `Job.addJob {queueName}`).
**`FlowProducer.add`** creates an **INTERNAL** parent span named `FlowProducer.add {queueName}`, with child spans for each job in the flow tree.
**`FlowProducer.addBulk`** creates an **INTERNAL** parent span named `FlowProducer.addBulk`, with child spans per flow.
### Consumer spans
**`Worker` processing** creates a **CONSUMER** span named `Worker.run {queueName}` for each job processed. The span captures job metadata, queue time, and any errors thrown during processing.
### Lifecycle events
Lock extensions, job removals, and retries are recorded as **events** on the active consumer span (not separate spans).
## BullMQ-specific attributes
In addition to the [standard job attributes](/features/jobs#span-attributes), the BullMQ instrumentation emits these extra attributes:
| Attribute | Description |
| - | - |
| `messaging.bullmq.job.name` | Job name |
| `messaging.bullmq.job.timestamp` | Job creation timestamp |
| `messaging.bullmq.job.delay` | Job delay in ms |
| `messaging.bullmq.job.attempts` | Total attempts made |
| `messaging.bullmq.job.failedReason` | Failure reason (set after processing) |
| `messaging.bullmq.job.finishedOn` | Completion timestamp |
| `messaging.bullmq.job.processedOn` | Processing start timestamp |
| `messaging.bullmq.job.repeatJobKey` | Repeat job key (for recurring jobs) |
| `messaging.bullmq.job.bulk.names` | Array of job names in bulk operations |
| `messaging.bullmq.worker.concurrency` | Worker concurrency setting |
| `messaging.bullmq.worker.lockDuration` | Lock duration in ms |
| `messaging.bullmq.worker.lockRenewTime` | Lock renew interval in ms |
| `messaging.bullmq.worker.rateLimiter.max` | Rate limiter max |
| `messaging.bullmq.worker.rateLimiter.duration` | Rate limiter duration |
# Drizzle ORM
Source: https://docs.monocle.sh/instrumentations/drizzle
Trace database queries made through Drizzle ORM
This guide shows how to instrument Drizzle for query monitoring. You will learn how to:
* Wrap your Drizzle database or underlying client
* Configure which query details are recorded
## Overview
`@monocle.sh/instrumentation-drizzle` creates OpenTelemetry spans for Drizzle queries. Initialize your OpenTelemetry SDK or agent before creating the database, then wrap the Drizzle instance with `instrumentDrizzleClient()`. The instrumentation is opt-in and is not enabled automatically by the Monocle agent.
## Installation
The package requires Drizzle ORM `0.28.0` or later. Install your database driver separately. This example uses `pg`.
```bash title="Terminal" theme={"theme":"vesper"}
npm install @monocle.sh/instrumentation-drizzle drizzle-orm pg
```
## Instrument your Drizzle database
Wrap the instance returned by `drizzle()` before using it. Set `dbSystem` so Monocle can identify the database in query details.
```typescript title="app/services/database.ts" theme={"theme":"vesper"}
import { drizzle } from "drizzle-orm/node-postgres";
import { instrumentDrizzleClient } from "@monocle.sh/instrumentation-drizzle";
import { Pool } from "pg";
const pool = new Pool({ connectionString: process.env.DATABASE_URL });
export const db = instrumentDrizzleClient(drizzle(pool), {
dbSystem: "postgresql",
dbName: "app",
});
```
Queries and prepared queries create database spans. Queries run inside a transaction are marked with `db.transaction`, and query errors are recorded on their spans.
If you want to instrument the underlying client directly, use `instrumentDrizzle()` before passing the client to Drizzle:
```typescript title="app/services/database.ts" theme={"theme":"vesper"}
import { drizzle } from "drizzle-orm/node-postgres";
import {
instrumentDrizzle,
type InstrumentDrizzleConfig,
} from "@monocle.sh/instrumentation-drizzle";
import { Pool } from "pg";
const config: InstrumentDrizzleConfig = { dbSystem: "postgresql" };
const pool = instrumentDrizzle(new Pool({ connectionString: process.env.DATABASE_URL }), config);
export const db = drizzle(pool);
```
Choose one entry point for a database. Wrapping both the Drizzle instance and its underlying client can create duplicate spans for a query.
## Configure query details
Pass these options to either instrumentation function:
* `dbSystem` identifies the database, for example `postgresql`, `mysql`, or `sqlite`. It defaults to `unknown`.
* `dbName`, `peerName`, and `peerPort` add database name and connection details to spans.
* `captureQueryText` includes SQL statements in spans by default. Set it to `false` to omit SQL text.
* `maxQueryTextLength` sets the maximum SQL statement length. It defaults to 1,000 characters.
The instrumentation creates `CLIENT` spans with database attributes such as the operation and table name when they can be read from the SQL statement. Transaction queries are marked with `db.transaction`.
# Kysely
Source: https://docs.monocle.sh/instrumentations/kysely
Trace database queries made through Kysely
`@monocle.sh/instrumentation-kysely` traces queries, streams, and transactions. Configure it manually after initializing OpenTelemetry; the Monocle agent does not enable it automatically.
## Installation
Supports Kysely `^0.28.16` and `^0.29.6`. This example uses `pg`:
```bash title="Terminal" theme={"theme":"vesper"}
npm install @monocle.sh/instrumentation-kysely kysely@^0.29.6 pg
```
## Configure query tracing
Wrap your dialect with `observeDialect()` and set `dbSystem` to your database type.
```typescript title="app/services/database.ts" theme={"theme":"vesper"}
import pg from "pg";
import { Kysely, PostgresDialect } from "kysely";
import { observeDialect } from "@monocle.sh/instrumentation-kysely";
interface Database {
users: { id: number; name: string };
}
const pool = new pg.Pool({ connectionString: process.env.DATABASE_URL });
export const db = new Kysely({
dialect: observeDialect({
dialect: new PostgresDialect({ pool }),
dbSystem: "postgresql",
}),
});
```
If your driver is also instrumented, queries can produce duplicate spans. Disable one instrumentation to keep a single span per query.
### Use postgres.js
For [postgres.js](https://github.com/porsager/postgres), use `PostgresJSDialect` from `kysely-postgres-js`:
```bash title="Terminal" theme={"theme":"vesper"}
npm install postgres kysely-postgres-js
```
```typescript title="app/services/database.ts" theme={"theme":"vesper"}
import postgres from "postgres";
import { PostgresJSDialect } from "kysely-postgres-js";
import { observeDialect } from "@monocle.sh/instrumentation-kysely";
const dialect = observeDialect({
dialect: new PostgresJSDialect({
postgres: postgres(process.env.DATABASE_URL!),
}),
dbSystem: "postgresql",
});
```
Pass this `dialect` to your Kysely instance. OpenTelemetry's `pg` instrumentation does not cover postgres.js, so it adds no duplicate spans here.
## Control captured data
SQL and errors are recorded by default and may contain sensitive data. Options for `observeDialect()`:
* `recordQueryText: false`: omit SQL text.
* `recordErrors: false`: omit error details while still marking failures.
* `requireParentSpan: true`: trace only within an active span.
* `connection`: add `host`, `port`, and `database` details.
## Pool metrics with pg
After initializing your metrics provider, call `registerPgPoolMetrics()` to track used and idle connections, pending requests, and the configured limit. Stop collection before closing the pool:
```typescript title="app/services/database.ts" theme={"theme":"vesper"}
import pg from "pg";
import { registerPgPoolMetrics } from "@monocle.sh/instrumentation-kysely";
const pool = new pg.Pool({ connectionString: process.env.DATABASE_URL, max: 10 });
const stopPoolMetrics = registerPgPoolMetrics({ pool, max: 10, poolName: "main" });
export async function shutdownDatabase() {
stopPoolMetrics();
await pool.end();
}
```
Pool metrics require `pg.Pool`. postgres.js does not expose the pool statistics this helper needs.
Query tracing still works with postgres.js and other Kysely drivers.
# Mastra
Source: https://docs.monocle.sh/instrumentations/mastra
Instrument Mastra agents and workflows for AI observability
This guide explains how to:
* Install and register the Mastra integration
* Connect Mastra's OpenTelemetry bridge
* Monitor agent runs, LLM calls, token usage, and evaluations
## Overview
Mastra already creates OpenTelemetry spans through `@mastra/otel-bridge`. Monocle adds a Mastra observability config helper and a span processor that normalizes Mastra attributes to the fields used by the AI Agents dashboard.
For the full list of required span attributes and what powers the AI dashboard, see the [AI Agents feature page](/features/ai-agents).
## Installation
```bash title="Terminal" theme={"theme":"vesper"}
npm install @monocle.sh/instrumentation-mastra
```
**Peer dependencies:** `@mastra/core` and `@mastra/otel-bridge`.
## With the AdonisJS agent
Registration through `integrations` requires AdonisJS agent 2.x. See the [migration guide](/migrations/adonisjs-2.0.0) when upgrading from 1.x.
Install this integration separately and add `mastra()` to `config/monocle.ts`. Installing the package alone does not activate it.
```typescript title="config/monocle.ts" theme={"theme":"vesper"}
import { defineConfig } from "@monocle.sh/adonisjs-agent";
import { mastra } from "@monocle.sh/instrumentation-mastra/integration";
import env from "#start/env";
export default defineConfig({
apiKey: env.get("MONOCLE_API_KEY"),
integrations: [mastra({ recordInputs: false, recordOutputs: false })],
});
```
The integration normalizes existing spans; configure the Mastra bridge below to emit them. Recording options here are independent of the Vercel AI integration.
## Usage with Mastra
Use `createMastraObservabilityConfig()` in your Mastra config. It creates the Mastra `OtelBridge` and excludes internal `MODEL_STEP` and `MODEL_CHUNK` spans by default because their data is already represented on the model generation span.
For other Node.js applications, add `new MastraSpanProcessor()` from `@monocle.sh/instrumentation-mastra` to your existing OpenTelemetry tracer provider. The AdonisJS `mastra()` integration registers this processor for you.
```typescript title="mastra/index.ts" theme={"theme":"vesper"}
import { Mastra } from "@mastra/core";
import { Observability, SamplingStrategyType } from "@mastra/observability";
import { createMastraObservabilityConfig } from "@monocle.sh/instrumentation-mastra";
export const mastra = new Mastra({
agents: {
// ...
},
observability: new Observability({
configs: {
monocle: createMastraObservabilityConfig({
serviceName: "api",
sampling: { type: SamplingStrategyType.ALWAYS },
}),
},
}),
});
```
## Normalized attributes
The Mastra processor preserves Mastra spans and adds the attributes Monocle expects:
| Mastra attribute | Monocle attribute |
| - | - |
| `gen_ai.provider.name` | `gen_ai.system` |
| `gen_ai.output.messages` | `gen_ai.response.text` |
| `gen_ai.tool.call.arguments` | `gen_ai.tool.input` |
| `gen_ai.tool.call.result` | `gen_ai.tool.output` |
| `gen_ai.agent.name` | `gen_ai.function_id` |
| `gen_ai.usage.cached_input_tokens` | `gen_ai.usage.input_tokens.cached` |
| `gen_ai.usage.reasoning_tokens` | `gen_ai.usage.output_tokens.reasoning` |
Model generation metadata is also copied to the parent `invoke_agent` span when Mastra emits it only on the child model span. This lets the Agents tab show the model, response preview, token usage, and cost on the agent run.
## Scorers and evaluations
Mastra scorer events are exported as `gen_ai.evaluation.result` span events. The event name is a Monocle convention; Monocle reads the standard OpenTelemetry evaluation attributes first:
| Attribute | Description |
| - | - |
| `gen_ai.evaluation.name` | Scorer name |
| `gen_ai.evaluation.score.value` | Numeric score value |
| `gen_ai.evaluation.score.label` | Optional score label |
| `gen_ai.evaluation.explanation` | Explanation for the score |
For richer scorer metadata, use the Monocle evaluation extensions documented on the [AI Agents feature page](/features/ai-agents#evaluation-result-attributes), such as `gen_ai.evaluation.scorer.id`, `gen_ai.evaluation.scorer.version`, and `gen_ai.evaluation.target.trace_id`.
# @modelcontextprotocol/sdk
Source: https://docs.monocle.sh/instrumentations/mcp-server
Instrument your Model Context Protocol server for observability
This guide covers how to instrument an MCP server built with `@modelcontextprotocol/sdk`. You will learn how to:
* Install and configure the instrumentation
* Track tool calls, resources, and prompts
* Capture inputs and outputs
## Overview
For the full list of required span attributes and what powers the MCP dashboard, see the [MCP feature page](/features/mcp-monitoring).
## Installation
```bash title="Terminal" theme={"theme":"vesper"}
npm install @monocle.sh/instrumentation-mcp
```
**Peer dependency:** `@modelcontextprotocol/sdk >= 1.0.0`
## With the AdonisJS agent
Install this package explicitly and use the direct import shown below. Initialize the agent before loading your MCP server module, then call `instrumentMcpServer()` before connecting the transport. No `integrations` entry is needed.
AdonisJS agent 2.x no longer bundles MCP or exports `@monocle.sh/adonisjs-agent/mcp`. Use `@monocle.sh/instrumentation-mcp` instead. See the [migration guide](/migrations/adonisjs-2.0.0) when upgrading from 1.x.
## Usage
Call `instrumentMcpServer()` on your server instance before connecting a transport. This wraps the server's internal handlers to create spans for every incoming request, notification, and outgoing message.
```typescript title="server.ts" theme={"theme":"vesper"}
import { z } from "zod";
import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
import { instrumentMcpServer } from "@monocle.sh/instrumentation-mcp";
const server = new McpServer({
name: "my-mcp-server",
version: "1.0.0",
});
instrumentMcpServer(server);
server.tool("search", { query: z.string() }, async ({ query }) => {
return { content: [{ type: "text", text: `Search requested: ${query}` }] };
});
const transport = new StdioServerTransport();
await server.connect(transport);
```
## Configuration
```typescript theme={"theme":"vesper"}
instrumentMcpServer(server, {
// Capture tool/prompt arguments in span attributes (default: false)
recordInputs: true,
// Capture tool results and prompt messages in span attributes (default: false)
recordOutputs: true,
// Override auto-detected server name/version
serverName: "my-server",
serverVersion: "1.0.0",
});
```
## Spans
The instrumentation creates spans for three categories of MCP activity.
### Request spans
A **SERVER** span is created for every incoming JSON-RPC request (`tools/call`, `resources/read`, `prompts/get`, `initialize`, etc.).
The span name includes the target when available: `tools/call search`, `resources/read config://app`, `prompts/get weather`.
### Notification spans
A **SERVER** span is created for incoming notifications (e.g., `notifications/initialized`, `resources/changed`).
### Outgoing notification spans
A **CLIENT** span is created for outgoing notifications sent by the server (e.g., `resources/changed`).
### Error spans
Transport errors produce an **INTERNAL** span named `mcp.transport.error` with the exception recorded. JSON-RPC errors set `rpc.jsonrpc.error_code` and `rpc.jsonrpc.error_message` on the request span.
## SDK-specific attributes
In addition to the [standard MCP attributes](/features/mcp-monitoring#span-attributes), this instrumentation emits these extra attributes when recording is enabled.
### Input attributes (when `recordInputs: true`)
Tool arguments are captured as `mcp.request.argument.{key}` (JSON-stringified per key).
### Output attributes (when `recordOutputs: true`)
| Attribute | Description |
| - | - |
| `mcp.tool.result.content_count` | Number of content items returned |
| `mcp.tool.result.{i}.content_type` | Content type of each item |
| `mcp.tool.result.{i}.content` | Content value of each item |
| `mcp.prompt.result.description` | Prompt description |
| `mcp.prompt.result.message_count` | Number of messages returned |
| `mcp.prompt.result.{i}.role` | Message role |
| `mcp.prompt.result.{i}.content` | Message content |
# Vercel AI SDK
Source: https://docs.monocle.sh/instrumentations/vercel-ai
Instrument the Vercel AI SDK for LLM observability
This guide covers how to instrument the Vercel AI SDK for monitoring LLM calls, token usage, and costs. You will learn how to:
* Install and configure the instrumentation
* Track AI operations (generateText, streamText, etc.)
* Group calls into conversations
## Overview
For the full list of required span attributes and what powers the AI dashboard, see the [AI Agents feature page](/features/ai-agents).
## Installation
```bash title="Terminal" theme={"theme":"vesper"}
npm install @monocle.sh/instrumentation-vercel-ai
```
**Peer dependency:** `ai >= 3.0.0 < 7`
## With the AdonisJS agent
Registration through `integrations` requires AdonisJS agent 2.x. See the [migration guide](/migrations/adonisjs-2.0.0) when upgrading from 1.x.
Install this integration separately and add `vercelAi()` to `config/monocle.ts`. Installing the package alone does not activate it.
```typescript title="config/monocle.ts" theme={"theme":"vesper"}
import { defineConfig } from "@monocle.sh/adonisjs-agent";
import { vercelAi } from "@monocle.sh/instrumentation-vercel-ai/integration";
import env from "#start/env";
export default defineConfig({
apiKey: env.get("MONOCLE_API_KEY"),
integrations: [vercelAi({ recordInputs: false, recordOutputs: false })],
});
```
The agent enables the import hooks and adds the span processor before starting its telemetry SDK. Keep the `otel.ts` import first in your application entry points. Omit `vercelAi()` to disable this integration.
## Without the AdonisJS agent
The instrumentation has two parts: the `VercelAiInstrumentation` that hooks into the AI SDK to enable telemetry, and the `VercelAiSpanProcessor` that normalizes span attributes to [OpenTelemetry GenAI Semantic Conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/).
```typescript title="instrumentation.ts" theme={"theme":"vesper"}
import { VercelAiInstrumentation, VercelAiSpanProcessor } from "@monocle.sh/instrumentation-vercel-ai";
import { NodeTracerProvider } from "@opentelemetry/sdk-trace-node";
// Add the span processor to your tracer provider
const provider = new NodeTracerProvider({
spanProcessors: [new VercelAiSpanProcessor()],
});
provider.register();
// Enable auto-instrumentation
const instrumentation = new VercelAiInstrumentation();
instrumentation.enable();
```
Once enabled, all `generateText`, `streamText`, `generateObject`, `streamObject`, `embed`, `embedMany`, and `rerank` calls are automatically instrumented.
```typescript title="example.ts" theme={"theme":"vesper"}
import { generateText } from "ai";
import { openai } from "@ai-sdk/openai";
// This automatically creates spans with token usage, model info, etc.
const result = await generateText({
model: openai("gpt-4o"),
prompt: "What is observability?",
});
```
## Configuration
```typescript theme={"theme":"vesper"}
const instrumentation = new VercelAiInstrumentation({
// Record prompt/input data in spans (default: true)
recordInputs: true,
// Record response/output data in spans (default: true)
recordOutputs: true,
});
```
You can also control recording per call via the AI SDK's `experimental_telemetry` option. Explicit `recordInputs` and `recordOutputs` options on the integration take priority. When an option is omitted, it falls back to the per-call setting, then to `true`. A per-call `experimental_telemetry.isEnabled: false` still disables telemetry for that call.
## Conversation tracking
Use `withConversationId()` to group multiple AI calls into a single conversation. This sets `gen_ai.conversation.id` on all spans created within the callback, enabling the Conversations tab in the Monocle AI dashboard.
```typescript title="chat.ts" theme={"theme":"vesper"}
import { withConversationId, getConversationId } from "@monocle.sh/instrumentation-vercel-ai";
await withConversationId("conv-abc-123", async () => {
// All AI calls here share the same conversation ID
const response = await generateText({
model: openai("gpt-4o"),
prompt: userMessage,
});
// You can also read the current conversation ID
const currentId = getConversationId(); // "conv-abc-123"
});
```
## Spans
The instrumentation does not create spans directly. It enables the Vercel AI SDK's built-in telemetry and processes those spans through the `VercelAiSpanProcessor` to normalize attributes.
### Pipeline spans
A span is created for each top-level AI call (`generateText`, `streamText`, etc.). The span name is normalized to `{operation} {functionId}` (e.g., `invoke_agent weather-app`).
### Inner LLM call spans
A child span is created for the actual LLM API call. The span name is `{operation} {modelId}` (e.g., `generate_text gpt-4o`).
### Tool call spans
Each tool invocation creates a span named `execute_tool {toolName}`. The instrumentation also detects Vercel AI SDK v5 tool errors embedded in result content and records them as exceptions on the corresponding span.
## Vercel AI-specific attributes
In addition to the [standard AI attributes](/features/ai-agents#span-attributes), the Vercel AI instrumentation emits these extra attributes.
### Input/output attributes (when recording is enabled)
| Attribute | Description |
| - | - |
| `gen_ai.system_instructions` | System prompt text |
| `gen_ai.input.messages` | Input messages (JSON) |
| `gen_ai.response.text` | Generated text |
| `gen_ai.response.tool_calls` | Tool calls made |
| `gen_ai.response.object` | Generated object |
| `gen_ai.tool.input` | Tool arguments |
| `gen_ai.tool.output` | Tool result |
### Provider-specific token breakdowns
The processor extracts detailed token metrics from provider metadata when available:
| Attribute | Providers |
| - | - |
| `gen_ai.usage.input_tokens.cached` | OpenAI, Anthropic, Bedrock, DeepSeek |
| `gen_ai.usage.input_tokens.cache_write` | Anthropic, Bedrock |
| `gen_ai.usage.input_tokens.cache_miss` | DeepSeek |
| `gen_ai.usage.output_tokens.reasoning` | OpenAI |
### Vercel AI SDK native attributes
Remaining Vercel-native attributes are preserved under the `vercel.ai.*` namespace (e.g., `vercel.ai.response.msToFirstChunk`, `vercel.ai.response.avgOutputTokensPerSecond`).
# Introduction
Source: https://docs.monocle.sh/introduction
Observability platform built on OpenTelemetry
![Monocle]()
Monocle is an observability platform that gives you full visibility into your applications. Distributed tracing, logs, metrics, exceptions — all in one place. Built on OpenTelemetry, so no vendor lock-in.
## Features
End-to-end request visualization across your entire stack
Smart grouping, stack traces, source code context, and AI analysis
Centralized aggregation with powerful filtering, correlated with traces
Response times, P95 latency, and slow endpoint detection
Query performance, N+1 detection across PostgreSQL, MySQL, Redis
Job execution and failures for `@adonisjs/queue` and BullMQ
Hit rates, miss rates, and latency per cache key
LLM calls, tool usage, token consumption, costs, and conversations
Health checks, incident detection, and configurable alerts
Public status pages with incident reports and uptime timelines
Ace command execution, failures, and duration tracking
Real-time CPU, memory, heap, and event loop monitoring
Connect Claude Code or Cursor to your observability data
Weekly recaps, notifications, team collaboration, multi-environment support
## Built on OpenTelemetry
Your telemetry data stays portable. Monocle uses standard OpenTelemetry protocols, so you can fan out data to Grafana, Honeycomb, or any OTLP-compatible backend alongside Monocle.
## Get started
Install the Monocle agent in your AdonisJS application
# Migrate to AdonisJS agent 2.0.0
Source: https://docs.monocle.sh/migrations/adonisjs-2.0.0
Upgrade from 1.x to explicit integrations and update your helper imports
This guide covers how to:
* Install and register the integrations your application uses
* Preserve recording settings and update helper imports
* Verify telemetry after upgrading
## Overview
Version `2.0.0` stops bundling Mastra, Vercel AI, BullMQ, Bentocache, and MCP integrations. Install and configure only the ones you use.
Node auto-instrumentation and built-in Adonis mail, queue, and CLI tracing stay included. Keep your `mail`, `queue`, `cli`, endpoint, API key, destinations, and sampling options.
## 1. Install the integrations you use
Upgrade the agent, then install the integrations your application needs. Keep their host libraries as direct application dependencies.
```bash title="Terminal" theme={"theme":"vesper"}
npm install @monocle.sh/adonisjs-agent@^2.0.0
```
```bash title="BullMQ" theme={"theme":"vesper"}
npm install @monocle.sh/instrumentation-bullmq
```
```bash title="Vercel AI SDK" theme={"theme":"vesper"}
npm install @monocle.sh/instrumentation-vercel-ai
```
```bash title="Mastra" theme={"theme":"vesper"}
npm install @monocle.sh/instrumentation-mastra
```
```bash title="Bentocache" theme={"theme":"vesper"}
npm install @monocle.sh/instrumentation-bentocache bentocache
```
```bash title="MCP" theme={"theme":"vesper"}
npm install @monocle.sh/instrumentation-mcp
```
## 2. Register integrations
Replace the top-level `ai`, `mastra`, `bullmq`, and `cache` options with entries in `integrations`. Keep only the imports and entries you use; omit an integration to disable it. Installing a package alone no longer activates it.
```diff title="config/monocle.ts" theme={"theme":"vesper"}
import { defineConfig } from "@monocle.sh/adonisjs-agent";
+import { bullmq } from "@monocle.sh/instrumentation-bullmq/integration";
+import { vercelAi } from "@monocle.sh/instrumentation-vercel-ai/integration";
+import { mastra } from "@monocle.sh/instrumentation-mastra/integration";
+import { bentocache } from "@monocle.sh/instrumentation-bentocache/integration";
import env from "#start/env";
export default defineConfig({
apiKey: env.get("MONOCLE_API_KEY"),
serviceName: env.get("APP_NAME"),
serviceVersion: env.get("APP_VERSION"),
environment: env.get("APP_ENV"),
- bullmq: { useProducerSpanAsConsumerParent: false },
- ai: { recordInputs: false, recordOutputs: false },
- mastra: {},
- cache: { includeKeys: false },
+ integrations: [
+ bullmq({ useProducerSpanAsConsumerParent: false }),
+ vercelAi({ recordInputs: false, recordOutputs: false }),
+ mastra({ recordInputs: false, recordOutputs: false }),
+ bentocache({ includeKeys: false }),
+ ],
});
```
* **AI recording:** Mastra no longer inherits `ai.recordInputs` or `ai.recordOutputs`. Set them on both `mastra()` and `vercelAi()` to preserve disabled recording, as above. Keep your [Mastra OpenTelemetry bridge](/instrumentations/mastra).
* **BullMQ:** The integration inherits the agent's `sampling` callback unless you override it in `bullmq()`. Omitting or disabling it does not bypass [generic job sampling](/features/job-sampling). Keep `useProducerSpanAsConsumerParent` for trace linking; `executionSpanLinkMode` is an Adonis queue option.
* **Cache:** Move existing `cache` options into `bentocache()`. Keep `bentocache` as a direct dependency even when using `@adonisjs/cache`. [Cache defaults and key sanitization](/instrumentations/bentocache) stay the same.
Built-in `mail`, `queue`, and `cli` options stay at the top level. Use `false` or `{ enabled: false }` to disable them. Underlying Node instrumentations, such as Redis tracing, can still emit spans independently.
## 3. Update helper imports
Replace the removed `/ai` and `/mcp` imports. The helpers keep their existing usage.
```diff title="app/services/telemetry.ts" theme={"theme":"vesper"}
-import { withConversationId, getConversationId } from "@monocle.sh/adonisjs-agent/ai";
+import { withConversationId, getConversationId } from "@monocle.sh/instrumentation-vercel-ai";
-import { instrumentMcpServer } from "@monocle.sh/adonisjs-agent/mcp";
+import { instrumentMcpServer } from "@monocle.sh/instrumentation-mcp";
-import type { InstrumentMcpServerOptions } from "@monocle.sh/adonisjs-agent/mcp";
+import type { InstrumentMcpServerOptions } from "@monocle.sh/instrumentation-mcp";
```
[MCP](/instrumentations/mcp-server) stays attached to a server instance: call `instrumentMcpServer()` before connecting a transport, without adding it to `integrations`.
## 4. Verify the upgrade
1. Run your application's type checker to catch old options and imports.
2. Keep `import '../otel.js'` first in `bin/server.ts` and `bin/console.ts` so instrumentation hooks load before application libraries.
3. Restart the application and workers. Exercise each enabled integration and confirm its spans appear in Monocle Cloud or Studio, with the expected input/output recording settings.
4. Remove unused integration packages from your application's dependencies.
If an `/integration` export is missing, upgrade the agent and integration packages together and reinstall dependencies. If initialization fails, the error identifies the integration: check its configuration and compatible host library, or remove the entry if unused.
# Migrate to DSN
Source: https://docs.monocle.sh/migrations/dsn
Update your existing AdonisJS connection in a few steps
A DSN replaces your API key and connection settings with a single value for each
service. Your existing API keys keep working, so you can migrate one service at a time.
## 1. Copy your DSN
Open your service in Monocle. Under **SDK connection**, click **Create DSN** and
copy it. You can find and copy it again from this page at any time.
## 2. Update your AdonisJS configuration
Use a version of the Monocle SDK that supports DSNs, then add your DSN to `.env`:
```dotenv title=".env" theme={"theme":"vesper"}
MONOCLE_DSN=https://mk_live_********@ingest.monocle.sh/00000000-0000-4000-8000-000000000001
```
Replace this example with the full DSN you copied. Add the variable to your environment schema:
```typescript title="start/env.ts" theme={"theme":"vesper"}
MONOCLE_DSN: Env.schema.string.optional(),
```
In `config/monocle.ts`, replace `apiKey`, `serviceName`, and `endpoint` with `dsn`:
```diff title="config/monocle.ts" theme={"theme":"vesper"}
export default defineConfig({
- apiKey: env.get('MONOCLE_API_KEY'),
- serviceName: env.get('APP_NAME'),
- endpoint: 'https://ingest.monocle.sh',
+ dsn: env.get('MONOCLE_DSN'),
environment: env.get('APP_ENV'),
serviceVersion: env.get('APP_VERSION'),
});
```
Keep your environment, integrations, sampling, and other options unchanged.
If you also export to another destination, you can keep `serviceName` for it.
## 3. Check that data arrives
Restart or redeploy your application, then make a request. Check that new data
appears under the expected service and environment in Monocle. You can also check
the DSN's **Last used** timestamp on the service page.
You can find your old keys under **Previous project API keys**. Only revoke a key
after every application using it has migrated.
# Pricing & Limits
Source: https://docs.monocle.sh/pricing-and-usage
Plans, usage limits, data retention, and how billing works
## Plans
Pick a plan that fits your needs. Both plans include all features — the difference is data volume and retention.
| | **Starter** | **Business** |
| - | - | - |
| **Price** | €25/mo | €79/mo |
| **Data included** | 50 GB | 200 GB |
| **Data retention** | 30 days | 90 days |
| **Overage** | €0.35/GB | €0.30/GB |
| | **Starter** | **Business** |
| - | - | - |
| **Price** | €250/yr | €790/yr |
| **Data included** | 50 GB | 200 GB |
| **Data retention** | 30 days | 90 days |
| **Overage** | €0.35/GB | €0.30/GB |
All plans include:
* Unlimited applications and environments
* Unlimited team members
* Unlimited uptime monitors
* Monocle Studio (local DevTools)
* All features (tracing, logs, metrics, exceptions, AI monitoring, etc.)
## How Usage Is Calculated
Usage is measured by the **total size of data ingested** (in GB) during your billing period. This includes all telemetry signals:
* **Traces** — spans from distributed tracing
* **Logs** — log events
* **Metrics** — data points (counters, gauges, histograms)
The total size is the sum of estimated bytes across all signals, converted to GB. You can view your current usage and breakdown by application in the **Billing** section of your organization settings.
### Billing Period
Your billing period is tied to your subscription start date. For example, if you subscribed on March 15, your billing period runs from the 15th of each month to the 15th of the next month. Usage resets at the start of each billing period.
## Overages
Paid plans (Starter and Business) use **soft limits** — your data ingestion is never blocked when you exceed your plan's included data.
When you go over your plan's data limit:
1. Monocle continues to ingest all your data normally
2. The overage amount is calculated as the difference between your actual usage and your plan's included data, rounded up to the nearest GB
3. Overage charges are reported daily and billed automatically at the end of your billing period
Overages are billed per GB at the rate specified in your plan (€0.35/GB for Starter, €0.30/GB for Business).
### Usage Alerts
Monocle sends email notifications to all organization members when your usage reaches certain thresholds:
| Threshold | Alert |
| - | - |
| **80%** of your plan's data limit | Warning — you're approaching your limit |
| **100%** of your plan's data limit | Over limit — overage charges will apply |
These alerts are sent once per billing period per threshold, so you won't be spammed.
## Data Retention
Data retention determines how long your telemetry data is stored and queryable in Monocle:
| Plan | Retention |
| - | - |
| Starter | 30 days |
| Business | 90 days |
After the retention period, data is automatically deleted. This applies to all signal types (traces, logs, metrics).
## Tracking Your Usage
Head to your **Organization Settings → Billing** to monitor your usage in real time. You'll see:
* **Overall usage** — how much data you've ingested this billing period, and what percentage of your plan's limit you've reached
* **Breakdown by application** — which app is sending the most data, in GB
* **Breakdown by environment** — drill down to see usage per environment within each app
This helps you identify which applications or environments are driving your data volume, so you can optimize instrumentation or plan upgrades accordingly.
## Managing Your Subscription
From the same **Billing** section, you can also:
* Upgrade or downgrade your plan
* Update your payment method
* Access your invoices through the Stripe billing portal
## FAQ
Yes. Every new organization gets a 14-day trial with full access to all features.
Yes. Paid plans use soft limits — your data is never blocked. You'll be charged for overage at your plan's per-GB rate. Free/trial plans do not have overage charges, but data ingestion may be limited.
Data size is based on the estimated bytes of all ingested telemetry (traces, logs, and metrics). This is the actual payload size of your OpenTelemetry data as stored in Monocle.
Yes. Plan changes take effect immediately. If you upgrade, you get access to the higher data limit and longer retention right away.
If you need higher limits or custom retention periods, contact us to discuss enterprise options.
# CLI Commands
Source: https://docs.monocle.sh/studio/cli
Query traces, logs, and errors from Monocle Studio directly in your terminal
Monocle Studio exposes CLI commands to query your local telemetry data without opening the browser. Useful for quick checks, scripting, or piping data into other tools.
These commands are also great for AI coding agents. Instead of copy-pasting logs from your terminal into Claude or Codex, your agent can call `monocle studio errors` or `monocle studio traces` directly and get full context on its own.
## Dev
Start the local Studio server. This is the main entry point for Monocle Studio.
```bash theme={"theme":"vesper"}
monocle studio dev
```
| Option | Default | Description |
| - | - | - |
| `--port` | `4200` | Port for the Studio UI and API |
| `--host` | `0.0.0.0` | Host to bind to |
| `--open` | | Open browser automatically |
| `--clean` | | Clear all stored data before starting |
| `--db-path` | | Custom path for the DuckDB database |
```bash theme={"theme":"vesper"}
monocle studio dev --port 3000 --open
monocle studio dev --clean # Fresh start, no past data
```
## Traces
Fetch recent traces from Studio. Each trace shows the HTTP method, path, status code, duration, and service name.
```bash theme={"theme":"vesper"}
monocle studio traces
```
You can filter by service name or trace ID, and control the output format.
```bash theme={"theme":"vesper"}
monocle studio traces --service my-api --format json
monocle studio traces --trace-id abc123 --limit 10
```
## Logs
Fetch recent logs with optional severity and search filters.
```bash theme={"theme":"vesper"}
monocle studio logs
monocle studio logs --severity error
monocle studio logs --search "payment failed" --limit 100
```
## Errors
Fetch recent exceptions with type, message, and stack trace preview.
```bash theme={"theme":"vesper"}
monocle studio errors
monocle studio errors --trace-id abc123
```
## Install agent instructions
Append a managed Monocle Studio block to `AGENTS.md` or `CLAUDE.md` so your agents know about Monocle Studio and how to use it.
```bash theme={"theme":"vesper"}
monocle studio install-agents
```
The command prompts for the target file, appends the block at the end, and updates only that managed block on later runs.
## Output formats
The `traces`, `logs`, and `errors` commands support three output formats via the `--format` flag.
| Format | Description |
| - | - |
| `pretty` | Human-readable with colors and indentation (default) |
| `short` | One line per entry, compact |
| `json` | Raw JSON, useful for piping to `jq` or other tools |
```bash theme={"theme":"vesper"}
monocle studio traces --format short
monocle studio logs --format json | jq '.[] | .body'
```
## Common options
The `traces`, `logs`, and `errors` commands share these options.
| Option | Default | Description |
| - | - | - |
| `-p, --port` | `4200` | Studio server port |
| `-f, --format` | `pretty` | Output format: `pretty`, `json`, `short` |
| `-n, --limit` | `25` | Number of entries to fetch |
| `--trace-id` | | Filter by trace ID |
Make sure Monocle Studio is running (`monocle studio dev`) before using the query commands. They connect to the local Studio server over HTTP.
# Desktop App
Source: https://docs.monocle.sh/studio/desktop-app
Run Monocle Studio as a standalone desktop application, no terminal needed
Monocle Studio also ships as a **standalone desktop app** for macOS, Linux, and Windows. Same features as the CLI version, packaged as a native application.
The desktop app is completely optional. It runs the exact same Studio server under the hood, just wrapped in a native window instead of your browser. Pick whichever you prefer.
## Why use the desktop app?
The CLI version works great. The desktop app adds a few quality-of-life improvements:
* **Always running in the background.** Studio lives in your system tray. The OTLP endpoint stays active even when you close the window. No terminal tab to keep open.
* **One click to open.** Click the tray icon instead of switching to a terminal and running a command.
* **Auto-updates.** New versions are downloaded and installed silently. No `npm update` needed.
* **Dedicated window.** Studio gets its own window in your taskbar/dock instead of being a browser tab you might accidentally close.
## No telemetry
The desktop app does not collect any telemetry, analytics, or usage data. Your traces and logs stay on your machine in `~/.config/monocle/studio.db`, exactly like the CLI version.
## Download
M1, M2, M3, M4
Intel-based Macs
AppImage (x64)
Installer (x64)
The app is not code-signed yet.
**macOS:** Run this command after downloading, then open the app normally:
```bash theme={"theme":"vesper"}
xattr -cr /Applications/Monocle\ Studio.app
```
**Windows:** Click **More info > Run anyway** in the SmartScreen dialog.
## How it works
The desktop app runs the same Studio server as `monocle studio dev`, an AdonisJS HTTP server with DuckDB storage listening on `localhost:4200`. Your instrumented app sends OTLP data to that port, and the Studio UI renders it.
When you close the window, the app stays in the system tray and the OTLP endpoint keeps running. Click the tray icon to reopen the window, or right-click to quit entirely.
## System requirements
| Platform | Minimum version |
| - | - |
| macOS | 11 (Big Sur) |
| Linux | Ubuntu 20.04 / Fedora 38 / any distro with glibc 2.31+ |
| Windows | 10 (64-bit) |
# MCP Server
Source: https://docs.monocle.sh/studio/mcp
Connect Claude Code, Cursor, or any MCP client to your local traces, logs, and queries
Monocle Studio includes a built-in [MCP](https://modelcontextprotocol.io/) server. This lets AI coding assistants like Claude Code, Cursor, or Codex query your local telemetry data directly — find errors, inspect traces, and debug performance issues without leaving your editor.
## Setup
The MCP server is available at `http://localhost:4200/mcp` when Monocle Studio is running.
If you also want your agents to know about Monocle Studio and how to use it, run:
```bash theme={"theme":"vesper"}
monocle studio install-agents
```
```bash theme={"theme":"vesper"}
pnpm dlx add-mcp http://localhost:4200/mcp -n monocle-studio -a claude-code
```
```bash theme={"theme":"vesper"}
pnpm dlx add-mcp http://localhost:4200/mcp -n monocle-studio -a cursor
```
```bash theme={"theme":"vesper"}
pnpm dlx add-mcp http://localhost:4200/mcp -n monocle-studio -a codex
```
`add-mcp` supports 15+ clients. Run `pnpm dlx add-mcp --all` to install for all detected clients, or configure manually:
```json theme={"theme":"vesper"}
{
"mcpServers": {
"monocle-studio": {
"type": "url",
"url": "http://localhost:4200/mcp"
}
}
}
```
## Available tools
| Tool | Description |
| - | - |
| `list_traces` | List recent traces with entry point, duration, and status. Supports filtering by trace ID. |
| `get_trace` | Get the complete span waterfall for a trace — parent/child relationships, durations, attributes. |
| `list_logs` | List logs filtered by severity, search text, or trace ID. |
| `list_errors` | List exceptions with type, message, stacktrace, and trace context. |
| `list_queries` | List database queries with SQL statement, duration, and system. Filter by database type. |
| `clear_data` | Clear all collected telemetry for a fresh debugging session. |
## Examples
Once connected, you can ask your AI assistant things like:
* *"What errors happened in the last few requests?"*
* *"Show me the trace for the slow /api/users endpoint"*
* *"Find all database queries slower than 200ms"*
* *"What's the overall health of my app right now?"*
* *"Clear all data and start a fresh debugging session"*
The assistant will call the appropriate MCP tools, query Monocle Studio, and give you the answer in context.
# Monocle Studio
Source: https://docs.monocle.sh/studio/overview
Local observability for development — traces, logs, exceptions, and queries in your browser
Monocle gives you full observability in production — traces, logs, exceptions, performance. But what about when you're developing locally?
**Monocle Studio** is a lightweight companion built for local development. One command, and you get a focused dashboard showing what's happening inside your app as you code. No setup, no external services — it just works.
![Monocle Studio]()
Think of it as browser DevTools, but for your backend. Every HTTP request, every database query, every log — visible in real-time as you develop.
* See the full trace waterfall for every request
* Track outgoing HTTP calls to external APIs
* Spot slow queries before they become production issues
* Browse structured logs with stack traces and JSON attributes
* Catch exceptions before they ever reach your users
* Connect your AI coding assistant via MCP to query your traces
## How it works
Your app uses the same Monocle agent as in production. In development, the agent automatically sends telemetry to Monocle Studio instead of the cloud. Studio receives it, stores it locally, and serves a dashboard — all on a single port (`localhost:4200`). Data persists between restarts so you can review past sessions.
## Features
HTTP requests with full span waterfall, timing, and attributes
Structured logs with severity filtering, JSON viewer, and stack traces
Exceptions with stack traces, grouped by type
SQL queries with duration and statement preview
Outgoing HTTP calls to external APIs with domain, status, and duration
Job execution, queue name, duration, and success/failure tracking
Traces for CLI scripts and scheduled tasks with custom spans
Let Claude Code or Cursor query your local traces and logs
New data appears instantly — no manual refresh needed
## Quick start
```bash theme={"theme":"vesper"}
npm install -D @monocle.sh/studio @monocle.sh/cli
```
Dev mode activates automatically when `NODE_ENV=development` (the default in AdonisJS during `node ace serve`). No extra config needed — the agent detects it and sends data to Monocle Studio instead of the cloud.
```bash theme={"theme":"vesper"}
monocle studio install-agents
```
This appends a managed Monocle Studio block to `AGENTS.md` or `CLAUDE.md` so your agents know about Monocle Studio and how to use it.
```bash theme={"theme":"vesper"}
monocle studio dev
```
This starts the Studio server on `http://localhost:4200`.
```bash theme={"theme":"vesper"}
node ace serve --hmr
```
Make some requests, then open `http://localhost:4200` to see your traces, logs, and exceptions.
## CLI options
```bash theme={"theme":"vesper"}
monocle studio dev # Start on default port 4200
monocle studio dev --port 3000 # Custom port
monocle studio dev --host localhost # Bind to localhost only
monocle studio dev --open # Open browser automatically
monocle studio dev --clean # Clear all stored data before starting
monocle studio dev --db-path ./my.db # Custom database path
```
## CLI commands
You can also query Studio data directly from your terminal without opening the browser. Fetch traces, logs, and errors with `monocle studio traces`, `monocle studio logs`, and `monocle studio errors`. See the [CLI reference](/studio/cli) for all options.
## Desktop app
Prefer a native app over a terminal command? Monocle Studio is also available as a [standalone desktop application](/studio/desktop-app) for macOS, Linux, and Windows — with system tray support and auto-updates.
## Data storage
Monocle Studio stores data locally at `~/.config/monocle/studio.db`. Data persists between restarts. Use `monocle studio dev --clean` to start fresh.
# Support
Source: https://docs.monocle.sh/support
Get help from the Monocle team
Setting up Monocle and something's not clicking? Found a bug that's driving you crazy? Got questions about pricing or billing, or just want to know if Monocle is the right fit before committing? Drop us a message, we'll get back to you quickly.
Reach out at **[support@monocle.sh](mailto:support@monocle.sh)** for any questions, bug reports, or feature requests.
Stay up to date with the latest news and updates.