Get automatic logging, distributed tracing, and real-time metrics including latency, token usage, and request data for agents deployed on Blaxel.
Deploying and running agents on Blaxel gives you total observability by design, without needing to install any additional library. When you either deploy from the console’s low-code builder or by using the Blaxel SDK to wrap your code for deployment, Blaxel will automatically implement logging & tracing on your requests.
Monitor from the Blaxel console
There are three ways you can observe and monitor your running workloads:
Metrics
Logs
Traces
Metrics
Metrics are aggregated data about workload executions. They include:
Number and rate of requests
End-to-end latency: average, p50, p90, p99
Number of tokens generated by model APIs: input, output and total
City and country of origin of all requests
Logs
Logs are timestamped data about what happens with your agents, functions and model APIs. Such data includes:
Status of all requests on agents, functions and model APIs
Logs generated when building your deployments
Custom logs generated by your agents and functions
Traces
Tracing helps you understand how your agent works by showing you all its parts in action. When you make a request, it creates a trace that contains multiple spans.Think of a span as a building block - it shows when something starts and ends, what went in, what came out, and other useful details. Spans can contain other spans, like when one function calls another. You’ll often see spans for things like LLM calls, tool calls, or steps in an agent’s process. You can click on any trace to see all its spans laid out clearly. This makes it much easier to follow how your agent works and fix any problems you find.Blaxel collects and saves the traces of a sampled 10% of all your executions. In order to force saving the trace on an execution, call the run API and add query parameter debug:true.
Add custom spans
You can add your own information to traces, such as inputs, outputs, and intermediate results. This is useful when you need to perform detailed debugging.
import "@blaxel/telemetry";import { startSpan } from "@blaxel/telemetry";const span = startSpan("span-name", { attributes: { "input": JSON.stringify(userInput), },});try { // ... your code here ... span.setAttribute("action.result", JSON.stringify(result));} finally { span.end();}
import blaxel.telemetryfrom opentelemetry import tracetracer = trace.get_tracer(__name__)with tracer.start_as_current_span("span-name", attributes={"input": user_input}) as span: # ... your code here ... span.set_attribute("action.result", str(result))
You can call startSpan anywhere inside a request handler, and the span will be attached to the active trace for that request.
Opt out
This feature is controlled by the BL_ENABLE_OPENTELEMETRY environment variable.When you deploy an agent to Blaxel, the platform automatically injects BL_ENABLE_OPENTELEMETRY=true into the environment.When developing locally, this environment variable is not set and therefore defaults to false.To explicitly disable telemetry:
export BL_ENABLE_OPENTELEMETRY=false
Setting DO_NOT_TRACK=1 has no effect on OpenTelemetry.
{"docsConfig":{"theme":"mint","$schema":"https://mintlify.com/docs.json","name":"Documentation","description":"Blaxel documentation: cloud sandboxes, AI agent hosting, MCP servers, batch jobs, and model gateway for building and deploying AI-powered applications.","colors":{"primary":"#F0652D","light":"#FD7B35","dark":"#F0652D"},"logo":{"light":"https://mintcdn.com/blaxel/Z-RIT2VoXMT2Aw7L/images/logo-light.svg?fit=max\u0026auto=format\u0026n=Z-RIT2VoXMT2Aw7L\u0026q=85\u0026s=7a7cf1e3c766c77f97f32e5c0d4fd209","dark":"https://mintcdn.com/blaxel/Z-RIT2VoXMT2Aw7L/images/logo-dark.svg?fit=max\u0026auto=format\u0026n=Z-RIT2VoXMT2Aw7L\u0026q=85\u0026s=17297eca3e92fe15a91eed262f17d659"},"favicon":"https://mintcdn.com/blaxel/nH6Nd4qmP2OsLhOO/favicon.svg?fit=max\u0026auto=format\u0026n=nH6Nd4qmP2OsLhOO\u0026q=85\u0026s=484b8f0ebe3fb56aef259d514eeaa18c","appearance":{"default":"system"},"background":{"color":{"dark":"#09090B"}},"navbar":{"links":[{"icon":"calendar","href":"/changelog","label":"Changelog"},{"icon":"robot","href":"/skills-mcp","label":"Agent Skill"},{"icon":"puzzle-piece","href":"https://blaxel.ai/integrations","label":"Integrations"},{"icon":"discord","href":"/Community","label":"Community"}],"primary":{"type":"button","label":"Blaxel Console","href":"https://app.blaxel.ai"}},"footer":{"socials":{"github":"https://github.com/blaxel-ai","linkedin":"https://www.linkedin.com/company/blaxel-ai","twitter":"https://x.com/blaxelai","discord":"https://discord.gg/CsWKUZUHFQ"}},"search":{"prompt":"Search Blaxel docs..."},"seo":{"metatags":{"og:type":"website","og:site_name":"Blaxel Documentation"},"indexing":"navigable"},"fonts":{"family":"Inter"},"integrations":{"posthog":{"apiKey":"phc_HGJ7t7Qa8cJJilL0CONU5se4vM6jU6nKFCuoCC9LSha","apiHost":"https://events.blaxel.ai"}},"banner":{"content":"Blaxel has joined Baseten to build the future of agentic infrastructure! [Read our announcement.](https://www.baseten.co/blog/blaxel-is-joining-baseten-to-build-the-future-of-agentic-cloud/)","dismissible":true},"contextual":{"options":["copy","view","chatgpt","claude","cursor","devin","download-spec"],"display":"header"},"metadata":{"timestamp":true},"styling":{"latex":false},"navigation":{"global":{"anchors":[{"anchor":"Changelog","icon":"calendar","href":"/changelog"},{"anchor":"Blaxel Skills \u0026 MCP","icon":"robot","href":"/skills-mcp"},{"anchor":"Integrations directory","icon":"puzzle-piece","href":"https://blaxel.ai/integrations"},{"anchor":"Community","icon":"discord","href":"/Community"}]},"pages":[]}},"deployment":{"deploymentMetadataValue":{"subdomain":"blaxel","hasAudioNarration":false,"actualSubdomain":"blaxel","customDomains":["docs.blaxel.ai"],"gitSource":{"type":"github","owner":"blaxel-ai","repo":"docs","deployBranch":"main","contentDirectory":"","isPrivate":true,"mountPath":"","label":""},"feedback":{"thumbs":true,"issues":true,"edits":true},"entitlements":{"AI_CHAT":{"status":"ENABLED"},"WEB_MCP":{"status":"ENABLED"},"NARRATION":{"status":"ENABLED"}},"buildId":"6afd038b9f3c20a72e93d2de","clientVersion":"0.0.4227","searchFilterCounts":[{"language":"default","tag":"Platform APIs Reference","count":180},{"language":"default","tag":"Management API","count":177},{"language":"default","tag":"Documentation","count":115},{"language":"default","tag":"Overview","count":75},{"language":"default","tag":"CLI Reference","count":73},{"language":"default","tag":"Commands","count":72},{"language":"default","tag":"Sandbox API Reference","count":46},{"language":"default","tag":"Troubleshooting","count":41},{"language":"default","tag":"Tutorials and Examples","count":35},{"language":"default","tag":"compute","count":35}]},"assistantConfig":{"enableStarterQuestions":true,"enableContextualStarterQuestions":true,"deflection":{"enabled":true,"email":"support@blaxel.ai","showHelpButton":true}},"basePath":"","hasChatPermissions":true,"hasWebMcpPermissions":true,"toggles":[{"name":"kill-isr","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"custom-rbac-roles","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"agent-mintlify-slack","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"agent-integrations-settings","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"rbac-expansion","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"automations-pr-grouping","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"custom-prefix","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"analytics-v7","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"user-flows-analytics","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"no-publish-flow","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workos-onboarding","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"snapshot-serving-live","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"stripTemperatureMiddleware","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"support-platform-pylon-zendesk","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"snapshot-serving","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"instant-rollback","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"granular-notifications-server","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"granular-notifications","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"automations-fable","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"fixed-ai-action-pricing","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"automation-resolve-merge-conflict","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"scoped-publishing","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"mintql","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"page-level-analytics","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"editor-version-history-v2","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"billing-page-v2","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"full-page-search","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workspace-projects","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"assistant-v5","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"fork-pr-previews","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"revalidation-jwt","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workos-sso-scim","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"hobby-abuse-scan","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"dashboard-inbox","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workos-signup","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"turbopuffer-dual-write","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workos-enabled","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"approve-review-flow","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"editor-metadata","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"sso-setup-deadline-banner","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"turbopuffer-serve","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"live-agents","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"starter-pricing","enabled":true,"variant":{"name":"control","enabled":true,"payload":{"type":"string","value":"control"},"feature_enabled":true},"impressionData":false},{"name":"llms-txt-v2","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"wildcard-email-auth","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false}],"googleFontsInlined":false},"customCss":[{"filePath":"style.css","content":"#navbar .nav-logo{height:2.5rem;width:auto}#navbar:has(#banner){display:flex;flex-direction:column}#navbar:has(#banner)>.max-w-8xl{width:100%}#navbar:has(#banner)>div[aria-hidden=\"true\"]{display:none}#banner{position:relative;inset:auto;order:1;padding:1.25rem 3rem;background:transparent;font-size:0.9375rem;font-weight:400}#banner>div{width:100%;max-width:70rem;margin:0;padding:1rem 3.5rem 1rem 1.75rem;border:1px solid #394655;border-radius:0.75rem;background:radial-gradient(ellipse at 50% 0, rgb(99 216 226 / 12%), transparent 70%),\n linear-gradient(110deg, #121b29, #081827 52%, #141925);box-shadow:0 12px 32px -16px rgb(7 11 17 / 40%),\n inset 0 1px 0 rgb(255 255 255 / 5%)}#banner>button{right:max(4rem, calc((100% - 70rem) / 2 + 1rem))}#banner p{color:#dbdbdb!important;line-height:1.5}#banner a{display:inline-flex;align-items:center;gap:0.5rem;margin-left:0.75rem;color:#fff!important;text-decoration:none;white-space:nowrap}#banner a::after{content:\"\";width:1.25rem;height:1.25rem;background-color:#eb7229;mask:url(\"data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 18 18' fill='none' stroke='black' stroke-width='1.25' stroke-linecap='round' stroke-linejoin='round'%3E%3Cpath d='M3 9h12M10 4l5 5-5 5'/%3E%3C/svg%3E\") center / contain no-repeat}#banner a:hover{text-decoration:underline;text-decoration-color:#fff}#banner a:focus-visible{border-radius:2px;outline:2px solid #eb7229;outline-offset:4px}@media (max-width: 767px){#banner{padding:1rem;text-align:left}#banner>div{padding:1rem 2.75rem 1rem 1rem}#banner a{display:flex;width:fit-content;margin:0.375rem 0 0}#banner>button{top:1.75rem;right:1.75rem;transform:none}}"}]}
{"page":{"pagePath":"Observability/Overview","slug":"/Observability/Overview","pageMetadata":{"title":"Logs \u0026 traces","description":"Get automatic logging, distributed tracing, and real-time metrics including latency, token usage, and request data for agents deployed on Blaxel.","href":"/Observability/Overview"},"description":{"compiledSource":"\"use strict\";\nconst {jsx: _jsx} = arguments[0];\nconst {useMDXComponents: _provideComponents} = arguments[0];\nfunction _createMdxContent(props) {\n const _components = {\n p: \"p\",\n ..._provideComponents(),\n ...props.components\n };\n return _jsx(_components.p, {\n children: \"Get automatic logging, distributed tracing, and real-time metrics including latency, token usage, and request data for agents deployed on Blaxel.\"\n });\n}\nfunction MDXContent(props = {}) {\n const {wrapper: MDXLayout} = {\n ..._provideComponents(),\n ...props.components\n };\n return MDXLayout ? _jsx(MDXLayout, {\n ...props,\n children: _jsx(_createMdxContent, {\n ...props\n })\n }) : _createMdxContent(props);\n}\nreturn {\n default: MDXContent\n};\n","frontmatter":{},"scope":{}},"pageType":"default","mdx":{"mdxSource":{"compiledSource":"\"use strict\";const{Fragment:_Fragment,jsx:_jsx,jsxs:_jsxs}=arguments[0];const{useMDXComponents:_provideComponents}=arguments[0];function _createMdxContent(props){const _components={a:`a`,code:`code`,em:`em`,li:`li`,p:`p`,pre:`pre`,span:`span`,strong:`strong`,ul:`ul`,..._provideComponents(),...props.components},{CodeBlock,CodeGroup,Heading,OptimizedImage,Warning}=_components;if(!CodeBlock)_missingMdxReference(`CodeBlock`,true);if(!CodeGroup)_missingMdxReference(`CodeGroup`,true);if(!Heading)_missingMdxReference(`Heading`,true);if(!OptimizedImage)_missingMdxReference(`OptimizedImage`,true);if(!Warning)_missingMdxReference(`Warning`,true);return _jsxs(_Fragment,{children:[_jsxs(_components.p,{children:[`Deploying and running agents on Blaxel gives you `,_jsx(_components.strong,{children:`total observability by design`}),`, without needing to install any additional library. When you either deploy from the console’s low-code builder or by using the Blaxel SDK to wrap your code for deployment, Blaxel will automatically implement logging \u0026 tracing on your requests.`]}),`\n`,_jsx(Heading,{level:`2`,id:`monitor-from-the-blaxel-console`,children:`Monitor from the Blaxel console`}),`\n`,_jsx(_components.p,{children:`There are three ways you can observe and monitor your running workloads:`}),`\n`,_jsxs(_components.ul,{children:[`\n`,_jsx(_components.li,{children:`Metrics`}),`\n`,_jsx(_components.li,{children:`Logs`}),`\n`,_jsx(_components.li,{children:`Traces`}),`\n`]}),`\n`,_jsx(Heading,{level:`3`,id:`metrics`,children:`Metrics`}),`\n`,_jsx(_components.p,{children:`Metrics are aggregated data about workload executions. They include:`}),`\n`,_jsxs(_components.ul,{children:[`\n`,_jsx(_components.li,{children:`Number and rate of requests`}),`\n`,_jsx(_components.li,{children:`End-to-end latency: average, p50, p90, p99`}),`\n`,_jsx(_components.li,{children:`Number of tokens generated by model APIs: input, output and total`}),`\n`,_jsx(_components.li,{children:`City and country of origin of all requests`}),`\n`]}),`\n`,_jsx(OptimizedImage,{src:`https://mintcdn.com/blaxel/opPwRTewCbLy-pVp/Observability/Overview/image.webp?fit=max\u0026auto=format\u0026n=opPwRTewCbLy-pVp\u0026q=85\u0026s=80107c67289f9a35637a6598df00bcb8`,alt:`image.webp`,\"data-og-width\":`1434`,width:`1434`,\"data-og-height\":`1226`,height:`1226`,\"data-path\":`Observability/Overview/image.webp`,\"data-optimize\":`true`,\"data-opv\":`3`,srcset:`https://mintcdn.com/blaxel/opPwRTewCbLy-pVp/Observability/Overview/image.webp?w=280\u0026fit=max\u0026auto=format\u0026n=opPwRTewCbLy-pVp\u0026q=85\u0026s=1d6173bc32f0af2ebfc5a6919fde9a29 280w, https://mintcdn.com/blaxel/opPwRTewCbLy-pVp/Observability/Overview/image.webp?w=560\u0026fit=max\u0026auto=format\u0026n=opPwRTewCbLy-pVp\u0026q=85\u0026s=c8f6efe6df1de291d82a3c8e6492a034 560w, https://mintcdn.com/blaxel/opPwRTewCbLy-pVp/Observability/Overview/image.webp?w=840\u0026fit=max\u0026auto=format\u0026n=opPwRTewCbLy-pVp\u0026q=85\u0026s=aacab39c3ad1e58e704f4f0c307cb3a2 840w, https://mintcdn.com/blaxel/opPwRTewCbLy-pVp/Observability/Overview/image.webp?w=1100\u0026fit=max\u0026auto=format\u0026n=opPwRTewCbLy-pVp\u0026q=85\u0026s=8ab35b9f6c3976eac1e7634fbbe5aa0b 1100w, https://mintcdn.com/blaxel/opPwRTewCbLy-pVp/Observability/Overview/image.webp?w=1650\u0026fit=max\u0026auto=format\u0026n=opPwRTewCbLy-pVp\u0026q=85\u0026s=babf41cc8dd2718eeaf23dcab58c0bc4 1650w, https://mintcdn.com/blaxel/opPwRTewCbLy-pVp/Observability/Overview/image.webp?w=2500\u0026fit=max\u0026auto=format\u0026n=opPwRTewCbLy-pVp\u0026q=85\u0026s=d5d8c46851f6f3ded855d9f157bcd571 2500w`}),`\n`,_jsx(OptimizedImage,{src:`https://mintcdn.com/blaxel/opPwRTewCbLy-pVp/Observability/Overview/image1.webp?fit=max\u0026auto=format\u0026n=opPwRTewCbLy-pVp\u0026q=85\u0026s=a3ec36312e2f3e85675d4c2dc95aac26`,alt:`image.webp`,\"data-og-width\":`1376`,width:`1376`,\"data-og-height\":`492`,height:`492`,\"data-path\":`Observability/Overview/image1.webp`,\"data-optimize\":`true`,\"data-opv\":`3`,srcset:`https://mintcdn.com/blaxel/opPwRTewCbLy-pVp/Observability/Overview/image1.webp?w=280\u0026fit=max\u0026auto=format\u0026n=opPwRTewCbLy-pVp\u0026q=85\u0026s=0cf5aa309e6ecdc9b7c4d6897ec08d44 280w, https://mintcdn.com/blaxel/opPwRTewCbLy-pVp/Observability/Overview/image1.webp?w=560\u0026fit=max\u0026auto=format\u0026n=opPwRTewCbLy-pVp\u0026q=85\u0026s=5a0245b70890c11f93e68d35884a5bf9 560w, https://mintcdn.com/blaxel/opPwRTewCbLy-pVp/Observability/Overview/image1.webp?w=840\u0026fit=max\u0026auto=format\u0026n=opPwRTewCbLy-pVp\u0026q=85\u0026s=acd49356d2ac0c5ecb9be14beb6a8aea 840w, https://mintcdn.com/blaxel/opPwRTewCbLy-pVp/Observability/Overview/image1.webp?w=1100\u0026fit=max\u0026auto=format\u0026n=opPwRTewCbLy-pVp\u0026q=85\u0026s=e737b4b6e541f6f15cb9853b1bc07a4f 1100w, https://mintcdn.com/blaxel/opPwRTewCbLy-pVp/Observability/Overview/image1.webp?w=1650\u0026fit=max\u0026auto=format\u0026n=opPwRTewCbLy-pVp\u0026q=85\u0026s=130d16500c183a2f4b09315b224ca1a1 1650w, https://mintcdn.com/blaxel/opPwRTewCbLy-pVp/Observability/Overview/image1.webp?w=2500\u0026fit=max\u0026auto=format\u0026n=opPwRTewCbLy-pVp\u0026q=85\u0026s=7782234bd1a0703d2a45b92d85d4ddf6 2500w`}),`\n`,_jsx(Heading,{level:`3`,id:`logs`,children:`Logs`}),`\n`,_jsx(_components.p,{children:`Logs are timestamped data about what happens with your agents, functions and model APIs. Such data includes:`}),`\n`,_jsxs(_components.ul,{children:[`\n`,_jsx(_components.li,{children:`Status of all requests on agents, functions and model APIs`}),`\n`,_jsx(_components.li,{children:`Logs generated when building your deployments`}),`\n`,_jsx(_components.li,{children:`Custom logs generated by your agents and functions`}),`\n`]}),`\n`,_jsx(OptimizedImage,{src:`https://mintcdn.com/blaxel/opPwRTewCbLy-pVp/Observability/Overview/image2.webp?fit=max\u0026auto=format\u0026n=opPwRTewCbLy-pVp\u0026q=85\u0026s=08b502cd812f35e6a47400c746683f2b`,alt:`image.webp`,\"data-og-width\":`1563`,width:`1563`,\"data-og-height\":`1163`,height:`1163`,\"data-path\":`Observability/Overview/image2.webp`,\"data-optimize\":`true`,\"data-opv\":`3`,srcset:`https://mintcdn.com/blaxel/opPwRTewCbLy-pVp/Observability/Overview/image2.webp?w=280\u0026fit=max\u0026auto=format\u0026n=opPwRTewCbLy-pVp\u0026q=85\u0026s=0f1f20ea50276046b590ec53518c42a6 280w, https://mintcdn.com/blaxel/opPwRTewCbLy-pVp/Observability/Overview/image2.webp?w=560\u0026fit=max\u0026auto=format\u0026n=opPwRTewCbLy-pVp\u0026q=85\u0026s=5536c1ccfbf0c71d0095718cdded60f5 560w, https://mintcdn.com/blaxel/opPwRTewCbLy-pVp/Observability/Overview/image2.webp?w=840\u0026fit=max\u0026auto=format\u0026n=opPwRTewCbLy-pVp\u0026q=85\u0026s=d7e9d204edfb61c202400b33bd0121a4 840w, https://mintcdn.com/blaxel/opPwRTewCbLy-pVp/Observability/Overview/image2.webp?w=1100\u0026fit=max\u0026auto=format\u0026n=opPwRTewCbLy-pVp\u0026q=85\u0026s=49a00d807c64733ebd922f2f4d248421 1100w, https://mintcdn.com/blaxel/opPwRTewCbLy-pVp/Observability/Overview/image2.webp?w=1650\u0026fit=max\u0026auto=format\u0026n=opPwRTewCbLy-pVp\u0026q=85\u0026s=3f5f9bc9caa7905f8c99d9822f09c8d2 1650w, https://mintcdn.com/blaxel/opPwRTewCbLy-pVp/Observability/Overview/image2.webp?w=2500\u0026fit=max\u0026auto=format\u0026n=opPwRTewCbLy-pVp\u0026q=85\u0026s=cacd80cc666a673e0e1f663757f1fed4 2500w`}),`\n`,_jsx(Heading,{level:`3`,id:`traces`,children:`Traces`}),`\n`,_jsxs(_components.p,{children:[`Tracing helps you understand how your agent works by showing you all its parts in action. When you make a request, it creates a trace that contains multiple `,_jsx(_components.em,{children:`spans`}),`.`]}),`\n`,_jsx(_components.p,{children:`Think of a span as a building block - it shows when something starts and ends, what went in, what came out, and other useful details. Spans can contain other spans, like when one function calls another. You’ll often see spans for things like LLM calls, tool calls, or steps in an agent’s process. You can click on any trace to see all its spans laid out clearly. This makes it much easier to follow how your agent works and fix any problems you find.`}),`\n`,_jsxs(_components.p,{children:[`Blaxel collects and saves the traces of a `,_jsx(_components.strong,{children:`sampled 10%`}),` of all your executions. In order to force saving the trace on an execution, call the run API and add query parameter `,_jsx(_components.code,{children:`debug:true`}),`.`]}),`\n`,_jsx(OptimizedImage,{src:`https://mintcdn.com/blaxel/opPwRTewCbLy-pVp/Observability/Overview/traces.webp?fit=max\u0026auto=format\u0026n=opPwRTewCbLy-pVp\u0026q=85\u0026s=5b4c5f205c985894d79dc57aaf02d332`,alt:`traces.webp`,\"data-og-width\":`1877`,width:`1877`,\"data-og-height\":`1389`,height:`1389`,\"data-path\":`Observability/Overview/traces.webp`,\"data-optimize\":`true`,\"data-opv\":`3`,srcset:`https://mintcdn.com/blaxel/opPwRTewCbLy-pVp/Observability/Overview/traces.webp?w=280\u0026fit=max\u0026auto=format\u0026n=opPwRTewCbLy-pVp\u0026q=85\u0026s=9fb3cd2537b08eb4165b6519cce6c282 280w, https://mintcdn.com/blaxel/opPwRTewCbLy-pVp/Observability/Overview/traces.webp?w=560\u0026fit=max\u0026auto=format\u0026n=opPwRTewCbLy-pVp\u0026q=85\u0026s=cd9dfbc30ef8c50e036fa0a1fea97a85 560w, https://mintcdn.com/blaxel/opPwRTewCbLy-pVp/Observability/Overview/traces.webp?w=840\u0026fit=max\u0026auto=format\u0026n=opPwRTewCbLy-pVp\u0026q=85\u0026s=81304a17f5a7705717ab8547068fc30b 840w, https://mintcdn.com/blaxel/opPwRTewCbLy-pVp/Observability/Overview/traces.webp?w=1100\u0026fit=max\u0026auto=format\u0026n=opPwRTewCbLy-pVp\u0026q=85\u0026s=3809a534edc1ac55394733e31d8e5671 1100w, https://mintcdn.com/blaxel/opPwRTewCbLy-pVp/Observability/Overview/traces.webp?w=1650\u0026fit=max\u0026auto=format\u0026n=opPwRTewCbLy-pVp\u0026q=85\u0026s=b0b0bd7f6bb285c7567d4a2380bea226 1650w, https://mintcdn.com/blaxel/opPwRTewCbLy-pVp/Observability/Overview/traces.webp?w=2500\u0026fit=max\u0026auto=format\u0026n=opPwRTewCbLy-pVp\u0026q=85\u0026s=81df249687ab84a43ab791e5f5825c8d 2500w`}),`\n`,_jsx(Heading,{level:`4`,id:`add-custom-spans`,children:`Add custom spans`}),`\n`,_jsx(_components.p,{children:`You can add your own information to traces, such as inputs, outputs, and intermediate results. This is useful when you need to perform detailed debugging.`}),`\n`,_jsxs(CodeGroup,{children:[_jsx(CodeBlock,{filename:`TypeScript`,numberOfLines:`15`,language:`typescript`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`typescript`,children:_jsxs(_components.code,{language:`typescript`,numberOfLines:`15`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`import`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"@blaxel/telemetry\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`;`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`import`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` { `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`startSpan`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` } `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`from`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"@blaxel/telemetry\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`;`})]}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`const`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4FC1FF`},children:` span`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:` =`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` startSpan`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"span-name\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, {`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` attributes:`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` {`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"input\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`:`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4FC1FF`},children:` JSON`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:`stringify`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`userInput`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`),`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` },`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`});`})}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`try`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` {`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` // ... your code here ...`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` span`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:`setAttribute`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"action.result\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4FC1FF`},children:`JSON`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:`stringify`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`result`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`));`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`} `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`finally`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` {`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` span`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:`end`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`();`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`}`})}),`\n`]})})}),_jsx(CodeBlock,{filename:`Python`,numberOfLines:`8`,language:`python`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`python`,children:_jsxs(_components.code,{language:`python`,numberOfLines:`8`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`import`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` blaxel.telemetry`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`from`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` opentelemetry `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`import`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` trace`})]}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`tracer `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` trace.get_tracer(`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#9CDCFE`},children:`__name__`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`with`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` tracer.start_as_current_span(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"span-name\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:`attributes`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`{`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"input\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: user_input}) `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`as`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` span:`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#6E7781`,\"--shiki-dark\":`#6A9955`},children:` # ... your code here ...`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` span.set_attribute(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"action.result\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4EC9B0`},children:`str`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(result))`})]}),`\n`]})})})]}),`\n`,_jsxs(_components.p,{children:[`You can call `,_jsx(_components.code,{children:`startSpan`}),` anywhere inside a request handler, and the span will be attached to the active trace for that request.`]}),`\n`,_jsx(Heading,{level:`2`,id:`opt-out`,children:`Opt out`}),`\n`,_jsxs(_components.p,{children:[`This feature is controlled by the `,_jsx(_components.code,{children:`BL_ENABLE_OPENTELEMETRY`}),` environment variable.`]}),`\n`,_jsxs(_components.p,{children:[`When you deploy an agent to Blaxel, the platform automatically injects `,_jsx(_components.code,{children:`BL_ENABLE_OPENTELEMETRY=true`}),` into the environment.`]}),`\n`,_jsxs(_components.p,{children:[`When developing locally, this environment variable is not set and therefore defaults to `,_jsx(_components.code,{children:`false`}),`.`]}),`\n`,_jsx(_components.p,{children:`To explicitly disable telemetry:`}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`1`,language:`shellscript`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`shellscript`,children:_jsxs(_components.code,{language:`shellscript`,numberOfLines:`1`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`export`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` BL_ENABLE_OPENTELEMETRY`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`false`})]}),`\n`]})})}),`\n`,_jsx(Warning,{children:_jsxs(_components.p,{children:[`Setting `,_jsx(_components.code,{children:`DO_NOT_TRACK=1`}),` has no effect on OpenTelemetry.`]})}),`\n`,_jsxs(_components.p,{children:[_jsx(_components.a,{href:`/Security/Data-collection-and-privacy`,children:`Read more about SDK and CLI data collection and privacy`}),`.`]})]})}function MDXContent(props={}){const{wrapper:MDXLayout}={..._provideComponents(),...props.components};return MDXLayout?_jsx(MDXLayout,{...props,children:_jsx(_createMdxContent,{...props})}):_createMdxContent(props)}return{default:MDXContent};function _missingMdxReference(id,component){throw new Error(`Expected `+(component?`component`:`object`)+\" `\"+id+\"` to be defined: you likely forgot to import, pass, or provide it.\")}","frontmatter":{},"scope":{"config":{},"pageMetadata":{"title":"Logs \u0026 traces","description":"Get automatic logging, distributed tracing, and real-time metrics including latency, token usage, and request data for agents deployed on Blaxel.","href":"/Observability/Overview"}}},"mdxSourceWithNoJs":null},"mdxExtracts":{"tableOfContents":[{"title":"Monitor from the Blaxel console","slug":"monitor-from-the-blaxel-console","depth":2,"children":[{"title":"Metrics","slug":"metrics","depth":3,"children":[]},{"title":"Logs","slug":"logs","depth":3,"children":[]},{"title":"Traces","slug":"traces","depth":3,"children":[]},{"title":"Add custom spans","slug":"add-custom-spans","depth":4,"children":[]}]},{"title":"Opt out","slug":"opt-out","depth":2,"children":[]}],"codeExamples":{}},"jsLinks":[],"subdomain":"blaxel","siteName":"Documentation","title":"Logs \u0026 traces - Documentation","contextualStarterQuestions":[{"_id":"6a057b8f071845739ab5ffbf","deploymentId":"672e5f6ba9a398341da2aaec","questionText":"What metrics does Blaxel track?","status":"active","generatedBy":"contextual","path":"Observability/Overview","createdAt":"2026-05-14T07:36:47.418Z","updatedAt":"2026-05-14T07:36:47.418Z","__v":0},{"_id":"6a057b8f071845739ab5ffc1","deploymentId":"672e5f6ba9a398341da2aaec","questionText":"Is tracing setup automatic?","status":"active","generatedBy":"contextual","path":"Observability/Overview","createdAt":"2026-05-14T07:36:47.418Z","updatedAt":"2026-05-14T07:36:47.418Z","__v":0},{"_id":"6a057b8f071845739ab5ffc0","deploymentId":"672e5f6ba9a398341da2aaec","questionText":"How do I monitor my agent logs?","status":"active","generatedBy":"contextual","path":"Observability/Overview","createdAt":"2026-05-14T07:36:47.418Z","updatedAt":"2026-05-14T07:36:47.418Z","__v":0}],"navMaps":{"firstHrefInVersion":{},"firstHrefInLanguage":{}},"lastModified":"2026-05-14T07:36:33.325Z","gitLastModified":"2026-05-14T07:36:08.000Z","originalFileLocation":"Observability/Overview.mdx"},"headHtml":"\u003cscript data-mint-head>(function(){var d=document.documentElement,p=null;if(!p){try{p=localStorage.getItem(\"isDarkMode\")}catch(e){}if(p==='true')p='dark';else if(p==='false')p='light';if(p!=='light'\u0026\u0026p!=='dark'\u0026\u0026p!=='system')p=\"system\"}var r=p==='system'?(matchMedia(\"(prefers-color-scheme: dark)\").matches?'dark':'light'):p;d.classList.remove(r==='dark'?'light':'dark');d.classList.add(r);d.setAttribute(\"data-theme-preference\",p)})();\u003c/script>\u003cmeta data-mint-head name=\"description\" content=\"Get automatic logging, distributed tracing, and real-time metrics including latency, token usage, and request data for agents deployed on Blaxel.\">\u003cmeta data-mint-head name=\"application-name\" content=\"Documentation\">\u003cmeta data-mint-head name=\"generator\" content=\"Mintlify\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"192x192\" href=\"/mintlify-assets/_mintlify/favicons/blaxel/uaLoRkh_ONjRStCE/_generated/favicon/android-chrome-192x192.png\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"apple-touch-icon\" type=\"image/png\" sizes=\"180x180\" href=\"/mintlify-assets/_mintlify/favicons/blaxel/uaLoRkh_ONjRStCE/_generated/favicon/apple-touch-icon.png\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"16x16\" href=\"/mintlify-assets/_mintlify/favicons/blaxel/uaLoRkh_ONjRStCE/_generated/favicon/favicon-16x16.png\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"32x32\" href=\"/mintlify-assets/_mintlify/favicons/blaxel/uaLoRkh_ONjRStCE/_generated/favicon/favicon-32x32.png\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"shortcut icon\" type=\"image/x-icon\" href=\"/mintlify-assets/_mintlify/favicons/blaxel/uaLoRkh_ONjRStCE/_generated/favicon/favicon.ico\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"192x192\" href=\"/mintlify-assets/_mintlify/favicons/blaxel/uaLoRkh_ONjRStCE/_generated/favicon-dark/android-chrome-192x192.png\" media=\"(prefers-color-scheme: dark)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"16x16\" href=\"/mintlify-assets/_mintlify/favicons/blaxel/uaLoRkh_ONjRStCE/_generated/favicon-dark/favicon-16x16.png\" media=\"(prefers-color-scheme: dark)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"32x32\" href=\"/mintlify-assets/_mintlify/favicons/blaxel/uaLoRkh_ONjRStCE/_generated/favicon-dark/favicon-32x32.png\" media=\"(prefers-color-scheme: dark)\">\u003clink data-mint-head rel=\"shortcut icon\" type=\"image/x-icon\" href=\"/mintlify-assets/_mintlify/favicons/blaxel/uaLoRkh_ONjRStCE/_generated/favicon-dark/favicon.ico\" media=\"(prefers-color-scheme: dark)\">\u003cmeta data-mint-head property=\"og:title\" content=\"Logs \u0026amp; traces - Documentation\">\u003cmeta data-mint-head property=\"og:description\" content=\"Get automatic logging, distributed tracing, and real-time metrics including latency, token usage, and request data for agents deployed on Blaxel.\">\u003cmeta data-mint-head property=\"og:site_name\" content=\"Blaxel Documentation\">\u003cmeta data-mint-head property=\"og:url\" content=\"https://docs.blaxel.ai/Observability/Overview\">\u003cmeta data-mint-head property=\"og:type\" content=\"website\">\u003cmeta data-mint-head property=\"og:image\" content=\"https://blaxel.mintlify.site/_mintlify/api/og/v2?division=Observability\u0026amp;title=Logs+%26+traces\u0026amp;description=Get+automatic+logging%2C+distributed+tracing%2C+and+real-time+metrics+including+latency%2C+token+usage%2C+and+request+data+for+agents+deployed+on+Blaxel.\u0026amp;theme=aa6a37f877c64d380f9dda80\u0026amp;signature=b346ee40ad84e6cc894043aa16c70192795fe212202f0771d2e77ec99600178b\">\u003cmeta data-mint-head property=\"og:image:width\" content=\"1200\">\u003cmeta data-mint-head property=\"og:image:height\" content=\"630\">\u003cmeta data-mint-head name=\"twitter:card\" content=\"summary_large_image\">\u003cmeta data-mint-head name=\"twitter:title\" content=\"Logs \u0026amp; traces - Documentation\">\u003cmeta data-mint-head name=\"twitter:description\" content=\"Get automatic logging, distributed tracing, and real-time metrics including latency, token usage, and request data for agents deployed on Blaxel.\">\u003cmeta data-mint-head name=\"twitter:image\" content=\"https://blaxel.mintlify.site/_mintlify/api/og/v2?division=Observability\u0026amp;title=Logs+%26+traces\u0026amp;description=Get+automatic+logging%2C+distributed+tracing%2C+and+real-time+metrics+including+latency%2C+token+usage%2C+and+request+data+for+agents+deployed+on+Blaxel.\u0026amp;theme=aa6a37f877c64d380f9dda80\u0026amp;signature=b346ee40ad84e6cc894043aa16c70192795fe212202f0771d2e77ec99600178b\">\u003cmeta data-mint-head name=\"twitter:image:width\" content=\"1200\">\u003cmeta data-mint-head name=\"twitter:image:height\" content=\"630\">\u003cmeta data-mint-head name=\"msapplication-config\" content=\"/mintlify-assets/_mintlify/favicons/blaxel/uaLoRkh_ONjRStCE/_generated/favicon/browserconfig.xml\">\u003cmeta data-mint-head name=\"apple-mobile-web-app-title\" content=\"Documentation\">\u003cmeta data-mint-head name=\"msapplication-TileColor\" content=\"#F0652D\">\u003clink data-mint-head rel=\"canonical\" href=\"https://docs.blaxel.ai/Observability/Overview\">\u003clink data-mint-head rel=\"alternate\" type=\"application/xml\" href=\"/sitemap.xml\">\u003clink data-mint-head rel=\"alternate\" type=\"text/markdown\" href=\"/Observability/Overview.md\">\u003cscript data-mint-head type=\"application/ld+json\">{\"@context\":\"https://schema.org\",\"@graph\":[{\"@type\":\"Organization\",\"@id\":\"https://docs.blaxel.ai/#organization\",\"name\":\"Documentation\",\"url\":\"https://docs.blaxel.ai\",\"logo\":{\"@type\":\"ImageObject\",\"url\":\"https://mintcdn.com/blaxel/Z-RIT2VoXMT2Aw7L/images/logo-light.svg?fit=max\u0026auto=format\u0026n=Z-RIT2VoXMT2Aw7L\u0026q=85\u0026s=7a7cf1e3c766c77f97f32e5c0d4fd209\"}},{\"@type\":\"WebSite\",\"@id\":\"https://docs.blaxel.ai#website\",\"name\":\"Documentation\",\"url\":\"https://docs.blaxel.ai\",\"publisher\":{\"@id\":\"https://docs.blaxel.ai/#organization\"}},{\"@type\":\"WebPage\",\"@id\":\"https://docs.blaxel.ai/Observability/Overview#webpage\",\"url\":\"https://docs.blaxel.ai/Observability/Overview\",\"name\":\"Logs \u0026 traces\",\"description\":\"Get automatic logging, distributed tracing, and real-time metrics including latency, token usage, and request data for agents deployed on Blaxel.\",\"dateModified\":\"2026-05-14T07:36:08.000Z\",\"isPartOf\":{\"@id\":\"https://docs.blaxel.ai#website\"},\"breadcrumb\":{\"@id\":\"https://docs.blaxel.ai/Observability/Overview#breadcrumb\"}},{\"@type\":\"BreadcrumbList\",\"@id\":\"https://docs.blaxel.ai/Observability/Overview#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Documentation\",\"item\":\"https://docs.blaxel.ai\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Logs \u0026 traces\",\"item\":\"https://docs.blaxel.ai/Observability/Overview\"}]},{\"@type\":[\"Article\",\"TechArticle\"],\"@id\":\"https://docs.blaxel.ai/Observability/Overview#article\",\"headline\":\"Logs \u0026 traces\",\"name\":\"Logs \u0026 traces\",\"description\":\"Get automatic logging, distributed tracing, and real-time metrics including latency, token usage, and request data for agents deployed on Blaxel.\",\"url\":\"https://docs.blaxel.ai/Observability/Overview\",\"mainEntityOfPage\":{\"@id\":\"https://docs.blaxel.ai/Observability/Overview#webpage\"},\"dateModified\":\"2026-05-14T07:36:08.000Z\",\"publisher\":{\"@id\":\"https://docs.blaxel.ai/#organization\"},\"isPartOf\":{\"@id\":\"https://docs.blaxel.ai#website\"}}]}\u003c/script>\u003cscript data-mint-head type=\"application/ld+json\">{\"@context\":\"https://schema.org\",\"@type\":\"WebSite\",\"name\":\"Documentation\",\"publisher\":{\"@type\":\"Organization\",\"name\":\"Documentation\"},\"provider\":{\"@type\":\"Organization\",\"name\":\"Mintlify\",\"url\":\"https://mintlify.com\"}}\u003c/script>","agentPointerHref":"/Observability/Overview.md","siteName":"Documentation","nav":{"global":{"anchors":[{"anchor":"Changelog","icon":"calendar","href":"/changelog"},{"anchor":"Blaxel Skills \u0026 MCP","icon":"robot","href":"/skills-mcp"},{"anchor":"Integrations directory","icon":"puzzle-piece","href":"https://blaxel.ai/integrations"},{"anchor":"Community","icon":"discord","href":"/Community"}]},"tabs":[{"tab":"Documentation","groups":[{"group":"Get Started","pages":[{"title":"Blaxel: infrastructure for autonomous agents","sidebarTitle":"Overview","href":"/Overview"},{"title":"Get started","href":"/Get-started"},{"title":"Blaxel Skills \u0026 MCP","icon":"robot","href":"/skills-mcp"},{"title":"Community","sidebarTitle":"Community","href":"/Community"}]},{"group":"Sandboxes","pages":[{"title":"Sandboxes","sidebarTitle":"Overview","href":"/Sandboxes/Overview"},{"group":"Expiration policies","pages":[{"title":"Expiration policies","href":"/Sandboxes/Expiration"},{"title":"Lifecycle signals","href":"/Sandboxes/Lifecycle-signals"}]},{"title":"Process execution","href":"/Sandboxes/Processes"},{"group":"File system","pages":[{"title":"File system","sidebarTitle":"Overview","href":"/Sandboxes/Filesystem"},{"title":"Code generation tools","href":"/Sandboxes/Codegen-tools"}]},{"title":"Log streaming","href":"/Sandboxes/Log-streaming"},{"title":"Snapshots and forking","sidebarTitle":"Snapshots \u0026 forking","tag":"private preview","href":"/Sandboxes/Fork"},{"title":"Archive and unarchive sandboxes","sidebarTitle":"Archive","tag":"private preview","href":"/Sandboxes/Archive"},{"group":"Ports, previews \u0026 sessions","pages":[{"title":"Ports","href":"/Sandboxes/Ports"},{"title":"Real-time previews","href":"/Sandboxes/Preview-url"},{"title":"Handle CORS on Blaxel","sidebarTitle":"CORS","href":"/Sandboxes/CORS"},{"title":"Client-side sessions","href":"/Sandboxes/Sessions"},{"title":"Standby control","href":"/Sandboxes/Standby-control"}]},{"title":"Sandbox schedules","sidebarTitle":"Schedule","href":"/Sandboxes/Schedules"},{"group":"Proxy","pages":[{"title":"Proxy","tag":"public preview","href":"/Sandboxes/Proxy"},{"title":"Proxy routing with secrets injection","tag":"public preview","href":"/Sandboxes/Proxy-secrets-injection"},{"title":"Domain filtering","tag":"public preview","href":"/Sandboxes/Proxy-domains"}]},{"title":"Volumes for sandboxes","sidebarTitle":"Volumes","href":"/Sandboxes/Volumes"},{"title":"Build reusable sandbox images with Dockerfiles","sidebarTitle":"Images","href":"/Sandboxes/Templates"},{"title":"Run sandbox workloads as a non-root user","sidebarTitle":"Non-root user","href":"/Sandboxes/Non-root-user"},{"title":"Sandbox variables and secrets","sidebarTitle":"Variables \u0026 secrets","href":"/Sandboxes/Variables-and-secrets"},{"title":"Sandbox MCP server","sidebarTitle":"MCP server","href":"/Sandboxes/MCP"},{"title":"Sandbox best practices","sidebarTitle":"Best practices","href":"/Sandboxes/best-practices"}]},{"group":"Agent Drive","pages":[{"title":"Agent Drive overview","sidebarTitle":"Overview","tag":"private preview","href":"/Agent-drive/Overview"},{"title":"Drive permissions","sidebarTitle":"Permissions","tag":"private preview","href":"/Agent-drive/Permissions"},{"title":"Access Agent Drive over S3","sidebarTitle":"S3 endpoint","tag":"private preview","href":"/Agent-drive/S3-endpoint"}]},{"group":"Volumes","pages":[{"title":"Volumes","sidebarTitle":"Overview","href":"/Volumes/Overview"},{"title":"Volume templates","href":"/Volumes/Volumes-templates"}]},{"group":"Batch Jobs","pages":[{"title":"Jobs","sidebarTitle":"Overview","href":"/Jobs/Overview"},{"title":"Develop a job","href":"/Jobs/Develop-a-job"},{"group":"Deploy a job","pages":[{"title":"Deploy jobs","href":"/Jobs/Deploy-a-job"},{"title":"Deploy jobs with a Dockerfile","sidebarTitle":"Deploy with a Dockerfile","href":"/Jobs/Deploy-dockerfile-jobs"},{"title":"Deploy multiple jobs from a mono-repo","sidebarTitle":"Deploy multiple resources","href":"/Jobs/Deploy-multiple-jobs"},{"title":"Deploy multiple jobs from a single image","href":"/Jobs/Deploy-shared-image"}]},{"group":"Manage job execution","pages":[{"title":"Manage job execution in TypeScript","href":"/Jobs/Manage-job-execution-ts"},{"title":"Manage job execution in Python","href":"/Jobs/Manage-job-execution-py"}]},{"title":"Variables and secrets for jobs","sidebarTitle":"Variables and secrets","href":"/Jobs/Variables-and-secrets-jobs"}]},{"group":"Applications","pages":[{"title":"Applications","sidebarTitle":"Overview","tag":"private preview","href":"/Applications/Overview"}]},{"group":"MCP Servers Hosting","pages":[{"title":"MCP servers","sidebarTitle":"Overview","href":"/Functions/Overview"},{"title":"Quickstart: Adapt an existing MCP server for Blaxel","href":"/Functions/Adapt-Existing-MCP-server"},{"title":"Quickstart: Develop a custom MCP server","href":"/Functions/Create-MCP-server"},{"title":"Deploy custom MCP servers","href":"/Functions/Deploy-a-function"},{"title":"Query MCP servers","href":"/Functions/Invoke-functions"},{"title":"Code mode","href":"/Functions/Code-mode"},{"title":"Variables and secrets for MCP servers","sidebarTitle":"Variables and secrets","href":"/Functions/Variables-and-secrets"},{"title":"OAuth flow for integrations","href":"/Functions/oauth-flow"}]},{"group":"Model Gateway","pages":[{"title":"Model APIs","sidebarTitle":"Overview","href":"/Models/Overview"},{"title":"External model APIs","href":"/Models/External-model-apis"},{"title":"Query a model API","description":"Make inference requests to model APIs via their global endpoints on the Global Agentics Network, with ChatCompletions and custom sub-endpoint support.","href":"/Models/Query-a-model"}]},{"group":"Observability","pages":[{"title":"Logs \u0026 traces","href":"/Observability/Overview"}]},{"group":"Integrations","pages":[{"title":"Integrations","sidebarTitle":"Overview","description":"Browse Blaxel integrations that connect your agents to LLM providers, SaaS apps, databases, and private APIs through pre-built MCP servers.","href":"/Integrations"},{"group":"Model providers integrations","pages":[{"title":"OpenAI integration","href":"/Integrations/OpenAI"},{"title":"Anthropic integration","href":"/Integrations/Anthropic"},{"title":"Mistral AI integration","href":"/Integrations/MistralAI"},{"title":"Cohere integration","href":"/Integrations/Cohere"},{"title":"xAI integration","href":"/Integrations/xAI"},{"title":"DeepSeek integration","href":"/Integrations/DeepSeek"},{"title":"HuggingFace integration","href":"/Integrations/HuggingFace"},{"title":"Azure AI Foundry integration","href":"/Integrations/Azure-AI-Foundry"}]},{"group":"MCP server integrations","pages":[{"title":"Google Maps","href":"/Integrations/Google-Maps"},{"title":"Gmail","href":"/Integrations/Gmail"}]},{"group":"Agent frameworks and tools","pages":[{"title":"OpenAI Agents API on Blaxel","sidebarTitle":"OpenAI Agents API","href":"/Integrations/OpenAI-Agents-API"},{"title":"AgentMail","href":"/Integrations/AgentMail"},{"title":"Rippletide","href":"/Integrations/Rippletide"}]}]},{"group":"Administration \u0026 security","pages":[{"group":"Workspaces and permissions","pages":[{"title":"Workspaces, users and roles","href":"/Security/Workspace-access-control"},{"title":"Baseten managed workspaces","href":"/Security/Baseten-managed-workspaces"},{"title":"Service accounts","href":"/Security/Service-accounts"}]},{"title":"Access tokens","href":"/Security/Access-tokens"},{"title":"auth.md","sidebarTitle":"Agent auth discovery (auth.md)","href":"/auth"},{"title":"Billing Explorer API","href":"/Security/Billing-Explorer-API"},{"group":"Identity \u0026 Access","pages":[{"title":"Domain capture","href":"/Security/Domain-capture"},{"title":"SAML \u0026 Directory Sync","tag":"preview","href":"/Security/SSO-Directory-sync"}]},{"title":"Usage and quotas","href":"/Security/Quotas"},{"title":"Data collection and privacy","href":"/Security/Data-collection-and-privacy"},{"title":"Policies","href":"/Model-Governance/Policies"},{"group":"Infrastructure","pages":[{"title":"Regions","href":"/Infrastructure/Regions"},{"title":"Generations","href":"/Infrastructure/Gens"},{"title":"Dedicated egress gateways","tag":"private preview","href":"/Infrastructure/Dedicated-egress-gateways"},{"title":"Custom domains","href":"/Infrastructure/Custom-domains"},{"title":"HTTP validation","tag":"private preview","href":"/Infrastructure/Custom-domains-http-validation"}]}]},{"group":"Troubleshooting","pages":[{"title":"Troubleshoot Blaxel deployments, runtime, and CLI errors","sidebarTitle":"Troubleshoot common errors","href":"/troubleshooting"},{"title":"Error codes reference","href":"/troubleshooting/error-codes"},{"group":"Knowledge base","pages":[{"title":"Knowledge Base","sidebarTitle":"Overview","href":"/knowledge-base/overview"},{"group":"Integrations","pages":[{"title":"Set up Next.js with Blaxel Sandboxes","href":"/knowledge-base/integrations/setting-up-nextjs-with-blaxel-sandboxes"},{"title":"Set up Astro with Blaxel Sandboxes","href":"/knowledge-base/integrations/setting-up-astro-with-blaxel-sandboxes"},{"title":"Set up Vite with Blaxel Sandboxes","href":"/knowledge-base/integrations/setting-up-vite-with-blaxel-sandboxes"},{"title":"Set up Expo with Blaxel Sandboxes","href":"/knowledge-base/integrations/setting-up-expo-with-blaxel-sandboxes"}]},{"group":"Configuration and Operation","pages":[{"title":"Using custom domains with sandboxes","href":"/knowledge-base/configuration-and-operation/using-custom-domains-with-sandboxes"},{"title":"Getting static IP addresses for Blaxel sandboxes","href":"/knowledge-base/configuration-and-operation/getting-static-ip-addresses-for-blaxel-sandboxes"},{"title":"Understanding sandbox active vs standby states and billing","href":"/knowledge-base/configuration-and-operation/understanding-sandbox-active-vs-standby-states-and-billing"},{"title":"Understanding sandbox TTL and expiration policies","href":"/knowledge-base/configuration-and-operation/understanding-sandbox-ttl-and-expiration-policies"},{"title":"Use template names instead of image IDs when creating sandboxes","href":"/knowledge-base/configuration-and-operation/use-template-names-instead-of-image-id-when-creating-sandboxes"},{"title":"Ensure sandboxes use the latest deployed image","href":"/knowledge-base/configuration-and-operation/ensure-sandboxes-use-the-latest-deployed-image"},{"title":"Volume resizing for running sandboxes","href":"/knowledge-base/configuration-and-operation/volume-resizing-for-running-sandboxes"},{"title":"Sandbox inactivity with long-running processes","href":"/knowledge-base/configuration-and-operation/sandbox-inactivity-with-long-running-processes"},{"title":"Batch delete sandboxes from the CLI","href":"/knowledge-base/configuration-and-operation/batch-delete-sandboxes-from-the-cli"}]},{"group":"Security and Authentication","pages":[{"title":"Setting up MCP servers with authentication on Blaxel","href":"/knowledge-base/security-and-authentication/setting-up-mcp-servers-with-authentication-on-blaxel"},{"title":"Multi-Factor Authentication (MFA) and compliance requirements","href":"/knowledge-base/security-and-authentication/multi-factor-authentication-mfa-and-compliance-requirements"},{"title":"How to authenticate the Blaxel CLI in CI environments","href":"/knowledge-base/security-and-authentication/how-to-authenticate-the-bl-cli-in-ci-environments"}]},{"group":"Troubleshooting","pages":[{"title":"Resolving sandbox creation issues","href":"/knowledge-base/troubleshooting/resolving-sandbox-creation-issues"},{"title":"Resolving sandbox creation race conditions and duplicate requests","href":"/knowledge-base/troubleshooting/resolving-sandbox-creation-race-conditions-and-duplicate-requests"},{"title":"Troubleshooting custom Docker image deployment issues","href":"/knowledge-base/troubleshooting/troubleshooting-custom-docker-image-deployment-issues"},{"title":"Sandbox limits and quota exceeded errors","href":"/knowledge-base/troubleshooting/sandbox-limits-and-quota-exceeded-errors"},{"title":"WebSocket connection timeouts and the 15-minute limit","href":"/knowledge-base/troubleshooting/websocket-connection-timeouts-and-15-minute-limit"},{"title":"Handling Preview not found 404 errors in sandboxes","href":"/knowledge-base/troubleshooting/handling-preview-not-found-404-errors-in-sandboxes"},{"title":"Resolving sandbox restarts caused by nc not found","href":"/knowledge-base/troubleshooting/sandbox-restarting-nc-not-found"},{"title":"When localhost goes silent in Debian-based sandboxes","href":"/knowledge-base/troubleshooting/when-localhost-goes-silent-in-debian-based-sandboxes"},{"title":"Fix Miniflare inspector startup failures on Carbon sandboxes","href":"/knowledge-base/troubleshooting/miniflare-inspector-startup-failures-on-carbon"},{"title":"SSL certificate verification errors when cloning repositories","href":"/knowledge-base/troubleshooting/ssl-certificate-verification-errors-when-cloning-repositories"},{"title":"Resolving ENOSPC no space left on device errors","href":"/knowledge-base/troubleshooting/resolve-enospc-no-space-left-on-device-errors"}]}]}]}]},{"tab":"Tutorials and Examples","pages":[{"title":"Tutorials and Examples","href":"/Tutorials/Overview"}]},{"tab":"Platform APIs Reference","pages":[{"title":"Platform APIs Reference","href":"/api-reference/introduction"}]},{"tab":"Sandbox API Reference","pages":[{"title":"Sandbox API Reference","href":"/api-reference/archive/export-the-filesystem-changes-to-a-presigned-url"}]},{"tab":"SDK Reference","pages":[{"title":"SDK Reference","href":"/sdk-reference/introduction"}]},{"tab":"CLI Reference","pages":[{"title":"CLI Reference","href":"/cli-reference/introduction"}]},{"tab":"Deployment Reference","pages":[{"title":"Deployment Reference","href":"/deployment-reference"}]}]},"basePath":"","title":"Logs \u0026 traces - Documentation","deploymentId":"dpl_vSEmv961Ncyvz1psVwgUnTXQMSYk","buildId":"dpl_vSEmv961Ncyvz1psVwgUnTXQMSYk","dynamics":["10e4515597","a74be98ba7"],"chunks":["BannerMarkdown-DnSAhvaM.js","dist-DIfYWxfm.js","lib-DMeTj1gb.js","lib-B8HbPgcC.js","katex-Dbtb6Wk7.js","html-void-elements-mCSPHezU.js","marked.esm-CZWythHk.js","zoom-image-lightbox-BguUeq2N.js","lightbox-handle-r17bVUz3.js"]}