OverArchitected

What is it? When you create an external location in Unity Catalog, Databricks now automatically provisions the cloud resources (SNS topics, SQS queues on AWS; ADLS queues on Azure) to detect file changes. Autoloader gets event-driven ingestion out of the box — no manual cloud infrastructure setup.

Is it for you? If you're ingesting files from cloud storage at any meaningful scale, this makes Autoloader significantly faster and more efficient. Previously you had to manually configure file event notifications — now it just works when you set up your external location.

Try it Create or update an external location in Unity Catalog. File events are enabled by default. Your existing Autoloader jobs will automatically benefit from event-driven file discovery.

What is it? Three new models on the Foundation Model API: Claude Opus 4.6 and Claude Sonnet 4.6 (Anthropic), plus Gemini 3.1 Pro (Google). All served through Databricks with instrumentation, inference tables, AI Gateway, and PII detection included.

Is it for you? If you're building anything with LLMs on Databricks — agents, data extraction, code generation — these are the latest and most capable models available. Sonnet 4.6 matches the previous Opus 4.5 at a lower price point, making it a strong default for most workloads.

Try it Model Serving > Create Endpoint > select from Foundation Models. Pay-per-token, no provisioning. Point any OpenAI-compatible client at the endpoint URL.

What is it? Three performance optimizations are now available for stateless streaming queries: Adaptive Query Execution (AQE) rebalances skewed partitions at runtime, dynamic shuffle partition adjustment right-sizes partitions based on actual data volume, and auto-optimized shuffle reduces overhead. All enabled by default — no configuration needed.

Is it for you? If you have stateless streaming jobs (map, filter, projection — anything without aggregations or joins) that process unevenly distributed data, these optimizations reduce processing time and resource waste automatically.

Try it Nothing to do — these are enabled by default on DBR 18.0+. Check your streaming query's execution plan to see AQE and optimized shuffle in action.

What is it? MLflow experiment traces can now sync directly to a Delta table in Unity Catalog. One click in the Experiments UI — pick a catalog and schema — and a sync pipeline is created automatically. Every trace (sessions, calls, responses, latencies) becomes a queryable, shareable table.

Is it for you? If you're running agents or LLM-powered features and need observability beyond the MLflow UI — dashboards, anomaly detection, cost analysis, or sharing trace data with other teams — this puts your traces where all your other governed data lives.

Try it Go to your MLflow Experiment, click the Delta Sync button, select a destination catalog/schema, and name your table. The sync pipeline starts automatically.

What is it? Execute multiple SQL statements against a single table as one atomic commit — no partial updates visible to readers. Now also works with Delta Sharing, meaning shared table consumers only ever see fully committed state.

Is it for you? If you need to update and delete from the same table in a single logical operation (data corrections, end-of-period closings, conditional cleanup), this prevents anyone from seeing an intermediate state. Essential for shared datasets where consistency matters.

Try it Wrap your statements in BEGIN TRANSACTION / COMMIT. Works in notebooks, SQL editor, and dbSQL. No configuration needed.

What is it? The Supervisor Agent in Agent Bricks is now GA. It's a routing layer that combines up to 20 different agents or tools — agent endpoints, Unity Catalog functions, Genie spaces, and external MCP servers — into a single conversational interface. You ask a question, it figures out which sub-agent to call.

Is it for you? If your workspace has multiple specialized agents, Genie spaces, or tool endpoints and you want a single entry point that intelligently routes between them, this is it. No code required — configure it entirely through the UI.

Try it Go to Machine Learning > Agent Bricks > Supervisor Agent. Select up to 20 agents or tools from your workspace, give it a system prompt, and deploy. Test it in the playground before serving.

What is it? Metric views — Databricks' semantic layer for defining how metrics are calculated — now support window functions (rolling averages, cumulative totals, trailing period counts) and filter clauses in aggregate functions (e.g., sum only women's events). There's also a new visual UI editor as an alternative to YAML definitions.

Is it for you? If you use Genie spaces or need consistent metric definitions across your organization, metric views ensure everyone computes things the same way. Window functions and filters expand what you can define without writing one-off SQL.

Try it Navigate to your catalog, find or create a metric view, and use the new UI editor to define measures with window functions. Connect the metric view to a Genie space for natural language querying.