Databricks
Connect Databricks SQL Warehouses to Kizen. Automatically read and write live Databricks data and use this data in Agentic Workflows.
Audience: Admins, Developers, Solution Architects
Purpose: Explains what the Databricks integration is, when to use it, and how it works within your Agentic Workflows
Overview
Connect Databricks to Kizen with the Databricks integration. Run SQL queries and statements against a Databricks SQL Warehouse directly from an Agentic Workflow and use the results in later steps of the automated process.
What Is the Databricks Integration?
The Databricks integration adds two action steps to the Agentic Workflow builder:
Read Data: Runs a
SELECTquery against a Databricks SQL Warehouse and returns the results for use in KizenWrite Data: Executes
INSERT,UPDATE, orDELETEstatements and returns the number of affected Records
You configure the connection to Databricks once, as an integration secret in Kizen. See Set Up Databricks in Kizen for details.
The integration supports a single Databricks environment or multiple environments, each with its own connection credentials. If you have more than one environment, you specify which to query when you call Read Data or Write Data from your Agentic Workflow. To learn more, see Add Databricks Data to an Agentic Workflow.
When to Use the Databricks Integration
Use the Databricks integration when an Agentic Workflow needs to check or update Databricks SQL Warehouse data before continuing to the next step. You don't need to copy the data into Kizen first. The integration treats Databricks as an extension of Kizen and handles the communication with the Databricks server.
Common Use Cases
Here are some common use cases of the Databricks integration:
Pulling a customer's current usage or eligibility figures from a Databricks Unity Catalog table and deciding which branch of the Agentic Workflow to follow on the basis of this information
Writing the outcomes of an Agentic Workflow into a Delta table in Databricks
Pointing the same Agentic Workflow at your development and production environments in Databricks without changing the underlying queries
How the Databricks Integration Works
Before you configure the Databricks integration, it's important to understand how it functions at a conceptual level.
Connection Model
Connecting to Databricks from Kizen requires a Unity Catalog workspace and a SQL Warehouse. Kizen stores your connection credentials to Databricks as an integration secret, which can hold the credentials for multiple Databricks environments, such as production and staging. Specify the environment in which to work at each Read Data or Write Data step of the Agentic Workflow.
Because the integration doesn't utilize an authorization protocol like OAuth, you must obtain a Personal Access Token (PAT) through Databricks and include the PAT in the Kizen secret. Your PAT must be configured for the Databricks SQL Warehouse that you want to use in Kizen, and it must have SELECT and MODIFY grants to read and write data on the target Unity Catalog, schema, and table.
For more information, see Set Up Databricks in Kizen.
Network
No additional network configuration is required to use the integration. You don't need to set your firewall to allow inbound TCP traffic from a specific port, and you don't need to add Kizen's egress IP address to any Databricks allowlist. Access is governed entirely through your PAT and Unity Catalog permission grants.
Query Execution
Queries and statements execute synchronously, so the Agentic Workflow waits at the Read Data or Write Data step until Databricks returns the result. The step times out on the Kizen side after 30 seconds and causes it to fail with a timeout error. Kizen doesn't automatically retry the step.
Rate Limiting
The integration uses Databricks' SQL Statement Execution API to send your queries to Databricks from Kizen. Databricks hasn't published rate limits for this API. Instead, your SQL Warehouse's own concurrency and cluster size creates an effective throughput boundary. If your query overloads capacity, the API responds that the request failed, and your Agentic Workflow fails at this step. Kizen doesn't automatically retry the request.
Results Handling
Both the Read Data and Write Data steps include the Return Single Value input flag, which, when yes, signifies that you expect to return exactly one row and one column from Databricks.
Set this flag to no to return multiple rows or columns. These come back as a single string in JSON format. To refer to the values of individual keys in later steps of the Agentic Workflow, parse the JSON string in a Code step.
Please also note the following:
Requests for results sets larger than 25 MB always fail. Use
LIMITin the query to narrow the results.A result with multiple rows isn't auto-paginated. If you need pages, use
LIMITin the query.A query that returns no rows returns an empty string, not the null value.
Data Security
If your Agentic Workflow writes data to your Databricks SQL Warehouse, be sure to validate its behavior thoroughly on non-production data first. You can't roll back a Write Data operation after it executes, and the operation might partially apply if an error occurs during execution.
To maximize the security of your data, grant the PAT the appropriate level of access to Databricks data and operations. If you don't intend to write data, for example, excluding the MODIFY grant is advised.
How to Use the Databricks Integration
For step-by-step instructions on using the Databricks integration, see the following topics:
What's Next
Continue to Set Up Databricks in Kizen to configure the connection between Databricks and Kizen.
Databricks and Unity Catalog are trademarks of Databricks, Inc. Delta Lake is a trademark of the Delta Lake project and its contributors. Kizen's Databricks integration is not affiliated with, sponsored by, or endorsed by Databricks, Inc. or the Delta Lake project.
Last updated
Was this helpful?