SmartConnectors

Audience: Administrators, Developers, Integrators, Solution Architects

Purpose: Introduces SmartConnectors as the batch data integration and ETL layer of the Kizen platform, establishes a mental model for how they work, and prepares readers to progress into Core Concepts, SmartConnector Types, SmartConnector SQL Processing, and API guidance.

Overview

SmartConnectors are the mechanism Kizen uses to process data in bulk, ingesting, transforming, and loading large datasets into Records reliably and efficiently. They accept data from spreadsheet files, webhooks, schedules, or external sources; optionally transform it using ClickHouse SQL; validate it through typed execution variables; and load it into one or more Kizen Objects. SmartConnectors are designed for batch operations, not real-time, single-Record processing.

SmartConnectors act only on data that is explicitly included in the current run's input. Existing Records in Kizen are unaffected unless they appear in the input data and match a load step's matching rules.

When To Use SmartConnectors

SmartConnectors are the right tool when you need to process data in bulk, apply transformation logic before ingestion, or build a repeatable Workflow that runs on a schedule or in response to incoming data. Common use cases include:

• Loading data in bulk from an external system on a recurring schedule. For example, ingesting a weekly carrier file or a nightly database export
• Transforming raw vendor or partner files before ingesting them into Kizen Records, using SQL to normalize, filter, or reshape the data
• Batching inbound webhook data for efficient processing rather than handling each event individually as it arrives
• Populating multiple related Objects from a single data source in a single run. For example, creating vendor Records and then creating product Records that reference them
• Running scheduled logic against reference data without requiring a file upload and using the Scheduler SmartConnector type with existing Kizen data as the input

When To Use SmartConnectors VS. The Kizen API

Both SmartConnectors and the Kizen API can write data to Kizen, but they are optimized for different use cases. Use the table below to determine which tool is the right fit for your scenario.

	SmartConnectors	Kizen API
Volume	Hundreds, thousands, or millions of Records	Single Records or small sets
Transformation	Complex SQL transformation before ingestion	Pass-through or simple field mapping
Frequency	Recurring imports, scheduled Workflows	Real-time, event-driven operations
Validation	Row-level type validation with per-row failure reporting	Immediate response per request

Note: If you are reading or writing more than ~1,000 Records, SmartConnectors are almost always the right choice. For real-time events at high volume, SmartConnectors can also be used to reduce processing overhead by grouping incoming webhook data into batches for scheduled processing.

When To Use SmartConnectors VS. Agentic Workflows

SmartConnectors and Agentic Workflows are both powerful tools in Kizen, but they are built for fundamentally different kinds of work. Choosing the right one depends on how many Records you are operating on and whether the work is event-driven or batch-oriented.

	SmartConnectors	Agentic Workflows
Volume	Hundreds, thousands, or millions of Records	One Record at a time, or no Record (global)
Trigger model	File upload, batched webhook queue, or time-based schedule	Record field changes, form submissions, individual webhook events, or time-based triggers on a single Record
Data transformation	Full SQL transformation using ClickHouse before ingestion	Field updates, conditions, and logic applied step by step
Execution model	Batch processing with chunked, sequential load steps	Asynchronous, Record-scoped or global step execution
Best for	Bulk imports, recurring data Workflows, and complex ETL Workflows	Workflow Agentic Workflow, notifications, field updates, and integrations on individual Records

Note: If you are operating on more than ~1,000 records at once, SmartConnectors are almost always the right choice. For single-Record, event-driven work, use Agentic Workflows. The two can also work together. A SmartConnector load step can explicitly trigger an Agentic Workflow on Records it creates or updates.

Key Differentiators

SmartConnectors are built for a specific kind of work. The following capabilities set them apart from other data integration approaches in Kizen.

• Batch-Optimized: SmartConnectors process data in chunks designed for reliability and efficiency at scale, making them the right choice when operating on hundreds, thousands, or millions of Records. Processing continues chunk by chunk even if individual rows fail.
• SQL-Powered Transformation: The integrated ClickHouse SQL environment gives developers and integrators the full power of a columnar analytical database to transform, join, and reshape data before it enters Kizen, without needing to pre-process files externally.
• Type-Safe by Design: Execution variables enforce data type validation before any data reaches the load step, failing individual rows without halting the entire run and producing a full audit report of what was processed, skipped, and errored. When a variable is declared as a Number type, the system extracts numeric values from the input and ignores any non-numeric characters. For example, $16.99 USD becomes 16.99 and 41 Apples becomes 41. This coercion happens silently rather than failing the row, so confirm that your input data produces the expected numeric values by reviewing the execution variable tab in the output report before processing live data.
• File-Oriented Workflow: Every SmartConnector type, whether triggered by a file upload, a webhook, a schedule, or a bulk action, ultimately passes data through the same file-oriented Workflow, meaning all SmartConnector types share identical behavior from variable mapping onward.

---

SmartConnectors Mastery Checklist

Explore the following topics to understand how SmartConnectors are configured and used in Kizen.

Core Knowledge

• What a SmartConnector is and how it differs from the Kizen API and Agentic Workflows
• The Input → SQL Processing → Execution Variables → Load Steps → Kizen Records mental model and how to apply it to a real Workflow
• The five SmartConnector types and when to use each

SQL Processing

• How the ClickHouse SQL processor works and what input tables are available
• How to write, test, and publish SQL within a SmartConnector
• How to use reference data, SQL parameters, and integration secrets in SQL
• How to select the right CHDB runtime version and avoid the SELECT * anti-pattern
• How to connect to external data sources including PostgreSQL, MySQL, and S3

Execution Variables

• What execution variables are and why every SQL output column must be declared with an explicit type
• How type validation works at the row level and what happens when a required variable is invalid or blank
• How variable scopes relate to SQL output tables

Load Steps

• How matching rules identify whether to create or update a Record
• How mapping rules write execution variable values to fields and how conflict resolution works
• How relationship fields work in SmartConnectors and why missing related Records are skipped rather than created
• How to configure post-load Agentic Workflow triggers and the entity variable pattern for multi-step Workflows

Running and Operating

• How diff checking works, when to use it, and when a custom diff approach is more appropriate
• How to activate a SmartConnector, run a dry run, interpret the XLS output report, and troubleshoot failed rows
• How to configure concurrent executions, notification emails, and the execution ended webhook

Webhook Connectors

• What makes Webhook SmartConnectors different
• How to access webhook payload data in SQL using input.webhooks and input.webhooks_raw
• How to capture sample webhooks during development using the inactive state

Advanced Topics

• How permissions govern SmartConnector access
• How to export, import, and duplicate SmartConnectors across businesses and environments
• How connectors, executions, and output are structured as data model entities

APIs & Developer Tooling

• How to trigger, monitor, and cancel SmartConnector executions programmatically via API
• How to upload an input file and trigger a run via API — see File Upload (Trigger Run) API (Topic Coming Soon)
• How to deliver webhook payloads to a Webhook SmartConnector programmatically
• How to use the sc CLI and dev package for local SQL development and deployment — see Build with SmartConnectors (Topic Coming Soon)

Design Best Practices

• How to design reliable, maintainable SmartConnectors that avoid common data integrity pitfalls
• When to use SQL parameters, integration secrets, and the entity variable pattern

---

What's Next

Continue to SmartConnector Core Concepts to build the foundational understanding you need before configuring SmartConnectors, designing load steps, or working with the API. The Core Concepts establishes the shared vocabulary, Workflow model, and component overview that every subsequent topic builds on.

Related Topics

• SmartConnector Core Concepts
• SmartConnector Types
• SmartConnector SQL Processing