Skip to main content
Version: 3.0

Transformations Overview

Teckel provides 45 transformation types for building data pipelines declaratively in YAML. Each transformation is an asset with a unique name and exactly one operation key.

Formal reference: Section 8 — Transformations in the Teckel Specification.

45 transformation types

transformation:
- name: myTransformation
<operation_key>:
...

Every transformation entry must have exactly one operation key. Unless otherwise stated, transformations consume one upstream asset via a from field and produce one output dataset.


Transformation Categories​

Filtering and Row Selection​

YAML KeySectionDescription
select8.1Project specific columns or expressions
where8.2Filter rows by a boolean condition
distinct8.9Remove duplicate rows
limit8.10Return at most N rows
sample8.19Return a random sample of rows
offset8.32Skip the first N rows (v3.0)
tail8.33Return the last N rows (v3.0)
dropNa8.35Drop rows with NULL values (v3.0)

Aggregation and Sorting​

YAML KeySectionDescription
group8.3Group rows and apply aggregate functions
orderBy8.4Sort rows by one or more columns
rollup8.23Hierarchical aggregation with subtotals
cube8.24Multi-dimensional aggregation (all combos)
groupingSets8.42Arbitrary grouping column combinations (v3.0)
describe8.43Compute descriptive statistics (v3.0)
crosstab8.44Frequency cross-tabulation (v3.0)

Joins​

YAML KeySectionDescription
join8.5Join datasets (7 types including cross)
asOfJoin8.39Temporal as-of join (v3.0)
lateralJoin8.40Correlated lateral join (v3.0)

Set Operations​

YAML KeySectionDescription
union8.6Combine rows from multiple datasets
intersect8.7Rows present in all source datasets
except8.8Rows in the first source but not the second

Column Operations​

YAML KeySectionDescription
addColumns8.11Add computed columns
dropColumns8.12Remove columns
renameColumns8.13Rename columns via a mapping
castColumns8.14Change column data types
fillNa8.34Replace NULL values (v3.0)
replace8.36Replace specific values (v3.0)
parse8.38Parse string columns to structured data (v3.0)

Window Functions​

YAML KeySectionDescription
window8.15Apply window functions over partitions

Reshaping​

YAML KeySectionDescription
pivot8.16Rotate rows into columns (long to wide)
unpivot8.17Rotate columns into rows (wide to long)
flatten8.18Flatten nested structures
conditional8.20Add a column with CASE WHEN logic
split8.21Split dataset into two based on condition
transpose8.41Transpose rows and columns (v3.0)

Advanced​

YAML KeySectionDescription
sql8.22Execute a raw SQL query
scd28.25Slowly Changing Dimension Type 2
enrich8.26Enrich records via external HTTP API
schemaEnforce8.27Validate or evolve dataset schema
assertion8.28Validate data quality rules
repartition8.29Change partition count (with shuffle)
coalesce8.30Reduce partitions (no full shuffle)
custom8.31Invoke a user-registered component
merge8.37MERGE (upsert) operation (v3.0)
hint8.45Optimizer hints for the engine (v3.0)

General Rules​

  • Each transformation entry must have exactly one operation key.
  • Unknown operation keys produce an error.
  • The pipeline forms a DAG — execution order is determined by data dependencies, not YAML order.
  • Asset names must be unique across all inputs and transformations.