Skip to Content
SourcesDelta Lake

Delta Lake

Delta Lake

Scan Delta Lake tables in S3-compatible storage — schema, versions, and row sampling, no Spark required.

Category
Warehouse & Lakehouse
Source type
DELTA_LAKE
Produces
table

Delta Lake tables are the storage layer under most lakehouse platforms. This source reads them where they live — as files in a bucket — so you can scan a warehouse path without a Databricks workspace, a Spark cluster, or a catalog service in front of it.

What you need to connect

A bucket and, unless the scanning environment already has ambient cloud credentials, an access key pair. For MinIO, Cloudflare R2, Backblaze B2 and other S3-compatible providers, add the custom endpoint URL and region.

Then either give a prefix and let table discovery find every Delta table under it (recognised by its _delta_log directory), or list exact table paths when you know them.

Shared behaviour · Lakehouse tables

No Spark, no JVM, and no catalog service is required. Delta table metadata is read directly out of the table's own files in object storage, and rows are sampled from the underlying Parquet — so the only thing you have to provide is object-storage access to the warehouse path.

Connection details are the same shape as the S3-Compatible Storage source, which means AWS S3, MinIO, Cloudflare R2 and the rest all work the same way.

What Classifyre reads

One asset per table. Its metadata is the table’s real structure — columns and types, row count, current version — and its content is a sample of rows drawn through the sampling strategy you chose.

Metadata on every asset

Asset kind · table

FieldTypeAlways presentWhat it is
databasestringYesDatabase or catalog name
table_namestringYesTable name
table_typestringYesObject type (TABLE/VIEW)
schemastringNoSchema name
columnsobject[]NoColumns as {name, type} objects
row_countintegerNoEstimated number of rows
format_versionintegerNoDelta protocol/reader version
num_filesintegerNoNumber of data files
partition_columnsstring[]NoPartition column names
history_lengthintegerNoNumber of commits in the table history

Lineage

Lineage

This source records no lineage. Nothing in the system it reads describes data moving from one place to another, so no FLOW edges are produced. Related items are still linked — see Lineage & Relationships for what those links mean and how they differ from lineage.

Worth knowing

  • Reads are non-destructive and version-aware — the table’s current snapshot is read; nothing is compacted, vacuumed or written.
  • Discovery costs list calls. On a very large bucket, a narrower prefix or an explicit table list is much faster than scanning from the root.

Configuration

Beyond the fields below, every source also has the settings shared by all of them: the sampling strategy, the detectors to run, the scan schedule, and the compute limits for its scan jobs.

Required

Without these, the source will not save.

FieldTypeRequiredWhat it doesDefault
requiredobjectYesno extra properties
bucketstringYesBucket holding the tables — AWS S3, MinIO, Cloudflare R2, Backblaze B2, Garage, and other S3-compatible endpoints

Secrets

Stored encrypted and never shown again after you save them. See Configuration & Fields.

FieldTypeRequiredWhat it doesDefault
maskedobjectNoOptional static credentials. Leave empty to use ambient AWS credentials chain.no extra properties
aws_access_key_idstringNoS3-compatible access key ID
aws_secret_access_keystringNoS3-compatible secret access key
aws_session_tokenstringNoOptional session token for temporary credentials

Optional

Everything you can tune. Sensible defaults apply when you leave them alone.

FieldTypeRequiredWhat it doesDefault
optionalobjectNono extra properties
connectionobjectNoS3-compatible storage connection options (AWS S3, MinIO, Cloudflare R2, Backblaze B2, Garage, ...). Mirrors the S3 Compatible Storage source connection settings.no extra properties
connection.endpoint_urlstringNoCustom endpoint URL for MinIO/R2/B2/Garage and other S3-compatible providersformat uri
connection.max_keys_per_pageintegerNoMaximum objects requested per provider list API call during table discoverymin 1, max 10001000
connection.region_namestringNoRegion (recommended for AWS; required by some S3-compatible providers)
connection.request_timeout_secondsnumberNoNetwork timeout in seconds for storage list/read operationsmin 1, max 30030
connection.verify_sslbooleanNoTLS certificate verification toggletrue
scopeobjectNoDelta Lake table selection scope within the bucket.no extra properties
scope.prefixstringNoKey prefix to search for Delta Lake tables (e.g. warehouse/). Tables are auto-discovered by their _delta_log/ directory.
scope.table_limitintegerNoOptional cap on number of table assetsmin 1
scope.table_pathsarrayNoExplicit Delta Lake table root keys or s3:// URIs. When set, auto-discovery under prefix is skipped.
scope.table_paths[]stringNo
Last updated on