S3-Compatible Storage
S3-Compatible Storage
Scan objects from AWS S3, MinIO, Cloudflare R2, Backblaze B2, and other S3-compatible endpoints.
- Category
- Warehouse & Lakehouse
- Source type
- S3_COMPATIBLE_STORAGE
- Produces
- fileimageaudiovideoarchive
Buckets are where data goes to be forgotten: exports, backups, log dumps, “temporary” copies from three years ago. This source scans any S3-compatible endpoint — AWS S3, MinIO, Cloudflare R2, Backblaze B2, Garage, Ceph, Wasabi and the rest — through the same connection.
What you need to connect
A bucket name, plus either an access key pair or nothing at all — leave the
credentials empty and the environment’s ambient AWS credential chain (instance
role, workload identity, ~/.aws) is used. For non-AWS providers, add the
endpoint URL and, where the provider needs it, a region.
Read-only credentials are enough: s3:ListBucket and s3:GetObject on the
prefix you want scanned.
What Classifyre reads
Every object under the prefix you choose, filtered by extension allow- and denylists if you want to narrow it further.
Shared behaviour · File and object sources
Every object in the bucket becomes one asset. What it is — a PDF, a spreadsheet, a screenshot, a video — is worked out from the bytes themselves, not from the file name, because storage systems routinely label everything as generic binary data. The full list of readable types is on File Formats.
Files hidden inside other files are pulled out and scanned in their own right: images embedded in a document, and every member of a ZIP, TAR, 7z or RAR archive. Images, audio and video need OCR and transcription switched on before their content can be read.
Large objects are streamed rather than loaded whole, so a multi-gigabyte file costs disk rather than memory, and only the part a sampling window asks for is read.
Objects from this bucket are read with the shared file pipeline: see Supported File Formats for everything it can open, and OCR & Transcription for reading text out of images, audio and video.
Metadata on every asset
Asset kind · file
| Field | Type | Always present | What it is |
|---|---|---|---|
| size_bytes | integer | Yes | Raw byte size of the content |
| mime_type | string | Yes | Resolved MIME type |
| parse_error | string | No | Set when content extraction failed |
| image_width | integer | No | Width in pixels |
| image_height | integer | No | Height in pixels |
| page_count | integer | No | Number of pages (pdf) |
| paragraph_count | integer | No | Number of paragraphs (docx) |
| table_count | integer | No | Number of tables (docx) |
| row_count | integer | No | Number of data rows |
| columns | object[] | No | Columns as {name, type} objects (type may be empty for csv/xlsx) |
| encoding | string | No | Detected character encoding |
| json_root_type | string | No | Root JSON type: object, array, or scalar |
| top_level_keys | integer | No | Number of top-level keys when the root is an object |
| array_length | integer | No | Length when the root is an array |
| provider | string | Yes | Storage provider label |
| object_key | string | Yes | Object key/path |
| etag | string | No | Object entity tag |
| source_hash | string | No | Hash of the parent asset (embedded files and archive members only) |
| location | string | No | Location within the parent (embedded file location or archive member path) |
Asset kind · image
| Field | Type | Always present | What it is |
|---|---|---|---|
| size_bytes | integer | Yes | Raw byte size of the content |
| mime_type | string | Yes | Resolved MIME type |
| parse_error | string | No | Set when content extraction failed |
| image_width | integer | No | Width in pixels |
| image_height | integer | No | Height in pixels |
| provider | string | No | Storage provider label |
| object_key | string | No | Object key/path |
| etag | string | No | Object entity tag |
| source_hash | string | No | Hash of the parent asset (embedded files and archive members only) |
| location | string | No | Location within the parent (embedded file location or archive member path) |
Asset kind · audio
| Field | Type | Always present | What it is |
|---|---|---|---|
| size_bytes | integer | Yes | Raw byte size of the content |
| mime_type | string | Yes | Resolved MIME type |
| parse_error | string | No | Set when content extraction failed |
| provider | string | Yes | Storage provider label |
| object_key | string | Yes | Object key/path |
| source_hash | string | No | Hash of the parent asset (embedded files and archive members only) |
| location | string | No | Location within the parent (embedded file location or archive member path) |
Asset kind · video
| Field | Type | Always present | What it is |
|---|---|---|---|
| size_bytes | integer | Yes | Raw byte size of the content |
| mime_type | string | Yes | Resolved MIME type |
| parse_error | string | No | Set when content extraction failed |
| provider | string | Yes | Storage provider label |
| object_key | string | Yes | Object key/path |
| source_hash | string | No | Hash of the parent asset (embedded files and archive members only) |
| location | string | No | Location within the parent (embedded file location or archive member path) |
Asset kind · archive
| Field | Type | Always present | What it is |
|---|---|---|---|
| size_bytes | integer | Yes | Raw byte size of the content |
| mime_type | string | Yes | Resolved MIME type |
| parse_error | string | No | Set when content extraction failed |
| provider | string | Yes | Storage provider label |
| object_key | string | Yes | Object key/path |
| etag | string | No | Object entity tag |
| source_hash | string | No | Hash of the parent asset (embedded files and archive members only) |
| location | string | No | Location within the parent (embedded file location or archive member path) |
Lineage
Lineage
This source records no lineage. Nothing in the system it reads describes data moving from one place to another, so no FLOW edges are produced. Related items are still linked — see Lineage & Relationships for what those links mean and how they differ from lineage.
Worth knowing
- Object type is detected from content. S3-compatible providers routinely label uploads as generic binary data; the format is worked out from the bytes, so those objects are still read properly.
- Archives are expanded into child assets, with caps on member count, member size, and total uncompressed bytes — the guard against a zip bomb.
- Big objects don’t blow up memory. Anything past the in-memory threshold is streamed to a temporary file or read by byte range instead.
- Metadata-only inventories are possible: turn off content preview and you get a catalogue of what’s in the bucket without downloading anything.
Configuration
Beyond the fields below, every source also has the settings shared by all of them: the sampling strategy, the detectors to run, the scan schedule, and the compute limits for its scan jobs.
Required
Without these, the source will not save.
| Field | Type | Required | What it does | Default |
|---|---|---|---|---|
| required | object | Yes | —no extra properties | — |
| bucket | string | Yes | Bucket name for AWS S3, MinIO, Cloudflare R2, Backblaze B2, Garage, and other S3-compatible endpoints | — |
Secrets
Stored encrypted and never shown again after you save them. See Configuration & Fields.
| Field | Type | Required | What it does | Default |
|---|---|---|---|---|
| masked | object | No | Optional static credentials. Leave empty to use ambient AWS credentials chain.no extra properties | — |
| aws_access_key_id | string | No | S3-compatible access key ID | — |
| aws_secret_access_key | string | No | S3-compatible secret access key | — |
| aws_session_token | string | No | Optional session token for temporary credentials | — |
Optional
Everything you can tune. Sensible defaults apply when you leave them alone.
| Field | Type | Required | What it does | Default |
|---|---|---|---|---|
| optional | object | No | —no extra properties | — |
| connection | object | No | —no extra properties | — |
| connection.endpoint_url | string | No | Custom endpoint URL for MinIO/R2/B2/Garage and other S3-compatible providersformat uri | — |
| connection.max_archive_member_bytes | integer | No | Maximum uncompressed bytes read from a single archive membermin 1024 | 10485760 |
| connection.max_archive_members | integer | No | Maximum member files expanded from one archive into child assets. Bounds fan-out, and is the guard against a zip bomb.min 1, max 10000 | 200 |
| connection.max_archive_total_bytes | integer | No | Maximum uncompressed bytes read across all members of one archive. The decompression-ratio ceiling: a zip bomb hits this before it hits memory.min 1024 | 104857600 |
| connection.max_embedded_files | integer | No | Maximum embedded files (parquet image/audio columns, office media) expanded from one container into child assets per runmin 1, max 10000 | 200 |
| connection.max_file_bytes | integer | No | Refuse any object larger than this many bytes. 0 or unset means no limit: an object above max_object_bytes is spooled to disk rather than held in memory, so file size is bounded by free disk, not by RAM.min 0 | — |
| connection.max_keys_per_page | integer | No | Maximum objects requested per provider list API callmin 1, max 1000 | 200 |
| connection.max_object_bytes | integer | No | Maximum bytes of one object held in memory. Larger objects are streamed to a temporary file (or read by byte range where the provider supports it), so this bounds memory rather than the size of file that can be scanned. See max_file_bytes to refuse large objects outright.min 1024 | 5242880 |
| connection.region_name | string | No | Region (recommended for AWS; required by some S3-compatible providers) | — |
| connection.request_timeout_seconds | number | No | Network timeout in seconds for list/download operationsmin 1, max 300 | 30 |
| connection.verify_ssl | boolean | No | TLS certificate verification toggle | true |
| scope | object | No | Object scope and filtering controls.no extra properties | — |
| scope.exclude_extensions | array | No | Optional extension denylist | — |
| scope.exclude_extensions[] | string | No | — | — |
| scope.include_content_preview | boolean | No | Download object bytes to infer MIME and extract detector-ready text previews | true |
| scope.include_empty_objects | boolean | No | Include zero-byte objects in extraction results | false |
| scope.include_extensions | array | No | Optional extension allowlist (for example, .pdf, .csv, .parquet) | — |
| scope.include_extensions[] | string | No | — | — |
| scope.include_object_metadata | boolean | No | Attach provider metadata (etag, size, content-type hints, timestamps) to asset checksums | true |
| scope.prefix | string | No | Object key prefix filter (for example, exports/2026/) | — |