Scan subreddits — posts, threaded comments, attached media, and linked video transcripts. Always read-only.
- Category
- Social Media
- Source type
- Produces
- postcommentimagevideofileexternal_video
Public discussion is where leaks surface first — a frustrated employee pasting a config file, a customer posting a screenshot of their invoice, a support thread quoting an internal ticket. Monitoring the subreddits that matter to you turns that into something you find rather than hear about.
What you need to connect
A Reddit app (client ID and secret), a user agent string, and the subreddits to watch. Three app types are supported: read-only, script (with a username and password), and web/installed (with a refresh token).
Scanning is always read-only — nothing is posted, voted or commented.
What Classifyre reads
- Posts from the listing you choose — new, hot, top, rising or controversial, with a time window for top and controversial.
- Comments, each as its own asset, linked parent-to-reply so a thread stays navigable. Capped per post, with optional expansion of “load more comments”.
- Media attached to a post — images, video, audio, documents, gallery items — each as a scannable asset.
- Linked external videos (YouTube, Vimeo and similar), analysed with the same captions, transcription and on-screen-text pipeline as the YouTube source.
NSFW and stickied posts can be excluded.
Media attached to posts are read with the shared file pipeline: see Supported File Formats for everything it can open, and OCR & Transcription for reading text out of images, audio and video.
Metadata on every asset
Asset kind · post
| Field | Type | Always present | What it is |
|---|---|---|---|
| submission_id | string | Yes | Reddit submission id (base-36, without the t3_ prefix) |
| subreddit | string | Yes | Subreddit the submission belongs to, without the r/ prefix |
| title | string | Yes | Submission title |
| author | string | No | Redditor who posted, or [deleted] |
| score | integer | No | Net score at extraction time |
| upvote_ratio | number | No | Fraction of votes that are upvotes |
| num_comments | integer | No | Comment count Reddit reports for the submission |
| flair | string | No | Link flair text |
| is_self | boolean | No | Whether the submission is a text post rather than a link |
| over_18 | boolean | No | Whether Reddit flags the submission as over-18 |
| spoiler | boolean | No | Whether the submission is marked as a spoiler |
| stickied | boolean | No | Whether the submission is pinned to the top of the subreddit |
| locked | boolean | No | Whether the submission is locked to new comments |
| domain | string | No | Domain of the outbound link, or self.<subreddit> for text posts |
| outbound_url | string | No | URL a link submission points at |
| listing | string | No | Subreddit listing this submission was sampled from |
| links_count | integer | Yes | Number of linked assets (comments and media) |
Asset kind · comment
| Field | Type | Always present | What it is |
|---|---|---|---|
| comment_id | string | Yes | Reddit comment id (base-36, without the t1_ prefix) |
| submission_id | string | Yes | Submission the comment belongs to |
| subreddit | string | No | Subreddit the comment belongs to, without the r/ prefix |
| author | string | No | Redditor who commented, or [deleted] |
| parent_fullname | string | No | Reddit fullname of the parent (t3_ for a top-level comment, t1_ for a reply) |
| parent_hash | string | No | Asset hash of the parent submission or comment |
| depth | integer | No | Nesting depth in the comment forest (0 for top-level) |
| score | integer | No | Net score at extraction time |
| is_submitter | boolean | No | Whether the commenter is the submission's author |
| stickied | boolean | No | Whether the comment is pinned to the top of the thread |
| reply_count | integer | No | Number of direct replies materialised for this comment |
| links_count | integer | Yes | Number of linked assets (parent plus replies) |
Asset kind · image
| Field | Type | Always present | What it is |
|---|---|---|---|
| filename | string | No | Original file name |
| mime_type | string | No | MIME type |
| size_bytes | integer | No | Raw byte size |
| image_width | integer | No | Width in pixels |
| image_height | integer | No | Height in pixels |
| submission_id | string | Yes | Submission the media was attached to |
| subreddit | string | No | Subreddit the submission belongs to, without the r/ prefix |
| file_name | string | Yes | File name derived from the Reddit media URL |
| source_url | string | No | Reddit-hosted URL the bytes are downloaded from |
| gallery_index | integer | No | Position within a gallery submission (0-based) |
| parent_hash | string | No | Asset hash of the submission the media belongs to |
Asset kind · video
| Field | Type | Always present | What it is |
|---|---|---|---|
| filename | string | No | Original file name |
| mime_type | string | No | MIME type |
| size_bytes | integer | No | Raw byte size |
| submission_id | string | Yes | Submission the media was attached to |
| subreddit | string | No | Subreddit the submission belongs to, without the r/ prefix |
| file_name | string | Yes | File name derived from the Reddit media URL |
| source_url | string | No | Reddit-hosted URL the bytes are downloaded from |
| gallery_index | integer | No | Position within a gallery submission (0-based) |
| parent_hash | string | No | Asset hash of the submission the media belongs to |
| duration_seconds | integer | No | Duration Reddit reports for a hosted video |
Asset kind · file
| Field | Type | Always present | What it is |
|---|---|---|---|
| filename | string | No | Original file name |
| mime_type | string | No | MIME type |
| size_bytes | integer | No | Raw byte size |
| submission_id | string | Yes | Submission the media was attached to |
| subreddit | string | No | Subreddit the submission belongs to, without the r/ prefix |
| file_name | string | Yes | File name derived from the linked URL |
| source_url | string | No | URL the bytes are downloaded from |
| gallery_index | integer | No | Position within a gallery submission (0-based) |
| parent_hash | string | No | Asset hash of the submission the file belongs to |
Asset kind · external_video
| Field | Type | Always present | What it is |
|---|---|---|---|
| submission_id | string | Yes | Submission that linked the video |
| subreddit | string | No | Subreddit the submission belongs to, without the r/ prefix |
| video_url | string | Yes | URL of the linked video |
| provider | string | No | Host that serves the video (e.g. youtube.com, vimeo.com) |
| title | string | No | Video title reported by the provider |
| parent_hash | string | No | Asset hash of the submission that linked the video |
| duration_seconds | integer | No | Video duration in seconds |
| transcript_available | boolean | No | Whether transcript text was obtained for the video |
| transcript_source | string | No | Origin of the transcript text: 'captions' (caption track) or 'whisper' (audio transcribed with faster-whisper) |
| transcript_language | string | No | Language code of the fetched transcript |
| transcript_is_generated | boolean | No | Whether the transcript is auto-generated (vs creator-uploaded) |
Lineage
Lineage
This source records no lineage. Nothing in the system it reads describes data moving from one place to another, so no FLOW edges are produced. Related items are still linked — see Lineage & Relationships for what those links mean and how they differ from lineage.
Worth knowing
- Comment forests are unbounded. The per-post comment cap is what keeps a busy subreddit from consuming an entire scan.
- Rate limits are absorbed by pacing requests rather than failing.
- Self-hosted Reddit-compatible instances can be pointed at by overriding the site endpoints.
Configuration
Beyond the fields below, every source also has the settings shared by all of them: the sampling strategy, the detectors to run, the scan schedule, and the compute limits for its scan jobs.
Required
Without these, the source will not save.
This section depends on which authentication method you pick — one of the following applies.
Reddit Read-Only App
| Field | Type | Required | What it does | Default |
|---|---|---|---|---|
| auth_mode | "READ_ONLY" | Yes | — | — |
| subreddits | array | Yes | Subreddits to scan, by name (e.g. datasets) or URL (e.g. https://www.reddit.com/r/datasets/). | — |
| user_agent | string | Yes | Unique application description required by Reddit's API rules. Recommended format: <platform>:<app ID>:<version string> (by u/<reddit username>). | — |
Reddit Script App
| Field | Type | Required | What it does | Default |
|---|---|---|---|---|
| auth_mode | "SCRIPT" | Yes | — | — |
| subreddits | array | Yes | Subreddits to scan, by name (e.g. datasets) or URL (e.g. https://www.reddit.com/r/datasets/). | — |
| user_agent | string | Yes | Unique application description required by Reddit's API rules. Recommended format: <platform>:<app ID>:<version string> (by u/<reddit username>). | — |
Reddit Web / Installed App
| Field | Type | Required | What it does | Default |
|---|---|---|---|---|
| auth_mode | "REFRESH_TOKEN" | Yes | — | — |
| subreddits | array | Yes | Subreddits to scan, by name (e.g. datasets) or URL (e.g. https://www.reddit.com/r/datasets/). | — |
| user_agent | string | Yes | Unique application description required by Reddit's API rules. Recommended format: <platform>:<app ID>:<version string> (by u/<reddit username>). | — |
Secrets
Stored encrypted and never shown again after you save them. See Configuration & Fields.
This section depends on which authentication method you pick — one of the following applies.
Reddit App Credentials
| Field | Type | Required | What it does | Default |
|---|---|---|---|---|
| client_id | string | Yes | App client id — the ~14-character string shown under the app name. | — |
| client_secret | string | Yes | App client secret — the ~27-character string shown next to 'secret'. | — |
Reddit Script Credentials
| Field | Type | Required | What it does | Default |
|---|---|---|---|---|
| client_id | string | Yes | App client id — the ~14-character string shown under 'personal use script'. | — |
| client_secret | string | Yes | App client secret — the ~27-character string shown next to 'secret'. | — |
| username | string | Yes | Reddit account name that registered the script app. | — |
| password | string | Yes | Password for that Reddit account. With 2FA enabled use password:token. | — |
Reddit Refresh Token
| Field | Type | Required | What it does | Default |
|---|---|---|---|---|
| client_id | string | Yes | App client id from https://www.reddit.com/prefs/apps. | — |
| client_secret | string | No | App client secret. Leave empty for installed apps, which are issued no secret. | — |
| refresh_token | string | Yes | Long-lived refresh token saved from a previous code-grant authorization. | — |
Optional
Everything you can tune. Sensible defaults apply when you leave them alone.
| Field | Type | Required | What it does | Default |
|---|---|---|---|---|
| optional | object | No | —no extra properties | — |
| connection | object | No | PRAW client behaviour: rate limiting, timeouts and startup checks.no extra properties | — |
| connection.check_for_async | boolean | No | Warn when PRAW is used from an asynchronous environment, recommending Async PRAW (PRAW default: true). | true |
| connection.check_for_updates | boolean | No | PRAW's own update check, which reports newer PRAW releases on standard error (PRAW default: true). Disabled by default here because it contacts PyPI on every run from an ephemeral scan job. | false |
| connection.ratelimit_seconds | integer | No | Longest Reddit-reported ratelimit PRAW will absorb by sleeping rather than raising. PRAW sleeps for the reported value plus one second. Reddit can report up to 14 minutes, so raising this trades run time for fewer failures.min 0, max 900 | 5 |
| connection.timeout | integer | No | Seconds PRAW waits for a single Reddit request before raising (PRAW default: 16).min 1, max 300 | 16 |
| connection.window_size | integer | No | Seconds between Reddit rate-limit resets, used by PRAW to pace requests (PRAW default: 600).min 1, max 3600 | 600 |
| scope | object | No | What is pulled out of each subreddit.no extra properties | — |
| scope.comment_sort | enum | No | Order Reddit returns a submission's comment forest in. TOP and BEST surface the most-engaged branches first, which is what a bounded max_comments_per_post should keep. Allowed: TOP, BEST, NEW, OLD, CONTROVERSIAL, Q_AND_A | TOP |
| scope.exclude_file_extensions | array | No | Never download attachments with these extensions. | — |
| scope.exclude_file_extensions[] | string | No | — | — |
| scope.exclude_nsfw | boolean | No | Skip submissions Reddit flags as over-18. | false |
| scope.exclude_stickied | boolean | No | Skip stickied/announcement submissions. | false |
| scope.expand_more_comments | boolean | No | Expand the 'load more comments' placeholders. Off by default: each expansion is an extra API round-trip and a busy thread can need hundreds. | false |
| scope.include_comments | boolean | No | Emit each comment as its own asset, linked to its parent submission or comment so the thread is navigable. | true |
| scope.include_external_videos | boolean | No | Analyse YouTube/Vimeo-style videos a submission links to using the same captions, Whisper transcription and frame-OCR pipeline as the YouTube source. | true |
| scope.include_file_extensions | array | No | Only download attachments with these extensions (e.g. ['.pdf', '.png']). Empty means all extensions. | — |
| scope.include_file_extensions[] | string | No | — | — |
| scope.include_media | boolean | No | Download images, video, audio and documents attached to a submission (including gallery items) and emit each as its own asset, parsed by the standard file pipeline. | true |
| scope.listing | enum | No | Subreddit listing that supplies candidate submissions. Leave unset to let the sampling strategy pick: LATEST/AUTOMATIC/ALL use NEW (chronological, so cursors stay meaningful) and RANDOM uses HOT. Allowed: NEW, HOT, TOP, RISING, CONTROVERSIAL | — |
| scope.max_comments_per_post | integer | No | Upper bound on comments materialised per submission. Reddit's comment forest is unbounded and each expansion costs an API call, so this caps both.min 0, max 5000 | 200 |
| scope.max_media_bytes | integer | null | No | Skip attachments larger than this many bytes. Null means no limit. | 104857600 |
| scope.time_filter | enum | No | Time window applied to the TOP and CONTROVERSIAL listings. Ignored by the other listings. Allowed: ALL, DAY, HOUR, MONTH, WEEK, YEAR | ALL |
| site | object | No | Endpoints and fullname prefixes of the Reddit instance. Only change these for a self-hosted or third-party Reddit-compatible site.no extra properties | — |
| site.comment_kind | enum | No | Type prefix for comments on the instance (default: t1_). Allowed: t1_, t2_, t3_, t4_, t5_, t6_ | t1_ |
| site.message_kind | enum | No | Type prefix for messages on the instance (default: t4_). Allowed: t1_, t2_, t3_, t4_, t5_, t6_ | t4_ |
| site.oauth_url | string | No | URL used to reach the instance's API.format uri | https://oauth.reddit.com |
| site.reddit_url | string | No | Base URL of the Reddit instance. PRAW assumes the OAuth authorization endpoints live under it.format uri | https://www.reddit.com |
| site.redditor_kind | enum | No | Type prefix for redditors on the instance (default: t2_). Allowed: t1_, t2_, t3_, t4_, t5_, t6_ | t2_ |
| site.short_url | string | No | URL used to generate short links on the instance.format uri | https://redd.it |
| site.submission_kind | enum | No | Type prefix for submissions on the instance (default: t3_). Allowed: t1_, t2_, t3_, t4_, t5_, t6_ | t3_ |
| site.subreddit_kind | enum | No | Type prefix for subreddits on the instance (default: t5_). Allowed: t1_, t2_, t3_, t4_, t5_, t6_ | t5_ |
| site.trophy_kind | enum | No | Type prefix for trophies on the instance (default: t6_). Allowed: t1_, t2_, t3_, t4_, t5_, t6_ | t6_ |
Last updated on