A Better Way to Search Years of Public Reddit Comments

Comments are where most of Reddit is. They are also the hardest thing to search, and the usual approach — broad query, hope for the best — fails slowly and uninformatively.

Why comment search is slow

Post search mostly matches titles: short, indexed, fast. Comment search runs full-text queries across bodies, which is a fundamentally more expensive operation.

At archive scale this is slow enough that broad queries time out rather than returning slowly. A timeout is not a service outage — it is the query being too expensive to complete, and the fix is a narrower query rather than a retry.

Keyword search requires a scope

Archives do not support searching all comments for a word. A keyword must be paired with a subreddit, an author, or a thread.

Working with this rather than around it improves the research. "Where was this discussed" is answered by community, not by keyword. Deciding the community first turns a fishing expedition into a question.

Window the time range

The single most effective technique is splitting a long period into shorter windows and searching each.

Searching 2019 to 2024 in one query asks the archive to consider everything. Searching year by year asks six cheaper questions. Each returns quickly, each either succeeds or fails visibly, and a failure tells you which period is problematic instead of failing the whole request.

It also produces a timeline for free — six counts rather than one total — which is usually more informative than the aggregate anyway.

Understand the pagination

Archives page by timestamp rather than offset. You ask for records before a moment, then before the oldest record you received.

The boundary is inclusive, so the last record of one page reappears as the first of the next. The tempting fix — advance the cursor by one second — silently drops every other record sharing that timestamp, and a busy community produces many. Take the duplicate and remove it by ID.

Search locally once loaded

Every query costs the archive something. Once records are loaded, searching them locally is free and instant.

So load a defensible set — one community, one period — and then explore within it. Trying three variations locally beats three round trips, and it is kinder to a free service.

Local search also permits things the archive cannot do: exact phrase matching, word-boundary matching so a search for one term does not silently include longer words containing it, and excluding a high-volume community from the results without refetching.

Expect the account-level ceiling

Searching one account's comments across years hits a different limit: very active accounts time out on aggregation regardless of how the query is phrased.

The approach is the same — window the time range — with the addition that a hundred records is usually enough to decide whether more is worth retrieving.

A workflow that holds up

1. Pick the community or author. This is the scope, and it is a research decision.
2. Split the period into years or quarters.
3. Load one window and read some of it. Confirm the query is finding the right thing.
4. Load the remaining windows, keeping the counts separate.
5. Search and filter locally from there.
6. Record what you loaded, from where, and when.

Slower than typing a keyword and pressing enter. It also completes, which the alternative frequently does not.