Working With Large Reddit Datasets in the Browser

Fifty thousand Reddit records is a few tens of megabytes of mostly short strings. Modern browsers handle that comfortably. Interfaces that struggle with it are usually struggling for reasons unrelated to the volume.

Measure before threading

The standard prescription is Web Workers: move heavy analysis off the main thread so the interface stays responsive.

Before doing that, measure. Here is a real result from timing an analysis layer over 50,000 records:

Operation	Time
Subreddit aggregation	15 ms
Heatmap for a year	14 ms
Score statistics	25 ms
Keyword search	72 ms
Deduplication	173 ms
Term frequency	353 ms
Monthly timeline	6,632 ms

Six and a half seconds for the timeline, against fifteen milliseconds for a superficially similar aggregation.

The cause was one line. Producing a bucket label called toLocaleString with a month format — once per record. Intl formatting costs on the order of 130 microseconds, so fifty thousand records spent over six seconds generating twelve distinct strings.

Replacing it with a lookup array of month names took the operation to about 35 milliseconds.

Threading that work would have moved six seconds of pointless computation onto another thread and declared the problem solved. The lesson generalises: the slow thing is often slow for a reason that has nothing to do with how much data there is.

Where the real costs sit

Formatting. Intl.DateTimeFormat and Number.prototype.toLocaleString are expensive relative to arithmetic. Called per row, they dominate. Hoist the formatter, or precompute the handful of strings you need.

The DOM. Rendering 50,000 rows is genuinely fatal — not the data, the elements. Windowed rendering keeps the DOM to what fits on screen plus a margin, and this is where the actual win is.

Repeated passes. Six functions each looping the full set is six full traversals. Usually fine at these sizes, and worth collapsing if it is not.

Serialisation. Posting records to a worker copies them. Below a few thousand records, the copy costs more than the analysis it avoids — which is an argument for a threshold rather than always threading.

When workers earn their place

After fixing the formatting, one operation remained slow enough to matter: term frequency, at around 350ms. That is long enough to feel like a freeze during typing.

So that one runs in a worker, above a threshold, with a main-thread fallback for environments without workers. Everything else stays synchronous because it finishes in under a quarter of a second and moving it would cost more than it saves.

The general rule: thread the thing you measured as slow, not the thing that sounds heavy.

Guard the fix with a test

An optimisation without a test is a temporary optimisation. A performance test over a large synthetic dataset, with thresholds set several times above the measured figures, catches an algorithmic regression without failing on a slow machine.

Had such a test existed earlier, the per-record formatting call would never have shipped.

Knowing the ceiling

Browsers are not databases. At some point — millions of records, complex joins, repeated queries over the same corpus — the right tool is a database, and Reddit archives publish bulk dumps for exactly that.

The useful thing is knowing where the boundary sits rather than assuming it is lower than it is. Tens of thousands of records, analysed locally with no upload and no server: that works, and works well, provided nobody calls a locale formatter in a loop.