Reading a 150 MB playlist on a 4 GB TV box without running out of memory

I build NewTV Player, a media player for people's own playlists (an M3U link or an Xtream login from their provider). It runs on phones, Android TV boxes, Fire TV and Windows, and it is written in React Native with Expo and SQLite. The hardest bug I have fixed in it had nothing to do with video. It was a JSON file. The bug: an empty playlist on a 4 GB box A user on an Android TV box with 4 GB of memory added their subscription. The progress bar moved, the app worked for a while, and then the playlist was simply empty. No crash report, no error on screen. The reason was size. Xtream panels answer three calls for a catalogue: live channels, films and series. Each one is a single JSON array. For real subscriptions we measured 72 MB in one list and 156 MB across the three. The first version did what most code does: const res = await fetch(url); const items = JSON.parse(await res.text()); That line holds the response body as a string, then builds a JavaScript object for every item, with all its fields, while the string is still alive. On a 72 MB list that costs several times the file size in memory. A phone with 8 GB shrugs. A TV box with 4 GB, sharing it with the launcher, the video decoder and everything else, kills the app quietly, and the playlist stays empty. Plain M3U files are worse. The largest one we have seen measured 340 MB with 1,234,673 entries. The rule we wrote down Never read a provider's list whole. Everything below follows from that one sentence. Step 1: put the list on disk first On Android the response goes straight to a file, never into a JavaScript string. The download writes bytes to storage as they arrive, so memory use during the download is a small buffer, whatever the list size. Step 2: read it back a megabyte at a time The file is then read back one megabyte at a time. A megabyte is in memory, never the list. Step 3: a streaming parser that hands over small batches A JSON array of objects can be split without a full parser: track string state (inside quotes or not, escaped or not) and brace depth, and every time depth returns to the array level you have one complete object. You can JSON.parse that one small object safely. The reader collects about 2,000 items and hands them over as a batch. The caller never sees more than that at once. M3U files get the same treatment with a line reader: one #EXTINF line plus its URL is one entry. Step 4: write to SQLite in big statements, not one row at a time Inserting 245,000 films one INSERT at a time is slow, because each statement has a fixed cost. Packing many rows into one statement is far faster, but SQLite limits how many bound values one statement may carry. We save 2,900 rows a statement, which keeps us under that limit with our column count. One trap here: the insert helper takes a fixed column count. If you add a column, every row builder that calls it has to change too. We have five (live, films, series, M3U, and a demo playlist), and missing one makes the inserts fail silently. Step 5: stage the update, then swap An update writes into staging tables while the old catalogue stays on screen. Only when everything has arrived does it swap. If the connection drops halfway, the viewer still has yesterday's playlist instead of half of today's. Results On a real box, a subscription with about 16,000 channels and 245,000 films now loads in roughly two minutes, without the system killing the app on the way. The 4 GB box that showed an empty playlist now shows all of it. What I would tell anyone parsing big responses on TV hardware Measure the real response sizes from real users. Ours were far bigger than we had guessed. Treat JSON.parse(await res.text()) as a red flag for anything that can grow. Disk is cheap on these boxes; memory is not. Spill to a file first. Batch everything: parsing, inserts, UI updates. Test on the weakest box you can buy, not on your phone. If you want to try it, NewTV Player is on Google Play and the Microsoft Store: https://newtvplayer.com. I'm happy to answer questions about any of this in the comments.