Keeping junk out of programmatic geo pages: an allow list for metro, a block list for districts

On podbor-minuta.ru we have listing pages grouped by metro station and by district. The idea is simple: every station and every district gets its own page with the new-build apartments near it. These pages are built ahead of time from the database, each with its own URL, title and a real list of apartments. The problem was the data we built those groups from. The metro and district values come from a scraper, and it copies them from other sites as they are. In the metro field, instead of a clean station name, we often had strings like "Tushinskaya (13 min)". The walking time was glued onto the station. For a person that is nothing, but for grouping it is a disaster: "Tushinskaya", "Tushinskaya (5 min)" and "Tushinskaya (13 min)" count as three different values. One station turned into several near-identical pages. District was worse. The scraper dumped anything into that field: project names ("ZhK Sezar City"), marketing fragments ("multifunctional complex..."), and sometimes whole sentences from the description, with prices, areas and build stages. Real districts were less than half of it. Grouping naively on these fields produced about 193 thin, near-duplicate pages. On a young domain that is a direct risk: search engines dislike a pile of empty lookalike pages and can push the whole site down. We wanted only pages with real stations and real districts, and we wanted variants of one station to collapse into one page. The first thing we understood: metro and district are two different cases, and they need different medicine. For metro we have a full reference of Moscow stations. So we can check against an allow list: if the cleaned value is in the reference, keep it, if not, drop it. And to collapse the variants, the page URL is built not from the raw value but from the normalized name. First we strip the walking-time tail, get a clean "Tushinskaya", and build the URL from that. All variants with different times land on the same URL. // "Tushinskaya (13 min)" -> "Tushinskaya" -> tushinskaya const station = normalizeMetroValue(raw); if (!isKnownMetroStation(station) && !KNOWN_TRANSIT_LINES.has(lineKey)) { return null; // not a real station - no page } const slug = transliterateSlug(station); Separately we kept transit lines as their own kind of page: MCD, MCK, BKL (Moscow's suburban and ring lines). They are not stations, but they have real search demand ("new builds near MCD") and thousands of apartments under them. So next to the station allow list we keep a small list of lines. const KNOWN_TRANSIT_LINES = new Set([ 'мцд', 'мцк', 'мцд-1', 'мцд-2', 'мцд-3', 'мцд-4', 'мцд-5', 'бкл', ]); District did not work this way, and here is why. We do not have a full district reference. Our districts table is missing the okrugs and the towns of the Moscow region. Those are real, living groups we do not want to lose. If we made an allow list, we would throw out half of the good data along with the junk. So for district we went the other way: we do not describe what is good, we describe what is clearly bad and reject that. The bad is well defined here. A project name almost always starts with "ZhK", "MFK", "residential complex", "tower", "residences". A fragment of a description almost always has numbers with "million", "thousand", "rub", "sq m", or "min", or words about construction like "being built", "handed over", "phase". A real district never looks like that. const JUNK_DISTRICT_RE = /(жилой\s+(?:комплекс|квартал)|(?:^|\s)(?:жк|мфк|башня)(?:\s|$)|(?:^|\s)(?:апарт|резиденц)[а-яё]*|\d[\d.,\s]*(?:млн|тыс|руб|кв\.?\s?м|мин\b)|возвод|строит|сдан|очеред)/i; export function isJunkDistrictValue(raw: string): boolean { const v = raw.trim().toLowerCase(); if (!v || v.length > 60) return true; // a district is never a full sentence return JUNK_DISTRICT_RE.test(v); } Two details that are easy to trip on. In JavaScript the "\w" class does not include Cyrillic, so word continuations had to be spelled out with "[а-яё]*". And stems must be bound to a word boundary: if you just search for "dom" (house), the regex will hit "Domodedovo" and kill a real district. So short words require a space or a line end around them, while prefixes like "apart" and "residenc" match the project forms directly. Both checks meet in one function. It takes a raw value and either returns a pair of "display name plus slug" or null, in which case the page simply does not exist. This is the single point through which geo reaches the catalog and the sitemap. export function resolveSeoGeoLabel(dim, rawDisp) { const raw = (rawDisp || '').trim(); if (!raw) return null; let display; if (dim === 'metro') { const station = normalizeMetroValue(raw); if (!isKnownMetroStation(station) && !KNOWN_TRANSIT_LINES.has(key)) return null; display = station; } else { if (isJunkDistrictValue(raw)) return null; display = raw; } const slug = transliterateSlug(display); if (!slug || slug.length > MAX_GEO_SLUG_LEN) return null; return { display, slug }; } On top of that there is one more simple safety margin. We capped the slug length at 64 characters. Each of these pages becomes a directory on disk at build time, and a directory name in the filesystem cannot be longer than 255 bytes. A single dirty value the length of a whole sentence once crashed our entire build. We have a separate article about that story, and here we just do not let long values through. And the last filter is count. We build a page only if it has at least a threshold of apartments under it. In SQL that is one line, "HAVING COUNT(*) >= minLots". This drops groups of one or two apartments, which would also be thin pages. What we took from this. Where you have a full reference, use an allow list: keep only what is in it. Where you have no reference but the junk is clear, use a block list: drop what is clearly bad and keep the rest. This is not symmetry for looks, it is about what you actually know about the data. And always clean the raw value before it becomes part of a URL, or one station variant will multiply into a dozen empty pages. The site is podbor-minuta.ru, daily price monitoring for Moscow new builds.