Our supplier moved its price file to a CDN and every accented product name came back garbled

Every night our importer downloads a price and description file from one of our largest suppliers and updates the four thousand products of theirs we sell online. On a Thursday morning in October our merchandising team found Crème brûlée moulds listed as Crème brûlée moulds, and the same damage in about eleven hundred other names and descriptions. Prices were fine. Every product with an accent, a pound sign or a curly quote in its text was not. The file had not changed. It was UTF-8, as it had always been. What had changed was where it came from. The supplier had moved its downloads behind a CDN that week, and the CDN served the file as text/csv without a charset parameter. Their old server had sent text/csv; charset=utf-8. Our importer is Python and fetched the file with requests, reading response.text. When requests sees a text content type with no charset, it does not guess. It uses ISO-8859-1, the default that the old HTTP/1.1 specification gave for text, and its documentation says that it follows the specification in this case. So every é, two bytes in UTF-8, was decoded as two separate Latin-1 characters, à and ©. Nothing raised an error, because every possible byte is a valid ISO-8859-1 character. A decoding that cannot fail cannot tell you it is wrong. The importer parsed a perfectly valid CSV full of perfectly valid nonsense and wrote it to the catalogue. We restored the names that afternoon from the previous night's catalogue snapshot and ran the import again with the encoding fixed. The importer now decodes response.content as UTF-8 in strict mode and ignores the header, because the supplier's contract says UTF-8 and their CDN is not part of the contract. A file in any other encoding now fails the import loudly instead of being read as something else. Before an import is applied, it is compared with the night before, and if more than two percent of the descriptions change it waits for a person to approve it. We also asked the supplier to add the charset to their CDN configuration, and they did, but we no longer depend on it. The bytes in that file were right all along. The header that said how to read them had disappeared, and our client filled the gap with a default from 1999. – Sergey Shinder