Skip to content

Option to recover from invalid byte sequences (with a warning handler) instead of aborting #400

Open

Description

@hadley

🤖 Written by AI, reviewed by me 🤖

Since a8b0466 ("SAV reader: Force all strings through iconv conversion"), any string that doesn't match the file's declared encoding aborts the entire parse with READSTAT_ERROR_CONVERT_BAD_STRING. That change was the right call (it stops clients receiving invalid UTF-8), but it means one bad cell makes the whole file unreadable. In practice this shows up in SPSS files that declare UTF-8 (character code 65001) but contain a handful of stray latin1 bytes, e.g. the ePIRLS 2016 and Afrobarometer files in tidyverse/haven#615. SPSS itself opens these files and just shows the bad characters as boxes. #239 looks like the same problem in Stata files.

I'd like to propose an opt-in way to recover, plus a way for ReadStat to tell the client that it did:

typedef enum readstat_invalid_bytes_e {
    READSTAT_INVALID_BYTES_ERROR,     // current behaviour, remains the default
    READSTAT_INVALID_BYTES_REPLACE,   // emit U+FFFD (or '?' for non-UTF-8 output)
    READSTAT_INVALID_BYTES_FALLBACK   // decode the offending bytes with a fallback charset
} readstat_invalid_bytes_t;

typedef struct readstat_warning_s {
    readstat_warning_code_t code;     // e.g. READSTAT_WARNING_INVALID_BYTES
    int         var_index;            // -1 if not tied to a variable
    int         obs_index;            // -1 if outside the data section (labels etc.)
    const char *message;
} readstat_warning_t;

typedef int (*readstat_warning_handler)(const readstat_warning_t *warning, void *ctx);

readstat_error_t readstat_set_warning_handler(readstat_parser_t *parser, readstat_warning_handler handler);
readstat_error_t readstat_set_invalid_bytes(readstat_parser_t *parser, readstat_invalid_bytes_t policy);
readstat_error_t readstat_set_fallback_encoding(readstat_parser_t *parser, const char *encoding);

The handler can return READSTAT_HANDLER_ABORT to stop, like the other handlers. FALLBACK with latin1 is the interesting one: in the haven#615 files it recovers the original text exactly (e.g. "VOTAÇÃO"), rather than just papering over it.

Implementation-wise, I think this can stay mostly inside readstat_convert_impl(): on EILSEQ, write the replacement, skip a byte, keep going, and call the warning handler. To avoid touching all ~50 readstat_convert() call sites, the per-format ctx structs could hold a small readstat_converter_t (iconv handle + policy + handler + current var/obs index) in place of the bare iconv_t, so the call sites only change type. The readers would just need to keep var/obs index up to date where they already loop over cells.

A warning code (rather than just a message) would also give a home for related cases that are currently silent, like the illegally truncated multi-byte strings that iconv() drops via the EINVAL branch.

Would you be open to something along these lines? If so, I'm happy to put together a PR, starting with the SAV reader. cc @gorcha, who prototyped a pass-through version in the invalid-bytes branch of gorcha/ReadStat.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions