You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Option to recover from invalid byte sequences (with a warning handler) instead of aborting #400
Since a8b0466 ("SAV reader: Force all strings through iconv conversion"), any string that doesn't match the file's declared encoding aborts the entire parse with READSTAT_ERROR_CONVERT_BAD_STRING. That change was the right call (it stops clients receiving invalid UTF-8), but it means one bad cell makes the whole file unreadable. In practice this shows up in SPSS files that declare UTF-8 (character code 65001) but contain a handful of stray latin1 bytes, e.g. the ePIRLS 2016 and Afrobarometer files in tidyverse/haven#615. SPSS itself opens these files and just shows the bad characters as boxes. #239 looks like the same problem in Stata files.
I'd like to propose an opt-in way to recover, plus a way for ReadStat to tell the client that it did:
typedefenumreadstat_invalid_bytes_e {
READSTAT_INVALID_BYTES_ERROR, // current behaviour, remains the defaultREADSTAT_INVALID_BYTES_REPLACE, // emit U+FFFD (or '?' for non-UTF-8 output)READSTAT_INVALID_BYTES_FALLBACK// decode the offending bytes with a fallback charset
} readstat_invalid_bytes_t;
typedefstructreadstat_warning_s {
readstat_warning_code_tcode; // e.g. READSTAT_WARNING_INVALID_BYTESintvar_index; // -1 if not tied to a variableintobs_index; // -1 if outside the data section (labels etc.)constchar*message;
} readstat_warning_t;
typedefint (*readstat_warning_handler)(constreadstat_warning_t*warning, void*ctx);
readstat_error_treadstat_set_warning_handler(readstat_parser_t*parser, readstat_warning_handlerhandler);
readstat_error_treadstat_set_invalid_bytes(readstat_parser_t*parser, readstat_invalid_bytes_tpolicy);
readstat_error_treadstat_set_fallback_encoding(readstat_parser_t*parser, constchar*encoding);
The handler can return READSTAT_HANDLER_ABORT to stop, like the other handlers. FALLBACK with latin1 is the interesting one: in the haven#615 files it recovers the original text exactly (e.g. "VOTAÇÃO"), rather than just papering over it.
Implementation-wise, I think this can stay mostly inside readstat_convert_impl(): on EILSEQ, write the replacement, skip a byte, keep going, and call the warning handler. To avoid touching all ~50 readstat_convert() call sites, the per-format ctx structs could hold a small readstat_converter_t (iconv handle + policy + handler + current var/obs index) in place of the bare iconv_t, so the call sites only change type. The readers would just need to keep var/obs index up to date where they already loop over cells.
A warning code (rather than just a message) would also give a home for related cases that are currently silent, like the illegally truncated multi-byte strings that iconv() drops via the EINVAL branch.
Would you be open to something along these lines? If so, I'm happy to put together a PR, starting with the SAV reader. cc @gorcha, who prototyped a pass-through version in the invalid-bytes branch of gorcha/ReadStat.
Activity