fix: convert CSV fields above the parser limit

[KennySimpson (KennyMcSimpson)]

A CSV field longer than Python's parser limit currently makes CsvConverter fail. Through the public MarkItDown API, conversion then falls back to plain text, so a valid CSV loses its Markdown table. With the standard limit, this happens at 131,073 characters.

Keep the existing limit for the first parsing attempt. If parsing raises csv.Error and the decoded input is longer than the current limit, retry with the limit bounded by the input length and restore the previous setting in finally. A module lock serializes CSV parsing across converter instances so their retries cannot overwrite each other's saved limit.

csv.field_size_limit is process-wide. The lock coordinates CsvConverter instances only: unrelated CSV readers can observe the temporary limit, and independent changes to that setting are not synchronized.

Validation

python -m pytest packages/markitdown/tests/test_csv.py -q
pre-commit run --all-files

43 tests passed, including public API checks at the standard limit, one character above it, and a long Unicode field. They also check that the previous parser limit is restored.

Separate checks passed for custom limits, quoted multiline fields, and 36 concurrent public API conversions. The full multi-format test suite was not run.

[KennySimpson (KennyMcSimpson)]

@microsoft-github-policy-service agree