Repository navigation
Description
Describe the bug, including details regarding any error messages, version, and platform.
arrow::Decimal128::FromString silently truncates (wraps mod 2¹²⁸) when the input string has more than 38 significant digits, producing a garbage unscaled value and returning Status::OK. The out-parameter precision does reflect the true parsed precision (> 38), but the Decimal128 value itself is already corrupted, and most callers do not validate precision against Decimal128::kMaxPrecision.
The same issue applies symmetrically in Decimal256::FromString when precision > 76 - both paths share DecimalFromString<T> in cpp/src/arrow/util/decimal.cc.
Why?
DecimalFromString feeds the digit string into ShiftAndAdd, which multiplies-and-adds into a fixed-size uint64_t array sized to the target decimal's bit width:
std::array<uint64_t, Decimal::kBitWidth / 64> little_endian_array{};
ShiftAndAdd(dec.whole_digits, little_endian_array.data(),
little_endian_array.size());
ShiftAndAdd(dec.fractional_digits, little_endian_array.data(),
little_endian_array.size());ShiftAndAdd carries high bits only through out_size limbs and drops the remaining carry on the floor - no overflow check. DecimalFromString then returns Status::OK() without validating parsed_precision <= kMaxPrecision (and also without validating parsed_scale <= kMaxScale).
Relevant code (paths as of apache/arrow main):
ShiftAndAdd-cpp/src/arrow/util/decimal.cc:747DecimalFromString-cpp/src/arrow/util/decimal.cc:857
Repro
// arrow_decimal_repro.cc
#include <arrow/util/decimal.h>
#include <iostream>
#include <string>
int main() {
const std::string input =
"1.55555555555555555555555555555555555555555555555555";
arrow::Decimal128 dec;
int32_t precision = 0, scale = 0;
auto st = arrow::Decimal128::FromString(input, &dec, &precision, &scale);
std::cout << "input: " << input << "\n"
<< "status: " << st.ToString() << "\n"
<< "precision: " << precision << "\n"
<< "scale: " << scale << "\n"
<< "dec (raw): " << dec.ToString(0) << "\n";
return 0;
}Observed output (Arrow main, macOS arm64, clang++ -std=c++17):
input: 1.55555555555555555555555555555555555555555555555555
status: OK
precision: 51
scale: 50
dec (raw): 151473969238762845885664163847966963939
That unscaled value is exactly the input (51-digit integer 155555555555555555555555555555555555555555555555555) reduced mod 2¹²⁸:
>>> 155555555555555555555555555555555555555555555555555 % (1 << 128)
151473969238762845885664163847966963939Desired (?) behavior
Return Status::Invalid when parsed precision exceeds kMaxPrecision (or parsed scale exceeds kMaxScale) for the target decimal type, matching the error-reporting contract of Decimal::Rescale and friends.
Impact (breaking change?)
Any caller that trusts FromString to fail on invalid input silently produces wrong answers for user-supplied decimal strings longer than 38 significant digits. Downstream, Gandiva's castDECIMAL_utf8 - and any compute kernel that routes VARCHAR-to-DECIMAL through FromString before rescaling via Decimal128::Rescale / gandiva::decimalops::Convert - returns numerically wrong results for perfectly legal SQL casts such as:
CAST('1.55555555555555555555555555555555555555555555555555' AS DECIMAL(38,37))
-- expected: 1.5555555555555555555555555555555555556 (HALF_UP)
-- observed: 0.0000000000015147396923876284588566416Component(s)
C++, Gandiva
Activity
dmitry-chirkov-dremio commentedon Apr 20, 2026
Burned few tokens so you don't have to. Feel free to ignore and apply your own reasoning.
Two layered fixes. They are independent but complementary:
Decimal{128,256}::FromStringa clear "lossless-or-bust" contract, so callers can trust the return value. Small blast radius, strong invariant.castDECIMAL_utf8the only Arrow caller that intentionally handles lossy SQL-style casts by rounding before parsing. Preserves the user-visible behavior of legal SQL likeCAST('1.555…' AS DECIMAL(38,37))after fix 1 tightens the parser.Without fix 2, fix 1 would regress queries that used to return wrong results but now return an error. Without fix 1, fix 2 only closes the Gandiva code path; other Arrow callers (compute kernels, CSV converter, Python bindings that land on
FromString, …) would still silently corrupt on out-of-range input.Fix 1 —
Decimal{128,256}::FromStringreturnsStatus::Invalidwhen input exceeds type rangeFile:
cpp/src/arrow/util/decimal.ccFunction:
DecimalFromString<Decimal>(and itsSimple*sibling for Decimal32/64)Change
After the existing
parsed_precision/parsed_scalederivation, validate against the target decimal type's compile-time limits before writing the value:Place the checks before the
ShiftAndAddloop so no bits are written tooutin the error path.Apply the same block to
SimpleDecimalFromString<T>so Decimal32 (precision ≤ 9) and Decimal64 (precision ≤ 18) get the same guarantee.Contract after fix
FromStringreturnsStatus::OK, the writtenDecimalvalue is an exact, lossless representation of the input string.Test additions
Target file:
cpp/src/arrow/util/decimal_test.ccRisk assessment
Callers that today trust
FromStringto succeed on any syntactically valid decimal string will now seeStatus::Invalidon out-of-range inputs. Audit needed:cpp/src/gandiva/gdv_function_stubs.cc::gdv_fn_dec_from_string— already propagates status correctly. (Fix ARROW-9: Replace straggler references to Drill #2 below makes this a non-issue in practice.)cpp/src/arrow/csv/converter.cc— already returns the status to the row reader; a CSV row with 50-digit decimals will now surface an error instead of garbage. Desired behavior.cpp/src/arrow/python/decimal.cc— does not callFromString(usesDecimalFromPyDecimal); unaffected.Existing tests that construct Decimals from strings like
"9999999999999999999999999999999999999999"(intentionally out-of-range) will flip from pass to error. Grepcpp/src/arrow/util/decimal_test.ccand fix expectations as part of this patch.Fix 2 — Gandiva
castDECIMAL_utf8rounds to target scale before parsingFile:
cpp/src/gandiva/precompiled/decimal_wrapper.ccFunction:
castDECIMAL_utf8(around line 398)Problem solved
After fix 1,
CAST('1.555…(50 fives)' AS DECIMAL(38,37))would raise an error instead of returning the correctly rounded value1.5555555555555555555555555555555555556. That's a behavior regression for legal SQL input.castDECIMAL_utf8is the one place where the target(out_precision, out_scale)is known before parsing, so it's the natural home for caller-side rounding.This fix also closes a separate, pre-existing bug: even when the input does fit in 128 bits (e.g. 38 significant digits), the current code uses
decimalops::Convertto rescale, which truncates instead of rounding half-up. Pre-rounding in the string domain makes Gandiva's VARCHAR→DECIMAL match the JavaCastVarCharDecimalpath exactly (Java usesRoundingMode.HALF_UPinjava.math.BigDecimal::setScale).Change
Introduce a helper that normalizes the digit string to fit the target scale, with HALF_UP rounding, before handing off to
Arrow::Decimal128::FromString:Then in
castDECIMAL_utf8:Helper contract (for review)
After
RoundDecimalStringHalfUp(in, target_scale):target_scalefractional digits.Arrow::Decimal128::FromString.target_scale + 1digit, if digit ≥ 5 the truncation point is rounded up; ties go away from zero.+/-) and exponent notation (e±N) are preserved / normalized."9.99"rounded to scale 1 →"10.0"), but never by more than one. The caller (Convert) handles cases where the resulting precision exceedsout_precisionTest additions
Target file:
cpp/src/gandiva/tests/decimal_test.ccUnit-level tests for
RoundDecimalStringHalfUp:End-to-end tests for
castDECIMAL_utf8(all expected results match whatjava.math.BigDecimal(str).setScale(scale, HALF_UP)would produce):Risk assessment
castDECIMAL_utf8is an LLVM-IR-compiled function. Adding a C++ helper means the bitcode module will pull instd::stringand digit-walking code — modest size increase but well within what Gandiva already includes (Arrow utilities are linked). No ABI change; the function signature is unchanged.Performance impact: one extra pass over the input string plus one small allocation. For VARCHAR→DECIMAL casts this is dwarfed by the cost of
Decimal128::FromStringitself, which already walks the string twice (component parsing +ShiftAndAdd).dmitry-chirkov-dremio commentedon Apr 20, 2026
Couple of pointers around historical evolution:
apacheGH-49817: [C++] Reject decimal strings that exceed the target p…
[-]arrow::Decimal128::FromString silently truncates when the input string has more than 38 significant digits[/-][+][C++] arrow::Decimal128::FromString silently truncates when the input string has more than 38 significant digits[/+]apacheGH-49817: [C++] Reject decimal strings that exceed the target p…
apacheGH-49817: [C++] Test max decimal value and clarify shift comments
apacheGH-49817: [C++] Reject decimal strings that exceed the target p…
apacheGH-49817: [C++] Test max decimal value and clarify shift comments
apacheGH-49817: [C++] Handle signed decimal overflow
apacheGH-49817: [C++] Allow minimum Decimal32 and Decimal64
apacheGH-49817: [C++] Validate metadata-only decimal parsing
apacheGH-49817: [C++] Reject overflowing decimal exponents
35 remaining items
apacheGH-49817: [Gandiva] Reject oversized normalized decimals
apacheGH-49817: [Gandiva] Limit retries to checked scale reductions
apacheGH-49817: [Gandiva] Cover long all-zero decimal inputs
apacheGH-49817: [C++] Detect decimal parsing overflow
apacheGH-49817: [C++] Handle signed decimal overflow
apacheGH-49817: [C++] Allow minimum Decimal32 and Decimal64
apacheGH-49817: [C++] Validate metadata-only decimal parsing
apacheGH-49817: [C++] Reject overflowing decimal exponents
apacheGH-49817: [C++] Fix decimal boundary tests on MSVC
apacheGH-49817: [Gandiva] Round long strings before decimal conversion
apacheGH-49817: [Gandiva] Preserve decimal exponent errors
apacheGH-49817: [Gandiva] Reject oversized normalized decimals
apacheGH-49817: [Gandiva] Limit retries to checked scale reductions
apacheGH-49817: [Gandiva] Cover long all-zero decimal inputs
apacheGH-49817: [C++] Reject decimal strings that overflow the bit wi…