Skip to content

fix(java): round tiny float32 values up to the smallest float16 subnormal - #4195

Open
NUAAwqy wants to merge 1 commit into
apache:mainfrom
NUAAwqy:fix/java-float16-subnormal-rounding
Open

NUAAwqy wants to merge 1 commit into
apache:mainfrom
NUAAwqy:fix/java-float16-subnormal-rounding

Conversation

@NUAAwqy

@NUAAwqy NUAAwqy commented Oct 25, 2026

Copy link
Copy Markdown

Why?

Float16.toBits(3.0e-8f) currently returns 0x0000. Every float strictly between 2^-25 and 2^-24 is above the midpoint between zero and the smallest float16 subnormal, so round-to-nearest-even — the rounding rule the project documents for float16 — must place it at 0x0001. Instead the conversion's flush-to-zero guard fires one binade too early and drops the whole interval to signed zero. Measured over the whole input space: 16,777,214 of the 2^32 float32 bit patterns are wrong (the entire 2^-25 binade, both signs) — e.g. toBits(4.4702e-8f) → 0x0000 instead of 0x0001, toBits(-3.0e-8f) → 0x8000 instead of 0x8001.

What does this PR do?

floatToFloat16Bits shifts the 24-bit float significand onto the subnormal grid and rounds with a round/sticky correction. The netShift >= 24 shortcut declares any shift of 24 a guaranteed flush, but at netShift == 24 the correction still has work to do: the exact tie 2^-25 rounds down to zero (unchanged behaviour, and correct under ties-to-even), while anything above it rounds up to MIN_VALUE. The guard becomes netShift > 24, which also keeps the very large shifts — where the round/sticky indexing would wrap on Java's masked shift counts — inside the flush branch. Decode (toFloat) is untouched.

Evidence

  • Exhaustive differential sweep against java.lang.Float.floatToFloat16 (JDK 20+) over all 2^32 float32 inputs: 16,777,214 disagreements before the fix — all of them this flush, both signs, every pattern of the 2^-25 binade except the exact 2^-25 tie — and 0 after. NaN payload canonicalisation differs from the JDK in 32,766 patterns before and after alike (sNaN → qNaN), so NaN behaviour is untouched.
  • Float16Test gains two tests: a boundary test (the 0x33000000 tie and the largest float below it stay zero; 0x33000001, 3.0e-8f, 4.4702e-8f, 0x337FFFFF, negatives and parse("3e-8") all give 0x0001/0x8001) and a sweep over all 2^23−1 mantissas of the 2^-25 binade for both signs asserting zero flushes. Red run: 2 failures; green run: 0.
  • Full fory-core module: 2348 tests, 1 failure — ExpressionTest.testLiteralSourceRoundTrip, which reproduces on pristine 8e93633 with this change reverted (pre-existing, unrelated package).
  • Style: mvn -pl fory-core spotless:check and checkstyle:check pass (spotless run under JDK 21, per .agents/languages/java.md).
  • The new tests are Java 8-clean on purpose — hardcoded bit patterns only, no Float.floatToFloat16 — so the JDK 8/11/17/21/25/26 CI matrix is safe.
  • Existing ASF license headers preserved; no new files.

User-facing change?

No public API or binary-protocol change: values that used to serialize as zero now serialize as the subnormal they should have been. Existing float16 payloads are unaffected because no float16 encoding decodes into that interval, so the xlang round-trip suites and docs/specification/** are unaffected — this is a Java-only rounding bug in a value range no other encoding produces.

AI Usage Disclosure

  • substantial_ai_assistance: yes
  • scope: code drafting (fix), tests, PR body drafting
  • affected_files_or_subsystems: java/fory-core/src/main/java/org/apache/fory/type/Float16.java, java/fory-core/src/test/java/org/apache/fory/type/Float16Test.java
  • ai_review: line-by-line self-review completed (each hunk checked against the IEEE-754 binary16 rounding rules; red→green confirmed). The §5 two-reviewer AI review loop with persisted artifacts has not been attached yet; it can be run and linked before maintainer review on request.
  • ai_review_artifacts: none persisted yet (see previous line)
  • human_verification: mvn -pl fory-core -am test -Dtest=Float16Test red (2 failures) → green (0); full module 2348 tests / 1 pre-existing failure reproduced on the base commit; exhaustive 2^32 differential sweep 16,777,214 → 0; spotless/checkstyle clean
  • performance_verification: N/A — one comparison widened on a value-conversion path; no serialization hot-path, allocation or wire-format change
  • provenance_license_confirmation: original work under Apache-2.0; no third-party code introduced; license headers preserved

…rmal

Float16.floatToFloat16Bits flushes a value to zero when the subnormal
rounding shift reaches 24 bits, but the shift only becomes a guaranteed
flush one bit later than that. At netShift == 24 the round and sticky
bits still decide, and every float strictly between 2^-25 (the midpoint
between zero and 2^-24) and 2^-24 is closer to 2^-24 than to zero, so it
has to round up to MIN_VALUE.

The guard as written drops all of them. An exhaustive sweep of every
2^32 float32 patterns against java.lang.Float.floatToFloat16 (JDK 20+)
finds 16777214 disagreements, and every one of them is this case: the
2^23 - 1 patterns with a non-zero mantissa in the 2^-25 binade, for both
signs. Each one encodes as a signed zero instead of 0x0001/0x8001, so a
value that is representable in float16 is silently lost. The rounding
rule that the conversion must follow is the one the docs already state
for the JavaScript codec: round to nearest, ties to the even
least-significant bit.

Relaxing the guard to netShift > 24 makes the routine agree with the JDK
conversion for all 2^32 patterns; the remaining 32766 differences are
signaling-NaN payload canonicalisation, which is unchanged by this
commit.

Nothing else is affected. The flush that the guard still performs for
netShift > 24 is what keeps the rounding path out of the shift-count
masking traps (netShift - 1 == 32 would otherwise index roundBit and
sticky with a masked shift), and the existing Float16Test suite never
covered this interval because no float16 bit pattern decodes into it.

Verification:
- red: Float16Test 20 tests, 2 failures, sweep test reports expected
  [0] but found [16777214]
- green: Float16Test 20/20 after the guard change
- full fory-core suite: 2348 tests, 1 failure
  (ExpressionTest.testLiteralSourceRoundTrip, reproduces without this
  change)
- exhaustive differential sweep against Float.floatToFloat16 over all
  2^32 inputs: 0 mismatches; Float16.toFloat over all 2^16 inputs: 0
  mismatches

Signed-off-by: Qingyuan Wang <162230229@nuaa.edu.cn>
@NUAAwqy
NUAAwqy requested a review from chaokunyang as a code owner October 25, 2026 13:13

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant