Skip to content

Too many fractional digits are removed when a number rounds up to a power of ten #16

@gibson042

Description

@gibson042

(originally posted by @gibson042 in #10 (comment))

Consider formatting "0.99" "0.999" with { maximumFractionDigits: 2 }, which AFAICT results in ToRawFixed(x=0.999, stringDigitCount=3, minFraction=0, maxFraction=2, roundingIncrement=1, unsignedRoundingMode="halfExpand"):

  1. fmaxFraction = 2
  2. n1 ← 99; r1 ← 0.99
  3. n2 ← 100; r2 ← 1
  4. xFinal ← ApplyUnsignedRoundingMode(x, r1, r2, unsignedRoundingMode) = 1
  5. [If xFinal is r1, …]. [Otherwise,] nn2 = 100
  6. [If n = 0, …]. [Otherwise,] m ← toDecimal(n) = "100"
  7. [If f ≠ 0, then]
    1. k ← getLength(m) = 3
    2. [If kf, then …]
    3. [Else,] zn ← 0
    4. a ← substring(m, 0, k - f = 3 - 2 = 1) "1"; b ← substring(m, k - f) = "00"
    5. int ← getLength(a) = 1
    6. [If n = 0, …]; [else] sfcstringDigitCount - int + zn = 3 - 1 + 0 = 2
    7. cutmaxFraction - max(sfc, minFraction) = 2 - max(2, 0) = 2 - 2 = 0
    8. [Repeat, while cut > 0 and …]
    9. [If b is the empty String, …]; [else] m = a + "." + b = "1" + "." + "00" = "1.00"
  8. [Else, …]
  9. Return the Record { [[FormattedString]]: m = "1.00", [[RoundedNumber]]: xFinal = 1, [[IntegerDigitsCount]]: int = 1, [[RoundingMagnitude]]: –f = -2 }

It seems to return "1.0" rather than the expected output of "1.00".

EDIT: Updating the input from "0.99" to "0.999" so transformation actually takes place demonstrates lack of an issue.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions