TIL: number precision in rails
Recently when building a hobby project for my mother, I wanted to get rid of unnecessary zeros from numbers; in one usage the numbers always ended with 0.0 for the latest 300 rows (numbers like 20.0, 30.0, 50.0, 80.0), and in another it was 1.0 about 80% of the time and 0.5 for the rest.
I hastily searched the rails api
and found a suitable method
number_to_human(number, options = {})
.
It seemed to work great at first,
at least in the few cases the site used.
Here are some of them:
rails console
The first bug
The code felt simple enough
and I didn’t even bother to test it properly.
But then we noticed something,
380.0 turned into 400.
I ran the code in the console to see what was passed into the function,
and got this #<BigDecimal:3cdf908,'0.38E3',9(18)>
.
My first guess was that it happened because of the scientific e notation and that the type was BigDecimal because i summed up multiple numbers. I thought it got rounded to 0.4 before multiplied with 1000, that didn’t really make sense and I should’ve test to send 380 to the function in literal notation.
I felt that I needed to make a bug-report, but then I would’ve need to research it a bit more, and I didn’t have the time so I just reverted the code.
The second bug
In the other place,
1.5 turned into 2.
I though precision: 1
should keep 1 decimal place not round it away.
This time it felt really wrong,
I went back to the documentation and found the problem.
Not only is the default for strip_insignificant_zeros true, but significant should be false if I want precision to control only fractional digits. This makes sense as this function is used to humanize and approximate big numbers, which is not what I was looking to do.
This time I also found number_with_precision, which already defaults to significant: false, but instead has strip_insignificant_zeros: false.
Lessons learned
When searching documentation try to not skim through the things your going to use for the first time. Take a moment and look up a bit more than what you need. This will help save time and frustration in the future. It’s good to know what your tools can do.
When not doing TDD fully, put some real data into the dev environment to help spot issues faster.
When investigating a bug, think (guess) less and look more.