Dataset Column Expression Math Error when using convert

Howard Arner 21 Reputation points
2020-11-20T17:11:50.57+00:00

OK, this has me stumped. I am just trying to round a column using an expression in datatable column. Here is a program that shows the behavior i am not expecting

    Dim T As New DataTable
    Dim Ty As System.Type
    Ty = System.Type.GetType("System.Double")
    T.Columns.Add("A", Ty)
    T.Columns.Add("B", Ty)
    T.Columns.Add("C", Ty, "A*B")
    T.Columns.Add("D", Ty, "Convert(A*B*100,System.Int64)/100")

    T.Rows.Add(3, 3.331)
    T.Rows.Add(3, 3.333)
    Dim S As String = String.Format("A: {0} B: {1} C:{2} D:{3}", T.Rows(0).Item(0), T.Rows(0).Item(1), T.Rows(0).Item(2), T.Rows(0).Item(3))
    Debug.Print(S)

    S = String.Format("A: {0} B: {1} C:{2} D:{3}", T.Rows(1).Item(0), T.Rows(1).Item(1), T.Rows(1).Item(2), T.Rows(1).Item(3))
    Debug.Print(S)

The output of the above is:

A: 3 B: 3.331 C:9.993 D:9.99
A: 3 B: 3.333 C:9.999 D:10

Notice what is happening on the second case, the number is being truncated. I have tried many variations on the formula and cannot get it to handle it correctly. some kind of voodoo is happening. I tried converting the 100 to a decimal, converting it to a double. I cannot get it to function. What am i missing?

.NET
.NET
Microsoft Technologies based on the .NET software framework.
3,373 questions
VB
VB
An object-oriented programming language developed by Microsoft that is implemented on the .NET Framework. Previously known as Visual Basic .NET.
2,569 questions
0 comments No comments
{count} votes

Accepted answer
  1. Viorel 112.1K Reputation points
    2020-11-20T17:53:49.34+00:00

    Show the expected result and try this expression: "Convert(AB*100 - 0.49, System.Int64) / 100"*.

    If negative numbers are possible, then try "Convert(AB*100 + iif(AB>=0, -0.49, +0.49), System.Int64) / 100".


0 additional answers

Sort by: Most helpful