Using Value.Convert with string source data to union containing string and non-string?

[colatkinson]

Suppose we have some code like the following:

import { Type } from "@sinclair/typebox"
import { Value } from "@sinclair/typebox/value"

const TExample = Type.Object({
    stringFirst: Type.Union([Type.String(), Type.Number()]),
    numberFirst: Type.Union([Type.Number(), Type.String()]),
})
const converted = Value.Convert(TExample, { stringFirst: "123", numberFirst: "123" })

console.log(JSON.stringify(converted, undefined, 4))

Between 0.34.37 and 0.34.38, the behavior changed due to #1282.

With 0.34.37:

{
    "stringFirst": "123",
    "numberFirst": 123
}

With 0.34.38:

{
    "stringFirst": "123",
    "numberFirst": "123"
}

I'd hesitate to call this a bug or regression in any way, since I don't think this left-to-right union evaluation was documented in any way. If this sort of thing just isn't something the project wants to support/deal with, that's absolutely understandable. If this is out of scope for Value.Convert, switching to an alternative validator for this particular use case is totally feasible.

To add some details, the specific use case I'm looking at is loading data from env vars -- thus the source values will always be strings.

Is there a way to denote the source values as "strings-but-not-strings", for example by making a wrapper class with a custom conversion? The current conversion functions seem fairly static, so doesn't seem to be support for such a thing but maybe I'm missing something?

typebox/src/value/convert/convert.ts

Lines 121 to 126 in c54a0f8

	function TryConvertString(value: unknown) {
	return IsSymbol(value) && value.description !== undefined ? value.description.toString() : IsValueToString(value) ? value.toString() : value
	}
	function TryConvertNumber(value: unknown) {
	return IsStringNumeric(value) ? parseFloat(value) : IsValueTrue(value) ? 1 : IsValueFalse(value) ? 0 : value
	}

Alternatively, could it make sense to allow explicitly marking certain union members as preferable? Or perhaps flagging some unions as left-to-right?

[sinclairzx81]

@colatkinson Hi,

To add some details, the specific use case I'm looking at is loading data from env vars -- thus the source values will always be strings. ......... Is there a way to denote the source values as "strings-but-not-strings", for example by making a wrapper class with a custom conversion? The current conversion functions seem fairly static, so doesn't seem to be support for such a thing but maybe I'm missing something?

As a general rule, TypeBox expects types to describe values exactly as an application would like to receive them, but it doesn't want to know the state in which values could be sent (which could be anything)

In the case you presented, you defined a string | number to suggest that the Input could be sent (env loaded) either be string or number, but TypeBox doesn't need to know what the Input "could" be, it only needs to know what the Output "should" be. If you omit the Union properties from the type, then Convert will try it's best to convert into the target type, yielding the 0.34.37 return value.

const TExample = Type.Object({
    stringFirst: Type.String(),      // remove unions. Just define the exact type
    numberFirst: Type.Number(),      // you want your application to receive.
})

const converted = Value.Convert(TExample, { stringFirst: "123", numberFirst: "123" })

console.log(JSON.stringify(converted, undefined, 4))
// {
//     "stringFirst": "123",
//     "numberFirst": 123
// }

There are of course cases where an application may need to receive structures that are Union, in which case TypeBox uses the Left to Right ordering to Test and Convert, returning the first matching if it can. The update on #1282 largely corrected union handling inline with this (which I think is correct)

---

I'd hesitate to call this a bug or regression in any way, since I don't think this left-to-right union evaluation was documented in any way. If this sort of thing just isn't something the project wants to support/deal with, that's absolutely understandable. If this is out of scope for Value.Convert, switching to an alternative validator for this particular use case is totally feasible.

I don't think it's a bug. Historically TypeBox has preferred value conversion to be an opaque process that either returns a valid value (if conversion was possible) or invalid (if a conversion was not possible). An application should trust that TypeBox has made an adequate attempt to convert a value, but where the value returned may still be entirely invalid (failed conversion)

Input -> Convert -> Check  // The return value of Convert MUST be checked

TypeBox prefers Convert to be opaque for the following reasons:

1. There is no rulebook for how Convert should work.
2. It frees TypeBox to change Convert behaviors and relying on Check to catch problems (steps are taken to avoid breaking changes however)
3. It allows Convert to evolve inline with real world requirements, without knowing all requirements up front.
4. It allows for Convert logic to assessed and optimized in future.

The lack of documentation to the internal processes of Convert is an unfortunate a consequence of the evolving nature of the function. This said, TypeBox loosely follows AJV coercion rules.

https://ajv.js.org/coercion.html

from type → / to type ↓	string	number	boolean	null	array*
string	-	x→""+x	false→"false" true→"true"	null→""	[x]→x
number / integer	Valid number / integer: x→+x	-	false→0 true→1	null→0	[x]→x
boolean	"false"→false "true"→true "abc"⇸ ""⇸	0→false 1→true x⇸	-	null→false	[false]→false [true]→true
null	""→null "null"⇸ "abc"⇸	0→null x⇸	false→null true⇸	-	[null]→null
array*	x→[x]	x→[x]	false→[false] true→[true]	null→[null]	-

Of course, this only accounts for primitive types. When considering deeply nested / logical (OR / AND) / structural schematics, the Convert function sits somewhere between real world pragmatism and academia. I'm using the former to try shed light on the latter, but given the myriad of ways users want to see values converted (especially union), it may forever remain in the former. We'll see.

---

Hope this brings some insight into the Convert function and some of the thinking around it. With respect to using another library to handle conversion, if it does what you need, that's the way to go.

I will say, I am looking at ways to make the value processing system more flexible in the long term, including letting users build and integrate their own processing functions (i.e. BYO Convert function) and wire them up into pipelines.

Current design is loosely the following.

// ------------------------------------------------------------------
// Standard Parse Pipeline
// ------------------------------------------------------------------
const Parse = Value.Pipeline([
  (_context, _type, value) => Value.Clone(value),
  (context, type, value) => Value.Clean(context, type, value),
  (context, type, value) => Value.Default(context, type, value),
  (context, type, value) => Value.Convert(context, type, value),  // <- replace with your own function here
  (context, type, value) => { Value.Assert(context, type, value); return value },
])

const parsed = Parse({}, Type.String(), 'hello world')

I will be introducing some of this infrastructure in the next version scheduled for next month.
Cheers
S