Formatting OverrideFormats

[hl662]

Posting this on behalf of @SheetalNair:

There are 2 major questions from our side:

1. Can we create a FormatterSpec from an OverrideFormat? … or more directly from a formatString?
   It was mentioned that this could be tricky because the result of OverrideFormat.parseFormatString is OverrideFormatProps which cannot be directly used to create a FormatterSpec.
   Though you suggested we could use serialize/deserialize methods to get Format Props, but I couldn’t find the method which would convert the result from OverrideFormatProps to OverrideFormat and then use it to get the formatProps we want.

2. Do we really need to supply our own UnitsProvider? … is there a way to get a standard one?
   I tried to use the SchemaUnitProvider in the backend and it worked, but I want to share the code I tried and confirm if this is the right way. The input to this code is FormatProps from core-quantity NOT the formatString from ecschema-metadata.

    const schemaContext = new SchemaContext();
    const schemaJson = iModel.getSchemaProps("Units");
    await Schema.fromJson(schemaJson, schemaContext);
    const unitProvider = new SchemaUnitProvider(schemaContext);
 
    const format = new Format("text");
    await format.fromJSON(unitProvider, formatProps).catch(() => {});
    const spec = await FormatterSpec.create("text", format, unitProvider,await unitProvider.findUnitByName(inputUnit) );
 
    let displayString = spec.applyFormatting(measuredValue);

For question 1: If we have a way to applyFormatting to value without using FormatterSpec and just by working with OverrideFormatProps from ecschema-metadata package, we can use the formatString to store formatters and work with that for quantity formatting. Is there already a way to do that? Am I missing something?

For question 2 : Why do we have QuantityFormatter class only on IModelApp which can be used only in frontend? If we had a similar class for backend, we won’t have to worry about UnitsProvider, as the QuantityFormatter will have all the data we need to get the FormatterSpec and applyFormating.

[hl662]

@SheetalNair
I'll answer question 2 first:

There is a standard one that is enabled by default through IModelApp.quantityFormatter.unitsProvider. It's a BasicUnitsProvider that supports very limited amounts of units, and is not extensible (nor is it instantiable on it's own, because its an internal API). Good practice is to avoid using that entirely, and to always use a SchemaUnitProvider when possible.

QuantityFormatter acts as a helper API for formatting quantities, you can still create formatterSpecs and do formatting/parsing without it (it'll take a few extra lines of code). We don't have a similar class for backend because previously it was thought that the concept of formatting quantities and parsing them are a frontend/visualization concept. The effort it takes to implement a UnitsProvider in the backend was reduced a lot in the upcoming iTwin 5.0 - IModelDb now has a schemaContext property from this PR, which can be used to initialize a SchemaUnitProvider easily.

As for question 1, and I'll also answer on your code snippet: Yes, that's the right way to do it, to supply a FormatProps from core-quantity and not a formatString. SchemaItemFormatProps in ecschema-metadata is an extension of FormatProps in core-quantity, and SchemaItemOverrideFormatProps is an extension of SchemaItemFormatProps, so these APIs are compatible between each other. Anyways, to the root of question 1: How to create a FormatterSpec from an OverrideFormat (or formatString).

It's clear that we did not expose an easy way to go from a formatString to a SchemaItemFormatProps (or SchemaItemOverrideFormatProps). The implementation is... long, and it's private - format overrides normally exist within KindOfQuantities so the logic for parsing them exist only in there, as a helper method.

We can open a GH issue requesting enhancements to the OverrideFormat API to help deserializing OverrideFormats outside of a KoQ. While OverrideFormat has a getJSONProps() for serializing into a SchemaItemOverrideFormatProps, there is no deserializing method . For now, we'll need to utilize a schemaContext, and a unitsProvider, and the constructor of an OverrideFormat to do so. You can refer to the linked implementation above (for the private method processPresentationUnitsSync), starting from line 113. You just need to replace any usage of this.schema with your own schemaContext, be it from iModelDb.schemaContext or like your code snippet shows. An example:

... // copy KindOfQuantity.ts 113-139
const overrideFormat = new OverrideFormat(format, precision, unitAndLabels) // unitAndLabels has it's first tuple value transformed from string to LazyLoadedUnit

const formatProps = overrideFormat.getJsonProps()

[SheetalNair]

@hl662 ,
Having schemaContext as an iModelDb property seems pretty exciting, glad to know about it.
Regarding requesting an API for deserialization in OverrideFormat, I need to have a discussion with my team about it before we request any changes in the API, I will get back to you soon.
Thank you for addressing the queries and providing good suggestions!

[SheetalNair]

@ColinKerr @hl662
I just wanted to restart this discussion with some additional context.
In drawing production, we are looking for a way to store formatting information in the iModel for dimension elements.

The above image shows a linear and an angular dimension, where the measurements 2250.479 ft and 90deg are not stored anywhere in the iModel, it gets generated in our backend geometry code and needs to be formatted to show the value in required units.
To format these values, we need to store the formatter information in our iModel.
We have a dimension style element corresponding to a bis class AnnotationDimensionStyle, that stores style data in form of json in a first class property named settings. We are planning to store the formatter information in this settings json property and need help in figuring out how to do that.
I found a link to an ongoing discussion #6746 (comment), where this code snippet shows how formatter spec can be created using the [UnitSchema-name].[koq-name] e.g AecUnits.LENGTH.
We are looking for the correct way to do this

• Do we store the value [UnitSchema-name].[koq-name] in our json data?
• If we are storing the above value, we should also store the required presentationUnit as formatString defined by the Kind of quantity, is that correct?
• Another idea is to just store a formatString and use the steps discussed here to create the formatterSpec out of it Formatting OverrideFormats #8069 (comment). [There seems to be no straightforward way to go from formatString to formatProps] - Is storing formatString a valid approach?
• Is there a way to serialize a formatterSpec and store it?

The above questions are just based on a primitive understanding of quantity formatter, please feel free to let me know if these assumptions are correct or not and also guide us in the right direction to implement this use case.

Adding @Josh-Schifter in case I missed out any detail.

[hl662]

Do we store the value [UnitSchema-name].[koq-name] in our json data?

This depends - If you have multiple dimension style elements, do you want them to share the same formatting options or can each of them be different? If they are sharing the same options, then to save space, you can store just the schema item name of the KindOfQuantity you want to associate your item with. Think of your element as an ECProperty - all ECProperties will have a KindOfQuantity associated to it. You would want to ask yourself if there's an existing KindOfQuantity in one of the common Schemas (like AecUnits, or RoadRailUnits) that would fit your formatting needs, or maybe create a new one.

For example, what another first party product did when they found an existing schema didn't have the KOQ they needed, they modified the schema to add new ones, and described the preesentationUnits for that i.e the formatStrings as you are asking in the question after this one.

If each of them are different, then perhaps it's better to store the FormatProps json in each of the style element.

I see that your element uses both ft and deg, so maybe your style element would store 2 kindOfQuantities, and 2 persistenceUnits. The persistenceUnits mean how you want your values be stored as within the iModel. 2250.479 would use Units.FT, and 90 would use Units.DEG, with your example above.

Is there a way to serialize a formatterSpec and store it?

No, the formatter spec contains information on unit conversion and that may be pretty large if you're using a unit that can be converted to many others. The FormatProps that represents your formatting options, and the UnitProps of the persistenceUnit is typically more than enough.

[ColinKerr]

It's not clear to me that you should be storing or referencing Formats at all. @hl662 is working on an API that stores the format specifications used by an application. Tools, UI displaying persisted properties, type editors, etc use this API so that there is a consistent place to store and configure formatting information. Each tool, property, etc just references a KindOfQuantity which is a key used to identify Format Specifications in the common store.

This would greatly simplify the drawing code because they could rely on this API to find/store/parse and return the unit conversion and formatting information you need. Another benefit is that the UI to set these formats could be standard and unified with all formatting preferences.

[pmconne]

It is important to note that this formatting will produce persistent geometry. So it can't be dependent upon formatting information that can be changed arbitrarily (e.g., through app/user settings). It could certainly use those settings as initial inputs though.

[Josh-Schifter]

@hl662, what I think you might be missing is that we are not persisting the measured values at all. When the dimension geometry is being created (on the backend), we have geometric logic to compute the measured value. We want to use the QuantityFormatter logic to convert this measure value into a string. We will the turn the string into a TextString which will be added to our geometryStream.

[Josh-Schifter]

@ColinKerr, I like the idea of each DimensionStyle storing the name of a KOQ. We will need to create a UI to show a filtered list of KOQs from the appropriate domain schema. We need to show a list of Length KOQs for use in formatting lengths, and Angle KOQs for angles, etc. This would be great because it allows the domain schema designers to control the options for formatting dimensions in their domain.

The problem I see is that each KOQ can have multiple presentation units. We need a way to also present these as options in the UI. And we need a way to persist the chosen presentation unit in the dimension style. Where I get stuck is how to store a persistent reference to a presentation unit. I think we need to store a format string, for example "f:DefaultRealU(2)[u:MM]". ???

There is a strangeness here in that I think we don't actually require the KOQ to do the formatting since the format string seems to be sufficient. Nevertheless, I do think that KOQ is the key idea here since that's what allows the domain schema to influence the dimensions.

If we agree on above, the rest is just implementation. I think we need help writing a backend function with inputs (measuredValue: number, formatString: string) which outputs formattedValue: string.

Note for myself: The term "presentation units" here is used to describe more than just units. It also includes the formatting options. For example, this KOQ the "persentation units" string is actually a semicolon separated list of formatStrings:
<KindOfQuantity typeName="LENGTH_SHORT" displayLabel="Short Length" persistenceUnit="u:M" relativeError="0.01" presentationUnits="f:DefaultRealU(2)[u:MM];f:DefaultRealU(4)[u:CM];f:AmerFI;f:AmerFI(16);f:DefaultRealU(4)[u:IN]"/>

[ColinKerr]

This is exactly the problem solved by the presentation format sets and associated API/UI @hl662 has been working on. It stores the format to be used for a given KOQ as specified by the user. All you do is pass in a KOQ name as key and you get a format back.

[ColinKerr]

Here are the docs for FormatSet: https://www.itwinjs.org/learning/quantity/#formatset

[hl662]

persist the chosen presentation unit in the dimension style

If you store the full name of the KindOfQuantity, you'd be able to query for the schemaItem of that KindOfQuantity and retrieve the persistence unit, as each KoQ has a persistenceUnit property on it. So by storing just the key for the KoQ, you would be implicitly storing a reference to the persistenceUnit as well, unless you intend to use different units?

[SheetalNair]

After further discussion on this, I have just noted down a tentative plan that we have for storing formatting information in dimensionStyle element.
The dimension style element stores json data in the iModel, and the formatting information can be a part of this jsondata as shown below

{
    ...otherStyleProps
    Length:
    {
        koq:"AeCUnits.LENGTH,
        unitSystem:"imperial"
    }
    Angle:
    {
        koq:"AeCUnits.RAD,
        unitSystem:"imperial"
    }

}

The koq property stores the {UnitSchemaName}.{koqName}, and the iModel we use, should have the UnitSchema already provisioned for this property to work well.

In our backend code for dimension text formatting, we can use the below code where AeCUnits.LENGTH can be replaced by value of koq property from the above style element and "metric" can be replaced by the value of unitSystem property from the above style element.

The above points are about how we store formatting information in the style element and use them in the backend for formatting.
For the frontend part of how we create the style element:

• Dimension style elements are created and stored in a catalogDb file (.bim file).
• The catalogDb will be provisioned with a unit schema [For now our focus is on OS+ users, so we will use the same unit schema as OS+ does]
• The StyleAdmin frontstage in drawing production has UI to create a dimension style in catalog db, this UI will now have a new input box for selecting appropriate koq and unit system for a style, the UI options will be populated by the list of koqs present in the catalogDb (that came from the unit schema we provisioned)
• After selecting appropriate options for a style, a dimension style element is inserted in the catalogDB
• Now, when using the dimension tool in drawing production, we select one of the styles from the catalogdb and copy the style element to the iModel, if the Unit schema mentioned in koq does not exist in our iModel, we display a warning stating that this style cannot be used here. [This scenario might not occur soon, as we are just targeting OS+ iModels for now, in future we might have to think about it in more details]

This is just a rough summary of all the discussions, please feel free to ask questions and give suggestions. @ColinKerr @pmconne

Adding @Josh-Schifter, in case I missed out any details

[hl662]

Looks good, AecUnits does have a KoQ called AecUnits.ANGLE you should use instead of AecUnits.RAD (RAD is a unit of measurement, not a way to format values).