Support embedded directions

[Hywan]

A string can contain both left-to-right and right-to-left text. We need a better algorithm to guess the current direction of a text :-).

[boast]

Hey there, coming from reddit :) Some suggestions for an algorithm to solve this issue:

• Check if the string contains LRM (0x200e) or RLM (0x200f) (and treat ARM 0x061c ‭arabic letter mark as "alias" for RLM), as they are specifically used to mark the string, in which it should be interpreted.
  • If it contains both directions, return BIDI (should add this as constant)
  • else if it only contains LRM, return LTR
  • else if it only contains RLM and / or ARM, return RTL
• Set default assumption on the first character
• Check if we find any markers (LRM, LRE, LRO (may LRI) and RLM, RLE, RLO (may RLI), ARM) which would imply a direction change compared to the first character, if so, return BIDI
• Check the string if it contains a character from the opposing direction, if so, return BIDI, if not, return the respective direction based on the assumption we have from the first string.

Does this sound reasonable? As I cannot think of any sane way to detect that "私 - is a japanese letter" "should" be LTR, the user has decide by himself what to do with BIDI text.

[Hywan]

@boast It sounds reasonable yes. I didn't check how other implemetation deals with it. Any PR :-)?

[boast]

As for reference implementations: https://github.com/waiting-for-dev/string-direction

Or http://en.wikipedia.org/wiki/Bi-directional_text on that topic (notice the table with the classifications). I'll work on it tonight 👍 However, probably need to refactor some methods into helper protected methods to do the checks more granulated.

[Hywan]

@boast Thank you! :-)

[boast]

I tried my best to adapt the coding style. No tests broken (or lets say: some tests failed on my Ubuntu Dev Machine before I changed anything, seems like those collator and normalizer tests (especially when they are not available) are broken?) and added a new one following more or less the spec described above.

[Hywan]

ping?

[boast]

Hey there, thank you for the ping. I was occupied this half year with doing my bachelor degree in CS. ;) We should define our definitive approach for this problem together and then I / we can work out the implementation. My knowledge about the problem comes specifically from these sources:

• http://unicode.org/reports/tr9/
• https://en.wikipedia.org/wiki/Bi-directional_text (extensive list)

IMHO, we should first decide on the actual "goal" and "usecase" of this method. Why and when is the information "which direction is this text going" needed? Because one can go crazy on the "strong", "weak" and "normal" characters and contexts...

[Hywan]

So far, we use getCharDirection to decide the behavior of append, prepend and other methods. This method only checks the first character. We must check the last character first. Second, it should be great to have a method to know if we have bi-directional text. I don't know really why it can be useful yet but I am sure it will be. We can also add methods to force to change the direction of the text (maybe we would like to write french in reverse order 😉). And a most useful usage is:

• Iterate over direction portions. It can be particularly useful when transforming it into HTML for instance (or PDF, text etc.).
• Also, with the append and prepend methods for instance, we can say: $str->append('text', $str::RTL); to force appending something in the opposite direction (to have bi-directionnal text thus).

[Hywan]

PS: How your bachelor goes 😉?

[Hywan]

Another use case:

• When comparing strings, we would compare portion of directions, not the whole string at once. This some usages I think of.