Impacts of Executor handling of 2a

[JosepMariaBlasco]

Legacy programs are impacted when

• <after number> is a valid variable symbol and the NOVALUE condition is enabled.
• A label starts with a number

The second item is too restrictive. All contexts in which a taken_constant is allowed are also affected. This includes labels, but also message names, user condition names, trap names, function and procedure names, annotation names, annotation values (when symbols), class, metaclass, mixin class, subclass, and inherited classes in ::CLASS directives, ::CONSTANT names, ::CONSTANT values (when symbols), ::METHOD names, ::REQUIRES program names, namespace names, ::RESOURCE names, ::RESOURCE end delimiters, ::ROUTINE names, ...

[JosepMariaBlasco]

The problem amounts to the following: due to the free syntax of (oo)Rexx and Executor, the scanner does not have enough information to know how to parse a construct like 2A. Ideally, it would be nice if taken constants were not affected. But the only way I see to implement that is to introduce some form of mutation in the higher levels of parsing. Something like the following: mark a constant symbol as "special" when it is of the form 2A. Later on, if we discover that it is a taken constant, leave it alone, and, if it occupies the place of a term, mutate it into (2 || A).

There are some subtle problems with exponential numbers, though. Executor accepts 2E+3i as ((2E+3)||i), but this cannot be a taken constant (because of the "+"). (Thinking). Maybe one would need, then, two cases instead of one. 2A would be "special", and 2E+3i would automatically be (2E+3)||i because it cannot be a taken constant.

[jlfaucher]

This is a long message, sorry. That helps me to clarify my ideas.

After your 1st message

I was thinking to experiment something like this:
when a symbol is splittable in <number><after number>, then the scanner returns this sequence of tokens

token1    <number><after number>
token2        <number>
token3        ||
token4        <after number>

A count of alternative tokens stored on token1. Here it would be 3.

And then the parser decides which tokens to use:

• Either the full symbol
• Or the alternative tokens

I name this approach "the simple alternative tokens", by opposition to "the generalized alternative tokens", where I think I went too far by mixing ooRexx5 and Executor logic.

Your second message shows a similar idea, if I'm not wrong.

But the only way I see to implement that is to introduce some form of mutation in the higher levels of parsing. Something like the following: mark a constant symbol as "special" when it is of the form 2A. Later on, if we discover that it is a taken constant, leave it alone, and, if it occupies the place of a term, mutate it into (2 || A).

But your second message, and your examples from a previous discussion, demonstrate that it's not so simple...

There are some subtle problems with exponential numbers, though. Executor accepts 2E+3i as ((2E+3)||i), but this cannot be a taken constant (because of the "+"). (Thinking). Maybe one would need, then, two cases instead of one. 2A would be "special", and 2E+3i would automatically be (2E+3)||i because it cannot be a taken constant.

Taken as constant

To have a precise idea of what is "taken as constant", I test

x~expression  -- Object "X" does not understand message "<taken as constant>"

and with Executor, I see that the "+" is included in the message name.
Is it against the definition of "taken as constant"?

In Executor, it's possible to generate a range of numbers like that:

1~9=                             -- [ 1, 2, 3, 4, 5, 6, 7, 8, 9]
1.001~1.009(by:0.001)=           -- [ 1.001, 1.002, 1.003, 1.004, 1.005, 1.006, 1.007, 1.008, 1.009]
1e10~2e10(for:10)=               -- [ 10000000000, 10000000001, 10000000002, 10000000003, 10000000004, 10000000005, 10000000006, 10000000007, 10000000008, 10000000009]
1e-10~2e-10(by:0.1e-10, for:10)= -- [ 0.0000000001, 0.00000000011, 0.00000000012, 0.00000000013, 0.00000000014, 0.00000000015, 0.00000000016, 0.00000000017, 0.00000000018, 0.00000000019]

As you can see, it's interesting to support message with "-" or "+" inside.

Generalization of alternative tokens

I try to visualize what would be the impact if I scan both à la ooRexx5 and à la Executor, and select the tokens in function of the context...
The flow of tokens becomes:
alternative ( ooRexx5 tokens ) | ( Executor tokens )

After review of all these test cases, I think I prefer to stay with a unique rule, instead of having a dual interpretation of tokens.
What I can do is to continue the investigation of the simple tokens alternative, and try to reduce the cases of errors by using the not-splitted symbol where it's more appropriate.

If you have counter examples, or if I made a mistake somewhere, let me know.

Test cases

2e+3i
ooRexx5: i = "sqrt(-1)" ; say 2e+3i -- Nonnumeric value ("2E") used in arithmetic operation
Executor: i = "sqrt(-1)" ; say 2e+3i -- 2E+3sqrt(-1)
( 2E + 3I ) | ( 2E+3 || I )
taken as constant = "2E" versus "2E+3"

2e+3 i
ooRexx5: i = "sqrt(-1)" ; say 2e+3 i -- 2E+3 sqrt(-1)
Executor: i = "sqrt(-1)" ; say 2e+3 i -- 2E+3 sqrt(-1)
in both cases: ( 2E+3 ' ' I ) so no alternative
taken as constant = "2E" versus "2E+3"

2.1e+3.2
ooRexx5: say 2.1e+3.2 -- Nonnumeric value ("2.1E") used in arithmetic operation
Executor: say 2.1e+3.2 -- 2.1E+3.2
( 2.1E + 3.2 ) | ( 2.1E+3 || .2 )
taken as constant = "2.1E" versus "2.1E+3"

2.1 e+3
ooRexx5: e = 2.71828 ; say 2.1 e+3 -- 2.1 5.71828
Executor: e = 2.71828 ; say 2.1 e+3 -- 2.1 5.71828
in both cases: ( 2.1 ' ' E + 3 ) so no alternative
taken as constant = "2.1" for both

2.1e +3
ooRexx5: e = 2.71828 ; say 2.1e +3 -- Nonnumeric value ("2.1E") used in arithmetic operation
Executor: e = 2.71828 ; say 2.1e +3 -- 2.15.71828 Problem of operator precedence? to investigate
( 2.1E + 3 ) | ( 2.1 || E + 3 )
( 2.1E ) | ( 2.1 || E ), followed by + 3
taken as constant = "2.1E" versus "2.1"

2.1e+ 3
ooRexx5: e = 2.71828 ; say 2.1e+ 3 -- Nonnumeric value ("2.1E") used in arithmetic operation
Executor: e = 2.71828 ; say 2.1e+ 3 -- Nonnumeric value ("2.12.71828") used in arithmetic operation
( 2.1E + 3 ) | ( 2.1 || E + 3 )
( 2.1E ) | ( 2.1 || E ), followed by + 3
taken as constant = "2.1E" versus "2.1"

Test cases reusing your examples from a previous discussion:

2e-3ab -> This cannot be rejoined, as this is not a valid symbol.
ooRexx5: ab = "ab value" ; say 2e-3ab -- Nonnumeric value ("2E") used in arithmetic operation
Executor: ab = "ab value" ; say 2e-3ab -- 2E-3ab value
( 2E - 3AB ) | ( 2E-3 || AB )
taken as constant = "2E" versus "2E-3"

2A -> number + simple variable; the parser differentiates between exposed variables or not, which creates problems when rejoining.
ooRexx5: A = "A value" ; say 2A -- 2A
Executor: A = "A value" ; say 2A -- 2A value
( 2A ) | ( 2 || A )
taken as constant = "2A" versus "2"

2A. -> number + stem; ditto.
ooRexx5: A. = "A. default" ; say 2A. -- 2A.
Executor: A. = "A. default" ; say 2A. -- 2A. default
( 2A. ) | ( 2 || A. )
taken as constant = "2A." versus "2"

2A.1 -> number + compound variable; ditto.
ooRexx5: A. = "A. default" ; say 2A.1 -- 2A.1
ooRexx5: A.1 = "A.1 value" ; say 2A.1 -- 2A.1
Executor: A. = "A. default" ; say 2A.1 -- 2A. default
Executor: A.1 = "A.1 value" ; say 2A.1 -- 2A.1 value
( 2A.1 ) | ( 2 || A.1 )
taken as constant = "2A.1" versus "2"

2.3.True -> number + environment symbol
ooRexx5: say 2.3.True -- 2.3.TRUE
Executor: say 2.3.True -- 2.31
( 2.3.True ) | ( 2.3 || .True )
taken as constant = "2.3.TRUE" versus "2.3"

2ab2 -> number + pure constant symbol
ooRexx5: drop ab2 ; say 2ab2 -- 2AB2
ooRexx5: ab2 = "ab2 value" ; say 2ab2 -- 2AB2
Executor: drop ab2 ; say 2ab2 -- 2AB2
Executor: ab2 = "ab2 value" ; say 2ab2 -- 2ab2 value
( 2AB2 ) | ( 2 || AB2 )
taken as constant = "2AB2" versus "2"

2.. -> number (2.) + period.
ooRexx5: say 2.. -- 2..
Executor: say 2.. -- 2..
( 2.. ) | ( 2. || . )
taken as constant = "2.." versus "2."

2e-3.22 -> number + number.
ooRexx5: say 2e-3.22 -- Nonnumeric value ("2E") used in arithmetic operation
Executor: say 2e-3.22 -- 2E-3.22
( 2E - 3.22 ) | ( 2e-3 || .22)
taken as constant = "2E" versus "2E-3"

[JosepMariaBlasco]

I think what you call the "simple alternative tokens" is the right way to go.

As you know, symbols can be variable (and then simple, compound or stems), numbers (integers, decimal and exponential), and "constant symbols". I use quotes because, since Object Rexx was released, constant symbols which are environment symbols are no longer "constant". So that, in reality, what we have is environment symbols, on the one hand, and really constant symbols, on the other hand. I'm using "symbol literal" in my parser to refer to the "true" constant symbols. Then you have ".", which is an exception, because it's a symbol literal starting with a dot which is not a valid number, but not an environment symbol because it has only 1 character. Then "..." is overloaded. Indeed, "..." is an environment symbol (for example .environment~".."), but in certain contexts it has a fixed meaning.

A mess :)

A symbol which is "taken as a constant" can be any of these, which includes numbers. That's why "1.2E+3" is a valid label, and also a valid message name.

Say 2.3e+4( 12 )  -- Prints "15". Not that I'd recommend writing code like that! :)
Exit

2.3e+4: Return Arg(1)+3

So, yes, I would store the alternative somewhere, and then I would reconstitute or split the symbol as appropiate. That way, you'll minimize the incompatibilities, while still retaining the use of things like 2i.

When writing my parser, I've had to mark all the identifiers and/or strings which have to be marked as "taken as a constant". I can send you a list of all the places where this is used if you'd like.

[JosepMariaBlasco]

(Edit: corrected bug in EBNFs below)

Coming back to this, after having slept on it. I think there are two separate issues here, which are somehow conflated, and this conflation is most probably causing some of the (recurring) difficulties. I will develop an analysis which might well be the final one. Please bear with me, this will also be long.

On the one hand, we have the practical problem of extending ooRexx with the new syntax, and, at the same time, preserving, as much as possible, compatibility with the existing code base. Reinterpreting "2i" as "(2||i)" in the context of a term involves a tiny breakage surface, because nobody (that I know of) uses constant symbols like "2i" as terms. To the contrary, labels like "2D", "3D", "10x", etc., are not completely unheard of. It would be nice, therefore, if "2i" was interpreted as "2i" (i.e., as (oo)Rexx says, "taken as a constant") in all the other, non-term, contexts. This can be easily achieved using the "simple alternative tokens" technique, with a minor modification: when a symbol can be split in this way, you store the splitting position and, if later on, at higher levels of parsing, you observe that the symbol it is being used as a term, you simply split it at that point. This means that the splitting should occur when parsing expressions, not at the scanner stage.

On the other hand, we are left with the "pathological" case of signed exponential numbers + extra. This is when things get philosophical. In the case above, we were simply splitting what ooRexx would call a constant symbol; but, when splitting "1.2E+3A", what are we splitting??? That's the crucial question. "1.2E+3A" does not have an entity, an ontology: it does not even have a denomination, a name: this amounts to saying that we don't understand well what it is (as, otherwise, it would have a name). And here comes, I think, the real difficulty. To try to tackle this problem, a possibility would be to extend the definition of a symbol. An ooRexx symbol can be a variable symbol, a constant symbol (with the proviso that this is an outdated denomination, since it includes environment symbols, which are not constants), and numbers. Rexx syntax does some contortions to accommodate numbers as symbols, because numbers may have a signed exponent. A Signed Exponential Number has the following syntax:

 SEN ::= decimal_part ("+"|"-") exponent

and this is the only case in which a sign is allowed as part of a symbol. In Executor, constructs like "1E+2i" are allowed. But, to be consistent with the split/don't split alternative, maybe the definition of a symbol in Executor should be extended to include Signed Exponent Constructs:

 SEC ::= decimal_part ("+"|"-") exponent [extra]

where "extra" is a classical (i.e., non-signed-exponential) symbol.

This would mean that some constructs that are not ooRexx symbols would be Executor symbols, for example "1E+2I". This would mean that you could use "1E+2I" as a term and get the current Executor behaviour. This would also mean that you could write things like

  Signal 1E+2i

and get control transferred to a label called "1E+2I" (uppercased), which of course wouldn't work in ooRexx. But then the breakage would really be minimal.

In all the non-term cases, the only breakage would be an extension of the ooRexx language, i.e., more programs would be accepted as valid than before. This is equivalent to non-breakage, except in the cases where a program dynamically depends on such breakage, which are extremely rare. Indeed, in most cases, backwards compatibility could be achieved by switching to the string form, for example, in the above example, one could write

  Signal "1E+2I"

if one wanted to be 100% compatible with ooRexx.

In the term cases, there would be breakage. But this is unavoidable when you want to have an extension to a language :)

If in principle you're OK with this analysis, I can program this semantics in my parser, so that we can take a look at some code highlighted following these concepts and evaluate it further. It wouldn't be a big effort.

[JosepMariaBlasco]

[Note to myself] Oh, and -- rereading my last comment, I realized that (oo)Rexx makes a total mess between "constant symbols" and "symbols taken as a constant". A constant symbol does not need to be constant (i.e., it can be an environment symbol), unless it is "taken as a constant" (e.g., used as a method or routine name). And a symbol that is "taken as a constant" does not need to be a constant symbol, i.e., it can be a variable symbol, or an environment symbol -- when it is "taken as a constant", it is simply uppercased, but not evaluated.