0.9.2+ New WML preprocessor
Moderator: Forum Moderators
Forum rules
- Please use [code] BBCode tags in your posts for embedding WML snippets.
- To keep your code readable so that others can easily help you, make sure to indent it following our conventions.
0.9.2+ New WML preprocessor
The new preprocessor uses a recursive grammar. This grammar was posted on the forum some times ago, so I will just outline some differences between the old and the new preprocessor. All the old scenarios should still work (except for text domains, see below): the new preprocessor is supposed to be strictly more powerful.
The old one could not parse this sentence {MACRO1 ({MACRO2 (A B)} C)} because the first right parentheses would have matched the first left parenthesis. See bug #10995 for a real scenario example (Wasvoid).
The old preprocessor was also unable to handle nested conditional directives:
If A was not defined, it would have simply output "yyy" (whatever B and without any error message) instead of outputting nothing as it could be expected.
Macro definitions are the only part of the grammar that is not recursive: you can't nest a #define...#enddef inside another one, and you can't quote or comment #enddef.
The #textdomain tracking doesn't follow my initial proposal, it is a bit clever now. Consider this example:
The translation of "Something in tda" won't be taken in the "tdb" domain (where it will be included), it will be correctly searched in the "tda" domain (where it is first defined). Moreover, the argument "Something in tdb" of the macro won't be translated in "tda", but in "tdb" (where it is defined).
Finally, text domains are no more WML-scoped, they last as long as no new #textdomain directive has been encountered, and they get reset at end of file and end of macro.
The old one could not parse this sentence {MACRO1 ({MACRO2 (A B)} C)} because the first right parentheses would have matched the first left parenthesis. See bug #10995 for a real scenario example (Wasvoid).
The old preprocessor was also unable to handle nested conditional directives:
Code: Select all
#ifdef A
#ifdef B
xxx#else
yyy#endif
#endif
Macro definitions are the only part of the grammar that is not recursive: you can't nest a #define...#enddef inside another one, and you can't quote or comment #enddef.
The #textdomain tracking doesn't follow my initial proposal, it is a bit clever now. Consider this example:
Code: Select all
#textdomain tda
#define A B
_"Something in tda" + {B}#enddef
...
#textdomain tdb
message={A _"Something in tdb"}
Finally, text domains are no more WML-scoped, they last as long as no new #textdomain directive has been encountered, and they get reset at end of file and end of macro.
-
- Posts: 873
- Joined: July 4th, 2004, 9:14 pm
- Location: My imagination
- Contact:
Some questions/wishes:
Does _"bar" get translated in textdomain foo?
Does the file get searched for from a.cfg's parent?
Note that if it does, it is possible and practical for a campaign to be completely unaware of which directory it is placed in. Otherwise, if for example it uses a separate directory for maps, the scenarios must use an 'absolute' (for Wesnoth) path to include things like that (which is indeed the current practice, though I don't like it).
Implementing {../} inclusion syntax is not a good soution to this problem, because it requires each file to know how deep it is in the directory hierarchy (which is likely to change during WML refactoring).
EDIT: wait... [binary_path] and [textdomain] tags still require absolute paths... hmm...
Are parentheses still required around that?
Does that work at all? Does it work without affecting the textdomain later in the file it is used in? Will the newline after the #textdomain line get destroyed so that the macro doesn't introduce any extra newlines when used?
I also wish I could have spaces in the names of macros
Code: Select all
#file a.cfg
#textdomain foo
{b.cfg}
Code: Select all
#file b.cfg
key=_"bar"
Code: Select all
#file a.cfg
#define INCLUDE_./_SOMEWHERE
{./somewhere.cfg}#enddef
Code: Select all
#file b.cfg
{INCLUDE_./_SOMEWHERE}
Note that if it does, it is possible and practical for a campaign to be completely unaware of which directory it is placed in. Otherwise, if for example it uses a separate directory for maps, the scenarios must use an 'absolute' (for Wesnoth) path to include things like that (which is indeed the current practice, though I don't like it).
Implementing {../} inclusion syntax is not a good soution to this problem, because it requires each file to know how deep it is in the directory hierarchy (which is likely to change during WML refactoring).
EDIT: wait... [binary_path] and [textdomain] tags still require absolute paths... hmm...
Code: Select all
{MACRO ({ANOTHER_MACRO argument})}
Code: Select all
#define USING_TEXTDOMAIN TEXTDOMAIN_NAME TEXT
#textdomain {TEXTDOMAIN_NAME}
{TEXT}#enddef
I also wish I could have spaces in the names of macros
Code: Select all
#define (RITUAL_IMAGE_Void Master)
Play a Silver Mage in the Wesvoid campaign.
No, text domains do not propagate into substituted macros or files. If they did, then wmlxgettext would get the wrong domains and the campaign would be untranslatable. Since no text domain is present before bar, the default text domain applies (which happens to be "wesnoth").Invisible Philosopher wrote:Code: Select all
#file a.cfg #textdomain foo {b.cfg}
Does _"bar" get translated in textdomain foo?Code: Select all
#file b.cfg key=_"bar"
This part of the preprocessor was not changed: the file would be looked up with respect to the parent of b.cfg.Invisible Philosopher wrote:Code: Select all
#file a.cfg #define INCLUDE_./_SOMEWHERE {./somewhere.cfg}#enddef
Does the file get searched for from a.cfg's parent?Code: Select all
#file b.cfg {INCLUDE_./_SOMEWHERE}
No, neither are they around quoted strings containing spaces.Invisible Philosopher wrote:Are parentheses still required around that?Code: Select all
{MACRO ({ANOTHER_MACRO argument})}
Yes, yes, no.Invisible Philosopher wrote:Does that work at all? Does it work without affecting the textdomain later in the file it is used in? Will the newline after the #textdomain line get destroyed so that the macro doesn't introduce any extra newlines when used?Code: Select all
#define USING_TEXTDOMAIN TEXTDOMAIN_NAME TEXT #textdomain {TEXTDOMAIN_NAME} {TEXT}#enddef
Hmm... the last "no" means that it won't work in fact. Because the newline is not destroyed, the parser will complain that there is a quoted string lying around in
Code: Select all
message={USING_TEXTDOMAIN the_domain _"the string"}
Code: Select all
message=""+{USING_TEXTDOMAIN the-domain _"the string"}
Hmm... spaces are allowed at the time of macro substitution, so you could indeed doInvisible Philosopher wrote:I also wish I could have spaces in the names of macrosCode: Select all
#define (RITUAL_IMAGE_Void Master)
Code: Select all
{(RITUAL_IMAGE_Void Master)}
The three wishes make sense to me, so I may modify the preprocessor (and the parser for the newline issue) if there is no objection. Speaking about new behavior: should the following lead to two macro arguments or only one (there is no space between the two parts)? At the moment, it is interpreted as two arguments, requiring parentheses for it to be interpreted as one single argument.
Code: Select all
{A {B}C}
-
- Posts: 873
- Joined: July 4th, 2004, 9:14 pm
- Location: My imagination
- Contact:
One argument. That is IMO much more obvious behavior; spaces should be and easily can be used to separate macro arguments.silene wrote:Speaking about new behavior: should the following lead to two macro arguments or only one (there is no space between the two parts)? At the moment, it is interpreted as two arguments, requiring parentheses for it to be interpreted as one single argument.Code: Select all
{A {B}C}
Oh, and does this work now? (having a macro argument that happens to have the same name as another macro)
Code: Select all
#define X Y
{Y}#enddef
#define Y Z
#enddef
{X Q}
Play a Silver Mage in the Wesvoid campaign.
What do you mean? Both with the old and the new preprocessor, the result would have been Q. Isn't it what you would expect?Invisible Philosopher wrote:Oh, and does this work now? (having a macro argument that happens to have the same name as another macro)Code: Select all
#define X Y {Y}#enddef #define Y Z #enddef {X Q}
-
- Posts: 873
- Joined: July 4th, 2004, 9:14 pm
- Location: My imagination
- Contact:
I was of the impression that it didn't work with the old preprocessor. I don't remember why I thought that. Now I don't know if it was true.silene wrote:What do you mean? Both with the old and the new preprocessor, the result would have been Q. Isn't it what you would expect?
Play a Silver Mage in the Wesvoid campaign.
-
- Posts: 873
- Joined: July 4th, 2004, 9:14 pm
- Location: My imagination
- Contact:
Using CVS, I have found some problems (and check out this patch of mine, which helped me pinpoint some of them):
Wesvoid.cfg, my campaign file in the normal place:
The inclusion fails. According to the output, it thought it was using
Also according to the output, in themes/default.cfg, which did,
turns intoso, it appears you will have to implement textdomain eating the newline or else all key={MACRO} constructs will be broken.
Also I have a comment which I don't know whether it's a problem: In (Campaign_Name).cfg, from my campaign template, this broke (and gave some nonsensical error messages much later because of it):
Putting parentheses around the whole thing fixed it:
And the result of this is quite a mess:
to
What's worse, it's different that the output of this, which should be identical (it has no parentheses around _"blah blah"):
, I think.
Wesvoid.cfg, my campaign file in the normal place:
Code: Select all
{./Wesvoid/public.cfg}
I have hilighted the error: ./ is broken currently because it doesn't have a '/' there where it should. It should only erase the '.', not the '/', when processing that path.line 1 /Users/isaac//Library/Preferences/Wesnoth/data/campaignsWesvoid/public.cfg 1 /Users/isaac//Library/Preferences/Wesnoth/data/campaigns/Wesvoid.cfg 30 data/game.cfg
Also according to the output, in themes/default.cfg, which did
Code: Select all
#define FONT_TINY
10 #enddef
Code: Select all
font_size={FONT_TINY}
Code: Select all
font_size=#line 17 data/themes/default.cfg 1 data/game.cfg 166 data/themes/default.cfg 1 data/game.cfg
#textdomain wesnoth
10 #line 166 data/themes/default.cfg 1 data/game.cfg
#textdomain wesnoth
Also I have a comment which I don't know whether it's a problem: In (Campaign_Name).cfg, from my campaign template, this broke (and gave some nonsensical error messages much later because of it):
Code: Select all
{@campaigns/(Campaign_Name)/external_units.cfg}
Code: Select all
{(@campaigns/(Campaign_Name)/external_units.cfg)}
And the result of this is quite a mess:
Code: Select all
#define USING_TEXTDOMAIN TEXTDOMAIN_NAME TEXT
#textdomain {TEXTDOMAIN_NAME}
{TEXT}#enddef
key={USING_TEXTDOMAIN wesnoth-test (_"blah blah")}
Code: Select all
#line 1 /Users/isaac/Library/Preferences/Wesnoth/data/test/test.cfg
#textdomain wesnoth
#line 4 /Users/isaac/Library/Preferences/Wesnoth/data/test/test.cfg
key=#line 2 /Users/isaac/Library/Preferences/Wesnoth/data/test/test.cfg 5 /Users/isaac/Library/Preferences/Wesnoth/data/test/test.cfg
#textdomain wesnoth
#textdomain {TEXTDOMAIN_NAME}
#line 5 /Users/isaac/Library/Preferences/Wesnoth/data/test/test.cfg
#textdomain wesnoth
_#line 5 /Users/isaac/Library/Preferences/Wesnoth/data/test/test.cfg
#textdomain wesnoth
"blah blah"#line 3 /Users/isaac/Library/Preferences/Wesnoth/data/test/test.cfg
#textdomain {TEXTDOMAIN_NAME}
#line 5 /Users/isaac/Library/Preferences/Wesnoth/data/test/test.cfg
#textdomain wesnoth
#textdomain wesnoth
#textdomain wesnoth
goes to#define USING_TEXTDOMAIN TEXTDOMAIN_NAME TEXT
#textdomain {TEXTDOMAIN_NAME}
{TEXT}#enddef
key={USING_TEXTDOMAIN wesnoth-test _"blah blah"}
Code: Select all
#line 1 /Users/isaac/Library/Preferences/Wesnoth/data/test/test.cfg
#textdomain wesnoth
#line 4 /Users/isaac/Library/Preferences/Wesnoth/data/test/test.cfg
key=#line 2 /Users/isaac/Library/Preferences/Wesnoth/data/test/test.cfg 5 /Users/isaac/Library/Preferences/Wesnoth/data/test/test.cfg
#textdomain wesnoth
#textdomain {TEXTDOMAIN_NAME}
_#line 5 /Users/isaac/Library/Preferences/Wesnoth/data/test/test.cfg
#textdomain wesnoth
"blah blah"#line 3 /Users/isaac/Library/Preferences/Wesnoth/data/test/test.cfg
#textdomain {TEXTDOMAIN_NAME}
#line 5 /Users/isaac/Library/Preferences/Wesnoth/data/test/test.cfg
#textdomain wesnoth
#textdomain wesnoth
#textdomain wesnoth
Last edited by Invisible Philosopher on May 20th, 2005, 10:48 am, edited 1 time in total.
Play a Silver Mage in the Wesvoid campaign.
Hmm... strange. I can't reproduce it, but it probably is because I use a zipped Wesnoth. I will have to take a look at it.Invisible Philosopher wrote:Using CVS, I have found some problems (and check out this patch of mine, which helped me pinpoint some of them):
Wesvoid.cfg, my campaign file in the normal place:The inclusion fails. According to the output, it thought it was usingCode: Select all
{./Wesvoid/public.cfg}
I have hilighted the error: ./ is broken currently because it doesn't have a '/' there where it should. It should only erase the '.', not the '/', when processing that path.line 1 /Users/isaac//Library/Preferences/Wesnoth/data/campaignsWesvoid/public.cfg 1 /Users/isaac//Library/Preferences/Wesnoth/data/campaigns/Wesvoid.cfg 30 data/game.cfg
Your preprocessed output is wrong. The preprocessor does not generate additional # characters. So the fact that #textdomain does not eat newline characters is irrelevant here (and same for #line).Invisible Philosopher wrote:Also according to the output, in themes/default.cfg, which did,Code: Select all
#define FONT_TINY 10 #enddef
turns intoCode: Select all
font_size={FONT_TINY}
so, it appears you will have to implement textdomain eating the newline or else all key={MACRO} constructs will be broken.Code: Select all
font_size=#line 17 data/themes/default.cfg 1 data/game.cfg 166 data/themes/default.cfg 1 data/game.cfg #textdomain wesnoth 10 #line 166 data/themes/default.cfg 1 data/game.cfg #textdomain wesnoth
Your file name contains parentheses? Then it's natural it broke, since the parentheses have a special meaning when preprocessing a bracketed block. And indeed, putting parentheses around the whole text will fix it.Invisible Philosopher wrote:Also I have a comment which I don't know whether it's a problem: In (Campaign_Name).cfg, from my campaign template, this broke (and gave some nonsensical error messages much later because of it):Putting parentheses around the whole thing fixed it:Code: Select all
{@campaigns/(Campaign_Name)/external_units.cfg}
Code: Select all
{(@campaigns/(Campaign_Name)/external_units.cfg)}
-
- Posts: 873
- Joined: July 4th, 2004, 9:14 pm
- Location: My imagination
- Contact:
Well actually I did not change the preprocessor. I just replace '\376' with '#' in the output because '\376' is not such a good character to look at.silene wrote:Your preprocessed output is wrong. The preprocessor does not generate additional # characters. So the fact that #textdomain does not eat newline characters is irrelevant here (and same for #line).
Play a Silver Mage in the Wesvoid campaign.
-
- Posts: 873
- Joined: July 4th, 2004, 9:14 pm
- Location: My imagination
- Contact:
Never mind about the ./ problem, I figured out what I was doing wrong.
Play a Silver Mage in the Wesvoid campaign.
-
- Posts: 873
- Joined: July 4th, 2004, 9:14 pm
- Location: My imagination
- Contact:
putting macro argument usages in parentheses is still necessary (using CVS of about an hour ago (I hope I actually was using it, not that I have any reason to believe I wasn't)). For an example of how it goes "wrong":
displays whereas some sort of error would be expected, or if not an error, at least
Code: Select all
#define TESTY ARG
{VARIABLE {ARG}}#enddef
[message]#in an event
speaker=narrator
message="{TESTY (a b)}"
[/message]
Code: Select all
[set_variable]
name=a
value=b
[/set_variable]
Code: Select all
[set_variable]
name=a b
value=
[/set_variable]
Play a Silver Mage in the Wesvoid campaign.
Yes, it is the expected behavior of macro argument substitution: they are purely textual, they don't carry any semantic. Now that I see your example, I understand this behavior could be quite error-prone. I have to take a deeper look, but I already know it won't be an easy fix: it requires context-sensitive substitution to add parentheses where it will make a difference. Another solution would be to switch to a more unified (macro argument / macro) substitution model; this is the fix I prefer.
But there is a drawback. When I rewrote the preprocessor, I tried as much as possible to not break any existing WML code. And as a matter of fact, none of the mainline campaigns had to be fixed, and the few user campaigns I had went along fine (UtBS had to be fixed, but it contained a typo the old preprocessor silently misinterpreted, so I still consider it a success). However, changing the way macro argument substitution is handled will break all the campaigns that have parentheses in the macro definitions in order to enforce argument locality.
Consequently, this is a strong incentive for me not to fix the problem, although I itch to. I will only do it if quite a few campaign designers ask me to.
But there is a drawback. When I rewrote the preprocessor, I tried as much as possible to not break any existing WML code. And as a matter of fact, none of the mainline campaigns had to be fixed, and the few user campaigns I had went along fine (UtBS had to be fixed, but it contained a typo the old preprocessor silently misinterpreted, so I still consider it a success). However, changing the way macro argument substitution is handled will break all the campaigns that have parentheses in the macro definitions in order to enforce argument locality.
Consequently, this is a strong incentive for me not to fix the problem, although I itch to. I will only do it if quite a few campaign designers ask me to.