'Missing closing tag'. Too many macros?

[Kernigh]

Some hackers like WML macros. They have macros that expand other macros that expand more macros, and so on. For example, SXCollection (the big collection of SurvivalXtreme maps) has thousands of lines of macros. But there seems to be a limit.

Many users report an error about 'Missing closing tag' to SXCollection thread. However, the error is different for each user, and some users never get the error. The error seems to change with different operating systems (Linux, Windows or OpenBSD) or different installed add-ons. Some hackers checked the WML code of SXC, but found no missing closing tags! I tried to delete the macro with my error, but my error moved to {SXC_ARMORY_LIMIT 4} in SXC_Attack, but there is no error for {SXC_ARMORY_LIMIT 4} in any of the scenarios included before SXC_Attack.

I seem to have hit the WML ceiling. The WML preprocessor seems to become 'full' and skip the last lines of a macro, thus the 'Missing closing tag'. To test this, I tried this...

Code: Select all

#define MACRO_0 var
    {VARIABLE_OP {var} add 1}
#enddef

#define MACRO_1 var
    {MACRO_0 {var}}
    {MACRO_0 {var}}
#enddef

#define MACRO_2 var
    {MACRO_1 {var}}
    {MACRO_1 {var}}
#enddef

#define MACRO_3 var
    {MACRO_2 {var}}
    {MACRO_2 {var}}
#enddef

...

#define MACRO_100 var
    {MACRO_99 {var}}
    {MACRO_99 {var}}
#enddef

When I put MACRO_13 in a scenario, it worked (by adding 2 ** 13 = 8192 to {var}). But when I put MACRO_14 in a scenario, I got another error about 'Missing closing tag':

Code: Select all

20101202 18:58:53 error config: error reading usermade add-on '/home/kernigh/.wesnoth1.8/data/add-ons/WML_Many_Macro_Test/_main.cfg'
20101202 18:58:53 error general: The following add-on had errors and could not be loaded:
/home/kernigh/.wesnoth1.8/data/add-ons/WML_Many_Macro_Test/_main.cfg
ERROR DETAILS:
Missing closing tag for tag event at ~add-ons/WML_Many_Macro_Test/scenarios/WML_Many_Macro_Test.cfg:28 included from ~add-ons/WML_Many_Macro_Test/_main.cfg:3, value '' at ~add-ons/WML_Many_Macro_Test/macros/gen.cfg:5 included from ~add-ons/WML_Many_Macro_Test/_main.cfg:2 included from ~add-ons/WML_Many_Macro_Test/macros/gen.cfg:10 included from ~add-ons/WML_Many_Macro_Test/_main.cfg:2 included from ~add-ons/WML_Many_Macro_Test/macros/gen.cfg:15 included from ~add-ons/WML_Many_Macro_Test/_main.cfg:2 included from ~add-ons/WML_Many_Macro_Test/macros/gen.cfg:20 included from ~add-ons/WML_Many_Macro_Test/_main.cfg:2 included from ~add-ons/WML_Many_Macro_Test/macros/gen.cfg:24 included from ~add-ons/WML_Many_Macro_Test/_main.cfg:2 included from ~add-ons/WML_Many_Macro_Test/macros/gen.cfg:29 included from ~add-ons/WML_Many_Macro_Test/_main.cfg:2 included from ~add-ons/WML_Many_Macro_Test/macros/gen.cfg:34 included from ~add-ons/WML_Many_Macro_Test/_main.cfg:2 included from ~add-ons/WML_Many_Macro_Test/macros/gen.cfg:40 included from ~add-ons/WML_Many_Macro_Test/_main.cfg:2 included from ~add-ons/WML_Many_Macro_Test/macros/gen.cfg:44 included from ~add-ons/WML_Many_Macro_Test/_main.cfg:2 included from ~add-ons/WML_Many_Macro_Test/macros/gen.cfg:50 included from ~add-ons/WML_Many_Macro_Test/_main.cfg:2 included from ~add-ons/WML_Many_Macro_Test/macros/gen.cfg:54 included from ~add-ons/WML_Many_Macro_Test/_main.cfg:2 included from ~add-ons/WML_Many_Macro_Test/macros/gen.cfg:59 included from ~add-ons/WML_Many_Macro_Test/_main.cfg:2 included from ~add-ons/WML_Many_Macro_Test/macros/gen.cfg:65 included from ~add-ons/WML_Many_Macro_Test/_main.cfg:2 included from ~add-ons/WML_Many_Macro_Test/macros/gen.cfg:69 included from ~add-ons/WML_Many_Macro_Test/_main.cfg:2 included from ~add-ons/WML_Many_Macro_Test/macros/gen.cfg:75 included from ~add-ons/WML_Many_Macro_Test/_main.cfg:2 included from ~add-ons/WML_Many_Macro_Test/scenarios/WML_Many_Macro_Test.cfg:46 included from ~add-ons/WML_Many_Macro_Test/_main.cfg:3

The closing tag [/event] is there, and the scenario works perfectly if I use {MACRO_13 number}. But if I change {MACRO_13 number} to {MACRO_14 number}, then I get the above error. MACRO_14 looks correct to me:

Code: Select all

#define MACRO_13 var
    {MACRO_12 {var}}
    {MACRO_12 {var}}
#enddef

#define MACRO_14 var
    {MACRO_13 {var}}
    {MACRO_13 {var}}
#enddef

No one needs MACRO_14, but SXC seems to need all its macros. This macro problem is the major reason why players are not playing SurvivalXtreme. I need help to fix this problem! Is there bug in Wesnoth C++? I am running OpenBSD 4.8 and Wesnoth 1.8.5.

How to reproduce this problem.

Method 1: using SXC

1. Start Wesnoth 1.8.5, click 'Get Add-ons', then install 'Mercenaries' and 'SXCollection'. (SXCollection requires Mercenaries.)
2. Now click 'Multiplayer' and then 'Local Game'.
3. You might get an error. But if you reach the map list (the 'Create Game' screen) without an error, then you did not reproduce the error. Try method 2 instead.

Method 2: using my test scenario

1. Unzip my attached WML_Many_Macro_Test.zip, and install it at ~add-ons/WML_Many_Macro_Test.
2. Start Wesnoth 1.8.5, click 'Multiplayer' then 'Local Game'.
3. You might get an error. But if you reach the map list, then choose 'WML Many Macro Test'. Move any unit to the 'Test' sign post. You will see the number 8192 calculated by MACRO_13.
4. Edit the '{MACRO_13 number}' of scenarios/WML_Many_Macro_Test.cfg. If there is no error, then change it to MACRO_14, MACRO_15, ..., until you get the error. But if you get the error, then change it to MACRO_12, MACRO_11, ..., until the error goes away. After each change, restart Wesnoth and try again the scenario. (Caution! If you go too high, like MACRO_20, then Wesnoth might take hours to expand all those macros. If this happens, then you can force Wesnoth to quit.)

Can anyone reproduce the problem with method 1 or method 2? Does the problem happen with all operating systems?

[Pentarctagon]

got the error just now with sxc/mercenary's on my 1.8.5, no other add-ons were installed. I'm on windows 7.

[silene]

Can anyone reproduce the problem with method 1 or method 2? Does the problem happen with all operating systems?

On my computer, the preprocessed file for MACRO_13 requires 211MB space. For MACRO_14, it is 462MB. MACRO_16 would cause any 32-bit operating systems to give up. Wesnoth is not a magical device: exponential is exponential. Did you really expect something else?

[zookeeper]

Yeah, MACRO_13 expands to...what, 32768 lines of WML, and MACRO_14 to 65536 lines, and so on? If anyone's doing something similar to that in an add-on then they really shouldn't.

[pauxlo]

About the error: There should be a more useful error message, like "out of memory" instead of "missing closing tag".

How to avoid this: Use custom events (with custom names) and [fire_event] to have a WML-level "function calling" instead of these preprocessor level macro expansion.

Here as an idea (untested):

Code: Select all

  [event]
    name=event 0
    first_time_only=no
    {VARIABLE_OP $var_name add 1}
  [/event]
  [event]
    name=event 1
    first_time_only=no
    [fire_event]
      name=event 0
    [/fire_event]
    [fire_event]
      name=event 0
    [/fire_event]
  [/event]
  ...
  [event]
    name=event 100
    first_time_only=no
    [fire_event]
      name=event 99
    [/fire_event]
    [fire_event]
      name=event 99
    [/fire_event]
  [/event]

And then to call this:

Code: Select all

    [event]
        name=moveto
        first_time_only=no
        [filter]
            x=3,10
            y=3,4
        [/filter]

        {VARIABLE number 0}
        {VARIABLE var_name number}
        [fire_event]
          name=event 100
        [/fire_event]
        {FLOATING_TEXT x,y=$x1,$y1 255,0,255 $number}
    [/event]

I think you could even generate these 100 similar events by WML, using a loop and [insert_tag].

(Of course, you would not really implement 2^n by addition.)

You can instead write your code in Lua and register it as a tag, which is much more comfortable, I think.

[Anonymissimus]

How to avoid this: Use custom events (with custom names) and [fire_event] to have a WML-level "function calling" instead of these preprocessor level macro expansion.
...
You can instead write your code in Lua and register it as a tag, which is much more comfortable, I think.

Yes, good demonstration of macros not being the same as functions. Only advantage I can think of is that they can take arbitrary code blocks as arguments what a normal function can't.
When limited to wml, custom events are often a much better alternative; although these event-functions can take only units as arguments (you can however store a variable previously).

[Kernigh]

On my computer, the preprocessed file for MACRO_13 requires 211MB space. For MACRO_14, it is 462MB. MACRO_16 would cause any 32-bit operating systems to give up. Wesnoth is not a magical device: exponential is exponential. Did you really expect something else?

How did you measure these numbers? Can you show me how to do this, so I can make my 211MB file for MACRO_13?

MACRO_14 is a very long, stupid macro that expands to 16384 copies of {VARIABLE_OP {var} add 1}. You say that MACRO_14 expands to 462MB = 462 * 1024 * 1024 = 484442112 bytes. If 16384 copies of {VARIABLE_OP {var} add 1} is 484442112 bytes, then a single {VARIABLE_OP {var} add 1} is 29568 bytes. This number seems wrong to me, do I really need 29568 bytes (more than 28 kilobytes) to put only one {VARIABLE_OP {var} add 1} in a scenario?

Yeah, MACRO_13 expands to...what, 32768 lines of WML, and MACRO_14 to 65536 lines, and so on? If anyone's doing something similar to that in an add-on then they really shouldn't.

SXC does not have one big macro. But ~add-ons/SXCollection/macros/SXCmacros.cfg defines 7182 lines of macros, plus other files have more macros.

One of the larger macros is SXC_ARMORY_LIMIT. The first scenario expands it five times, like this:

Code: Select all

    {SXC_ARMORY_LIMIT 1}
    {SXC_ARMORY_LIMIT 2}
    {SXC_ARMORY_LIMIT 3}
    {SXC_ARMORY_LIMIT 4}
    {SXC_ARMORY_LIMIT 5}

And there are 24 scenarios, with 3 to 5 expansions of SXC_ARMORY_LIMIT per scenario. In total, SXCollection has 118 expansions of SXC_ARMORY_LIMIT. The definition of SXC_ARMORY_LIMIT is 158 lines. To do all the expansions, we get 118 * 158 = 18644 lines. But SXC_ARMORY_LIMIT calls many other macros, so we probably get more than 50_000 lines.

In total, SXCollection might expand to hundreds of thousands (100_000s) of lines of WML.

About the error: There should be a more useful error message, like "out of memory" instead of "missing closing tag".

Yes, I agree now, Wesnoth needs a better error message.

Yes, good demonstration of macros not being the same as functions.

I would say, that macros are functions, but macros are inline functions, which is why there are 118 copies of the SXC_ARMORY_LIMIT 'function'. If I changed SXC_ARMORY_LIMIT to an event, as pauxlo suggests, then there would be only 24 copies of the SXC_ARMORY_LIMIT 'function'. (I assume that each of the 24 scenarios needs a copy of the event.)

The conclusion is that Wesnoth needs a better error message, and also that SXC needs to change some macros to custom events (or custom Lua tags).

[zookeeper]

I don't think anyone disagrees that better error messages would be really nice; a quick look at the WML workshop reveals that a whole lot of newbies have little idea why their code produces errors or how to figure it out themselves. However, someone needs to actually do it, and I doubt it's anywhere near trivial.

[-stf-]

Please can you send me a PM with the code for SXC_ARMORY_LIMIT changed to an event so I can see how to do it? I have not much time recently to find it out and when it will be working, it can help solve at least part of the problem and maybe I can apply it also to other macros. I have an idea but that needs time to do it and test it whether it's correct.
Also, I never worked with lua so it could take more time to rewrite the code in lua (when it's possible).
The other question is: Why is all the code for all scenarios loaded into the cache while loading it only for the selected scenario after it's selection couldn't take so long time? It couldn't take more time to parse the code after loading than loading the saved game. I think it could solve this problem too but needs the change in engine to not create cache for scenarios but only for units and I am not too familiar with the Wesnoth source code to help with that.

[Kernigh]

Please can you send me a PM with the code for SXC_ARMORY_LIMIT changed to an event so I can see how to do it?

I sent a PM with my attempt to change SXC_ARMORY_LIMIT to an event.

----

With my machine, I know that Wesnoth can expand MACRO_13 but fails to expand MACRO_14. So I decided to try a different macro language. These are my macros for m4 + sh.

Code: Select all

define(macro_0,
    `$1=$(dc -e "$$1 1 + p")')dnl
dnl
define(macro_1,
    `macro_0($1)
    macro_0($1)')dnl
dnl
define(macro_2,
    `macro_1($1)
    macro_1($1)')dnl
dnl
define(macro_3,
    `macro_2($1)
    macro_2($1)')dnl
dnl
...
define(macro_100,
    `macro_99($1)
    macro_99($1)')dnl

I am using two programs from OpenBSD 4.8: m4 is a generic macro preprocessor, and sh is a shell for running other programs. I decided to try macro_18.

Code: Select all

number=0
macro_18(number)
echo $number

My m4 expands these three lines to a sh script that repeatedly launches the dc calculator program to add 1 to a number:

Code: Select all

number=0
number=$(dc -e "$number 1 + p")
    number=$(dc -e "$number 1 + p")
    number=$(dc -e "$number 1 + p")
    number=$(dc -e "$number 1 + p")
    number=$(dc -e "$number 1 + p")
    ...
    number=$(dc -e "$number 1 + p")
echo $number

My sh can run this script! It slowly uses addition to calculate that 2 ** 18 is 262144.

Code: Select all

$ time m4 main.m4 | sh
262144
    7m35.74s real     2m5.10s user     5m41.38s system

My sh script is slow (about 7 minutes 30 seconds), because of the system calls to launch the dc program 262144 times. The preprocessor is fast; m4 needs less than 1 second to expand my call to macro_18. Also, the expansion of macro_18 requires 9437203 bytes (about 9 MB).

I am attaching my m4 + sh example.

M4_Many_Macro_Test.zip

m4 + sh using macro_18

(1.8 KiB) Downloaded 124 times

The big contrast is between WML and m4. My WML can barely expand MACRO_13, and fails to expand MACRO_14. My m4 easily handles macro_13 and macro_14, and rises to macro_18 without problem. I must wonder how MACRO_14 overloads Wesnoth, but macro_14 is only a tiny job for m4.