Working With and Around Self-Super Code Layering - A Best Practice (?)

[Minty23185Fresh]

Issue
Local script, those .lua files referenced in XML windowclasses, templates and controls, because of code layering, are difficult to override without wholesale function or even complete script file replacement. Having to replace whole functions or files bloats extensions, may make the use of multiple extensions by users impossible and can cause code maintenance to be more difficult or time consuming than necessary. The greatest single contribution to this difficulty is function and variable scope.

Solution
To mitigate the issue of local (modular or file level) variable scope, get and set functions, at the ruleset level, can be employed to expose the variables to other code layers. This methodology helps mitigate the issue of functions executing in an extension layer rather than the ruleset layer. By using a side-by-side existence for functions rather than function replacement code duplication can be substantially reduced.

Methodology
An agent or interface is used to host the get and set functions. The agent can and probably should reside in its own extension, a code set exposed to modification by all developers. This allows multiple extension employment by users without demanding the use and loading of a particular extension, other than the agent.

Implementation

Agent
The agent, or interface, is a wholesale replacement of specific ruleset code. This typically involves the XML object (windowclass, template, etc.) that references the .lua script affected by a developer's extension as well as the .lua file itself. The portion of the XML file containing the target object is copied, verbatim, to a like-named file in the agent and referenced in the extension's extension.xml file. The only change is the reference within the object to its .lua script file. Instead of referring to the ruleset LUA script, the XML will reference an exact copy of the ruleset .lua file residing in the agent. The .lua file, as just stated, is an exact copy of the ruleset file, unmodified except for one get function and one set function. The two functions provide a single conduit of variable exposure from the agent script to and from extensions employing the agent.

Functions
Using side-by-side function existence, at the global level, was discussed here. In essence, whenever the developer needs to modify a ruleset function, rather than just copying the function to the developer's extension, then modifying it as a replacement for the ruleset function, the developer should leave the ruleset function in place but rename it as a helper function. Then the ruleset function is made to point to the extension's like-named function. A minimal amount of code is added to the extension's function to augment the ruleset code. The extension's function calls the ruleset (helper) function either before or after its own code execution. When properly implemented this methodology allows multiple extensions to be developed by independent developers without regard to the contents of other extensions nor which ones the user decides to employ. As each extension is loaded it sets aside any other extension's functionality as a helper and substitutes it own. At runtime all extension functionality is performed one after the other via the side-by-side helpers.

Discussion

Conventions
1) A directory structure within the Agent should mirror that of the ruleset. So an .xml file from the ruleset's "common" folder should reside in a "common" folder within the Agent. A copied .lua file in the "common/scripts" folder of the ruleset should be placed in the same named Agent subfolder. This helps later developer's know where files came from and if a file that they need to work with already exists within an Agent.
2) Though not necessary, I'd suggest that XML and LUA file names in the Agent be prefaced with "ERA_" (Extension-Ruleset Agent). This will help developers recognize the agent files, those that are minimally modified, from their own extension files.
3) Helper functions should be named the same as the ruleset function they are replacing with a prefix unique to the developer's extension and a suffix of "_helper". Example: if one is developing the "MINT extension one might use: "MINT_theFunction_helper( )". This allows multiple helpers to reside side-by-side, as in EXT1_funct_helper, EXT2_funct_helper, EXT3_funct_helper.
4) Given the helper naming convention, one might consider naming the other functions in their extension similarly: MINT_updateDisplay( ), MINT_roll( ), MINT_getRandomNumber( ).
5) The added get and set functions in the Agent LUA file contain a root name of "Super", as in getSuperVar( ) and setSuperVar( ) as a simple indication of where those functions live, in the Agent, the "super" layer to all subsequent extension layers.
6) Be sure to pay attention to <loadorder> in the extension.xml files. Agent should load before extensions (have lower load order number).

Agent
At a minimum the agent will have three code files: the extension.xml, an .xml file containing the affected template, windowclass or control, and a .lua file.

The pertinent code within the extension.xml file would reference the XML object's file, something like this for a template_buttons.xml file in the ruleset (CBA stands for "Common Buttons Agent"):

   <base>
      <includefile source="common/CBA_template_buttons.xml" />
   </base>

The template that is being augmented, is the only object that need be in the CBA_template_buttons.xml file. The template should be an exact copy of the template in the ruleset. And then a single change, the script reference. The Agent's reference, should point to the like-named .lua file in the Agent.

   <root>
      <template name="button_iconcycler">
         <genericcontrol>
            <anchored height="20" />
            <script file="common/scripts/CBA_button_iconcycler.lua" />
         </genericcontrol>
      </template>
   </root>

And the referenced CBA_button_iconcycler.lua file would be an exact copy of the ruleset LUA file with two functions added to the bottom of the file, a "getter" and a "setter":

local sString;
local nInteger;

-- all of the functions would be here......

function getSuperVar(sVarName)
   if sVarName == "sString" then
      return sString;
   elseif sVarName == "nInteger" then
      return nInteger;
   else
      -- something here to deal with unhandled argument values
   end
end

function setSuperVar(sVarName, aVarValue)
   if sVarName == "sString" then
      sString = aVarValue;
   elseif sVarName == "nInteger" then
      nInteger = aVarValue;
   else
      -- something here to deal with unhandled argument values
   end
end

LUA Functions
Side-by-side existence of LUA script, the functions that need augmenting is accomplished in the extension's onOnit( ) function in the .lua file. Recall, the ruleset (actually the Agent's) function is renamed as a helper. Then the ruleset's function name is made to point to the extension's like-named function. The extension's function calls the helper, to maintain all the ruleset functionality, without a bunch of duplicate code, and executes the augmentation code it contains:

function onInit()
 -- get these in place before initializing super
  super.EIC1_synchData_helper = super.synchData;
  super.synchData = EIC1_synchData;
  super.onInit();
end

function EIC1_synchData()
  -- some augmentation might happen here, then call the helper
  super.EIC1_synchData_helper();
  -- and or some augmentation might happen here (too)  
end

(Continued in next post)

[Minty23185Fresh]

(Continued from above)

Code Layering
A diagrammatic illustration of LUA code layering is presented here.

Agent atop Ruleset
When the Agent extension is loaded, because of the way it is defined and its intent, one can think of the ruleset layer as being destroyed. One might be able to access it with super.super but one should refrain from doing so. It serves no purpose.

============================================================================
Agent Layer - ERA extension
I am self, ruleset is super
(but for all intents and purposes ruleset layer is void)

============================================================================
Ruleset Layer
I am self, I have no super

============================================================================

Extension atop Agent
The extension should load after the Agent.

============================================================================
Extension Layer
I am self, Agent is super
(All "super" functionality should reside in the Agent.)

============================================================================
Agent Layer - ERA extension
I am self, ruleset is super (but moot).  All "super" functionality resides here.

============================================================================
Ruleset Layer
I am self, I have no super

============================================================================

Second extension atop that
This the second extension should load after the Agent. Loadorder of the two extensions, one before the other, might not be important because of the predication of this methodology, but if it is found to be, it is available. If one extension uses the methodology, but the other doesn't, the one using the methodology should load after the one that does not.

============================================================================
Second Extension Layer
I am self, First Extension is super, if First does not have the desired super
functionality, execution is passed to its super (which is the Agent)
(Almost all "super" functionality should reside in the Agent.)

============================================================================
First Extension Layer
I am self, Agent is super
(All "super" functionality should reside in the Agent.)

============================================================================
Agent Layer - ERA extension
I am self, ruleset is super (but moot).  All "super" functionality resides here.

============================================================================
Ruleset Layer
I am self, I have no super.

============================================================================

Example Project
This methodology was developed and tested over several months as I implemented the Field Filters for All Libraries Extension. That extension contains the Agent, a base Field Field Filters extension, a Dragline extension and an Advanced Filters extension all compressed into one extension file. I developed them that way, a piece at a time, adding one layer upon another to show proof of concept. The multi-extension concept is maintained via a directory structure instead of delivering four extensions to the user.

For the purposes of this post a small project is presented based on work discussed in the button_iconcycler thread. In the attached .zip file the ruleset files are archived so that they're available after the next release of Fantasy Grounds. My original work is included (from the thread) and then a deconstructed, working project is presented that employs the methodology presented in this article.

Folders in Project.zip
1) rulesets (v3.3.4) - the two ruleset files of the current release for comparison purposes
2) extension_ICC - my final work presented in button_iconcycler thread mentioned above. Presented for comparison purposes, the wholesale copy and modify approach.
3) extensions - contains the Agent in subfolder CBA_ERA and the working extension in subfolder EIC1.
At some point I may add some more functionality to the iconcycler button to show multiple extension use of the agent.

[Bidmaron]

Minty, this is really good head work. Thanks for the framework, and I nominate you for "treasure to the community!"

[Steeleyes]

Minty this is a very timely post, I had been looking for input on this very subject and your post helps me tidy up my lua syntax (e.g by avoiding mess like super['syncData']) and gives advice on XML best practice.

LUA Functions
I'm not sure you want to call "super.onInit()" inside your new onInit function. I suspect this is likely to result in bad things like handlers being registered multiple times. I was expecting that super.onInit might have already been called before my extension's onInit and so it probably is best not to call it again? [Edit: I guess you are fine because you aim for a low loadorder and find a way to skip the original module loading in the xml - I've been using a high loadorder to avoid a duplicate of the super class]

LUA scope
Having tried this I was also a little surprised by the number of references to my super class that I needed. I had been expecting 'super.' to be implicit once I'd injected my new functions into my super class. I guess I'm missing some lua knowledge here. Would you put other supporting functions into the super class too or leave them out?

function onInit()
    PowerManager.UUP_performPCPowerAction_orig = PowerManager.performPCPowerAction;
    PowerManager.performPCPowerAction = performPCPowerAction;
    PowerManager.UUP_funThings = funThings;
end

function funThings(nodeAction)
end

function performPCPowerAction(draginfo, nodeAction, sSubRoll)
    PowerManager.UUP_funThings(nodeAction);
    PowerManager.UUP_performPCPowerAction_orig(draginfo, nodeAction, sSubRoll);
end

[Minty23185Fresh]

I'm glad someone is getting some use out of this.

Calling super.onInit()
In my case I had to do it. I put Debug.console("arrived"); in both the super and the self onInits. And ran the project. The super.onInit() never executed. I had to have much of the super functionality in place and initialized, so it was imperative that I called it. Your project may differ. I have written five or six extensions and many of them do not need the super.onInit call. In my Field Filters for All Libraries extension I also had to time the call exactly. I had to initialize some stuff in self.onInit, call the super.onInit then continue with more initialization in self.

More functions. Scope:
Your question here causes me to believe I have failed to express the key point for the reason I developed and expounded on the proposed methodology: Personally I abhor duplicate code. It is a real pain to maintain. Bloats your project. And in my opinion only encourages laziness.
That said, if I understood your question correctly - should more ruleset code be moved to the extension? I would emphatically say No. There is rarely a good reason to do so. One should supplement or augment the ruleset code, calling it to add functionally that is already in place. If that is impossible then replace the ruleset code, which might involve some copying.

Take all of this as soap boxing!!! Of course you will and should do as you please, the rant above is simply my opinion. It took me quite a few months work to develop, thoroughly test and then present the methodology. So I know it works but it might not be useful for everyone. There's overhead involved and a bit of difficulty to implement. For a tiny project it might not be worthwhile.

I truly appreciate your interest and your questions. I hope I've answered them. If not please ask again, sometimes I'm a little obtuse.

[Moon Wizard]

If you override a function at a higher level, it completely obscures any lower level implementations of that function. If you need to call the lower level version (such as to make sure that any lower level functionality works as expected), then you use the super.<function_name> call.

Normally, in Lua, if you replace a function, it is permanently replaced. The super/self hierarchy is written into the FG engine to provide greater flexibility and inheritance options.

Regards,
JPG

[Minty23185Fresh]

Thank you Moon Wizard for a concise answer I circuitously danced around.

Much appreciated.

[Steeleyes]

Thanks folks for the responses. Minty I think your goals are very clearly expressed and admirable. Forgive my poor attempt at paraphrasing... "as far as possible - avoid duplication, be robust to ruleset releases and allow multiple extensions to build on each other without clashes". That's music to my ears.

The super.onInit() in your example modules are required and correct, you don't need me to tell you that because you tested them, but apologies for my ill-founded concern (I had a slightly different situation in mind that has less clean inheritance but that's a poor excuse).

Do you think an extension that augments a high level (global) script may need a slightly different template?
My interest in this topic has practical application in a small extension I'm playing with. It works as it is but I was looking for the cleanest way to insert a one line hook into PowerManager and I don't think it's coincidence that the lua part of the extension is looking more and more like your extension/agent code above.

I'll go off study your examples again, run some experiments with 'super' and think about it some more.

[Minty23185Fresh]

Do you think an extension that augments a high level (global) script may need a slightly different template?

I did this in a previous post, it's mentioned in the first sentence of the "function" section near the top of the first post of this thread. This methodology springboards off that work, so a few of the ideas in this thread aren't new, just expanded upon.

The idea for the development of this methodology came from another thread where Moon Wizard and I discussed the difficulties of working around code layering. I (respectfully) took the Wizard's statements as a challenge to develop a generic way of using code layering to my advantage - the proverbial thrown gauntlet, so to speak. His help to educate me about code layering was intrumental.

@Steeleyes: Thanks for your kind words and your interest in this thread.

[Steeleyes]

Minty yes that's exactly right, how to cleanly override global functions is covered in the other thread (which I hadn't spotted). If anything the case in this thread (overriding XML window classes) is cleaner and I look forward to using it the next time it applies. Hopefully more extensions will use these templates and we can sticky your templates/explanations or combine the notes into other how-to threads.

I've done a minor tweak to my latest tiny extension and so now at least there is less chance of a clash.