Query Regarding How Extensions Load

[UrsaTeddy]

Greetings,

I have the following situation - and I would like to see if anyone can explain why it is happening ...

Two extensions (loadorders 105 and 110).

Both have a common code base (included vial <includefile>) which contains strings, scripts and templates.

When I load a single extension, everything works as expected ... no problems ... everything is working ... YAY.

However when I load both extensions, the common code base is not loaded for 105 (and hence it fails to setup/execute) however it does load for 110.

From my own experimentation (using Debugs), if the same <script> is encountered during load, it is ignored (or seems to be ignored) which is perfectly fine.

However 105 should be loading these in first, then 110.

Can anyone enlighten me please? The only other options I have are 1) Create a common extension that must be loaded first or 2) duplicate all my code with different tags etc (not ideal at all).

As an aside, if I include the same string file - it should override the previous one correct?

[Trenloe]

The thing to remember is that the code is not executed as the extensions are loaded, the code is being instantiated in FG's memory and that code will be executed once the ruleset and all extensions have been loaded.

Additionally, how scripts are instantiated depends on where they're defined.

If they are in extension.xml then yes, the last <script> file to be instantiated and will be the script that is used when the FG instance executes the loaded code as these are global script files and the <script> command on its own will override previously configured global script packages of the same name.

If the <script> is within a windowclass or control, then the scripts will be layered - in the order of them being defined. You can access the code in an earlier windowclass/control <script> definition by using the super variable. See script block scope in the Wiki here: https://fantasygroundsunity.atlassian.net/wiki/spaces/FGCP/pages/996644496/Ruleset+-+Scripting#Script-Block-Scope

[UrsaTeddy]

So from what you are saying is the following :

Extension 105 loads ... it loads the common code via <includefile source="100/100.xml" />

In this case it is the very first line in <base> for extension 105.

Extension 110 loads ... it loads the common code via <includefile source="100/100.xml" />

And once again it is the very first line in <base> for extension 110.

in 100.xml there is a series of includefiles (for strings) and scripts (for common libraries and utilities - for example a serialisation library, extensions to common LUA types and so on).

They get loaded and execution of those extensions begins ... now since 110 essentially overrides 105 in the case of this common library, the code execution of the common library will not occur until 110 executes ... have I got that correct?

If that is so then the only solution is a common extension that MUST be loaded first to get all the common code loaded first (which makes the extensions a little kludgey), or I have to include all of the common code in each extension, ensuring the scripts that are loaded use unique names (even though the code is identical).

[Trenloe]

They get loaded and execution of those extensions begins ... now since 110 essentially overrides 105 in the case of this common library, the code execution of the common library will not occur until 110 executes ... have I got that correct?
As mentioned previously, code doesn't get executed until all of the rulesets (usually CoreRPG plus a layered ruleset) and extensions are loaded into memory - with the code overwriting/layering based on the order and type of the code.

So, something like this:

CoreRPG code is loaded into FG's memory, no code is executed.
The layered ruleset code is loaded into FG's memory, no code is executed. Let's assume this is the 5E ruleset for example purposes.
The first extension code is loaded into FG's memory, no code is executed.
The code of subsequent extensions is loaded into FG's memory, no code is executed.

The LUA code and XML definitions can be modified by subsequent steps in the above list. XML could merge or overwrite depending on the layering tags used - some info here: https://fantasygroundsunity.atlassian.net/wiki/spaces/FGCP/pages/996644522/Ruleset+-+Layering LUA code will either overwrite global script packages or layer the code (if embedded in windowclasses or controls).


If that is so then the only solution is a common extension that MUST be loaded first to get all the common code loaded first (which makes the extensions a little kludgey), or I have to include all of the common code in each extension, ensuring the scripts that are loaded use unique names (even though the code is identical).
This is all theoretical and I can't give firm recommendations without knowing the code. There are ways of overriding individual functions in a global script package and scripts in windowclasses/controls are layered and so the lower down layer code can be executed via the super variable. But, if I understand you correctly, if you have a set of frequently used functions then you do essentially have two options for use in multiple extensions: 1) Have a library extension that needs to be loaded to use the extensions (which can cause use/support issues) or include that code in every extension - and maintain it in every extension. You wouldn't need to use different global script package names, just ensure the code is compatible in each extension - with the last extension loaded being the extension whose global script package will be used.

[UrsaTeddy]

I must not have been clear with my previous query.

What I want to know is that if the common code - which is 100% compatible with both extensions individually - is executed with the last extension to load it rather than executing the code (overwritten or otherwise) in the order of extensions.

Therefore 105 loads with common code loading
110 then loads, overwriting all the common code with the common code (since it is including the exact same code)

After loading, execution begins in the order of extension load order (assumed).

Since extension 105 has had its "common code" overwritten by 110 I am understanding that the "common code" is not executed until 110 is executed.

[damned]

code that is overwritten is not executed

[UrsaTeddy]

@damned ... actually in my case it is executed

my common code goes something like this ....

common type 1
common type 2
common type 3
common manager 1

extension 105 loads the common code and sets up options and some other things that are 105 exclusive. On its own it works perfectly.
extension 110 loads the common code and sets up options etc. On its own it works perfectly.

When I attempt to load 105 and 110 in the same campaign (currently development so only those 2 extensions)

extension 105 fails to load because it cannot find common type 1 etc that it needs
extension 110 succeeds because it finds everything it needs has been loaded

so all i have ascertained is that somehow 105 has "lost" its common code because 110 overwrote it and hence cannot execute it.

[damned]

I really dont follow

you say the common code IS executed by extension 105 but doesnt work because its overwritten by 110?
its either executed or not executed

[Moon Wizard]

Overwritten global script code is not run, nor fully replaced windowclass or template scripts.

Scripts used in template/control definitions and scripts in merged windowclass definition are merged together in layers.

Priority order for window classes:
Last defined merge in reverse order.

Priority order for templates/controls:
Window class control definition is top level, then templates are resolved by replacement with scripts stacked in decreasing priority.

Regards,
JPG

[UrsaTeddy]

@damned

Okay I have two extensions 105 and 110.

Both extensions use a common code base.

Individually (only extension loaded) all is 100% okay, extensions work as intended, including connection detection, OOB, etc.

When loaded together, both extensions load and then the execution phase begins.

105 tries to execute code that references the common code and fails ... cannot find commontype1.blah()
110 tries to execute code that references the common code and succeeds ... it can find commontype1.blah()

to test if it succeeded I was using Debug.chat("BLAH IS EXECUTING").

[UrsaTeddy]

@Moon Wizard

So what is my best solution here?

... I want to keep a single common code base for all of my extensions
... I do not want to have to require a user to load a "common" extension ... this is cumbersome IMHO

By your description of the loading system, I could not even put code in saying "if I have previously defined this item then ignore this code" because the code would have been overwritten during the loading phase.

[Moon Wizard]

Make sure your common code is in a separate extension with a lower load order than your extensions that rely on them.

Also, make sure that the scripts are fully defined, and not attempting to rewrite functions in onInit, since that order is not guaranteed.

JPG

[UrsaTeddy]

@Moon Wizard

So handing out my extension (for example on the Forge) would require me to put up two extensions with a warning that the common extension is required for the other extensions.

So my hopes of these extensions being atomic is a dream - unless I replicate the codebase ensuring no overwriting happens (therefore function BLAH() would have to be rewritten as BLAH105 and BLAH110).

I am assuming that I have it correct that string resources are not affected by the overwriting since they are are just named strings.

[Trenloe]

@damned

Okay I have two extensions 105 and 110.

Both extensions use a common code base.

Individually (only extension loaded) all is 100% okay, extensions work as intended, including connection detection, OOB, etc.

When loaded together, both extensions load and then the execution phase begins.

105 tries to execute code that references the common code and fails ... cannot find commontype1.blah()
110 tries to execute code that references the common code and succeeds ... it can find commontype1.blah()

to test if it succeeded I was using Debug.chat("BLAH IS EXECUTING").
How are you defining these commontype1.blah() functions? How are you calling them and from where within the FG file hierarchy? Please provide examples of actual code.

[damned]

I am assuming that I have it correct that string resources are not affected by the overwriting since they are are just named strings.

If you define a string and then you redefine a string it is overwriting the initial definition...

[UrsaTeddy]

@trenloe ...

for example i have a file called string.lua

in that file i have the following sample code ... that extends the lua string type with functions whose contents do not matter



_100_string_initialised = false

function onInit( )
Debug.chat("Setting up 100 STRING",_100_string_initialised)

if not _100_string_initialised
then
Debug.chat("STRING","Actual Initialising")

-- Extend LUA string
string[ 'mid' ] = self.mid
string[ 'left' ] = self.left
string[ 'right' ] = self.right
string[ 'split' ] = self.split
string[ 'trim' ] = self.trim
string[ 'concat' ] = self.concat

-- Convenience alias
string[ 'fget' ] = Interface.getString

_100_string_initialised = true
end
end


in both 105 and 110 there is a <script _100_string_lib file="types/string.lua"> in the extension.xml

loading each extension individually causes no problem ... however 110 essentially overwrites script_100_string_lib which from previous posts indicates that the code will not be accessible by 105 once 110 is loaded.

[Trenloe]

@trenloe ...

for example i have a file called string.lua

in that file i have the following sample code ... that extends the lua string type with functions whose contents do not matter



_100_string_initialised = false

function onInit( )
Debug.chat("Setting up 100 STRING",_100_string_initialised)

if not _100_string_initialised
then
Debug.chat("STRING","Actual Initialising")

-- Extend LUA string
string[ 'mid' ] = self.mid
string[ 'left' ] = self.left
string[ 'right' ] = self.right
string[ 'split' ] = self.split
string[ 'trim' ] = self.trim
string[ 'concat' ] = self.concat

-- Convenience alias
string[ 'fget' ] = Interface.getString

_100_string_initialised = true
end
end


in both 105 and 110 there is a <script _100_string_lib file="types/string.lua"> in the extension.xml

loading each extension individually causes no problem ... however 110 essentially overwrites script_100_string_lib which from previous posts indicates that the code will not be accessible by 105 once 110 is loaded.
How are you calling Global Script Package and from where within the FG file hierarchy? Where doesn't it run correctly?

Please provide more information to help us troubleshoot this issue. So far we've been very theoretical and this is the first post where we've seen some actual code, but we're still not seeing the whole picture of the code hierarchy, where the Global Script Package is being called and where it fails - please provide details of the actual code, where it fails, any error encountered, etc..

[UrsaTeddy]

@Trenloe ...

i am not 100% sure what you want ... there is a common code file string.lua that is loaded in both extensions.

In onInit( ) of the first extension - 105 in this case ... it calls string.concat to create a string.

When it is loaded on its own - the only extension loaded that uses the common code file - it works as expected.

When I load 105 and 110, the onInit() of 105 has an error on the string.concat call - it cannot locate string.concat, however 110 can find it and its string.concat call works.

That is the simplest I can describe it.

[Trenloe]

I'm not looking for descriptions - I'm looking for actual code and actual error messages. It's very easy to misunderstand high level discussions of low level code, without seeing that actual code, actual error messages etc..

If you could provide a couple of skeleton extensions that can be used to recreate/investigate the issue, that would be great. Otherwise we're pretty much shooting in the dark and going back and forwards without full details.

It may be that extending the base LUA string object doesn't work within the way script scope operates within FG - Moon Wizard would have to comment on that. The way FG normally does things like this is to leave the base LUA object alone and extend with a Global Script Package - see the CoreRPG StringManager Global Script Package as an example (scripts\manager_string.lua). I think this is a better way to go in FG, especially within a multi extension extension environment where it could be difficult to track down exactly where base LUA objects are being extended when multiple extensions are being ran. Is it as elegant a programming solution as extending the base LUA string object? No, but I think it's easier to maintain and troubleshoot in a multi extension, fairly uncontrolled (i.e. multi developer) environment.

[UrsaTeddy]

Extending LUA objects is expected in LUA development AFAIK.

However I understand your concern and if I ever get to the publishing stage, I would most likely change the extending of the LUA objects to something else if it was an issue.

I have created an extremely stripped down pair of extensions that show the issue very simply.

Load each one individually you will see output in the Console.

Load them together and the Console will open with an error.

DSC_SWADE_105.ext (https://david.vega.id.au/attachments/fgu/DSC_SWADE_105.ext)

DSC_SWADE_110.ext (https://david.vega.id.au/attachments/fgu/DSC_SWADE_110.ext)

For some reason the URL tags do not load the attachments, however if you copy the URL and paste it in a new window it does.

[SilentRuin]

You can do the brute force way I do it with common code. You make sure its copied everywhere you need it. That way an extension is self contained. Nothing is more irritating and confusing (and likely to cause problems for users) than making something dependent on something else to run when it could run on its own. Especially the load order works (someone can always sneak in between your code and your "common" code).

Now if you plan on always delivering your common code with your other code and have that code validate that it is out there before it blows up trying to do something with a message stating it can't run because extension xyz has to also be loaded and your careful to call the original code your overriding (if possible) - then you can get away with what your doing without confusing things.

Personally I like brute force stupid way I do it and just copy any shared functionality - especially because experience has taught me even shared functionality eventually diverges. In my world anyway. Good luck with whatever you decide to do, choose what YOU are most comfortable with.

[And of course I can imagine no greater nightmare than having one up to date and one out of date - safer to just put in common .lua and include with extension.xml]

[Trenloe]

Extending LUA objects is expected in LUA development AFAIK.

However I understand your concern and if I ever get to the publishing stage, I would most likely change the extending of the LUA objects to something else if it was an issue.

I have created an extremely stripped down pair of extensions that show the issue very simply.

Load each one individually you will see output in the Console.

Load them together and the Console will open with an error.

DSC_SWADE_105.ext (https://david.vega.id.au/attachments/fgu/DSC_SWADE_105.ext)

DSC_SWADE_110.ext (https://david.vega.id.au/attachments/fgu/DSC_SWADE_110.ext)

For some reason the URL tags do not load the attachments, however if you copy the URL and paste it in a new window it does.
Thanks for providing this - it really helps in troubleshooting and being on the same page.

If you change the 105.lua code to use the _100_string Global Script Package, it works fine (running the code from 110), that is: Debug.chat(_100_string.concat('---','one','two','three'))

EDIT: see next post.

[UrsaTeddy]

Sorry, I should have added in another library file instead of just string and table.

It occurs with my own libraries, not just extended LUA types.

In this case any aliases I create within my own libraries fail when loading both extensions.

The updated extensions are in the previous locations ...

In the updated code, if I access inc.constant (which is an alias to inc.add) it fails when both extensions are loaded ... if I access the original method it succeeds.

This is definitely not expected behaviour ...

As Moon Wizard suggested, I think this has to do with the "overwriting" of the common library code when the second extension loads with the "new/old" common code. Thus the onInit() of the original loaded code from 105 is not executed since it was replaced by 110.

[Trenloe]

Follow up... thinking about it, the issue is probably related to the order of overridden script being executed - not related to overriding the LUA string object. Moon Wizard should be able to comment on that.

If all you want to do is override the LUA string object then you can do that in the onInit function of Global Script Packages with different names - you can still use the functions in the same Global Script Package _100_string, that would be overridden by the later loaded extension but the public functions would be accessible, but you'd need to run the onInit code that extends the LUA string object outside of _100_string

[Trenloe]

So, do something like this in a LUA file that won't get overwritten:


function onInit( )
-- Extend LUA string
string[ 'mid' ] = _100_string.mid
string[ 'left' ] = _100_string.left
string[ 'right' ] = _100_string.right
string[ 'split' ] = _100_string.split
string[ 'trim' ] = _100_string.trim
string[ 'concat' ] = _100_string.concat

-- Convenience alias
string[ 'fget' ] = Interface.getString
end

You could even check if the extended function has already been instantiated. For example:

if not string[ 'mid' ] then
string[ 'mid' ] = _100_string.mid;
end

[UrsaTeddy]

Yes, however that would mean that every extension I write would have that specific onInit code written multiple times in non-overwritable files.

One question.

You will note that in my files I have a semaphore indicating if that file has been executed or not ... that semaphore is not set in the global space, so if I renamed the file and loaded it under another name, all the code is executed again and the semaphore is reset to false.

Is there a global space to do this in? Or do I have to make my own system?

[Trenloe]

This all goes back to what I said in my earlier posts - everything is loaded into FGs memory first - with Global Script Packages being overwritten by later loaded files, and windowclass/controls having layered scripts; then once all files have been loaded, the files are executed in order.

In this instance, as the string['mid'] is only setup in the onInit function within the _100_string Global Script Package, and the active file for _100_string is in the 110 extension, the onInit function of _100_string only gets executed when the code in 110 is executed (not when it's loaded into memory, when it's executed later) which is after the not overridden code in 105 is executed. So, making a call to string.mid in 105 results in the error. However a call to _100_string.mid in 105 works, as the public functions of _100_string are loaded into memory and accessible.

[Trenloe]

Yes, however that would mean that every extension I write would have that specific onInit code written multiple times in non-overwritable files.
Yes, or move to using Global Script Packages rather than overriding LUA objects. Then you won't be instantiating overrides in onInit functions, you'll be making calls to the a specific Global Script Pacakge name (e.g. _100_string) which will point to the last Global Script Package of that name loaded into memory.

[UrsaTeddy]

It doesn't matter if I am using LUA objects or not ... as in the newest version I uploaded, it happens in any global object that gets setup in an onInit() ... in my case the inc object I created and any aliases to it which are setup in the onInit().

This strictly has no connection to LUA objects but rather the method in which FGU overwrites scripts - which is perfectly fine and now that I know about it clearly enough, I have found a workaround that gives me what I need - a centralised code base - and handles the overwriting part of loading extensions.

[Trenloe]

It doesn't matter if I am using LUA objects or not ... as in the newest version I uploaded, it happens in any global object that gets setup in an onInit() ... in my case the inc object I created and any aliases to it which are setup in the onInit().

This strictly has no connection to LUA objects but rather the method in which FGU overwrites scripts - which is perfectly fine and now that I know about it clearly enough, I have found a workaround that gives me what I need - a centralised code base - and handles the overwriting part of loading extensions.
Sorry, I didn't spend the time to look into your newer files in detail. I was still carrying on the thought from the previous threads. I was just saying that the specific errors from the earlier thread aren't present if you move to using _100_string.mid instead of trying to use the extended string.mid before it's created.

[UrsaTeddy]

No problems ...

thank you to all for the help and guidance.

[Trenloe]

Yes, however that would mean that every extension I write would have that specific onInit code written multiple times in non-overwritable files.

One question.

You will note that in my files I have a semaphore indicating if that file has been executed or not ... that semaphore is not set in the global space, so if I renamed the file and loaded it under another name, all the code is executed again and the semaphore is reset to false.

Is there a global space to do this in? Or do I have to make my own system?
The _100_string._100_string_initialised boolean variable is available globally.

You could do this in your earlier loaded extensions to trigger the _100_string.onInit function when first needed:


if not _100_string._100_string_initialised then
_100_string.onInit();
end

For example, in your first posted extensions, make the onInit function in 105.lua as follows:


function onInit( )

if not _100_string._100_string_initialised then
_100_string.onInit();
end

Debug.chat("INITIALISING 105")
Debug.chat(string.concat('---','one','two','three'))

end

[UrsaTeddy]

Yes I have come up with something similar as my solution.

I have removed all onInit( ) functions and replaced them with an initialise( ) function.

I then manually call the initialise( ) functions triggered on the extension onInit( ).

This means that overridden code is only executed after all extensions are loaded and hence 105 will find the common library code after 110 has overridden it.

Achieves basically what you describe and works in my end.

[Trenloe]

Yes I have come up with something similar as my solution.

I have removed all onInit( ) functions and replaced them with an initialise( ) function.

I then manually call the initialise( ) functions triggered on the extension onInit( ).

This means that overridden code is only executed after all extensions are loaded and hence 105 will find the common library code after 110 has overridden it.

Achieves basically what you describe and works in my end.
Nice solution! :)

[UrsaTeddy]

I find it a little kludgy, however it lets me keep my common code base which as a developer makes me happy.

Thank you once again for all of your help.

[Trenloe]

Thank you once again for all of your help.
No worries! I learned some new things as part of this thread too. :)

[Moon Wizard]

This is why I defined a separate StringManager script in CoreRPG to avoid this issue of initialization order.

Regards,
JPG

[UrsaTeddy]

@Moon Wizard ...

My original question was answered by your description of the overwrite phase of the extension loading overwriting an onInit() which then was not executed in the space of one extension when another "replaced" the code with the exact same code.

Your StringManager script does not have an onInit() and other rulesets that layer on top do not overwrite your StringManager but rather create another one with another name - so it avoids what happened in my extensions.

I specifically wanted to share the same code base with all of my extensions and because I did not 100% understand the overwriting concept during load/execution I was essentially hampering my own work.

Now that you clearly explained everything, I have a better grasp of this subsystem and was able to work around it and successfully achieve what I wanted/needed.

In the end I had to remove all the onInit() code my common code base and manually initialise everything to solve the problem.

Thank you once again for all the help.