Grimrock

Happy 2021 everyone! As you may know I've been working on a 64-bit version of LOG2 recently. For a while I thought I had hit a dead-end, because the 64-bit version of Lua bytecode is not compatible with the 32-bit version. Savegames contain bytecode, which would break compatibility between the 32-bit and 64-bit versions.

I was ready to abandon the 64-bit project when I found an acceptable solution: the 64-bit version should simply ignore any bytecode contained in the savegames. The 32-bit LOG2 stores the bytecode of hooks and script functions to savegames, but this is ultimately unnecessary. Hooks can be copied from arch object definitions and script functions can be loaded from the dungeon file. I haven't yet fully worked out all the details, but I think this should work.

However, there are some (rare) cases where this could break. If scripts treat functions as first-class values, e.g. if they copy/rename/delete functions at runtime (for example to implement state machines), the savegame loader will load the state of those functions as they were when the dungeon was started. I have checked the main dungeon and I've found no cases which would fail.

But, for example, the following script would fail because the variable 'state' would always point to 'state1' after loading a savegame:
As a bonus the new version would also allow patching functions after starting a new game (at least to some extent), because the game would always use the hooks and scripts from the dungeon.dat.

Questions:

1. Are there any mods that rely on the old behavior (serialization of functions)?

2. Can you think of any reasons why this is a bad idea? (It's possible that I've missed some detail)

Damn, I forgot an obvious thing: loading scripts has side-effects, so this won't work. Back to square 1...

I really hope you can make it work

Hoo boy...I'll take a stab at this. Apologies if this is just stuff you've already thought of.

If Isle of Nex save compatibility is enough, I can think of a very gross and hacky, but relatively easy, way of doing it: parse ScriptComponent sources from dungeon.lua for function definitions, run those definitions on their own, and match them to the saved variable names. Your parser "only" needs to find blocks, so it shouldn't be too bad to write.
Since as you say, Isle of Nex never treats them as first-class values and all the functions in your ScriptComponents are defined in their main functions, you can reliably do this, gross as it is...unless of course I've missed something...

If compatibility with most custom dungeons is necessary, I think you could do a similar, but harder and even grosser thing:
1. parse every function in the entire Lua source code of the mod. A function could be defined in init.lua that eventually makes its way into a ScriptComponent.
2. string.dump() all those functions...but with the old version of LuaJIT. This is the step that sucks for you because the only easy way of doing it is to include two versions of LuaJIT. It is only about 350 extra KB, but it still feels dirty.
3. Now you can match the functions to the ones in the saved game by checking if the bytecode is identical, without ever having to actually run or translate any bytecode.

You don't have to worry about closures because those can't be saved in the first place. Although for full compatibility you would need to also track down the ways a custom dungeon can get non-closure function from an external source (e.g. the first return value of SurfaceComponent:contents()) and save it. But I don't think any dungeon ever actually does that.

Yeah, I've went through many variants of similar ideas but help is welcome!

Isle of Nex is easy. I don't even need to parse anything, because I could just compute the hash of bytecode at loadtime and lookup new bytecode from a library of precompiled 64-bit bytecode chunks.

Savegame compatibility with mods is the hairy one as you mentioned. Parsing functions is actually quite brittle and requires an AST-based static code analysis framework. Due to dynamic nature of Lua it's possible to have cases which fail even with such machinery...

The most promising option so far seem to be to write a bytecode patcher. I have mostly solved it, but there are some details with some opcodes that are not clear.

In any case if this goes forward (the chances are 50-50%) I would welcome some help in the form of testing etc. Quite a lot of changes that could break! =D

Ah crap, you're right, I was thinking that parsing for the "function()" and "function nameOfFunction()" syntaxes would be sufficient because the script interface doesn't give access to loadstring() or the like...but ScriptComponent:setSource() lets you produce the same effect as loadstring(), and at least one dungeon, Magic of Grimrock, does exactly that.

About the bytecode patcher, if you're interested... It's mostly simple as the opcodes are mostly same in the old and new versions. There's just a few new opcodes that have been added in the middle of the opcode list, so basically I just need to remap the opcodes.

BUT, there's one nasty detail. In the new 64-bit LuaJIT there's a new mode called "FR2" (two slot frame info). Normally when a function is called the function is in slot A followed by its arguments in slots A+1, A+2, A+3,... But with FR2 there's an empty gap between the function and its arguments. This becomes quite hard to patch, because all slot assignments would have to be considered and it could also potentially break some opcodes that work for some range of slots.

So I've been thinking of another approach: patch each call instruction to jump to stubs added at the end of each bytecode chunk.

For example, the stub for a CALL instruction would work like this:
1. Copy function to a new slot S
2. Copy arguments to slots S+2, S+3, ... (S+1 = gap)
3. Call the original function
4. Copy return values back to original slots
5. Jump back to the main program, to the instruction following the original CALL

There are other call opcodes, CALLT, CALLM and CALLMT, for varargs calls and tailcalls which would require slightly different stubs.

There are also special iterator call opcodes ITERC and ITERN which I haven't dealt with yet.

I'm going to hit the bed now. Later!

ITERC and ITERN are the same in 2.1 as they are in 2.0; they don't have the empty slot. So hopefully those won't be any trouble. (Famous last words...)

Damn, bytecode patching is a dead-end. As usually the devil is in the details... More specifically in multiple return values in this case. I managed to get simple cases working, but calls that pass around multiple return values are basically impossible to handle with this approach. The problem is that it's not possible to know the number of returned values in bytecode level when functions are chained like this:
How disappointing

The author of LuaJIT also confirmed this so the options seem to be either:
A) Break savegame compatibility, or
B) Limit max memory to 2 GB with the 64-bit version (this does not require FR2 mode)

With option B it would be possible to place resources not used by Lua code in memory areas beyond 2 GB (for example textures) but this requires some hacking...

Well, there's a third option: redo register allocations for all bytecode instructions. That requires quite a bit of data flow analysis. This would be the ideal solution, but I'm going to pass on that as this "christmas project" has already got well out of hand.