[MOD] - Eye of the Atlantis - version 3j

[Drakkan]

just small update. I had some time recently, almost all remaining corridors briefly added, now to fill it with content, which is still quite a lots of work.

Unfortunately I am hitting with low memory problem even in the editor and it is quite a pain.
I have removed almost all sx_city content and replaced it with urban asset, I removed whole pirate ship and replaced it with ordinary dungeon and I simplified some other areas as well, but it still not helped. Dungeon file has 150 mb now, which is almost 100 mb cut from the original size...
I will try do some more polishing, but seems number of used objects / content overall is just too huge for this game to handle it, which is really sad.

[Jhaelen]

I'm sorry to hear this. I'm exactly in the same case (but my dungeon is smaller in size, only 50 mb but 43 levels). I removed the winter tileset (undoing hours of works, I'm happy), and now I'm staying between 1400 and 1500 mb RAM used in the game. I'm still on the edge, but it's a small advance in the right direction.

Yesterday I found a custom asset that didn't have the "minimalSaveState" set to true. It's a fire pit. I fixed this problem and it seems that the RAM is more stable, because before this modification each time I saved the game the RAM bumped up 30 to 40 mb.

[Drakkan]

I'm sorry to hear this. I'm exactly in the same case (but my dungeon is smaller in size, only 50 mb but 43 levels). I removed the winter tileset (undoing hours of works, I'm happy), and now I'm staying between 1400 and 1500 mb RAM used in the game. I'm still on the edge, but it's a small advance in the right direction.

Yesterday I found a custom asset that didn't have the "minimalSaveState" set to true. It's a fire pit. I fixed this problem and it seems that the RAM is more stable, because before this modification each time I saved the game the RAM bumped up 30 to 40 mb.

how is possible to check, how much memory is used ? somewhere in windows processes I guess ?

[Jhaelen]

Yes I use Windows 8.1, so when I press Windows + X, I have an option "Gestionnaire des tâches" in french, I think maybe "Tasks manager" in english ?

In this I can see the amount of RAM used by each process. It's there that I understand that when Grimrock2.exe is hitting the 1600 mb RAM used, the "out of memory" exception isn't far away...

[Drakkan]

Yes I use Windows 8.1, so when I press Windows + X, I have an option "Gestionnaire des tâches" in french, I think maybe "Tasks manager" in english ?

In this I can see the amount of RAM used by each process. It's there that I understand that when Grimrock2.exe is hitting the 1600 mb RAM used, the "out of memory" exception isn't far away...

I am balancing on about 1432 in some areas but when I enter some other areas, it sometimes start to increase...and when it hits about 1480, it crash...
And I cant properly preview-play in the editor as well (it crash after each second preview). Although I am editing in this state longer than few months, it starting to REALLY piss me off and unfortunately I will be forced to make the whole ending part short-cutted and simplified...

[minmay]

Well, what's the total size of the textures used? How many objects are in the mod, and how many components? For saving, how many objects with minimalSaveState are there, and how many components on objects without minimalSaveState are there?

edit: here's a basic object/component counter you can paste into the console:

Code: Select all

do
 local objs = 0
 local comps = 0
 for l=1,Dungeon.getMaxLevels() do
 for e in Dungeon.getMap(l):allEntities() do
 objs = objs+1
 for _,c in e:componentIterator() do
 comps = comps+1
 end
 end
 end
 print(string.format("objects: %d, components: %d",objs,comps))
 end

Run that in your mod. For Isle of Nex, the number of objects starts at 58920, and the number of components at 121962. The difference between that and your mod may not be as big as you think, in which case the real problem is excessive textures/models and/or a lack of use of minimalSaveState.

(This counter will miss objects that are not on the map such as items inside containers but that's not really big enough to care about.)

[Drakkan]

Well, what's the total size of the textures used? How many objects are in the mod, and how many components? For saving, how many objects with minimalSaveState are there, and how many components on objects without minimalSaveState are there?

edit: here's a basic object/component counter you can paste into the console:

Code: Select all

do
 local objs = 0
 local comps = 0
 for l=1,Dungeon.getMaxLevels() do
 for e in Dungeon.getMap(l):allEntities() do
 objs = objs+1
 for _,c in e:componentIterator() do
 comps = comps+1
 end
 end
 end
 print(string.format("objects: %d, components: %d",objs,comps))
 end

Run that in your mod. For Isle of Nex, the number of objects starts at 58920, and the number of components at 121962. The difference between that and your mod may not be as big as you think, in which case the real problem is excessive textures/models and/or a lack of use of minimalSaveState.

(This counter will miss objects that are not on the map such as items inside containers but that's not really big enough to care about.)

objects: 103651 components: 214713
I am twice the size, which is quite a huge difference...

[minmay]

objects: 103651 components: 214713
I am twice the size, which is quite a huge difference...

Then you ought to have about twice the memory usage from saving. (The relationship isn't quite linear but it's close.) If the memory balloon is much bigger than that, you almost certainly have one or more very common objects that's missing minimalSaveState. If you are interested in solving this problem, perhaps you should share your object definitions?

Here are the most used objects in the first version of the mod I downloaded:

Code: Select all

forest_elevation_edge	11690
dungeon_pillar	8121
dungeon_wall_01	5712
swamp_grass_01	3369
castle_pillar_01	3033
dungeon_floor_dirt_01	2618
mine_floor_01	2382
mine_elevation_edge	2150
dungeon_wall_02	1955
mine_ceiling_01	1318
mine_ceiling_02	1272
castle_secret_door	1224
castle_floor_01	1110
castle_wall_01	1076
sx_winter_elevation_edge	1072
floor_trigger	864
castle_wall_02	848
castle_wall_03	757
invisible_rocky_wall	736
catacomb_ceiling	735
mine_floor_sandpile	709
mine_wall_01	701
forest_oak_cluster	651
dungeon_floor_01	649
dungeon_ceiling_0000	619
sx_town_floor_02	494
sx_town_floor_01	487
sx_town_floor_03	481
sx_town_wall_indoor_01	472
sx_town_floor_inside	452
castle_ceiling_0000	435
sx_winter_elevation_ledge	434
sx_town_ceiling_indoor_02	399
forest_ruins_fallen_bricks	382
castle_pillar_candle_holder	379
mine_chasm_edge	368
dungeon_ceiling_1010	353
script_entity	329
sx_winter_ice_ground_01	301
mine_chasm	283
fire_emitter	277
mine_moss_wall_01	276
forest_seaweed_floor_02	273
beach_rock_2x1	268
dungeon_ceiling_0101	267
castle_wall_outside_01	263
sx_winter_elevation_edge_ice	258
forest_seaweed_floor_01	242
dungeon_ceiling_1000	227
castle_wall_outside_tall_01	225
mine_support_pillar_01	221
sx_town_pillar_wood_fullsize	220
forest_heather	211
mine_pillar_01	209
dungeon_ceiling_0010	209
magic_bridge	208
mine_pillar_02	201
sx_winter_ground_01	201
dungeon_ceiling_0100	197
sx_wall_wood_edge_floor	196
forest_elevation_edge_overhang	191
sx_winter_grass_planes_01	190
castle_wall_bookshelf_02	188
dungeon_ceiling_0001	187
sx_town_wall_indoor_old	181
mine_pillar_03	179
mine_moss_ceiling_03	176
sx_town_brick_pillar	176
dungeon_secret_door	174
timer	169
spawner	161
mine_moss_ceiling_02	156
mine_moss_ceiling_01	154
forest_bridge	151
dungeon_ceiling_0011	148
beach_rock_3x1	143
forest_ruins_ground_tile_01	142
dungeon_wall_drain	140
sx_winter_forest_pillar_01	138
dungeon_ceiling_1001	138
sx_forest_fence	136
sx_wood_pillar_old	135
dungeon_ceiling_0110	132
sx_town_housewall_slate_01	132
dungeon_ceiling_1100	131
forest_spruce_01	130
castle_wall_bookshelf_01	130
coin_copper	130
forest_ruins_ground_tile_02	129
sx_town_wall_indoor_old_lev2	128
ctomb_pillar	127
forest_fireflies	127
pushable_block_floor	125
beach_ground_01	123
forest_seaweed_wall_decoration_02	119
castle_bridge_grating	116
mine_ceiling_shaft_edge	115
forest_seaweed_wall_decoration_01	115
blocker	113
forest_grass_01	111
floor_spike_trap	110
rock	108
forest_ground_01	106
mine_moss_pillar_03	105
mine_pillar_crystal	104
sx_autumn_forest_oak_cluster	102
mine_moss_pillar_01	101

You have 1224 instances of castle_secret_door, which does NOT have minimalSaveState. This is a major burden to save! I strongly recommend making a separate version of that object with minimalSaveState, for use in places where you don't need it to open. (I assume you are making big walls out of it.) Or, if it's easier, redefine castle_secret_door with minimalSaveState and make a separate copy of it without minimalSaveState to use where you do need it to open.

Here's the output of the same code for Isle of Nex, btw.

Code: Select all

dungeon_pillar	6898
dungeon_wall_01	3797
dungeon_floor_dirt_01	3226
castle_pillar_01	2219
dungeon_wall_02	1879
tomb_pillar	1545
tomb_ceiling_01	986
mine_floor_01	956
mine_floor_sandpile	940
mine_wall_01	933
catacomb_ceiling	845
castle_floor_01	813
beach_rock_pillar_01	778
swamp_grass_01	661
mine_support_pillar_01	627
dungeon_ceiling_0000	618
forest_elevation_edge	581
forest_pillar_01	543
mine_support_wall_01	514
castle_wall_01	508
tomb_floor_01	505
castle_wall_02	502
forest_oak_cluster	501
forest_heather	500
castle_wall_03	497
tomb_floor_02	490
mine_ceiling_01	460
forest_border_rocks_01	433
floor_trigger	429
mine_ceiling_02	418
tomb_wall_01	407
tomb_wall_02	395
mine_support_beam_01	366
tomb_wall_04	347
tomb_wall_03	345
mine_ceiling_shaft_edge	342
forest_spruce_01	315
grass_planes_01	301
spawner	283
beach_rock_wall_02	279
swamp_oak_cluster	277
forest_ruins_pillar_03	271
forest_wall_02	270
mine_pillar_crystal	259
beach_rock_wall_04	258
forest_hedge_01	257
beach_rock_wall_01	254
forest_wall_01	251
dungeon_ceiling_0101	251
mine_ceiling_shaft	248
dungeon_ceiling_1010	232
cemetery_pillar_01	229
mine_rockpile_01	229
dungeon_ceiling_0010	220
mine_pillar_03	211
teleporter	210
dungeon_ceiling_1000	209
dungeon_floor_01	209
mine_pillar_01	208
dungeon_ceiling_0001	204
mine_chasm_edge	192
mine_pillar_02	190
dungeon_ceiling_0100	190
blocker	189
mine_moss_wall_01	184
forest_oak	182
forest_plant_cluster_01	179
mine_chasm	179
mine_elevation_edge	170
floor_spike_trap	165
forest_willow	165
dungeon_wall_height_difference	164
swamp_heather	159
beach_rock_2x1	157
cemetery_wall_01	155
castle_ceiling_0000	154
mine_support_ceiling_01	151
forest_spruce_sapling_01	146
dungeon_ceiling_1100	142
forest_ruins_wall_01	142
castle_wall_outside_01	138
dungeon_ceiling_0011	138
forest_ruins_ceiling	138
magic_bridge	133
forest_ruins_pillar_01	130
dungeon_ceiling_1001	129
torch	128
swamp_oak	127
dungeon_ceiling_0110	127
dungeon_wall_broken_01	123
forest_seaweed_floor_02	122
dungeon_wall_grating	119
ladder	118
script_entity	116
dungeon_wall_drain	115
catacomb_alcove_01	111
forest_spruce_sapling_02	108
tomb_wall_ornament	107
forest_old_oak	107
forest_bridge	103
castle_bridge_grating	102
forest_oak_trunk	101
forest_ruins_ground_tile_03	100
forest_blocker_stone	100
catacomb_alcove_02	100

[bongobeat]

So if you have considerable number of object without minimalSaveState = true, that isn't good?
How do you extract that list? I'm interested to do the same for my mod.

I got 116633 objects and 253863 components

[minmay]

So if you have considerable number of object without minimalSaveState = true, that isn't good?

Yes, if an object lacks minimalSaveState then its entire state is serialized, including its world position and rotation, the state of all of its components, etc. This is not expensive for a single object, but with thousands of objects it gets significant. With minimalSaveState only the name, id, level, x, y, facing, and elevation of the object are saved; it is simply respawned at that location when the game is reloaded. This is much less expensive to serialize but obviously will not preserve any of the object's state except for its id and map location. In short, if your object is going to change in any way other than being destroyed or moved with GameObject:setPosition(), then it should NOT have minimalSaveState. Enabling or disabling a component in the editor is a change!
Otherwise, it SHOULD have minimalSaveState. Using minimalSaveState on objects that are used tons of times, like trees, floor/wall/ceiling pieces, etc. is essential, and missing even one can be a major problem.

How do you extract that list? I'm interested to do the same for my mod.

I wrote this extremely quick-and-dirty routine and pasted it into the console:

Code: Select all

    do
      local objs = 0
      local comps = 0
      local objList = {}
      for l=1,Dungeon.getMaxLevels() do
       for e in Dungeon.getMap(l):allEntities() do
        objs = objs+1
        objList[e.name] = objList[e.name] and objList[e.name]+1 or 1
         for _,c in e:componentIterator() do
          comps = comps+1
         end
       end
      end
      local sortedobjList = {}
      for name,count in pairs(objList) do
       table.insert(sortedobjList,{n=name,c=count})
      end
      table.sort(sortedobjList,function(a,b) return a.c > b.c end)
      for _,t in ipairs(sortedobjList) do
       print(t.n,t.c)
      end
      print(string.format("objects: %d, components: %d",objs,comps))
     end

Again, this does not bother to look inside containers (though it is easy to add that if you want it). It will print every object name in the world followed by its count, in descending order. Obviously this will greatly exceed the number of lines in the in-game console, so you should run the game from the terminal ("Command Prompt" on Windows) to view the full output; everything printed to the in-game console in Grimrock is also printed to the terminal.