These are tips for how to make your models as Grimrock-friendly as possible. This is not a modeling tutorial OR a tutorial for translating models into Grimrock. If you do not know a way to do that already, try part 12 of Skuggasveinn's editor tutorials.
The Most Common Mistake: Giving every vertex flat shading
If you don't know what flat shading and smooth shading are, an interactive picture says a thousand words. Here are some more words: when two polygons are connected to each other, if the vertices shared between them have flat shading, they will appear, well, flat. They'll be completely independent of each other with regards to lighting. This results in the viewer seeing a sharp edge between them. If, on the other hand, these vertices use smooth shading, the renderer will interpolate the lighting between the two to make the polygons appear to be part of a smooth, curved surface.
In a vacuum, smooth shading is more computationally expensive than flat shading. However, this is usually not a problem in modern contexts, and is especially not a problem with Grimrock. In fact, I tested it with a 330,000 triangle model and smooth shading is actually less expensive in Grimrock than flat shading, at least on my computers. (I wanted to use a 2 million triangle model but the exporter couldn't handle it.)
Why could that be? Well, I think it has to do with how Grimrock determines whether to use flat or smooth shading in the first place. It's very simple: if two triangles share a vertex, they get smooth shading on that vertex. So smooth shading is the default. If you want flat shading between two adjoining triangles, you have to make them not adjoining anymore. That means the model has more vertices. That means the model file is bigger on the disk and bigger in memory.
The direct implication of this, of course, is that smooth shading should generally be preferred because it's actually better for performance - takes up less disk space, less RAM/video RAM (I think), and actually renders slightly faster. Never mind the fact that smooth shading also generally looks better!
Many vertices will look better using flat shading - sharp edges like on the blade of a sword, for example. And in those cases, go ahead and use it. But there are practically no cases where you want to use flat shading on every vertex, and yet I see more custom Grimrock models with flat shading on every vertex than without!
Note: Remember that texcoords are stored in vertices and not faces, so if two faces appear to be adjoining in your modeling program, but the UV coordinates of their vertices don't match, they will have to be separated at some point during the conversion to .model, and will not be adjoining in Grimrock (and therefore cannot be smooth shaded). (Your UV map is usually contiguous anyway so this almost never matters, but there are a few cases where it can be annoying, like if you want to seamlessly tile a region of a texture instead of the whole texture.)
It's similar with materials, triangles with different materials won't ever be adjoining in the .model file so you will never get smooth shading between them.
In Blender you can switch an entire object between flat and smooth shading by clicking "Smooth" or "Flat" in the "Tools" tab of the transform menu (press t). Then, when you want flat shading, you can separate the appropriate region of triangles from the rest of the mesh by selecting them in edit mode and pressing 'y'. Alternatively, mark the edges as sharp and use the EdgeSplit modifier.
If you want to change those vertices to have smooth shading w.r.t. the rest of the mesh again, you can just select the appropriate vertices, hit Ctrl+v, and select "Remove Doubles". This will merge all the doubled vertices. It's very handy when importing a model with too much flat shading!
The models in the asset packs offer lots of good examples of where to use flat vs. smooth shading. (Some go way overboard with flat shading, like cudgel.model which has every single quad separated or power_gem.model which actually has coplanar triangles separated, but in general it's a good resource.)
So why do I see so many models with flat shading on every triangle, when it greatly inflates the size of the .model file and is actually worse for performance? Generally it's the product of people making and exporting models without thinking about the shading issue at all.
Exporting to .model:
bitcpy's Blender exporter: The exporter will translate Blender's shading to the Grimrock model. This means you should configure every object to use smooth shading, NEVER flat. If the object has flat shading in Blender, every individual triangle will get flat shading, which is not what we want; remember that means every shared vertex becomes multiple vertices resulting in worsening size, so you always want smooth shading on e.g. coplanar triangles. A sphere mesh with flat shading will be more than QUADRUPLE the size of the same mesh with smooth shading.
When you want some vertices to have flat shading, separate them as I described above.
Grimrock Model Toolkit: As far as I can tell, doesn't screw with the vertices at all. Import a model with two triangles connected and you'll export a model with two triangles connected. In other words you always get smooth shading where possible. I might be wrong on this because I haven't used GMT a lot.
GrimFBX: It's not my intention to badmouth GrimFBX. For a long time it was the best (read: only) option for getting custom animations into Grimrock, bar writing your own program. And the bugs are forgivable, being that it's an early unfinished version. That said...no matter what kind of FBX model I feed it, I get one of two results.
1. It separates every triangle. This is what happened to all the Lost Halls of the Drinn models. (I should really release an update fixing that...)
2. It keeps adjoining triangles adjoined, but turns the UV map into a garbled mess and appears to kill the tangents as well. (This seems to happen if the fbx has smoothing information.)
Therefore for models you should use bitcpy's Blender exporter or GMT instead, unless you have somehow found a way to not encounter either of these two issues. The Blender exporter works great for rigged, weight painted models. Feel free to keep using GrimFBX to convert animations, since obviously the above two issues only apply to models. But the Blender exporter can do Grimrock 1 animations too, and you'll need to use it anyway for Grimrock 2 animations as the format is different.
Merge your meshes (usually)
Grimrock .model files can contain multiple mesh objects. For example, door models often contain a "gate" mesh which moves when the DoorComponent opens/closes and is used to compute the door's projectile collider, along with a "frame" mesh which doesn't move.
However, two meshes are slower to render than one mesh (and result in a slightly larger file), even if the total triangle count is the same. This is especially true if the two meshes use the same material(s). You might think the impact of this is vanishingly small if only one of that model is being rendered, and you'd be completely correct to think that! But multiply it by 1000 models (think of outdoor levels) and it becomes significant.
So you should merge your meshes into one except when you specifically need the functionality of "gate", "lod0_mesh", "lod1_mesh" nodes. Or when you're animating the mesh objects (e.g. almost all secret buttons have a button mesh and a static mesh, and the entire button mesh is animated as an object instead of using bones).
All the .model exporters I'm aware of will keep separate mesh objects separate, so merging them is something to do in your modeling program.
What to do when this happens:
The altar on the left is the standard Grimrock 2 altar. The altar on the right is the same model, but with the scale inverted and the normals flipped. It has the exact same geometry, but it will be lit inside-out. The torch behind it is actually illuminating the front of it facing away from the light source, but not the surface.
So, when this happens, it means your model has negative scale. A model with negative scale will never be lit properly in Grimrock, and you need to edit it. (You can invert the scale again using the transformation matrix (getWorldRotation(), setWorldRotation()) but then you still have normals facing the wrong way, and using a double-sided material won't hack you around the problem either because only the front side of double-sided materials gets lit properly).
To fix negative scale you just need to invert the scale again. Doing this is easy: is the current scale of the mesh negative? If so, apply it and flip the normals and you're done. If not, invert the scale, then apply the newly inverted scale, then invert the scale again without applying, flip the normals, and you're done.
Note: If using bitcpy's Blender exporter, the mesh should have negative scale overall in Blender because of the game_matrix hack. You can tell by going into edit mode for the mesh and UV unwrapping, then immediately hitting ctrl+Z to undo so you don't lose your UV map. If the message "Object has negative scale, unwrap will operate on a non-flipped version of the mesh" appears, then your mesh's scale is correct.
If you are not using this exporter, then the opposite is true, and that message means your scale is wrong.
Further note: Normals should still appear to be on the outside in Blender!
Tangents
Now for an easy one. If your model ever uses a material with a normal map, you need to export tangents or else the normal map won't work. (Practically everything does this by default). If your model doesn't use a normal mapped material, then tangents will not be used by Grimrock and you can elect not to export them to save a little bit of space. This isn't a big deal at all, the savings are pretty small; some exporters won't give you the option to skip exporting tangents at all. But if you forget to export tangents on a model that DOES need tangents, it is a big deal and you'll notice quickly ingame.
