Hi trinsic and welcome to LoG modding.
What your are asking can't be done in the way you are probably hoping for, but it can be done with workarounds and limitations
When you open up the LoG editor you will see that its grid based,
and everything in the game is based around squared grids, the tileset, the movement of the party, the animations of monsters etc etc.
If we take a closer look at the tilesets and how they are done, you will find out that they have 4 basic parts.
you got the ceiling
you also have walls and floors
and then you have pillars
This is what the engine uses to create the basics of a dungeon layout, and when you are drawing the dungeon in the editor these are the models that get stitched together (there are many more models, I'm just talking about the basic ones)
There is nothing stopping you from creating something outside the box, like making a 45° turn,
but remember that everything in the engine is based of 2 axis movement, so that the party or monsters can't make this movement
This is where workarounds and limitations come into play
You can make a circular wall, that is just a decoration in front of the tileset, you can block the party and monsters from moving into that circular wall.
you can use this same method to create circular tunnels, but remember that movement through that tunnel will always have to by zigzag in nature, and therefore spells and projectiles can't be shoot down a circular tunnel.
You would also be taking up a lot of space inside the map just for your curves, every level is limited to the available grids you have per floor.
Regarding the programs, you will need to have a look at the Grimrock Model Toolkit (GMT) and just for tilesets you can use any program you like to create the models themselfs, the GMT can import obj and convert them into the model format the game uses so any program that can export to obj should do, I use Blender myself.
What you are asking is a tall order for sure
But making something thats outside the normal for a game engine is what modding is all about in my books
good luck.
Skuggasveinn.