Indeeed... ok!
We will probably need to go back to the earlier levels and find the gems there and then return to where you are stuck.
Your earlier assertion, that this is a bug - is very close. I did not foresee this being an issue and it clearly is possible to end up in your position.
But not to worry, we will find the gems!
Gem locations for Amadacia Ruins
level 3 -blue
1 - chest in north west corner of map - door opened by button on pit near "dont throw rock" wall text
2 - fireball/poison bolt puzzle near middle of the map - found with leather armour - secret opened by button in the north east section of the room
3 - chest behind gate in fireball corridor area - opened by lever near the south corridor of the area
4 - sack behind spider eggs in south east part of the map
level 3 - green
1 - skeleton crypt (through gate) - secret opened by button near middle of the room
2 - in a drainage alcove next to tentacles in flooded dragon statue area
level 4/5 - blue
1 - 2nd chest secret in teleporter puzzle room, opened by button next to first chest secret in the same room
2 - inside chest in the middle of the pit/alcove cavern near middle of the map - access via closed pit - closed by button in very south western section of the cavern/crypt area
3 - in a book alcove (bookcase) in flooded library area - accessed by bashing down some breakable walls
4 - in a chest near Amadacia Forge area - in a room accessed via a gate which can only be opened by a lever that is found in the KruShaul "only fallen may pass" area - accessed by a pit found at the beginnning of the game (north west section of level 3)