I have no idea what happened... and I didn't get any screen shots so I will try to sum this up the best way I can..
I originally had 58 gems, and I went to make a payment on my castle. I guess I must've input 9920 as the Gem payment, but I meant it as the gold payment.
It then stated that I did not have enough gems as payment and on my "personal info" stats I have -107 Gems... which later went to 0 gems.
The gem payment on the castle seems to be finished, although I entered 9920 for gems....
I really need those gems back. o.o