Implementing AES (Rijndael) into Torque?
by Stefan Lundmark · in Torque Game Engine · 04/13/2006 (12:09 pm) · 19 replies
Hello,
I have been trying to implement AES (Rijndael) into Torque, for a few days.
A few points before we begin:
1. Rijndael, is a block cipher adopted as an encryption standard by the US government. It is expected to be used worldwide and analysed extensively, as was the case with its predecessor, the Data Encryption Standard (DES).
2. I am using Szymon Stefanek's C++ class.
3. I have implemented this class into a regular Console Application, without the problems described in this thread.
I got three ConsoleFunctions, only one of them produces correct results, see below.
The following code does not always produce the same results. In my tests the rate of failure is ~15%. This is why I post the snippit here to see if anyone of you skilled programmers can help solve this for everyone's benefit.
The following code DOES always produce the same results.
Szymon Stefanek's C++ Class together with the above code, can be found here.
Any assistance is greatly appreciated!
Edit: Fixed link.
Sincerely,
Stefan Lundmark
I have been trying to implement AES (Rijndael) into Torque, for a few days.
A few points before we begin:
1. Rijndael, is a block cipher adopted as an encryption standard by the US government. It is expected to be used worldwide and analysed extensively, as was the case with its predecessor, the Data Encryption Standard (DES).
2. I am using Szymon Stefanek's C++ class.
3. I have implemented this class into a regular Console Application, without the problems described in this thread.
I got three ConsoleFunctions, only one of them produces correct results, see below.
The following code does not always produce the same results. In my tests the rate of failure is ~15%. This is why I post the snippit here to see if anyone of you skilled programmers can help solve this for everyone's benefit.
ConsoleFunction( encrypt, void, 3, 3, "")
{
// NOTES:
// argv[1] is the key.
// argv[2] is the string.
// Our Rijndael object.
Rijndael AES;
// We need to set the size based on how long the string is.
// We add another 16 because padding might happen in padEncrypt and we want to make sure
// we have enough memory allocated for that.
int size = (dStrlen(argv[2]) + 16);
// Allocate some memory for the output buffer.
char *output = new char[size + 1];
// Null the string.
dMemset(output, 0, (size + 1));
// Initialize Rijndael.
AES.init(Rijndael::ECB, Rijndael::Encrypt, ((unsigned char*)argv[1]), Rijndael::Key32Bytes);
// Encrypt the string. padEncrypt takes care of the padding.
AES.padEncrypt((unsigned char*)argv[2], size, (unsigned char*)output);
// I am using a console callback instead of returning the values, because
// I found no way to delete output[] after I have returned it, thus the hack below.
Con::executef(3, "encryptCallback", output, Con::getIntArg(size));
// Cleanup.
delete[] output;
}ConsoleFunction( decrypt, void, 4, 4, "")
{
Rijndael AES;
// This is ugly. I suspected that you had to supply the correct length of the original string to decrypt
// but boy was I wrong.
int size = dAtoi(argv[3]);
char *output = new char[size + 1];
dMemset(output, 0, (size + 1));
AES.init(Rijndael::ECB, Rijndael::Decrypt, ((unsigned char*)argv[1]), Rijndael::Key32Bytes);
AES.padDecrypt((unsigned char*)argv[2], size, (unsigned char*)output);
Con::executef(2, "decryptCallback", output);
delete[] output;
}The following code DOES always produce the same results.
ConsoleFunction( test, void, 3, 3, "")
{
Rijndael AES;
int size = (dStrlen(argv[2]) + 16);
char *output = new char[size + 1];
char *temp = new char[size + 1];
dMemset(output, 0, (size + 1));
dMemset(temp, 0, (size + 1));
AES.init(Rijndael::ECB, Rijndael::Encrypt, ((unsigned char*)argv[1]), Rijndael::Key32Bytes);
AES.padEncrypt((unsigned char*)argv[2], size, (unsigned char*)output);
Con::executef(2, "encryptCallback", output);
AES.init(Rijndael::ECB, Rijndael::Decrypt, ((unsigned char*)argv[1]), Rijndael::Key32Bytes);
AES.padDecrypt((unsigned char*)output, size, (unsigned char*)temp);
Con::executef(2, "decryptCallback", temp);
delete[] output;
delete[] temp;
}Szymon Stefanek's C++ Class together with the above code, can be found here.
Any assistance is greatly appreciated!
Edit: Fixed link.
Sincerely,
Stefan Lundmark
About the author
#2
I added a static char, and stored the input. Then I compared it to the decryption output.
This is the modification (apart from the static buffer addition) in the end of decrypt();
And this is the content log when trying the functions again:
[quote]
Match.
Match.
Match.
Match.
9F91502855A2354370589C9E706F5D87
9F91502855A23543?]
04/13/2006 (2:00 pm)
Thanks Orion.I added a static char, and stored the input. Then I compared it to the decryption output.
This is the modification (apart from the static buffer addition) in the end of decrypt();
int cmp = dStrcmp (output, buffer);
if (cmp == 0)
{
Con::printf("Match.");
Con::executef(1, "testEncrypt");
}
else
{
Con::printf(buffer);
Con::printf(output);
Con::printf("Mismatch.");
}And this is the content log when trying the functions again:
[quote]
Match.
Match.
Match.
Match.
9F91502855A2354370589C9E706F5D87
9F91502855A23543?]
#3
- i was thinking of comparing the string coming *in* to decrypt with the string which was sent out from encrypt.
eg
at the end of encrypt:
dStrncpy(buffer, output); // copy output into buffer
at the begining of decrypt:
dStrcmp(argv[2], buffer); // compare input to buffer
04/13/2006 (2:08 pm)
Quote:This is the modification (apart from the static buffer addition) in the end of decrypt();
- i was thinking of comparing the string coming *in* to decrypt with the string which was sent out from encrypt.
eg
at the end of encrypt:
dStrncpy(buffer, output); // copy output into buffer
at the begining of decrypt:
dStrcmp(argv[2], buffer); // compare input to buffer
#4
They look the same even when decryption fails, though.
04/13/2006 (2:16 pm)
Doh.They look the same even when decryption fails, though.
Quote:
*r?*0lpkVW'?!IzB4?lC.De{2pi¶~o'q
*r?*0lpkVW'?!IzB4?lC.De{2pi¶~o'q
Original string = Decrypted string.
uB?\|-Lu¦r???h(HU8DARSn?^3'Z?T#w[?'q
uB?\|-Lu¦r???h(HU8DARSn?^3'Z?T#w[?'q
Original string = Decrypted string.
O?^ 4??.a/}D[VTiew8N'q
O?^ 4??.a/}D[VTiew8N'q
Original string = Decrypted string.
l'M-3-
l'M-3-
Original string != Decrypted string.
Failed.
#5
04/13/2006 (2:19 pm)
Given some original input string, does that string always either pass or fail ?
#6
If I use the key from the failed test, but with a different string - it also works.
If I use BOTH the failing key, and the string.. and put them in manually in the code, bypassing the random MD5 generator, it will fail.
If I use BOTH the failing key, and the string.. and try them in the non-TGE console application - they do not fail, but pass.
Quite odd?
04/13/2006 (2:49 pm)
Nope. If I use the same input string but with a different key - it works.If I use the key from the failed test, but with a different string - it also works.
If I use BOTH the failing key, and the string.. and put them in manually in the code, bypassing the random MD5 generator, it will fail.
If I use BOTH the failing key, and the string.. and try them in the non-TGE console application - they do not fail, but pass.
Quite odd?
#7
04/13/2006 (10:39 pm)
You might be having an encoding problem. I always base64 encode strings before encrypting them to prevent problems that can arise.
#8
04/14/2006 (4:33 am)
I must have missed something here... why use AES with Torque?
#9
Would this even be needed, considering that I'm using MD5 strings as input?
Anyway, I implemented base64 and used it before passing any strings to the cipher. I do still get the same results with blank output from padEncrypt.
It's been 5 days and I'm pretty confident in that I won't be able to solve this.
Thanks everyone for your help, tons appreciated!
04/14/2006 (11:52 am)
Thank you Philip.Would this even be needed, considering that I'm using MD5 strings as input?
Anyway, I implemented base64 and used it before passing any strings to the cipher. I do still get the same results with blank output from padEncrypt.
It's been 5 days and I'm pretty confident in that I won't be able to solve this.
Thanks everyone for your help, tons appreciated!
#10
As far as, is base64 encoding necessary I can't give you a solid answer, but my experience has been to have problems without it. The thing to remember is that an encrypted string my cause unexpected escape sequences, so it is better not to convert it to a string and store it as a byte array.
After looking closer at you problems, it definitely looks like a padding problem. My guess is something weird is going on with the strings as they cross the different boundaries in torque. It may be due to the "everything is a string" approach torquescript has.
Don't give up... encryption can be a bear, trying breaking the steps down and testing the results.
04/14/2006 (12:13 pm)
You might want to look at http://www.opentnl.org/ source code and see how they deal with encryption. It uses "AES symmetric encryption with SHA-256 message authentication."As far as, is base64 encoding necessary I can't give you a solid answer, but my experience has been to have problems without it. The thing to remember is that an encrypted string my cause unexpected escape sequences, so it is better not to convert it to a string and store it as a byte array.
After looking closer at you problems, it definitely looks like a padding problem. My guess is something weird is going on with the strings as they cross the different boundaries in torque. It may be due to the "everything is a string" approach torquescript has.
Don't give up... encryption can be a bear, trying breaking the steps down and testing the results.
#11
OpenTNL uses TomCryptLib, which is not available from Tom's site anymore, and the version TNL uses is too advanced for me to understand.
I'll try again without the pad addition and only use blocks of 16 bytes.
Thanks alot.
Edit:
Phil, I can tell you as much as it's not the padding.
I now tried to use blockEncrypt/Decrypt and got to the point where I had the same results as before with the padfunctions.
So I made everything static to 16 bytes, ie.. all the memory allocations at 17, the input string was always 16 characters long.. and I still do get good results ~85% of the time only.
It's most likely, as you and Orion have been saying, TorqueScript mangling the strings and removing/adding characters at the end of the string.
Thanks again for your help.
04/14/2006 (1:00 pm)
Quote:
You might want to look at http://www.opentnl.org/ source code and see how they deal with encryption. It uses "AES symmetric encryption with SHA-256 message authentication."
OpenTNL uses TomCryptLib, which is not available from Tom's site anymore, and the version TNL uses is too advanced for me to understand.
Quote:
After looking closer at you problems, it definitely looks like a padding problem. My guess is something weird is going on with the strings as they cross the different boundaries in torque. It may be due to the "everything is a string" approach torquescript has.
I'll try again without the pad addition and only use blocks of 16 bytes.
Quote:
Don't give up... encryption can be a bear, trying breaking the steps down and testing the results.
Thanks alot.
Edit:
Phil, I can tell you as much as it's not the padding.
I now tried to use blockEncrypt/Decrypt and got to the point where I had the same results as before with the padfunctions.
So I made everything static to 16 bytes, ie.. all the memory allocations at 17, the input string was always 16 characters long.. and I still do get good results ~85% of the time only.
It's most likely, as you and Orion have been saying, TorqueScript mangling the strings and removing/adding characters at the end of the string.
Thanks again for your help.
#12
One thing, that just dawned on me just now, is that your mistmatch strings "looks" like they match and that makes me wonder if it is a size issue. dstrcmp might first check size before anything else? Or it is a trunkcation issue and a buffer is too small?
Oh yeah... and keeping things base64 encoded until they are on the torquescript side of things might help too.
04/14/2006 (2:03 pm)
Sounds like you are on the right path. One thing, that just dawned on me just now, is that your mistmatch strings "looks" like they match and that makes me wonder if it is a size issue. dstrcmp might first check size before anything else? Or it is a trunkcation issue and a buffer is too small?
Oh yeah... and keeping things base64 encoded until they are on the torquescript side of things might help too.
#13
The mismatch string does not always come out like the one you're probably thinking of from the above posts, sometimes it's just jibberish..
Anyway, it was a good learning experience I guess. I noticed how AES was implemented in a resource for T2D, so it should work in TGE - but that's without making it go trough TorqueScript (it's just a stream, as far as I can see) so I guess my point is moot.
04/14/2006 (3:46 pm)
I've tried with all different kinds of static buffers, up to 4096 bytes large. Should definatly not be that big :)The mismatch string does not always come out like the one you're probably thinking of from the above posts, sometimes it's just jibberish..
Anyway, it was a good learning experience I guess. I noticed how AES was implemented in a resource for T2D, so it should work in TGE - but that's without making it go trough TorqueScript (it's just a stream, as far as I can see) so I guess my point is moot.
#14
To quote from Advanced 3D Game Progrmming All in one:
expandEscape(text) - Escapes all of the escape characters in text. For example, \n becomes \\n.
Might want to try it just to see if it makes a difference.
04/15/2006 (10:29 am)
So I came across a function that probably would solve your problem if it is the escape sequence thing:To quote from Advanced 3D Game Progrmming All in one:
expandEscape(text) - Escapes all of the escape characters in text. For example, \n becomes \\n.
Might want to try it just to see if it makes a difference.
#15
But as my tests involved the normal block cipher (without the zero-padding) this shouldn't matter as the strings I used always were 16 bytes long, and never contained any spaces. (Which means, it's one block)
Of course there's still the case with the encrypted string which might contain spaces in the end of the string, and that it might get crushed while going trough TorqueScript.
Sadly, I've had a computer crash today with some W32/Sanit virus, and I cannot touch my source before it's fixed :/
Thanks again Philip, you've been tons of help and I really do appreciate it.
04/15/2006 (1:14 pm)
From what I understand, encrypting a string with padding, if the string is a multiple of 16 bytes, that would generate an additional 16 bytes of padding to the string.But as my tests involved the normal block cipher (without the zero-padding) this shouldn't matter as the strings I used always were 16 bytes long, and never contained any spaces. (Which means, it's one block)
Of course there's still the case with the encrypted string which might contain spaces in the end of the string, and that it might get crushed while going trough TorqueScript.
Sadly, I've had a computer crash today with some W32/Sanit virus, and I cannot touch my source before it's fixed :/
Thanks again Philip, you've been tons of help and I really do appreciate it.
#16
04/15/2006 (6:33 pm)
My pleasure, and good luck with the recovery!
#17
04/17/2006 (11:04 am)
Stefan, I implemented an encryption algorithm in a Director Xtra, and I found that sometimes a byte of cyphertext would be "\0" which, on the scripting side, had the effect of ending the string at that point. I have no idea if that's what happening for you here, but it's worth looking into. If so, a way to get around it would be to have the C function perform the base 64 encoding/decoding, so that in TorqueScript the string itself will never contain a zero byte.
#18
04/17/2006 (11:13 am)
Rob, don't you think using the expandEscape() function would solve that problem?
#19
I implemented Base64 into the above functions, and they were used in the engine side of code. Still, what you're saying is that base64 encoding the CIPHER text could work?
I never thought of that. I just base64'd the original string before encrypting it, and vice versa when decrypting it, but I never really took any measures on the actual ciphered string.
I did notice how encrypted strings sometimes occupied several lines, but it never striked me that it could have been the problem.
I'll take a look at that, I guess.
04/18/2006 (11:10 am)
Hello Rob,I implemented Base64 into the above functions, and they were used in the engine side of code. Still, what you're saying is that base64 encoding the CIPHER text could work?
I never thought of that. I just base64'd the original string before encrypting it, and vice versa when decrypting it, but I never really took any measures on the actual ciphered string.
I did notice how encrypted strings sometimes occupied several lines, but it never striked me that it could have been the problem.
I'll take a look at that, I guess.
Associate Orion Elenzil
Real Life Plus
and then comparing it against when gets passed in at the begining of decrypt,
just to make sure that your string isn't getting munged by torquescript.
(eg, maybe something is escaping/unescaping stuff ?)
by "static buffer" i mean something like this, above the declaration of ConsoleFunction encrypt:
static sMyBuffer[4096];