Definition of long in writeLongString
by Ingo Seidel · in Torque Game Engine · 08/08/2008 (7:46 am) · 17 replies
Hi,
I am currently trying to send arbitrary length strings (in the sense of some hundred characters) between client and server. I am using NetEvents and the writeLongString method of the Stream class. Although the documentation says that writeLongString is "slightly unstable" - whatever that means - I started using it. However, it seems that it is not possible to send long strings with this method. When using bitstreams, WriteLongString calls the method Bitstream::writeBits which throws an exception as soon as the bit count is above a certain count:
In my Torque environment the maxWriteBitNum is set to 12000, causing the writeLongString method to fail for strings that are larger than about 1500 characters.
So, what's the purpose of this maxWriteBitNum? Is it possible to set it to a much higher value? And why isn't any of this mentioned in the API of the writeLongString method?
I am currently trying to send arbitrary length strings (in the sense of some hundred characters) between client and server. I am using NetEvents and the writeLongString method of the Stream class. Although the documentation says that writeLongString is "slightly unstable" - whatever that means - I started using it. However, it seems that it is not possible to send long strings with this method. When using bitstreams, WriteLongString calls the method Bitstream::writeBits which throws an exception as soon as the bit count is above a certain count:
void BitStream::writeBits(S32 bitCount, const void *bitPtr)
{
...
if(bitCount + bitNum > maxWriteBitNum)
{
error = true;
AssertFatal(false, "Out of range write");
return;
}
}In my Torque environment the maxWriteBitNum is set to 12000, causing the writeLongString method to fail for strings that are larger than about 1500 characters.
So, what's the purpose of this maxWriteBitNum? Is it possible to set it to a much higher value? And why isn't any of this mentioned in the API of the writeLongString method?
#2
The more packets it gets split in to, the faster your chances of losing the entirety of the message via one lost UDP packet goes up.
Depending on your application and your planned deployment (is it only intended to be played/run by users connected to each other or a server all on the same LAN), it might be perfectly okay to increase the constant for the UDP packet size.
In any case, BitStream::writeString at least Huffman encodes the string, which might work for you (I don't even know why writeLongString exists... I'm going to rip it out).
If neither of those options is possible for you, you'll probably have to investigate writing them over TCP, or consider shipping the strings already in the client (if that's possible).
What sort of strings are they? What kinda of application/game is it?
08/08/2008 (1:17 pm)
There's no easy way around hitting this problem. Torque doesn't want to send packets larger than 1500 or so bytes because everything is sent via UDP, and this is typically the MTU or close to it, meaning that your message/event/whatever will need to split into two or more UDP packets.The more packets it gets split in to, the faster your chances of losing the entirety of the message via one lost UDP packet goes up.
Depending on your application and your planned deployment (is it only intended to be played/run by users connected to each other or a server all on the same LAN), it might be perfectly okay to increase the constant for the UDP packet size.
In any case, BitStream::writeString at least Huffman encodes the string, which might work for you (I don't even know why writeLongString exists... I'm going to rip it out).
If neither of those options is possible for you, you'll probably have to investigate writing them over TCP, or consider shipping the strings already in the client (if that's possible).
What sort of strings are they? What kinda of application/game is it?
#3
All the times I've used it, it has worked. But as Tim said, it doesn't employ any compression and you'll hit the 500 bytes packet limit sooner rather than later. What I did was to make a function which split the string into sizes of 400 and then pass them out as seperate NetEvents. If you go down this path, remember that you got packet headers and Torque internal stuff which takes a few bits as well, so stay safe.
On the other hand, what are you trying to accomplish here? Sending brutally long strings sounds like bad design, or a file-transfer protocol.
08/08/2008 (2:00 pm)
WriteLongString () is not a BitStream function, but since it is a child of Stream you can use it.All the times I've used it, it has worked. But as Tim said, it doesn't employ any compression and you'll hit the 500 bytes packet limit sooner rather than later. What I did was to make a function which split the string into sizes of 400 and then pass them out as seperate NetEvents. If you go down this path, remember that you got packet headers and Torque internal stuff which takes a few bits as well, so stay safe.
On the other hand, what are you trying to accomplish here? Sending brutally long strings sounds like bad design, or a file-transfer protocol.
#4
Thanks for the suggestions, it will probably boil down to splitting up long strings into smaller ones. My first attempt in this direction was to split the string and to call the writeLongString method consecutively in the NetEvent's pack method, i.e. bstream->write(first 1000 characters); bstream->write(next 1000 characters); ....; This method currently creates "invalid packets" even for strings that are smaller than 1000 characters - I do not know where this problem is coming from right now, but will have to figure it out sooner or later.
As you were talking about using multiple NetEvents, I wondered if the mentioned approach is possible at all. Can I call the writeLongString or writeString methods as often as I want on the same stream, or will this again reach the defined bit count after writing some 1400 characters? Is using multiple NetEvents the only way to transmit long strings? Sounds a bit cumbersome, as the string has to be split across multiple NetEvents and then pasted together again from the information in different NetEvents.
08/09/2008 (7:11 am)
Well, I just want to send some XML messages between client and server and they will be some hundred or even thousand characters long. Is there a better method/design than sending it via NetEvents? And, I am actually not developing a game, but working on a research project in the domain of multi agent systems, 3D virtual worlds and tourism.Thanks for the suggestions, it will probably boil down to splitting up long strings into smaller ones. My first attempt in this direction was to split the string and to call the writeLongString method consecutively in the NetEvent's pack method, i.e. bstream->write(first 1000 characters); bstream->write(next 1000 characters); ....; This method currently creates "invalid packets" even for strings that are smaller than 1000 characters - I do not know where this problem is coming from right now, but will have to figure it out sooner or later.
As you were talking about using multiple NetEvents, I wondered if the mentioned approach is possible at all. Can I call the writeLongString or writeString methods as often as I want on the same stream, or will this again reach the defined bit count after writing some 1400 characters? Is using multiple NetEvents the only way to transmit long strings? Sounds a bit cumbersome, as the string has to be split across multiple NetEvents and then pasted together again from the information in different NetEvents.
#5
When you do bstream->write () you write into the current BitStream. In your example above, you're writing 2000 bytes + packet headers, which is too much. Two reasons:
1. Torque limits your packet size to 1500 bytes, which is AFAIK, the maximum ethernet packet size most routers support.
2. Any packets above 576 bytes have a chance to get split (and hence dropped totally if one of the two pieces fail to deliver) if the path you're sending across uses a lower MTU.
If you want to play it safe, use 450. Also, you really should drop writeLongString. It's not intended for streams across the internet, but for writing to files.
08/09/2008 (10:10 am)
That's not how it works.When you do bstream->write () you write into the current BitStream. In your example above, you're writing 2000 bytes + packet headers, which is too much. Two reasons:
1. Torque limits your packet size to 1500 bytes, which is AFAIK, the maximum ethernet packet size most routers support.
2. Any packets above 576 bytes have a chance to get split (and hence dropped totally if one of the two pieces fail to deliver) if the path you're sending across uses a lower MTU.
If you want to play it safe, use 450. Also, you really should drop writeLongString. It's not intended for streams across the internet, but for writing to files.
#6
With tinyXML you should be able to load XML into a binary form and transmit only that. Then even if you need the actual XML code on the other side you can write it back out.
But it looks like tinyXML is a library only integrated with TGB and not TGE. In any case, compressing the xml to a binary form for transmition is the proper way to do ths.
08/09/2008 (12:42 pm)
How is the server/client using this xml? With tinyXML you should be able to load XML into a binary form and transmit only that. Then even if you need the actual XML code on the other side you can write it back out.
But it looks like tinyXML is a library only integrated with TGB and not TGE. In any case, compressing the xml to a binary form for transmition is the proper way to do ths.
#7
08/09/2008 (1:12 pm)
It doesn't solve his issue with payloads being larger than the MTU, however. Binary or not.
#8
08/09/2008 (1:19 pm)
It will reduce the total bits he has to write by a ton, how does that not solve his problem? Well, its still possible for it to be too large but that limit is going to be reached much much slower.
#9
AFAIK TinyXML was in TGEA 1.7.1. Shouldn't be difficult to get into TGE, in any case.
As for...
You could use TCPObject. You won't have to worry about these things as TCP should handle it for you transparently.
08/09/2008 (1:47 pm)
Sure, I'm just saying it's not scaleable if he ever wants to send more data.AFAIK TinyXML was in TGEA 1.7.1. Shouldn't be difficult to get into TGE, in any case.
As for...
Quote:
Is using multiple NetEvents the only way to transmit long strings? Sounds a bit cumbersome..
You could use TCPObject. You won't have to worry about these things as TCP should handle it for you transparently.
#10
08/10/2008 (1:01 am)
If you have arbitrary length strings and you need a simple solution, TCPObject is the way to go. Otherwise you can reimplement TCPObject using guaranteedordered events and breaking the long strings up into many small events of 100-500 bytes.
#11
A last comment about NetEvents. If I understood correctly, NetObjects can be used to send objects over the wire from the Torque server to the client and vice versa. Since the limitation on the maximum number of bytes is very strict, this approach only works for very small objects. If I have a NetObject with 10 string parameters of which each is 200 charachters long, I will no longer be able to send this object via the pack and unpack methods, right? Isn't this a very limiting restriction? Or is my reasoning wrong and it works in the case of NetObjects?
08/11/2008 (11:32 pm)
Wow, good to see so many suggestions, I finally used "the multiple NetEvents" approach as it required the least amount of code modifications. Concerning the usage of TCPObject, I was not sure whether this technique is encouraged or not. If I use TCPObjects, I will have to establish a new communication channel between the Torque server and all clients - thus bypassing the Torque internal communication facilities. Also, it is more complicated as you will need additional connection managers and handling components on each side.A last comment about NetEvents. If I understood correctly, NetObjects can be used to send objects over the wire from the Torque server to the client and vice versa. Since the limitation on the maximum number of bytes is very strict, this approach only works for very small objects. If I have a NetObject with 10 string parameters of which each is 200 charachters long, I will no longer be able to send this object via the pack and unpack methods, right? Isn't this a very limiting restriction? Or is my reasoning wrong and it works in the case of NetObjects?
#12
Not exactly, the NetEvent IS the object sent across the network.
@if I have a NetObject with 10 string parameters of which each is 200 charachters long, I will no longer be able to send this object via the pack and unpack methods,
If you are refering to the pack/unpack you implement in your derived NetEvent then yes the same restrictions apply. But it sounds like you might be confused and thinking that you send a NetEvent through a pack/unpack in some other object... To send/transmit a NetEvent you do connection->postEvent and it will be transmitted to the client represented by that connection object. The NetEvent header file has some good comments on using them.
08/11/2008 (11:53 pm)
@If I understood correctly, NetObjects can be used to send objects over the wire from the Torque server to the client and vice versa.Not exactly, the NetEvent IS the object sent across the network.
@if I have a NetObject with 10 string parameters of which each is 200 charachters long, I will no longer be able to send this object via the pack and unpack methods,
If you are refering to the pack/unpack you implement in your derived NetEvent then yes the same restrictions apply. But it sounds like you might be confused and thinking that you send a NetEvent through a pack/unpack in some other object... To send/transmit a NetEvent you do connection->postEvent and it will be transmitted to the client represented by that connection object. The NetEvent header file has some good comments on using them.
#13
Lets say you have 2000 characters to write. Write as many as you can, then save how many you wrote, and continue from there in the next pack. The unpack side would have to concatenate / append to file as it receives them.
08/12/2008 (2:44 am)
Actually here's an idea that just struck me, which might allow you to have just one object send it all and its simple, spread it across multiple pack/unpacks. Lets say you have 2000 characters to write. Write as many as you can, then save how many you wrote, and continue from there in the next pack. The unpack side would have to concatenate / append to file as it receives them.
#14
Of course, you *can* write 1400 packets and hope it gets trough - but chances are it won't. You could do MTU path discovery and see how much you can send, but that amount can change anytime during your connection, especially so if you're on wireless, and it can get complicated.
TCPObject is good because you don't have to get dirty. It will do the packet splitting for you, transparently.
However, it adds another quite complex class and opens yet another port and socket.
08/12/2008 (5:19 am)
What James describes is a common way to do this. Write as much as you can, and check BitStream::getStreamSize () so you're within bounds. You will not be able to fit more than 1500 into any single BitStream, so it won't matter if you use NetObject or not.Of course, you *can* write 1400 packets and hope it gets trough - but chances are it won't. You could do MTU path discovery and see how much you can send, but that amount can change anytime during your connection, especially so if you're on wireless, and it can get complicated.
TCPObject is good because you don't have to get dirty. It will do the packet splitting for you, transparently.
However, it adds another quite complex class and opens yet another port and socket.
#15
08/12/2008 (6:47 am)
Oh, I thought NetObject derives from NetEvent. Nevertheless it would be a good idea to mention these size restrictions somewhere in the API.
#16
Packet loss is extemely minimal on modern low-latency, high-speed LANs that are not saturated.
Even if you're using higher latency connections across the internet, I'm guessing you'll find that it works (it doesn't sound like you're spewing a ton of these messages).
08/12/2008 (10:36 am)
Hubertus, it sounds like you might have this all running on machines that are on a local LAN or seperated by at most one or two network hops. If that's the case, I'd figure out how long you really need your messages to be... if you can live with the UDP hard limit of 65K, I'd just go increase the packet size limit in Torque to be something arbitrarily large (10K or 20K, let's say), and try sending guaranteed net events that write everything into the buffer at once. And test it a bunch.Packet loss is extemely minimal on modern low-latency, high-speed LANs that are not saturated.
Even if you're using higher latency connections across the internet, I'm guessing you'll find that it works (it doesn't sound like you're spewing a ton of these messages).
#17
Some crappy VPN connectors artificially limit UDP packet size, and some (much) older routers will silently drop UDP beyond a certain size (we discovered that the Cisco zero-install VPN client causes issues with packet sizes beyond about 1200 bytes, at layer 4, even though the MTU at layer 3 is something like 1428 bytes).
Anyways, it all depends on where you are deploying. If you don't have those kinds of stacks to worry about in your architecture, then you probably won't have an issue.
08/12/2008 (10:42 am)
Ah, I should also caveat that... you might have a limit imposed on you by your network path at layer 4 (this is different than the MTU), where you get packets silently dropped because the implementation is old, buggy, etc.Some crappy VPN connectors artificially limit UDP packet size, and some (much) older routers will silently drop UDP beyond a certain size (we discovered that the Cisco zero-install VPN client causes issues with packet sizes beyond about 1200 bytes, at layer 4, even though the MTU at layer 3 is something like 1428 bytes).
Anyways, it all depends on where you are deploying. If you don't have those kinds of stacks to worry about in your architecture, then you probably won't have an issue.
Associate James Ford
Sickhead Games