Libcurl implementation
by Fyodor -bank- Osokin · in Torque Game Engine · 11/17/2007 (7:44 pm) · 63 replies
this is my own libcurl implementation into TGE 1.5.2 "as is". Feel free to use it for your own games.
---------------------------
What implemented:
download works for http and ftp (without login/password - tested)
download works for http and ftp (with login/password - not tested)
you can set to resume previously interrupted download/upload (tested)
you can cancel download (tested)
upload works, but not tested heavily
---------------------------
See the available script functions.
If you need upload, don't forget to call .setUploadFlag(true); :)
---------------------------
Instructions:
Download "full" curl package (I've used 1.17.1).
Unpack it into /lib/ folder (so you have the /lib/curl-7.17.1/* )
For "Torque Demo" (or your own project):
Add ../lib/curl-7.17.1/include to the project's "Addition Include Directories".
Add ;CURL_STATICLIB to project's "Preprocessor Definitions".
Add ;"../lib/curl-7.17.1/lib/Release" to project's "Additional Library Directories".
Add curllib.lib to project's "Addition Dependencies".
Add libcmt.lib to project's "Ignore Specific Library" (without this it won't link the EXE).
Add lib/curl-7-17-1/lib/curllib.vcproj project file into your solution.
In curllib project's properties:
Add ";../../zlib" to the project's "Addition Include Directories".
Add ";HAVE_ZLIB_H;HAVE_ZLIB;HAVE_LIBZ;CURL_STATICLIB;CURL_DISABLE_LDAP" to the "Preprocessor Definitions".
engine changes:
game/main.cc:
at the top where all other includes are, add:
Rebuild the "curllib" project.
Rebuild your project (Torque Demo).
If it don't compile and giving errors like "templates can't be used when "C" linkage.." for wspiapi.h (see this page for explanation), you need to change the curl/curl.h file in includes folder (changes in BOLD):
See next post for code :)
---------------------------
What implemented:
download works for http and ftp (without login/password - tested)
download works for http and ftp (with login/password - not tested)
you can set to resume previously interrupted download/upload (tested)
you can cancel download (tested)
upload works, but not tested heavily
---------------------------
See the available script functions.
If you need upload, don't forget to call .setUploadFlag(true); :)
---------------------------
Instructions:
Download "full" curl package (I've used 1.17.1).
Unpack it into /lib/ folder (so you have the /lib/curl-7.17.1/* )
For "Torque Demo" (or your own project):
Add ../lib/curl-7.17.1/include to the project's "Addition Include Directories".
Add ;CURL_STATICLIB to project's "Preprocessor Definitions".
Add ;"../lib/curl-7.17.1/lib/Release" to project's "Additional Library Directories".
Add curllib.lib to project's "Addition Dependencies".
Add libcmt.lib to project's "Ignore Specific Library" (without this it won't link the EXE).
Add lib/curl-7-17-1/lib/curllib.vcproj project file into your solution.
In curllib project's properties:
Add ";../../zlib" to the project's "Addition Include Directories".
Add ";HAVE_ZLIB_H;HAVE_ZLIB;HAVE_LIBZ;CURL_STATICLIB;CURL_DISABLE_LDAP" to the "Preprocessor Definitions".
engine changes:
game/main.cc:
at the top where all other includes are, add:
#include "sim/simCurl.h"change the beginning of initLibraries function:
static bool initLibraries()
{
[b]if (!SimCurl::initialize())
{
Platform::AlertOK("libcurl Error", "Unable to initialize the libcurl... aborting.");
return false;
}
[/b]
if(!Net::init())
...change the end of shutdownLibraries function:Net::shutdown(); [b]SimCurl::deinitialize();[/b] }
Rebuild the "curllib" project.
Rebuild your project (Torque Demo).
If it don't compile and giving errors like "templates can't be used when "C" linkage.." for wspiapi.h (see this page for explanation), you need to change the curl/curl.h file in includes folder (changes in BOLD):
#if !(defined(_WINSOCKAPI_) || defined(_WINSOCK_H))
/* The check above prevents the winsock2 inclusion if winsock.h already was
included, since they can't co-exist without problems */
[b]#ifdef __cplusplus
}
#endif[/b]
#include <winsock2.h>
#include <ws2tcpip.h>
[b]#ifdef __cplusplus
extern "C" {
#endif[/b]
#endifSee next post for code :)
About the author
Game developer.
Recent Threads
#42
08/12/2009 (2:52 am)
I think you will still have one issue, as the line after the if test will try to put a value to a NULL pointer... so maybe make an exit with something like thatconst U8 *SimCurl::readLine()
{
if(!storagedesc.mem.ptr)
{
static U8 retBuf[1] = "";
storagedesc.mem.ptr = NULL;
storagedesc.mem.size = 0;
storagedesc.mem.c = NULL;
storagedesc.mem.rsize = 0;
return &retBuf[0];
}
// Should not be here after NULL setting
storagedesc.mem.c[storagedesc.mem.rsize] = 0;
U8 *tokPos = storagedesc.mem.c;
...
#43
I'm terribly sorry for being so sloppy. That will teach me. ;)
08/12/2009 (3:08 am)
You're right.. I dumped my code, and added a pointer to yours instead. :)I'm terribly sorry for being so sloppy. That will teach me. ;)
#44
I can't get it to compile at all under Windows.
This line:
Throws errors if I'm using GCC or VC++, but only under Windows, not under OS X.
[edit]
Fixed by using the placenew macro, and making sure placenew() is defined in platform.h before new itself is redefined as a macro.
[edit]
Not fixed. It just tried recompiling everything. Silly me for thinking it was moving on to the next step. I still need help.
[edit]
Fixed. For real. Conditionally undef the new macro used by the memory manager. around the function using placement new.
09/09/2009 (10:05 pm)
Hey guys, any word on what might be going wrong with the placement new being used in SimCurl::write_data?I can't get it to compile at all under Windows.
This line:
new(s) MemStream(nsz, ptr);
Throws errors if I'm using GCC or VC++, but only under Windows, not under OS X.
[edit]
Fixed by using the placenew macro, and making sure placenew() is defined in platform.h before new itself is redefined as a macro.
[edit]
Not fixed. It just tried recompiling everything. Silly me for thinking it was moving on to the next step. I still need help.
[edit]
Fixed. For real. Conditionally undef the new macro used by the memory manager. around the function using placement new.
#45
09/18/2009 (10:08 am)
I wasnt going to go through 44 messages on this topic, but I did want to point something out. That as this option has upload can we not beable to setlimit on bandwith and instead of loading all missions ata time. set a trigger places around where the player goes into the next mis and just download files before he even reaches it? this would create a VERY faster load time, or would it cause a crash in the system when switching missions ?
#49
and your "instructions" part up top was very handy, thanks Bank !
fwiw, i used curl 7.19.7, and there were a couple differences:
* the library is "libcurl.lib", not "curllib.lib".
* i didn't have to add "libcmt.lib" to the ignored-libs list.
.. of course, that's just compiling and linking. i haven't actually used it so far! should be fine.
11/05/2009 (5:26 pm)
i'm doing another integration of libCurl into T3D,and your "instructions" part up top was very handy, thanks Bank !
fwiw, i used curl 7.19.7, and there were a couple differences:
* the library is "libcurl.lib", not "curllib.lib".
* i didn't have to add "libcmt.lib" to the ignored-libs list.
.. of course, that's just compiling and linking. i haven't actually used it so far! should be fine.
#50
So if you are really willing to use extensively, you need to address all those issues.
I hope I will be able to show you some videos of my new code where the system download all models when required from the server inside the mission without halting the player and so on.
11/06/2009 (1:19 am)
One important comment as I did a lot of work with this in order to implement a fully data streaming system (for mission objects and so on) inside T3D, this implementation is not thread-safe at all; there are a lot of issue with multiple parts of T3D code and mainly everything around the Con::execute and FileStream...So if you are really willing to use extensively, you need to address all those issues.
I hope I will be able to show you some videos of my new code where the system download all models when required from the server inside the mission without halting the player and so on.
#51
aye, i'm aware of the thread-safety gotchas. this particular instance is not super general-purpose and doesn't have to worry about them so much, but they're definitely something to keep very-much in mind. ie, i'm not even exposing it to script, and blocking operations on curl's part are fine. but yr absolutely right: to implement a flexible curl system for general use within torque definitely requires some attention to detail w/r/t thread-safety. eg, don't jump into script and then back into C and then back into script all as part of a single call-chain.
also, i was mistaken: i did need to add libcmt to the ignore list.
11/06/2009 (3:55 am)
hey frank, thanks.aye, i'm aware of the thread-safety gotchas. this particular instance is not super general-purpose and doesn't have to worry about them so much, but they're definitely something to keep very-much in mind. ie, i'm not even exposing it to script, and blocking operations on curl's part are fine. but yr absolutely right: to implement a flexible curl system for general use within torque definitely requires some attention to detail w/r/t thread-safety. eg, don't jump into script and then back into C and then back into script all as part of a single call-chain.
also, i was mistaken: i did need to add libcmt to the ignore list.
#52
12/28/2009 (8:25 pm)
After T3D 1.01 or 1.1 my downloads have stopped working. All text tasks work just as before but when I try to create a new download, it just doesnt start. Anyone know what could be happening?
#53
After you create the Thread, make sure the start() method is being called.
12/30/2009 (11:09 pm)
It's possible you're getting bit by a change to the Thread class constructor.After you create the Thread, make sure the start() method is being called.
#54
www.ignitiongamespm.com/uploads/Adam/simCurl.cpp
www.ignitiongamespm.com/uploads/Adam/simCurl.h
Would really appreciate if someone was able to look into this.
01/04/2010 (12:33 pm)
I checked over the code and start seems to be getting called when creating the thread. Just in case here is the code:www.ignitiongamespm.com/uploads/Adam/simCurl.cpp
www.ignitiongamespm.com/uploads/Adam/simCurl.h
Would really appreciate if someone was able to look into this.
#55
If you're downloading multiple objects, it's possible that one of the objects won't get downloaded or will get downloaded more than once.
I've made a suggestion or two on how to solve this, but I get ignored as usual :P
One suggestion was to change _G.head to volatile, and the proper way to do that is:
Not:
I strongly recommend moving away from this naive implementation and doing something a little more robust.
Consider taking a look at how I implemented this resource.
It only uses one worker thread, but it wouldn't be terribly difficult to adjust it so it uses more than one.
01/04/2010 (2:24 pm)
The code has several potential race conditions.If you're downloading multiple objects, it's possible that one of the objects won't get downloaded or will get downloaded more than once.
I've made a suggestion or two on how to solve this, but I get ignored as usual :P
One suggestion was to change _G.head to volatile, and the proper way to do that is:
SimCurl* volatile head;
Not:
volatile SimCurl* head;
I strongly recommend moving away from this naive implementation and doing something a little more robust.
Consider taking a look at how I implemented this resource.
It only uses one worker thread, but it wouldn't be terribly difficult to adjust it so it uses more than one.
#56
Anyone have a vc2005 or vc2008 priject file for TGEA / AFX Combo.
Thanks
02/12/2010 (4:20 pm)
Compiling stuff is so much of an head ake for me. I can never seem to get it working lol.Anyone have a vc2005 or vc2008 priject file for TGEA / AFX Combo.
Thanks
#57
I am using TGE Pro v1.7 and still new to this.
I managed to compile the source with libCurl after overcoming some errors.
I have not performed the update by Thomas "elfprince13" Dickerson - (http://www.torquepowered.com/community/resources/view/18545)
I have also added the script file provided by (Fyodor "bank") into my main.cs with exec.
Somehow i can't seem to run the script helper function startBinaryDownload().
I keep getting "Not Finished, status: 10" in my game console.
I also observe the code:
curler.addDownloadTask("http://www.garagegames.com/", "starter.fps/cache/gg_index.html, -1, "", "cbOK", "cbError", "cbProgress");
So the 1st parameter i assume is the URL of the website? eg. "http://www.torquepowered.com/"
And the 2nd parameter is the exact path to the file? eg."index.php"
I assume the file should appear in the directory where i run the game exe?
Thanks for any heads up on this.
Have been trying for half a day, but not getting it to work.
05/16/2010 (3:59 am)
Hi,I am using TGE Pro v1.7 and still new to this.
I managed to compile the source with libCurl after overcoming some errors.
I have not performed the update by Thomas "elfprince13" Dickerson - (http://www.torquepowered.com/community/resources/view/18545)
I have also added the script file provided by (Fyodor "bank") into my main.cs with exec.
Somehow i can't seem to run the script helper function startBinaryDownload().
I keep getting "Not Finished, status: 10" in my game console.
I also observe the code:
curler.addDownloadTask("http://www.garagegames.com/", "starter.fps/cache/gg_index.html, -1, "", "cbOK", "cbError", "cbProgress");
So the 1st parameter i assume is the URL of the website? eg. "http://www.torquepowered.com/"
And the 2nd parameter is the exact path to the file? eg."index.php"
I assume the file should appear in the directory where i run the game exe?
Thanks for any heads up on this.
Have been trying for half a day, but not getting it to work.
#58
05/16/2010 (4:05 am)
@brian: I wonder how you managed to compile this - you are not a licensee of any of the Torque engines. Care to explain?
#60
05/18/2010 (2:57 am)
@Brian: Oh, ok. Where did you get TGB?
Associate Konrad Kiss
Bitgap Games
Thanks again. I've corrected the code above.