Game Development Community

Libcurl implementation

by Fyodor -bank- Osokin · in Torque Game Engine · 11/17/2007 (7:44 pm) · 63 replies

this is my own libcurl implementation into TGE 1.5.2 "as is". Feel free to use it for your own games.
---------------------------
What implemented:
download works for http and ftp (without login/password - tested)
download works for http and ftp (with login/password - not tested)
you can set to resume previously interrupted download/upload (tested)
you can cancel download (tested)
upload works, but not tested heavily
---------------------------
See the available script functions.
If you need upload, don't forget to call .setUploadFlag(true); :)

---------------------------
Instructions:

Download "full" curl package (I've used 1.17.1).
Unpack it into /lib/ folder (so you have the /lib/curl-7.17.1/* )

For "Torque Demo" (or your own project):
Add ../lib/curl-7.17.1/include to the project's "Addition Include Directories".
Add ;CURL_STATICLIB to project's "Preprocessor Definitions".
Add ;"../lib/curl-7.17.1/lib/Release" to project's "Additional Library Directories".
Add curllib.lib to project's "Addition Dependencies".
Add libcmt.lib to project's "Ignore Specific Library" (without this it won't link the EXE).

Add lib/curl-7-17-1/lib/curllib.vcproj project file into your solution.

In curllib project's properties:
Add ";../../zlib" to the project's "Addition Include Directories".
Add ";HAVE_ZLIB_H;HAVE_ZLIB;HAVE_LIBZ;CURL_STATICLIB;CURL_DISABLE_LDAP" to the "Preprocessor Definitions".

engine changes:
game/main.cc:
at the top where all other includes are, add:
#include "sim/simCurl.h"
change the beginning of initLibraries function:
static bool initLibraries()
{
   [b]if (!SimCurl::initialize())
   {
      Platform::AlertOK("libcurl Error", "Unable to initialize the libcurl... aborting.");
      return false;
   }
[/b]
   if(!Net::init())
   ...
change the end of shutdownLibraries function:
Net::shutdown();
   [b]SimCurl::deinitialize();[/b]
}

Rebuild the "curllib" project.
Rebuild your project (Torque Demo).

If it don't compile and giving errors like "templates can't be used when "C" linkage.." for wspiapi.h (see this page for explanation), you need to change the curl/curl.h file in includes folder (changes in BOLD):
#if !(defined(_WINSOCKAPI_) || defined(_WINSOCK_H))
/* The check above prevents the winsock2 inclusion if winsock.h already was
   included, since they can't co-exist without problems */
  [b]#ifdef __cplusplus
    }
  #endif[/b]
#include <winsock2.h>
#include <ws2tcpip.h>
  [b]#ifdef __cplusplus
    extern "C" {
  #endif[/b]
#endif

See next post for code :)
#41
08/12/2009 (2:31 am)
@Frank: No reason at all, sorry about that. I shouldn't be allocating memory at all. It was some late night copy-paste programming - I meant to change it, but somehow forgot.

Thanks again. I've corrected the code above.
#42
08/12/2009 (2:52 am)
I think you will still have one issue, as the line after the if test will try to put a value to a NULL pointer... so maybe make an exit with something like that
const U8 *SimCurl::readLine()  
{  

   if(!storagedesc.mem.ptr)
   {
      static U8 retBuf[1] = "";
      storagedesc.mem.ptr = NULL;
      storagedesc.mem.size = 0;
      storagedesc.mem.c = NULL;
      storagedesc.mem.rsize = 0;
      return &retBuf[0];
   }

   // Should not be here after NULL setting
   storagedesc.mem.c[storagedesc.mem.rsize] = 0;  
  
   U8 *tokPos = storagedesc.mem.c;  

   ...
#43
08/12/2009 (3:08 am)
You're right.. I dumped my code, and added a pointer to yours instead. :)
I'm terribly sorry for being so sloppy. That will teach me. ;)
#44
09/09/2009 (10:05 pm)
Hey guys, any word on what might be going wrong with the placement new being used in SimCurl::write_data?

I can't get it to compile at all under Windows.

This line:
new(s) MemStream(nsz, ptr);

Throws errors if I'm using GCC or VC++, but only under Windows, not under OS X.


[edit]

Fixed by using the placenew macro, and making sure placenew() is defined in platform.h before new itself is redefined as a macro.

[edit]
Not fixed. It just tried recompiling everything. Silly me for thinking it was moving on to the next step. I still need help.

[edit]

Fixed. For real. Conditionally undef the new macro used by the memory manager. around the function using placement new.
#45
09/18/2009 (10:08 am)
I wasnt going to go through 44 messages on this topic, but I did want to point something out. That as this option has upload can we not beable to setlimit on bandwith and instead of loading all missions ata time. set a trigger places around where the player goes into the next mis and just download files before he even reaches it? this would create a VERY faster load time, or would it cause a crash in the system when switching missions ?
#46
09/21/2009 (8:31 am)
Thanks for this. It helped me a lot.

Regards,
Blue
Pret travaux
#47
09/30/2009 (11:17 am)
Added an update to it for T3D v1.0 here.
#48
10/21/2009 (2:49 am)
Added an update comprising my changes, with Cookie jars, form post data, more Torque Memory Manager friendly code surrounding the placement new, and fixes to all of the memory leaks mentioned in this thread, here.
#49
11/05/2009 (5:26 pm)
i'm doing another integration of libCurl into T3D,
and your "instructions" part up top was very handy, thanks Bank !

fwiw, i used curl 7.19.7, and there were a couple differences:
* the library is "libcurl.lib", not "curllib.lib".
* i didn't have to add "libcmt.lib" to the ignored-libs list.

.. of course, that's just compiling and linking. i haven't actually used it so far! should be fine.
#50
11/06/2009 (1:19 am)
One important comment as I did a lot of work with this in order to implement a fully data streaming system (for mission objects and so on) inside T3D, this implementation is not thread-safe at all; there are a lot of issue with multiple parts of T3D code and mainly everything around the Con::execute and FileStream...

So if you are really willing to use extensively, you need to address all those issues.

I hope I will be able to show you some videos of my new code where the system download all models when required from the server inside the mission without halting the player and so on.

#51
11/06/2009 (3:55 am)
hey frank, thanks.
aye, i'm aware of the thread-safety gotchas. this particular instance is not super general-purpose and doesn't have to worry about them so much, but they're definitely something to keep very-much in mind. ie, i'm not even exposing it to script, and blocking operations on curl's part are fine. but yr absolutely right: to implement a flexible curl system for general use within torque definitely requires some attention to detail w/r/t thread-safety. eg, don't jump into script and then back into C and then back into script all as part of a single call-chain.

also, i was mistaken: i did need to add libcmt to the ignore list.
#52
12/28/2009 (8:25 pm)
After T3D 1.01 or 1.1 my downloads have stopped working. All text tasks work just as before but when I try to create a new download, it just doesnt start. Anyone know what could be happening?
#53
12/30/2009 (11:09 pm)
It's possible you're getting bit by a change to the Thread class constructor.

After you create the Thread, make sure the start() method is being called.



#54
01/04/2010 (12:33 pm)
I checked over the code and start seems to be getting called when creating the thread. Just in case here is the code:

www.ignitiongamespm.com/uploads/Adam/simCurl.cpp
www.ignitiongamespm.com/uploads/Adam/simCurl.h

Would really appreciate if someone was able to look into this.
#55
01/04/2010 (2:24 pm)
The code has several potential race conditions.

If you're downloading multiple objects, it's possible that one of the objects won't get downloaded or will get downloaded more than once.

I've made a suggestion or two on how to solve this, but I get ignored as usual :P

One suggestion was to change _G.head to volatile, and the proper way to do that is:

SimCurl* volatile head;

Not:

volatile SimCurl* head;

I strongly recommend moving away from this naive implementation and doing something a little more robust.

Consider taking a look at how I implemented this resource.

It only uses one worker thread, but it wouldn't be terribly difficult to adjust it so it uses more than one.
#56
02/12/2010 (4:20 pm)
Compiling stuff is so much of an head ake for me. I can never seem to get it working lol.

Anyone have a vc2005 or vc2008 priject file for TGEA / AFX Combo.

Thanks
#57
05/16/2010 (3:59 am)
Hi,

I am using TGE Pro v1.7 and still new to this.

I managed to compile the source with libCurl after overcoming some errors.
I have not performed the update by Thomas "elfprince13" Dickerson - (http://www.torquepowered.com/community/resources/view/18545)
I have also added the script file provided by (Fyodor "bank") into my main.cs with exec.

Somehow i can't seem to run the script helper function startBinaryDownload().
I keep getting "Not Finished, status: 10" in my game console.

I also observe the code:
curler.addDownloadTask("http://www.garagegames.com/", "starter.fps/cache/gg_index.html, -1, "", "cbOK", "cbError", "cbProgress");

So the 1st parameter i assume is the URL of the website? eg. "http://www.torquepowered.com/"
And the 2nd parameter is the exact path to the file? eg."index.php"

I assume the file should appear in the directory where i run the game exe?

Thanks for any heads up on this.
Have been trying for half a day, but not getting it to work.
#58
05/16/2010 (4:05 am)
@brian: I wonder how you managed to compile this - you are not a licensee of any of the Torque engines. Care to explain?
#59
05/18/2010 (2:54 am)
Hi Konrad,

I think i got things mixed up.
I use TGB actually.

Brian
#60
05/18/2010 (2:57 am)
@Brian: Oh, ok. Where did you get TGB?