How to: Enable Vehicle->Vehicle(&Player!) Rigid Collison Solver

***UPDATE 1/6/2011: Now with Vehicle->Player collision and some minor improvements, as this resource:
www.torquepowered.com/community/resources/view/20727

Here's a quick video of it in action: www.youtube.com/watch?v=hpsKIL0a5h8

This info may more or less already be out there, but I don't think it's common knowledge since all of the old discussions are burried deep in the TGE forums. Feel free to skip these two paragraphs and go straight to the instructions if you don't care. ;)

Some of us may remember various attempts over the years to fix the Rigid class's collision solvers. In the end most of these attempts were abandoned when research indicated that the problems were 1) that the solvers didn't handle certain collision cases well to begin with and 2) that the formulas used in the solvers were intended to be paired with a time-reversal mechanism to solve collisions that occured between ticks.

Anyway, my understanding is that a physics abstraction layer solution for vehicles is on the table, possibly even being worked on, and that this will finally allow us to ditch Rigid in favor of PhysX or Bullet. However, since there's no concrete timeline on that, here's how you can use Rigid's vehicle vs. vehicle solver and have it be relatively stable (at least as stable as existing vehicle vs. static collisions, anyway):

(note: instructions based on T3D, however this should work with any version of Torque)
Go to T3D/vehicles/vehicle.cpp
Go to the function bool Vehicle::resolveCollision

Replace it with this:

bool Vehicle::resolveCollision(Rigid&  ns,CollisionList& cList)
{
   PROFILE_SCOPE( Vehicle_ResolveCollision );

   // Apply impulses to resolve collision
   bool collided = false;
   for (S32 i = 0; i < cList.getCount(); i++) 
   {
      Collision& c = cList[i];
      if (c.distance < mDataBlock->collisionTol) 
      {
         // [HNT] Skip converge test and contact constraints
         if (c.object->getTypeMask() & VehicleObjectType) // ADDED
         { // ADDED
            Vehicle *tmpShape = (Vehicle*)c.object; // ADDED
            ns.resolveCollision(cList[i].point, cList[i].normal, // ADDED
               &tmpShape->mRigid); // ADDED
            collided  = true; // ADDED

            // Keep track of objects we collide with // ADDED
            if (!isGhost())  // ADDED
            { // ADDED
               Point3F v,r; // ADDED
               ns.getOriginVector(c.point,&r); // ADDED
               ns.getVelocity(r,&v); // ADDED
               queueCollision(tmpShape,v - tmpShape->getVelocity()); // ADDED
            } // ADDED
         } // ADDED
         else // ADDED
         { // ADDED
            // Velocity into surface
            Point3F v,r;
            ns.getOriginVector(c.point,&r);
            ns.getVelocity(r,&v);
            F32 vn = mDot(v,c.normal);

            // Only interested in velocities greater than sContactTol,
            // velocities less than that will be dealt with as contacts
            // "constraints".
            if (vn < -mDataBlock->contactTol) 
            {
               // Apply impulses to the rigid body to keep it from
               // penetrating the surface.
               ns.resolveCollision(cList[i].point,
                  cList[i].normal);
               collided  = true;

               // Keep track of objects we collide with
               if (!isGhost() && c.object->getTypeMask() & ShapeBaseObjectType) 
               {
                  ShapeBase* col = static_cast<ShapeBase*>(c.object);
                  queueCollision(col,v - col->getVelocity());
               }
            }
         } // ADDED
         // [/HNT]
      }
   }

   return collided;
}

New lines marked with "// ADDED" so you can see what I changed.
Pretty simple, if the collided object is a Vehicle then use the version of the solver function that takes a second Rigid object as a param. I skip the converging velocity check because 1) that check only makes sense if you're colliding with a static object and 2) the solver function already does a version of this check which properly accounts for the velocity of the second Rigid body.

Now you need to go to T3D/rigid.cpp
Function is bool Rigid::resolveCollision(const Point3F& p, const Point3F &normal, Rigid* rigid)
Should be around line 126 (note this is the first Rigid::resolveCollision function, there are 2).

Depending on your version of Torque you may or may not need to fix the second to last line to say:
"rigid->applyImpulse(r2,impulse);" instead of "applyImpulse(r2,impulse);"
This fix eventually made it in, but I'm not sure at what point.

Now, to beat the potential for two Rigid objects to "slide" into each other, I suggest slightly exaggerating the force produced by the solver. It'll make your collisions a little bumpier at low speeds, but it will keep objects from becoming stuck in each other in all but the most extreme cases (the same cases where Vehicles get stuck in static objects). To do this, go to line 137 (right above the line "if (nv > 0.0f)") and add:

nv -= 3.0f;

Yeah, I know, that's a terrible hack. But it works better than any more complex method I've tried at various times like checking for negative collision distance or changing the updateForces/updateCollisions/integrate order. Adjust the value to taste; I found 1 to be too weak (could still lodge cars into each other) and 5 to be too bouncy. You're effectively adding to the converging velocity used to calculate the impulse, so a collison of velocity 20 will be treated as 23 (little notable difference) and a collision of velocity 0.5 will be treated as 3.5 (some bouncing, but keeps the solver from totally failing).

Ugly, but it's easy to implement, stable, and far better than the stock vehicle vs. vehicle collision (each vehicle treats the other as a static object). Note that certain types of collision hulls are better than others from the solver's perspective; boxes produce too many planar collisions which can produce unsolvable scenarios. You want to have a mesh that favors point and line collisions, like the collision mesh the default Buggy comes with (I actually just keep rescaling that collision mesh onto most of my vehicles).

RigidShape
Don't try to collide RigidShapes with Vehicles using this code, it will crash. RigidShapes don't have their own typeMask (they piggyback on VehicleObjectType) so the code will try to recast the RigidShape pointer to a Vehicle pointer, destroying the universe in the process. Even if you give the RigidShape its own typeMask, you'll still need to make mRigid public in order for Vehicles to access the RigidShape's mRigid (and the other way around.

The simple workaround is to use Wheel-less WheeledVehicles instead of RigidShapes. Take your existing RigidShape datablocks and replace "RigidShapeData" with "WheeledVehicleData" and you're good to go. IE, datablock WheeledVehicleData( BouncingBoulder ) { ...everything that was in the DB originally ... };