Rotating a single object selection with the World Editor gizmo gradually causes an error in rotation angle

If a single object is selected in the World Editor and rotated along an axis using the gizmo, the object's angle gradually changes to be aligned with the axis of rotation. This occurs only when alignment mode (mCurrentAlignment) is set to World, where the object is rotated about the world axis.

It looks like what is causing this is that WorldEditor::Selection::rotate(const EulerF & rot, const Point3F & center) always processes single selected objects using an object axis rotation (ie about the object's own axis). However, if World alignment mode is set, other areas of code calculate rotation for the object as if it was going to rotate using the world axis.

I made the following changes to try to prevent the above bug from occuring, using the format (previous code) changed to (modified code):

1. In worldEditor.h

void rotate(const EulerF &, const Point3F &);

changed to

void rotate(const EulerF &, const Point3F &, Align);

2. In worldEditor.cpp

void WorldEditor::Selection::rotate(const EulerF & rot, const Point3F & center)
{
   // single selections will rotate around own axis, multiple about world
	if(size() == 1)
   {
      SceneObject* object = dynamic_cast< SceneObject* >( at( 0 ) );
      if( object )
      {
         MatrixF mat = object->getTransform();

         Point3F pos;
         mat.getColumn(3, &pos);

         // get offset in obj space
         Point3F offset = pos - center;
         MatrixF wMat = object->getWorldTransform();
         wMat.mulV(offset);

         //
         MatrixF transform(EulerF(0,0,0), -offset);
         transform.mul(MatrixF(rot));
         transform.mul(MatrixF(EulerF(0,0,0), offset));
         mat.mul(transform);

         object->setTransform(mat);
      }
   }
   else
   {
      for( iterator iter = begin(); iter != end(); ++ iter )
      {
         SceneObject* object = dynamic_cast< SceneObject* >( *iter );
         if( !object )
            continue;
            
         MatrixF mat = object->getTransform();

         Point3F pos;
         mat.getColumn(3, &pos);

         // get offset in obj space
         Point3F offset = pos - center;

         MatrixF transform(rot);
         Point3F wOffset;
         transform.mulV(offset, &wOffset);

         MatrixF wMat = object->getWorldTransform();
         wMat.mulV(offset);

         //
         transform.set(EulerF(0,0,0), -offset);

         mat.setColumn(3, Point3F(0,0,0));
         wMat.setColumn(3, Point3F(0,0,0));

         transform.mul(wMat);
         transform.mul(MatrixF(rot));
         transform.mul(mat);
         mat.mul(transform);

         mat.normalize();
         mat.setColumn(3, wOffset + center);

         object->setTransform(mat);
      }
   }

   mCentroidValid = false;
}

changed to

void WorldEditor::Selection::rotate(const EulerF & rot, const Point3F & center, Align singleSelectAlignment)
{
   if(size() == 1 && singleSelectAlignment == Align::Object)
   {
     // Object axis rotation
      SceneObject* object = dynamic_cast< SceneObject* >( at( 0 ) );
      if( object )
      {
         MatrixF mat = object->getTransform();

         Point3F pos;
         mat.getColumn(3, &pos);

         // get offset in obj space
         Point3F offset = pos - center;
         MatrixF wMat = object->getWorldTransform();
         wMat.mulV(offset);

         //
         MatrixF transform(EulerF(0,0,0), -offset);
         transform.mul(MatrixF(rot));
         transform.mul(MatrixF(EulerF(0,0,0), offset));
         mat.mul(transform);

         object->setTransform(mat);
      }
   }
   else
   {
      // World axis rotation
      for( iterator iter = begin(); iter != end(); ++ iter )
      {
         SceneObject* object = dynamic_cast< SceneObject* >( *iter );
         if( !object )
            continue;
            
         MatrixF mat = object->getTransform();

         Point3F pos;
         mat.getColumn(3, &pos);

         // get offset in obj space
         Point3F offset = pos - center;

         MatrixF transform(rot);
         Point3F wOffset;
         transform.mulV(offset, &wOffset);

         MatrixF wMat = object->getWorldTransform();
         wMat.mulV(offset);

         //
         transform.set(EulerF(0,0,0), -offset);

         mat.setColumn(3, Point3F(0,0,0));
         wMat.setColumn(3, Point3F(0,0,0));

         transform.mul(wMat);
         transform.mul(MatrixF(rot));
         transform.mul(mat);
         mat.mul(transform);

         mat.normalize();
         mat.setColumn(3, wOffset + center);

         object->setTransform(mat);
      }
   }

   mCentroidValid = false;
}

3. In void WorldEditor::on3DMouseDragged(const Gui3DMouseEvent & event) :

case RotateMode:
   {
      Point3F centroid = getSelectionCentroid();
      EulerF rot = mGizmo->getDeltaRot();

      mSelected.rotate(rot, centroid);
      updateClientTransforms(mSelected);

      break;  
   }

changed to

case RotateMode:
   {
      Point3F centroid = getSelectionCentroid();
      EulerF rot = mGizmo->getDeltaRot();
	  Align alignment = mGizmo->getAlignment();

      mSelected.rotate(rot, centroid, alignment);
      updateClientTransforms(mSelected);

      break;  
   }

4. In void Gizmo::on3DMouseDragged( const Gui3DMouseEvent & event, const Point3F* cursorPos3D ) :

if ( mCurrentAlignment == World )
      {
         MatrixF mat = mObjectMat;
         mat.inverse();
         mat.mulV(mDeltaTotalRot);
         mat.mulV(mDeltaRot);
      }

changed to be commented out

/*
      if ( mCurrentAlignment == World )
      {
         MatrixF mat = mObjectMat;
         mat.inverse();
         mat.mulV(mDeltaTotalRot);
         mat.mulV(mDeltaRot);
      }
      */

5. In void WorldEditor::transformSelection(bool position, Point3F& p, bool relativePos, bool rotate, EulerF& r, bool relativeRot, bool rotLocal, S32 scaleType, Point3F& s, bool sRelative, bool sLocal) :

mSelected.rotate(r, centroid);

changed to

mSelected.rotate(r, centroid, Align::Object);

In changes #1 and #2, an extra parameter was added to the rotate function so that if World alignment mode is set and a single object is selected, the object is rotated about the world axis rather than the object axis. Change #3 passes in the alignment type from the gizmo to the rotate function when the rotate function is called.

Change #4 comments out code that appears to be rotating the gizmo's deltaAngle based on the object's axis. This code has no effect when multiple objects are selected because multiple object selections are always rotated about the world axis and have mObjectMat set to the identity matrix. However, single object selection has mObjectMat set to the object's transform. It looks like object axis rotation changes may already be handled in the rotate function, so it may be necessary to comment it out in on3DMouseDragged. However, I'm not sure if it is safe or necessary to comment out this code.

Change #5 is another area where the rotate function is called and the alignment mode parameter is passed in. This shouldn't cause any changes to the functionality of using TransformSelection.


The above changes have the effect that single object selection don't have a gradual angle error when rotating in World alignment mode, with other rotation types being the same. However, I'm still not completely sure if change #4 is OK.