collada import problem
by SylHar · in Torque 3D Beginner · 10/21/2009 (5:29 am) · 15 replies
Hi.
I'm trying Torque3d right now, and I've got a problem with some collada file.
When trying to import them, the program simply crash. no error message, nothing.
The file has been create from code, not converted by a modeling plugin. it pass the coherency checker for collada file and it is quite big ( 5mo )
I'm not sure what should I do. I've bug reported this, but I've been told to post it there so,...
If you want the file for a test, tell me and I will send it.
I'm trying Torque3d right now, and I've got a problem with some collada file.
When trying to import them, the program simply crash. no error message, nothing.
The file has been create from code, not converted by a modeling plugin. it pass the coherency checker for collada file and it is quite big ( 5mo )
I'm not sure what should I do. I've bug reported this, but I've been told to post it there so,...
If you want the file for a test, tell me and I will send it.
#2
10/21/2009 (3:11 pm)
I has the same problem in Softimage. All I had to do is make sure that the Materials COLLADA check-box is selected. If I didn't choose that, it would crash on me.
#3
10/22/2009 (3:14 am)
Hi SylHar - I'd definitely like to take a look at that model if you can email it to me.
#4
Short answer: the model has too many nodes.
Long answer: Torque uses a fixed size bit vector to indicate whether a particular node is animated by a given sequence. This is part of a scheme used to minimise memory usage by only storing node transforms that actually change during the sequence. Eg. If nodes 3, 4, and 5 were animated, we would only set bits 3,4,5 in the vector.
The size of this bit vector is (arbitrarily) set at 2048, meaning an animated model can have a maximum of 2048 nodes. You can easily change this by changing the MAX_TS_SET_DWORDS value at the top of tsIntegerSet.h. In your case, it would have to be at least 195 (195 32-bit words gives up to 6240 nodes).
A better solution would be to make this vector dynamic.
10/22/2009 (11:40 pm)
The model triggers an assert (in debug builds) when imported.Short answer: the model has too many nodes.
Long answer: Torque uses a fixed size bit vector to indicate whether a particular node is animated by a given sequence. This is part of a scheme used to minimise memory usage by only storing node transforms that actually change during the sequence. Eg. If nodes 3, 4, and 5 were animated, we would only set bits 3,4,5 in the vector.
The size of this bit vector is (arbitrarily) set at 2048, meaning an animated model can have a maximum of 2048 nodes. You can easily change this by changing the MAX_TS_SET_DWORDS value at the top of tsIntegerSet.h. In your case, it would have to be at least 195 (195 32-bit words gives up to 6240 nodes).
A better solution would be to make this vector dynamic.
#5
I can't test that now ( I'm making these tests with the demo version of torque 3d so I don't have the .h file ) But if you can import the mesh, that good enough for me.
While I'm at it, I have few questions about how the collada is handled:
- Most of our meshes use the reference feature of collada: we had one 'scene' collada file that reference and position others collada file.
This feature don't seems to be supported right ?
- After few try, I don't have any texture on my meshes ( even if I change the collada, put them in the same folder,...)
Like you have seen, We use a simple common profile with only one texture. Is there other thing to create to get them ?
- I've tried with a very small scene, with everything included in the file, everything triangulated. But a certain number of element don't appear. It seems that each element appear only once, even if it placed multiple time. We're again in the case of reference, but in the case it's reference of geometry inside the same file. Is that feature supported ?
Edit: ( I've just check, I use instance Node is these case, while the first time I create a node with an instance geometry in it)
Is instance Node not supported ?
10/23/2009 (5:06 am)
thank.I can't test that now ( I'm making these tests with the demo version of torque 3d so I don't have the .h file ) But if you can import the mesh, that good enough for me.
While I'm at it, I have few questions about how the collada is handled:
- Most of our meshes use the reference feature of collada: we had one 'scene' collada file that reference and position others collada file.
This feature don't seems to be supported right ?
- After few try, I don't have any texture on my meshes ( even if I change the collada, put them in the same folder,...)
Like you have seen, We use a simple common profile with only one texture. Is there other thing to create to get them ?
- I've tried with a very small scene, with everything included in the file, everything triangulated. But a certain number of element don't appear. It seems that each element appear only once, even if it placed multiple time. We're again in the case of reference, but in the case it's reference of geometry inside the same file. Is that feature supported ?
Edit: ( I've just check, I use instance Node is these case, while the first time I create a node with an instance geometry in it)
Is instance Node not supported ?
#6
This is via the URL attribute of the <instance_visual_scene> and <instance_geometry> elements, right? You are correct, this is not yet supported. It's not a major task to get it working, but it probably won't make it into T3D 1.1.
I took a look at the file you sent me, and the textures are specified with relative paths. This is perfectly ok, and the T3D collada loader tries to help you by using these relative paths in the generated materials.cs file.
Take a look at the paths in this file and make sure your textures are where they should be (or change the paths to suit).
instance_node is definitely supported, so I'm not sure why geometry instances are missing. Could you take a look in the ShapeEditor and check that the nodes instantiated using instance_node are present in the shape hierarchy? Note that 3space requires that node names are unique, so the loader will automatically append a suffix to the end of multiply-instantiated nodes.
10/25/2009 (11:04 pm)
Quote:Most of our meshes use the reference feature of collada: we had one 'scene' collada file that reference and position others collada file.
This feature don't seems to be supported right ?
This is via the URL attribute of the <instance_visual_scene> and <instance_geometry> elements, right? You are correct, this is not yet supported. It's not a major task to get it working, but it probably won't make it into T3D 1.1.
Quote:After few try, I don't have any texture on my meshes ( even if I change the collada, put them in the same folder,...)
Like you have seen, We use a simple common profile with only one texture. Is there other thing to create to get them ?
I took a look at the file you sent me, and the textures are specified with relative paths. This is perfectly ok, and the T3D collada loader tries to help you by using these relative paths in the generated materials.cs file.
Take a look at the paths in this file and make sure your textures are where they should be (or change the paths to suit).
Quote:I've tried with a very small scene, with everything included in the file, everything triangulated. But a certain number of element don't appear. It seems that each element appear only once, even if it placed multiple time. We're again in the case of reference, but in the case it's reference of geometry inside the same file. Is that feature supported ?
Edit: ( I've just check, I use instance Node is these case, while the first time I create a node with an instance geometry in it)
Is instance Node not supported ?
instance_node is definitely supported, so I'm not sure why geometry instances are missing. Could you take a look in the ShapeEditor and check that the nodes instantiated using instance_node are present in the shape hierarchy? Note that 3space requires that node names are unique, so the loader will automatically append a suffix to the end of multiply-instantiated nodes.
#7
We use only <instance_node>
in the first node, we reference a node in a url outside the file
in the second, we reference a node inside the file
both don't seems to work, both show a node in the shape editor ( in the shape editor I see no difference in the hierarchy between the working node and the other)
I'm doing some test for the texture. Right now event with the correct path ( in same folder, with correct value in the collada, I don't see them appear)
EDIT: I forgot to precise: It seams that if there a instance_node with a reference to another file, the collada can't be loaded ( the menu never appear)
10/26/2009 (8:14 am)
Here part of the collada file with reference.We use only <instance_node>
<node type="NODE" id="placeholder3393" name="placeholder3393">
<matrix>1 0 0 466.305 0 1 0 503.535 0 0 1 0.241586 0 0 0 1 </matrix>
<node type="NODE" id="model3394" name="model3394">
<matrix>1 0 0 0 0 1 0 0 0 0 1 0 0 0 0 1 </matrix>
<instance_node url="../../../old_street/decorations_street/sidewalk/old_sidewalk_around.dae#old_sidewalk_around.egg"/>
</node>
</node>
<node type="NODE" id="placeholder3395" name="placeholder3395">
<matrix>1 0 0 463.256 0 1 0 494.39 0 0 1 0.241586 0 0 0 1 </matrix>
<node type="NODE" id="model3396" name="model3396">
<matrix>1 0 0 0 0 1 0 0 0 0 1 0 0 0 0 1 </matrix>
<instance_node url="#model3394"/>in the first node, we reference a node in a url outside the file
in the second, we reference a node inside the file
both don't seems to work, both show a node in the shape editor ( in the shape editor I see no difference in the hierarchy between the working node and the other)
I'm doing some test for the texture. Right now event with the correct path ( in same folder, with correct value in the collada, I don't see them appear)
EDIT: I forgot to precise: It seams that if there a instance_node with a reference to another file, the collada can't be loaded ( the menu never appear)
#8
Inside the texture problem:
I tried to create a collada file with an folder-architecture like google stretch-up. ( main.dae texture/lost_of_texture.png)
( I can send you the file if you want).
If I import that file, the import stop ( never had the menu)
but if I move the .dae and import then it's working ( without the texture of course).
I should add that there a lot of texture referenced ( 117 files for 93mo)
I tried with a simple model using one texture only, same problem. ( texture in the same folder)
( file to your disposal if need it)
10/28/2009 (6:34 am)
Ok, some more information.Inside the texture problem:
I tried to create a collada file with an folder-architecture like google stretch-up. ( main.dae texture/lost_of_texture.png)
( I can send you the file if you want).
If I import that file, the import stop ( never had the menu)
but if I move the .dae and import then it's working ( without the texture of course).
I should add that there a lot of texture referenced ( 117 files for 93mo)
I tried with a simple model using one texture only, same problem. ( texture in the same folder)
( file to your disposal if need it)
#9
That means either a huge freaking skinned mesh which will perform horribly.
Or is a static model with as many as 6240 sub-meshes which means at least one draw call each.... again horrible for performance.
10/28/2009 (10:55 pm)
So is having 6240 nodes a normal thing for a single model?That means either a huge freaking skinned mesh which will perform horribly.
Or is a static model with as many as 6240 sub-meshes which means at least one draw call each.... again horrible for performance.
#10
Odd. I wouldn't expect the file with the external reference to load, but the internal reference to an instance_node should work just fine.
Could you email me this file? BTW I very much appreciate these test cases; every collada exporter has its own quirks but the T3D importer needs to be able to handle them all!
The key to getting materials to work is the materials.cs file that is generated in the same folder as the collada model. The T3D importer tries to be clever about guessing texture paths, but given the wide variance (ie. crap) that collada exporters generate it is almost impossible to get it right every time. Fortunately you can always fixup the materials.cs file manually, or via the Material Editor to point at the correct texture path.
If you wouldn't mind sending me the simple model, or posting the section of the DAE file that shows the texture paths I will take a look to see if the T3D importer could do a better job.
10/29/2009 (12:48 am)
Quote:in the first node, we reference a node in a url outside the file
in the second, we reference a node inside the file
both don't seems to work, both show a node in the shape editor ( in the shape editor I see no difference in the hierarchy between the working node and the other)
Odd. I wouldn't expect the file with the external reference to load, but the internal reference to an instance_node should work just fine.
Could you email me this file? BTW I very much appreciate these test cases; every collada exporter has its own quirks but the T3D importer needs to be able to handle them all!
Quote:I tried with a simple model using one texture only, same problem. ( texture in the same folder)
( file to your disposal if need it)
The key to getting materials to work is the materials.cs file that is generated in the same folder as the collada model. The T3D importer tries to be clever about guessing texture paths, but given the wide variance (ie. crap) that collada exporters generate it is almost impossible to get it right every time. Fortunately you can always fixup the materials.cs file manually, or via the Material Editor to point at the correct texture path.
If you wouldn't mind sending me the simple model, or posting the section of the DAE file that shows the texture paths I will take a look to see if the T3D importer could do a better job.
#11
10/29/2009 (6:31 am)
I send you a zip file with a complete scene, with both texture link and local instance_node.Quote:every collada exporter has its own quirks but the T3D importer needs to be able to handle them all!I'm exactly in the opposite view. And It's really hard to create an exporter that work with every importer ! So don't hesitate to point out the too strange thing that we do.
Quote:So is having 6240 nodes a normal thing for a single model?We're talking of a whole city here. The number of node just show that. I do hope that either the engine or the user will make adjustment to improve the performance after the import.
That means either a huge freaking skinned mesh which will perform horribly.
Or is a static model with as many as 6240 sub-meshes which means at least one draw call each.... again horrible for performance.
#12
I took a look at the file you sent.
1. First, it wouldn't load at all => not even to give a summary in the collada import gui. Examining the file in a text editor revealed that all of the XML element names were prefixed with "ns0:". Not sure what this was for? I did a find/replace to remove this string, and the file loaded into Torque........
2. and crashed. The <instance_node> URLs were set to (for example):
This is interpreted by the collada loader as an external reference. Internal references are just the node's ID prefixed with '#' (should have been #model21972 in the case above). Again, a simple find/replace fixed this problem, and the model now loads into Torque without issues.......
3. The building model is textured as expected, and a look at the generated materials.cs file showed the texture filenames to be correct (ie. they were correctly pointing into the 'textures' folder). I'm not sure why you weren't seeing this. Possibly because the collada loader only creates Materials the first time you load the model (since otherwise you might have created/edited the materials with the Material Editor and we don't want to overwrite those changes). You can force the loader to update materials.cs by selecting that option in the collada import gui.
I noticed a couple of minor things in the model you sent:
- almost every model I have encountered so far has used either <triangles>, <polygons> or <polylist> elements to describe the geometry. Your model contained some <tristrips> elements. Initially I had to add support for these to the collada loader to see them (this will be in the next update of T3D).
- I noticed that you had 'normal' and 'glow' textures specified as <image>s in the collada file, but no reference to them in the relevant <effect>s. The following snippet shows how you can specify them, and this will cause the T3D loader to automatically setup the Material for you:
The "FCOLLADA" profile used to specify the normal map is not part of the collada spec, but seems to be in common use (I have seen several exporters use it).
11/01/2009 (11:49 pm)
Quote:I send you a zip file with a complete scene, with both texture link and local instance_node.
I took a look at the file you sent.
1. First, it wouldn't load at all => not even to give a summary in the collada import gui. Examining the file in a text editor revealed that all of the XML element names were prefixed with "ns0:". Not sure what this was for? I did a find/replace to remove this string, and the file loaded into Torque........
2. and crashed. The <instance_node> URLs were set to (for example):
<instance_node url="./models/#model21972" />
This is interpreted by the collada loader as an external reference. Internal references are just the node's ID prefixed with '#' (should have been #model21972 in the case above). Again, a simple find/replace fixed this problem, and the model now loads into Torque without issues.......
3. The building model is textured as expected, and a look at the generated materials.cs file showed the texture filenames to be correct (ie. they were correctly pointing into the 'textures' folder). I'm not sure why you weren't seeing this. Possibly because the collada loader only creates Materials the first time you load the model (since otherwise you might have created/edited the materials with the Material Editor and we don't want to overwrite those changes). You can force the loader to update materials.cs by selecting that option in the collada import gui.
Quote:I'm exactly in the opposite view. And It's really hard to create an exporter that work with every importer ! So don't hesitate to point out the too strange thing that we do.
I noticed a couple of minor things in the model you sent:
- almost every model I have encountered so far has used either <triangles>, <polygons> or <polylist> elements to describe the geometry. Your model contained some <tristrips> elements. Initially I had to add support for these to the collada loader to see them (this will be in the next update of T3D).
- I noticed that you had 'normal' and 'glow' textures specified as <image>s in the collada file, but no reference to them in the relevant <effect>s. The following snippet shows how you can specify them, and this will cause the T3D loader to automatically setup the Material for you:
<library_materials>
<material id="MyMaterial">
<instance_effect url="#MyEffect"/>
</material>
</library_materials>
<library_effects>
<effect id="MyEffect">
<profile_COMMON>
<newparam sid="diffuse-surface">
<surface type="2D">
<init_from>diffuse-image</init_from>
</surface>
</newparam>
<newparam sid="diffuse-sampler">
<sampler2D>
<source>diffuse-surface</source>
</sampler2D>
</newparam>
<newparam sid="normal-surface">
<surface type="2D">
<init_from>normal-image</init_from>
</surface>
</newparam>
<newparam sid="normal-sampler">
<sampler2D>
<source>normal-surface</source>
</sampler2D>
</newparam>
<newparam sid="specular-surface">
<surface type="2D">
<init_from>specular-image</init_from>
</surface>
</newparam>
<newparam sid="specular-sampler">
<sampler2D>
<source>specular-surface</source>
</sampler2D>
</newparam>
<technique sid="common">
<phong>
<diffuse><texture texture="diffuse-sampler" texcoord="TEX0"/></diffuse>
<specular><texture texture="specular-sampler" texcoord="TEX0"/></specular>
</phong>
<extra>
<technique profile="FCOLLADA">
<bump>
<texture texture="normal-sampler" texcoord="TEX0"></texture>
</bump>
</technique>
</extra>
</technique>
</profile_COMMON>
</effect>
</library_effects>The "FCOLLADA" profile used to specify the normal map is not part of the collada spec, but seems to be in common use (I have seen several exporters use it).
#13
In the meantime, it's a perfectly valid xml/collada value. It should not create problems.
2) those where effectively external reference.
It should have both. ( this file has not, I must send you a bad version)
local reference is not something we 'want'. It's something we had to use: in the colladaMaya plugin for maya, each time there is a external reference, it reload the whole file, so we use local reference to stop this problem.
3) I restart my test then. I must I made mistake in the torque editor.
tristrips: yes, we support that. ( and 'trifans' too, but there just aren't any in the files that I send you.)
I'm a little surprised you didn't see that before, I though it was a common way of optimise meshes.
I had just included a option to export only triangles for compatibility purpose.
lots of textures: thank for the report. I should clean this. it's from the time we tried to export our material in colladaFX ( and paused this feature since nobody support it )
ps: Did you intend to support the external reference in the future ? I understand it's not something easy to do, but it allow a lots of possibility.
What we do between other: using the collada as an entire scene/world that place model.
EDIT: I cut out the namespace. I managed to have the material/texture working. thank
11/02/2009 (7:50 am)
1) It's supposed to be a namespace. ns0 is the namespace by default for xml file. I'm not sure why the library we use add this and if we can stop creating it.In the meantime, it's a perfectly valid xml/collada value. It should not create problems.
2) those where effectively external reference.
It should have both. ( this file has not, I must send you a bad version)
local reference is not something we 'want'. It's something we had to use: in the colladaMaya plugin for maya, each time there is a external reference, it reload the whole file, so we use local reference to stop this problem.
3) I restart my test then. I must I made mistake in the torque editor.
tristrips: yes, we support that. ( and 'trifans' too, but there just aren't any in the files that I send you.)
I'm a little surprised you didn't see that before, I though it was a common way of optimise meshes.
I had just included a option to export only triangles for compatibility purpose.
lots of textures: thank for the report. I should clean this. it's from the time we tried to export our material in colladaFX ( and paused this feature since nobody support it )
ps: Did you intend to support the external reference in the future ? I understand it's not something easy to do, but it allow a lots of possibility.
What we do between other: using the collada as an entire scene/world that place model.
EDIT: I cut out the namespace. I managed to have the material/texture working. thank
#14
Many XML parsers do not support namespaces, even though they are valid XML. The parser used by T3D (TinyXML) is one of these.
They were a bit odd is all. Your comment here showed what I would expect to see for external (a path and filename followed by a URL) and internal (a URL only) references. But the instance_node references in the file you sent me had a path followed by a URL ("./models/#model21972"), but no filename. It's not clear to me how the collada loader should interpret that?
Yes - I don't know why other exporters don't use them, but your model was the first I have ever seen that used the <tristrips> element, and I've never seen <trifans> used. By far the most common are <polygons> and <polylist>, even for exporters that have a 'Triangulate' option (they just generate triangle <polygons>).
I do intend to support external references, and it's not a major task to do so. There's just a lot of other stuff in the pipeline for T3D 1.1 so I'm not sure if I'll get to it in time. I'm happy to point you in the right direction in the T3D source if you want to take a crack at it yourself.
11/02/2009 (6:32 pm)
Quote:In the meantime, it's a perfectly valid xml/collada value. It should not create problems.
Many XML parsers do not support namespaces, even though they are valid XML. The parser used by T3D (TinyXML) is one of these.
Quote:those where effectively external reference.
They were a bit odd is all. Your comment here showed what I would expect to see for external (a path and filename followed by a URL) and internal (a URL only) references. But the instance_node references in the file you sent me had a path followed by a URL ("./models/#model21972"), but no filename. It's not clear to me how the collada loader should interpret that?
Quote:tristrips: yes, we support that. ( and 'trifans' too, but there just aren't any in the files that I send you.)
I'm a little surprised you didn't see that before, I though it was a common way of optimise meshes.
Yes - I don't know why other exporters don't use them, but your model was the first I have ever seen that used the <tristrips> element, and I've never seen <trifans> used. By far the most common are <polygons> and <polylist>, even for exporters that have a 'Triangulate' option (they just generate triangle <polygons>).
Quote:Did you intend to support the external reference in the future ? I understand it's not something easy to do, but it allow a lots of possibility.
I do intend to support external references, and it's not a major task to do so. There's just a lot of other stuff in the pipeline for T3D 1.1 so I'm not sure if I'll get to it in time. I'm happy to point you in the right direction in the T3D source if you want to take a crack at it yourself.
#15
It's effectevily a bug. I tried to have a collada with local reference and not external. It seem my exporter don't like that. the file should look like that:
maybe. I have a result in torque3d it was the most important.
11/03/2009 (5:34 am)
Quote:They were a bit odd is all. Your comment here showed what I would expect to see for external (a path and filename followed by a URL) and internal (a URL only) references. But the instance_node references in the file you sent me had a path followed by a URL ("./models/#model21972"), but no filename. It's not clear to me how the collada loader should interpret that?
It's effectevily a bug. I tried to have a collada with local reference and not external. It seem my exporter don't like that. the file should look like that:
<node type="NODE" id="placeholder4110" name="placeholder4110">
<matrix>0.939693 -0.34202 0 547.209 0.34202 0.939693 0 541.643 0 0 1 0.102 0 0 0 1 </matrix>
<node type="NODE" id="model4111" name="model4111">
<matrix>1 0 0 0 0 1 0 0 0 0 1 0 0 0 0 1 </matrix>
<instance_node url="../../../old_street/decorations_street/block/old_block.dae#old_block.egg"/>
</node>
</node>
<node type="NODE" id="placeholder4112" name="placeholder4112">
<matrix>-0.309017 -0.951057 0 549.467 0.951057 -0.309017 0 534.596 0 0 1 0.102 0 0 0 1 </matrix>
<node type="NODE" id="model4113" name="model4113">
<matrix>1 0 0 0 0 1 0 0 0 0 1 0 0 0 0 1 </matrix>
<instance_node url="#model4111"/>Quote:I do intend to support external references, and it's not a major task to do so. There's just a lot of other stuff in the pipeline for T3D 1.1 so I'm not sure if I'll get to it in time. I'm happy to point you in the right direction in the T3D source if you want to take a crack at it yourself.
maybe. I have a result in torque3d it was the most important.
Associate David Montgomery-Blake
David MontgomeryBlake
Note: if you open the console and type "trace(1)" and then load the file, you will see the various errors that I saw when it was parsing the nodes. Typing trace(0) will stop the in-depth trace.
EDIT:
I thought I should also note that he imported the Collada file into Maya, triangulated, and then re-exported it with a similar crash.