real time asset pipeline updates
by steven pearce · in Torque 3D Professional · 01/31/2010 (1:22 pm) · 12 replies
As I make changes to models and maps, I'm not getting those changes in T3D. I've had to completely close down T3D and reopen it and reload the objects. Is there something I'm missing? Is it a Mac issue?
About the author
#2
Not sure whether this problem is with the particular change notification mechanism used here (BSD kevents) and whether using the Cocoa File Manager API for this will work more reliably. Will try my luck later.
PS: BTW, this rules my "not affecting models" statement above wrong.
//Edit:
So turns out that the source of the problem is my usage of kevent which does not work that way. The current code directly monitors the directory for changes which, however, will not be synchronized to file access time changes of files it contains.
So, apparently, with kevent the only option is to monitor each file in a directory individually.
The only other option OSX seems to provide is a Cocao-based API that relies on applications voluntarily notifying the system about changes they have made to files.... uggh...
02/01/2010 (7:40 am)
Ok, the core problem seems to be that OSX is not triggering file change notifications reliably, i.e. Torque is never notified about the files being changed. I have to save out the same file 10 times or so until finally the change notification triggers. I've seen similar behavior in other Mac applications that rely on change notifications (I have committed a good deal of incomplete revisions thanks to this).Not sure whether this problem is with the particular change notification mechanism used here (BSD kevents) and whether using the Cocoa File Manager API for this will work more reliably. Will try my luck later.
PS: BTW, this rules my "not affecting models" statement above wrong.
//Edit:
So turns out that the source of the problem is my usage of kevent which does not work that way. The current code directly monitors the directory for changes which, however, will not be synchronized to file access time changes of files it contains.
So, apparently, with kevent the only option is to monitor each file in a directory individually.
The only other option OSX seems to provide is a Cocao-based API that relies on applications voluntarily notifying the system about changes they have made to files.... uggh...
#3
02/01/2010 (9:50 am)
You have a deeper knowledge of the problem and possible solution than myself...whatever additional info and help you can provide is much appreciated.
#4
02/01/2010 (5:18 pm)
Same problem here with Vista. It only updates the models when the T3D window is minimized.
#5
02/01/2010 (8:51 pm)
Interesting...I haven't tried minimizing then checking if they've updated.
#6
Short status update: I've rewritten the Mac-side filesystem code to use per-file notifications. While these now trigger with complete reliability, the problem is that they are too 'perfect', i.e. as soon as a process starts writing to a file, Torque will be notified about the file change. However, for most files, a sequence of write operations will take place on a file before the file is actually fully updated and in a state to be reloaded, so the reloading takes place too early and fails. Working on a solution to this.
Directory-level notifications make this easier since they don't monitor for individual file writes but rather react to file attribute changes.
02/04/2010 (10:28 am)
Short status update: I've rewritten the Mac-side filesystem code to use per-file notifications. While these now trigger with complete reliability, the problem is that they are too 'perfect', i.e. as soon as a process starts writing to a file, Torque will be notified about the file change. However, for most files, a sequence of write operations will take place on a file before the file is actually fully updated and in a state to be reloaded, so the reloading takes place too early and fails. Working on a solution to this.
Directory-level notifications make this easier since they don't monitor for individual file writes but rather react to file attribute changes.
#7
Ok, rewrote once more and works reliably now. Minimizing the window won't help since the previous implementation definitely was at least unreliable.
My fault was not to right away choose the right API out of the usual variety of APIs offered by OSX to do the same thing in different ways. Finally stuck with the FSEventStream API rather than the BSD stuff or the 'voluntary' File Manager nonsense. In retrospect I wonder what kept me from doing this right away. The implementation is clean and simple.
The changes will probably be rolled out during the beta cycle.
BTW, there's also a crash in the OpenGL with texture reloading, apparently due to the code trying to load the data into a released texture object. Fixing this next.
02/05/2010 (3:13 pm)
Ok, rewrote once more and works reliably now. Minimizing the window won't help since the previous implementation definitely was at least unreliable.
My fault was not to right away choose the right API out of the usual variety of APIs offered by OSX to do the same thing in different ways. Finally stuck with the FSEventStream API rather than the BSD stuff or the 'voluntary' File Manager nonsense. In retrospect I wonder what kept me from doing this right away. The implementation is clean and simple.
The changes will probably be rolled out during the beta cycle.
BTW, there's also a crash in the OpenGL with texture reloading, apparently due to the code trying to load the data into a released texture object. Fixing this next.
#8
02/07/2010 (8:29 pm)
I'm glad it worked, but I do not have a clue what you are talking about or where to begin to implement the fix you are talking about....sorry.
#9
As stated, the changes will roll out during the beta cycle. I haven't posted the changes since they are rather large.
It's still not perfect (depending on how an application handles file writes, the notification may still trigger too early and end up in a failed reload), but the notifications do appear to trigger reliably.
02/07/2010 (8:38 pm)
As stated, the changes will roll out during the beta cycle. I haven't posted the changes since they are rather large.
It's still not perfect (depending on how an application handles file writes, the notification may still trigger too early and end up in a failed reload), but the notifications do appear to trigger reliably.
#10
02/08/2010 (5:52 pm)
Maybe you can set a timer to perform the reload after a few seconds then? The filesystem event notifies of a change, and then a timer delays the real work. Unless there is a proper way, like checking for files being closed or having no other users with a lock.
#11
02/08/2010 (7:38 pm)
Do you happen to know how to create/alter that timer?
#12
I had indeed tried the other way you suggest with checking for file locks (more precisely, getting an exclusive write file lock so that the notifier knew when the write op had finished) but there is no useful file locking mechanism on OSX. Darwin has "advisory locks", i.e. applications may choose to set and respect locks many applications seem to not do it so this is useless functionality.
02/09/2010 (2:22 am)
Yeah, a timer does indeed help some. There's already such a timer built into this but maybe I increase the latency somewhat. Currently it's at three seconds.I had indeed tried the other way you suggest with checking for file locks (more precisely, getting an exclusive write file lock so that the notifier knew when the write op had finished) but there is no useful file locking mechanism on OSX. Darwin has "advisory locks", i.e. applications may choose to set and respect locks many applications seem to not do it so this is useless functionality.
Associate Rene Damm
Can confirm the problem for textures but not for models on OSX. Looking into this.