Share your wisdom .. How to manage huge amounts of content?
by Tim Reinermann · in General Discussion · 09/28/2009 (6:29 pm) · 7 replies
Hi everyone,
I wonder how you are managing huge amounts of binary content like lots of big texture files.
The Problem:
As far as I know it is not advisable to store huge content files in SVN as anything thats checked in once, will never be deleted due to Subversions ability to track the whole history of the repository. If modellers replace or rename huge files, the repository will grow very fast. Additionally it could become very annoying if you update your local copy for a few codelines changed ... and have to download 100 MB of content just checked in by a modeller ...
So I thought about alternatives:
1. A simple FTP storage. FTPClients allows to sync local and remote folders.
Pros:
- Modern browsers have integrated FTP Clients.
- Need no Serverside application
- easy to use for modellers
Cons:
- No history at all
- No relation to Subversion. Switching back to an old SVN revision does not switch content to that state ...
- Sync of FTP folder with local folder raises problems, because some parts of an art directory may still be located in SVN (scripts, profiles, ...). FTP sync will always ask you to upload these files ...
- even small modifications to a file will resault in a complete retransmission of the file ...
2. FTPvc
Pros:
- FTP as Repository
- Need no Serverside application
- version control and history
Cons (possible):
- Need to install Clients
- Would FTPvc interfere with SVN?! (did not yet test it)
- Also no relation to Subversion. Switching back to an old SVN revision does not switch content to that state. You have to keep track of Subversion and FTPvs revisions that fit together.
3. Live with exploding SVN repository ...
Nothing that convinces me ... how do you do it? Share your wisdom ^^
regards!
I wonder how you are managing huge amounts of binary content like lots of big texture files.
The Problem:
As far as I know it is not advisable to store huge content files in SVN as anything thats checked in once, will never be deleted due to Subversions ability to track the whole history of the repository. If modellers replace or rename huge files, the repository will grow very fast. Additionally it could become very annoying if you update your local copy for a few codelines changed ... and have to download 100 MB of content just checked in by a modeller ...
So I thought about alternatives:
1. A simple FTP storage. FTPClients allows to sync local and remote folders.
Pros:
- Modern browsers have integrated FTP Clients.
- Need no Serverside application
- easy to use for modellers
Cons:
- No history at all
- No relation to Subversion. Switching back to an old SVN revision does not switch content to that state ...
- Sync of FTP folder with local folder raises problems, because some parts of an art directory may still be located in SVN (scripts, profiles, ...). FTP sync will always ask you to upload these files ...
- even small modifications to a file will resault in a complete retransmission of the file ...
2. FTPvc
Pros:
- FTP as Repository
- Need no Serverside application
- version control and history
Cons (possible):
- Need to install Clients
- Would FTPvc interfere with SVN?! (did not yet test it)
- Also no relation to Subversion. Switching back to an old SVN revision does not switch content to that state. You have to keep track of Subversion and FTPvs revisions that fit together.
3. Live with exploding SVN repository ...
Nothing that convinces me ... how do you do it? Share your wisdom ^^
regards!
#2
\artwork
\engine
\servers
\game
and below \artwork each of the artists has their own folder so they could just checkout their own files.
09/29/2009 (5:56 pm)
We do much the same as Orion and use SVN for it all, and have folders like\artwork
\engine
\servers
\game
and below \artwork each of the artists has their own folder so they could just checkout their own files.
#3
thx for your reply. Unfortunately I did not fully understand your approach. I'm used to have the classical trunk/branches/tags setup for my SVN repositories, where trunk holds the current projectstate including code and art as they were found in a Torque3D Vanilla install.
I see the use of a special art folder, where modellers can place content that is not intended for release, but if its within the same repository it will still grow in size fast, if there is not certain discipline on what and when to commit. Perhaps a seperate Repository would do the job better. Its history could then be trunkated regularly.
I have heard good things about Perforce (lately in the twitter of one of my favourite game projects www.unknownworlds.com/ns2/). But Perforce license prices are a pain compared to a free (or almost free) solution ^^
09/29/2009 (6:35 pm)
Hi Orion,thx for your reply. Unfortunately I did not fully understand your approach. I'm used to have the classical trunk/branches/tags setup for my SVN repositories, where trunk holds the current projectstate including code and art as they were found in a Torque3D Vanilla install.
I see the use of a special art folder, where modellers can place content that is not intended for release, but if its within the same repository it will still grow in size fast, if there is not certain discipline on what and when to commit. Perhaps a seperate Repository would do the job better. Its history could then be trunkated regularly.
I have heard good things about Perforce (lately in the twitter of one of my favourite game projects www.unknownworlds.com/ns2/). But Perforce license prices are a pain compared to a free (or almost free) solution ^^
#4
thx for your reply too. :)
How is the content transfered from the artists folders to the "trunk" ? By Hand, from the artists or an art director or anything of that kind?
09/29/2009 (6:40 pm)
Hi Andy,thx for your reply too. :)
How is the content transfered from the artists folders to the "trunk" ? By Hand, from the artists or an art director or anything of that kind?
#5
yes, the repo will grow,
but you have engineers only check out release/ and eng/,
and artists only check out release/ and art/.
if you want to check out from the root of the repo,
you can also arrange it so that SVN 'checks out' a dummy (empty) folder for certain folders. let's see..
i think what you would do is setup something like this -
in the repo:
trunk/release
trunk/art
trunk/eng
trunk/empty
and then an artist would check out repo::trunk/empty to drive::trunk/eng.
if that makes sense. then when you update trunk/,
the local folder "eng" is empty.
a bit janky, yes.
09/29/2009 (6:41 pm)
i was thinking three folders under trunk. (and branches, & tags ? i'm a bit unfamiliar with those guys).yes, the repo will grow,
but you have engineers only check out release/ and eng/,
and artists only check out release/ and art/.
if you want to check out from the root of the repo,
you can also arrange it so that SVN 'checks out' a dummy (empty) folder for certain folders. let's see..
i think what you would do is setup something like this -
in the repo:
trunk/release
trunk/art
trunk/eng
trunk/empty
and then an artist would check out repo::trunk/empty to drive::trunk/eng.
if that makes sense. then when you update trunk/,
the local folder "eng" is empty.
a bit janky, yes.
#6
The tags folder is something like your release folder I think. Everytime you have a stable version, or reached a milestone you make a copy/branch of the trunk to the tags folder. Unlike branches, tags are not used for further development.
This Image illustrates it quite well
09/30/2009 (3:51 am)
Well branches are a "copy" of the trunk (or a part of it) with their own history from the branching point on. Branching allows to implement larger features that require a lot of commits (which might not build) without interfering with the trunk. When they are finished they have to be merged with the trunk again.The tags folder is something like your release folder I think. Everytime you have a stable version, or reached a milestone you make a copy/branch of the trunk to the tags folder. Unlike branches, tags are not used for further development.
This Image illustrates it quite well
#7
You are also able to create a 1.0 branch and svn cp sub-directories under your trunk as they reach a code/asset freeze.
My repository/Trac instance ends up having a structure like this:
dev/
1.0/
1.0.1/
1.1/
...and so on, with src/, gfx/ and other directories below each.
Just because I'm a lazy git, I also have "commit", "list" and "update" scripts in the root, which save me from typing the password when I'm not using a graphical Subversion client. It seems some GUI clients just don't handle self-signed SSL well.
09/30/2009 (1:28 pm)
I use branching (svn cp src dest) to create release versions when code is locked down for publishing. So when I hit 1.0, I do "svn cp dev 1.0". I prefer "dev" to "trunk" personally. Less typing, still gets the message across :)You are also able to create a 1.0 branch and svn cp sub-directories under your trunk as they reach a code/asset freeze.
My repository/Trac instance ends up having a structure like this:
dev/
1.0/
1.0.1/
1.1/
...and so on, with src/, gfx/ and other directories below each.
Just because I'm a lazy git, I also have "commit", "list" and "update" scripts in the root, which save me from typing the password when I'm not using a graphical Subversion client. It seems some GUI clients just don't handle self-signed SSL well.
Associate Orion Elenzil
Real Life Plus
* release
- this is where the final, shipping versions of all the files go.
* art
- this is where art does their thing: big .psd files, full-res models, etc.
* eng
- this is where engineering does their thing: source code, 3rd party libs, etc.
both art & eng may touch files directly in release,
but at least for eng you probably want the build process (eg ant) to produce build artifacts from eng into release.
the main problem with this is that life sort of sucks for Art,
due to the size of the files. And if the revision control system is a pain to use, people aren't going to use it, and sooner or later you will lose some work.
the artists i've worked with all seemed much happier with Perforce than with SVN, altho as an engineer i prefer SVN. (altho i haven't checked out perforce for a while)