[SDL] [OT] Resource file

Bob Pendleton bob at pendleton.com
Wed Feb 20 12:34:44 PST 2008

On 2/19/08, Andre de Leiradella <leiradella at bigfoot.com> wrote:
> Hi All,
> I'm designing a resource file format to hold game data (scripts, images,
> musics, sound effects etc.) and would like some input from the community.
> First, why another resource file? Because I want two things other
> resource files don't offer (at least that I'm aware of):
> A. Fast access through mmap (the file can be mmapped to the process
> address space and used with very little setup)

Well, mmap is a bit faster than reading/writing a file. I'm rather fond of
it myself. Not to mention that having the data in a binary format is very
nice. Do remember that addresses may not stay meaningful when mmaped into
memory. The worst case is when they do stay meaningful on your machine, and
on some other machines, but are not meaningful on every machine. This can
lead you to putting addresses into the file and then having to redo
everything so that you store offsets.

B. Spare space where I could insert the digital signature of the file.
> This space must be filled with zeroes while computing the hash for the
> signature.

That isn't really a big deal, you can always create a zeroed out item in any
resource file that can be used to store the signature after it is computed.
Even better, you can just put the signature in a wrapper so that you don't
actually change the signature of the file by adding the signature to the
file. A simple header consisting of the length of the signature followed by
the signature can be prepended to the file to sign it. Or, even simpler, you
can just append the signature to the file and never care about the format.

The preliminary version is working quite well. Data is accessed by name
> (char *), with a reader that supports basic data input operations being
> returned. Since the file is mmaped, the location of a given chunk of
> data within the file is quickly found via a bsearch call. Data can be
> stored without compression (good for mp3, ogg, jpeg...) or bzipped. The
> resource file can even be part of another file, the most common use
> being to append the resource file to an executable.
> There is also support for transparently using a directory instead of a
> real resource file just like zziplib, and reading entries via SDL_RWops.
> So my questions are:
> 1. Is there any thing bad about mmapping a resource file? The file size
> can easily be greater than 4 GiB.

Well, by allowing your file to be so large you restrict yourself to 64 bit
architectures. In general  32 bit machines can not address more than 4 gigs
of process space. In reality they can rarely address more than 2 gigs of
process space. If you don't care about 32 bit machines then there is really
nothing wrong with what you are doing. Just remember that your addresses and
offsets need to be 64 bits.

2. Are there other resource file formats that provide A and B above?

Well, your "A" requirement is a requirement of the implementation of the
access library and has absolutely nothing to do with the file format. So
basically all and/or no file format gives you that. You could take any
existing file format and create a library for accessing it that uses mmap.
If you look deep down in the file code for your favorite compiler you might
find that ti already uses mmap to implement read and write in which case all
libraries have this feature.

 And your "B" requirement can be met by simply appending the signature to an
existing file or by writing it to a different file, so pretty much all other
file formats meet this requirement.

3. Are there other important characteristics for a resource file format
> I'm missing?

That it is available right now and  you don't have to write it from scratch?

4. Do you have requirements for a new resource file?


The exposed interface so far is:

Hmmm, you aren't being consistent about the use of uint32_t and int. The
interface as written may blow up if you try to do arithmetic on offsets or
lengths because you are mixing unsigned and signed integers for lengths and
offsets. It depends on whether the "int" variables are 32 bits or 64 bits.

Above you mentioned that the file size can easily be bigger than 4 gigs but
you seem to only have 32 bits of offset in this format which restricts you
to <= 4 gigs. You need to changes this interface to use 64 bit offsets.

Bob Pendleton

> /* Seek modes. */
> #define AF_SEEK_SET    0
> #define AF_SEEK_CUR    1
> #define AF_SEEK_END    2
> /* The opaque file type. */
> struct af_file_t;
> typedef struct af_file_t af_file_t;
> /* The opaque reader type. */
> struct af_reader_t;
> typedef struct af_reader_t af_reader_t;
> /* The callback type used to return the address of needed functions,
>    i.e. BZ2_bzDecompress and friends. This allow the application
>    to statically link support libraries as bzip2 or dinamically load them
>    from anywhere in the file system. */
> typedef void *(*af_dynamic_link_t)(const char *library_name, const char
> *function_name);
> /* Information filled by af_entry_get_info and af_iterate. */
> typedef struct {
>     const char *name;
>     uint8_t compression[4];
>     uint32_t compressed_size;
>     uint32_t uncompressed_size;
>     const void *data;
> } af_entry_info_t;
> /* Sets the callback for the load of support libraries. */
> void af_set_dynamic_link_callback(af_dynamic_link_t call_back);
> /* Opens the resource file, possibly with an offset from the
>    beginning of the named file. Size defaults to the number
>    of bytes left in the file starting at offset, but can be specified
>    if the resource file is embedded in the middle of some
>    other file. */
> af_file_t *af_open(const char *name, uint32_t offset, uint32_t size);
> /* Closes the resource file, freeing all allocated resources. */
> void af_close(af_file_t *file);
> /* Calls the process callback for all entries in the resource file. */
> int af_iterate(af_file_t *file, int (*process)(af_entry_info_t *info,
> void *udata), void *udata);
> /* Opens an entry for reading. */
> af_reader_t *af_entry_open(af_file_t *file, const char *name);
> /* Reads size bytes into buffer. */
> int af_entry_read(af_reader_t *reader, void *buffer, int size);
> /* Changes the location of the next read operation. */
> int af_entry_seek(af_reader_t *reader, int offset, int whence);
> /* Closes the reader, freeing all allocated resources. */
> int af_entry_close(af_reader_t *reader);
> /* Returns the current position within the entry. */
> int af_entry_tell(af_reader_t *reader);
> /* Returns information of the entry associated with the reader. */
> int af_entry_get_info(af_file_t *file, af_entry_info_t *info, const char
> *name);
> --------------------8<--------------------
> Cheers,
> Andre
> _______________________________________________
> SDL mailing list
> SDL at lists.libsdl.org
> http://lists.libsdl.org/listinfo.cgi/sdl-libsdl.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.libsdl.org/pipermail/sdl-libsdl.org/attachments/20080220/8f7ff75b/attachment-0008.htm>

More information about the SDL mailing list