File subsystem
From NeoWiki
The file subsystem provides an abstraction of the underlying OS-specific file system, package (archive) files and network resources, and allows the game to treat all these in a uniform manner without worrying about platform-specific problems. All I/O is done through streams, matching the STL stream interfaces and improving these by providing services for endianess abstraction and text/binary mode I/O. The file subsystem streams also provide the base functionality for the networking subsystem which is also stream-based.
The file subsystem acts on a single virtual file system with a specified root path in the underlying OS file system. This enables you to use the same paths in all tools when developing your game as you later use in the actual game, eliminating all potential path errors when developing on multiple machines (artists, programmers and testers all have the same layout in the virtual file system and all tools act in this virtual file system).
Packages (file archives) are completely integrated in the virtual file system and are used transparently to the application. Real files override package files by default, but packages can be configured to override in a specific order, making patching a simple task of droppping a higher-priority package in the correct directory (later patches can override these by even higher assigned priority).
Table of contents |
Stream I/O
The IStream and OStream provide the base stream interfaces for input and output, respectively. All streams are derived from the StreamBase, providing the base functionality for endianess and text/binary mode control.
Endianess
Each stream has an associated endianess, either little endian or big endian, denoting the byte order of multi-byte data. All streams default to little endian order, and you can use the setEndianess method to change the file endianess. The endianess is only important when dealing with binary files or when reading unicode text files.
If the endianess of a stream and the system endianess does not match, the stream classes automatically switches the byte order when reading multi-byte data, allowing you to use a single code path for reading data regardless of the byteorder combination of stream and system. This is very important when dealing with network streams, as it allows you to write a single code path and not have to worry about what combination of systems that are communicating.
To control the endinaness (byte order) of a stream you use the setByteorder method, and to query the current endinaness use getByteorder.
Text/binary mode
Each stream has an associated text/binary mode flag.
In text mode, all stream I/O is based on string I/O, so when streaming a 32-bit integer with value 42 to a text-mode stream will output the string "42" (without the quotes, naturally) to the stream.
In binary mode, all stream I/O is done in raw byte arrays and the size of the data read or written determines how many bytes is read/written to/from the stream. Streaming the same 32-bit integer to a binary stream will output four bytes to the stream in the byte order set on the stream.
To control the text/binary mode of a stream you use setBinary and to query the current mode you use isBinary.
By abstracting the text/binary I/O into the stream classes controlled by a flag allows, for example, the chunk I/O subsystem to use the same code path for reading and writing both text-based and binary-based chunk files.
State
Each stream has a state associated matching the STL stream state. You can query and control the state with the methods good, bad (but not ugly ;-)), fail, eof and clear.
Note that, for example, when dealing with network streams over non-blocking UDP/TCP sockets you probably need to call clear before (or directly after) reading as you usually read until there is no more data, causing the eof (End Of File) flag to be set. Not calling clear before reading more incoming data will then cause the read request to fail since the stream has the eof flag set.
I/O
Input and output to/from streams are done either by streaming operators << and >>, or by raw read and write calls. The preferred method is the stream operators, as this makes the I/O completely transparent to endianess and text/binary mode issues.
The normal STL stream modifiers can be used to control the stream I/O, such as std::endl, std::hex and others. See the STL reference for more information on stream control modifiers.
Packages
A package is a file archive providing a local file system subtree, and the actual package management is handled by package codecs. Currently the engine provides built-in support for ZIP packages, but it is very easy to provide other (proprietary) codec implementations. Package files and which codec to use is identified by the file extension (for ZIP packages, the extensions .zip and .ZIP are recognized).
With "local subtree" we mean that if a package has a file with (local) path "/foo/bar/file.type", and the package itself has the path "/some/path/archive.zip" in the engine virtual file system, the file "file.type" provided by the package has the full path in the engine virtual file system of "/some/path/foo/bar/file.type".
Files in packages are used automatically and transparently by the engine when the application requests to open the corresponding file. When a ambiguity occurs, such as the real file system and a package provides a file with the same path, or two packages provides files with the same path, the priority between them determines which file gets used.
The default priority is that the real file system has higher priority over any package, and that all packages have the same priority. If two packages provide the same file and both packages have the same priority, the package with the lowest alphabetical sort order gets used.
Packages only gets loaded and searched for files when needed, so in the example above the package in "/some/path" will only be initialized when the first request for a file in (at least) that paths is made, and that request is not fulfilled by previous higher priority packages or the real file system.
Since packages are used transparently, you usually don't have to deal with any API for them. However, the priority of packages is set by a special "dot file" named ".priority" inside the package. All priorities are integers with the value 0 representing the lowest possible priority and 100 representing the priority of the real file system. Package have by default a priority of 50. In the ".priority" file, a single integer should be present. This file is read when a package is first opened.
Networked resources
You can request packages to be downloaded over a network protocol such as HTTP, useful for automatic patching and in web deployments. It can also be used for centralized version management during development. The protocol is an implementation of the PackageNetworkProtocol protocol class, and the engine provides built-in support for fetching packages over the HTTP protocol through the HTTPPackageProtocol class.
When you request a networked package to be managed by the engine file system with a call to insertNetworkPackage, the engine first checks if the file has previously been fetched over the same protocol and only downloads the file if there is an updated version available on the network server (this must be supported by the package network protocol requested). It then proceeds with downloading the package file if needed (updated, missing or invalid previous downloaded package file).
You can receive progress callbacks if you provide a pointer to a PackageNetworkCallback object, useful for progress bar updates and other maintenance tasks.
File subsystem
The engine virtual file system is managed by and accessed through the FileSystem singleton object. An application usually initializes the file system by three steps:
- Add any needed package codecs
- Add any needed network package protocols
- Set the root path
To add package codecs you use the insertCodec method. The object given to this method will be maintained and deallocated by the engine. Other methods related to package codecs are getCodecs and getCodecExtensions.
To add a package network protocol you use the insertProtocol method (given protocol object will be maintained and deallocated by the engine). Other related methods are getProtocols.
File system layout
The root of the virtual file system (VFS) is the path in the real underlying OS filesystem where the VFS is "mounted". All files in the given directory of the OS file system will be present in the root directory of the VFS, as are all files residing in the root of any packages present in this directory (such packages will always be parsed, naturally, since they can potentially contain any requested file). By using a single root directory, tools and game can reference files inside the VFS with a unified path, no matter of where in the actual OS file system the VFS resides. This makes maintenance between multiple workstations and multiple tools during development a simple task, and reduces errors such as "texture 'C:/foo/bar.png' not found". To set the root path, use the setRootPath method (query it with getRootPath). The root path probably will be read from a project/installation configuration file, registry or command line argument, or calculated from the application executable path to allow customizable installation paths and development workstation setups.
Files/Directories
To open a file or directory in the VFS, without having to worry about potential package files and priorities, use the openFile and openDirectory. Returned objects should be deallocated by the caller. If the file or directory is not found, a null pointer is returned to indicate the file/directory not found error.
The returned "file" is an IOStream, even if the requested open mode is in or out only. However, only the requested in/out mode will be valid on the returned stream. If the file was not found in the real file system or in any package, it will be created if the open mode say so (see the STL reference for combinations of open modes and their meaning).
The returned "directory" is a view inside the VFS, and provides access to file in both the real file system and in any packages providing files in the requested directory. This can potentially be very useful as it eliminates the need to enumerate potential packages and iterating over them. The directory provides the same methods to open files and subdirectories, and to query information, as the file system.
You can create a new path in the file system with the makePath method.
To remove a file, use removeFile (can also remove all files of the given path in all packages). To remove a package file, use removePackageFile.
Queries
You can query the file system for information such as if a file exists (isFile), to get all files in a given path (getFiles) and to get all subdirectories in a given path (getSubdirs]). To get files with a matching extension in a given path you can use getMatchingFiles.
All query methods can also be used on directory objects returned by openDirectory.
Special streams
Memory buffer stream
For I/O to and from a memory buffer the engine provides the BufferStream class. It automatically handles buffer resizes when writing over the current buffer limits and supports all the operations on normal IOStreams. Note that the initial buffer given to the constructor will be adopted by the object and managed internally, so it must be allocated on the heap with the new[] operator. It also means the buffer pointer is not guaranteed to remain static, so don't store this in a local variable, instead use the getBuffer method.
Debug streams
The engine offers a debug stream to output to the Microsoft Visual Studio debugger output window, implemented in the MSVCDebugStream class. It is present even in non-Windows builds (empty dummy implementation doing nothing) so it is safe to use even without surrounding #ifdef:s. This debug stream might be most useful as a log target.
Standard streams
The standard in, out and error streams are wrapped in the engine classes CIn, COut, CErr.
