Network and downloading

1. Loading and saving files using URLs

All methods in our engine take URL as the parameter, not just a FileName. Although in most cases you can also pass a filename (absolute or relative to the current directory), and it will also work as expected.

All loading and saving routines (for 3D models, images, sounds, and all other resources) automatically deal with URLs. To actually load network URLs (like http or https) you only need to set CastleDownload.EnableNetwork to true.

To directly load or save your own binary file formats (as ObjectPascal TStream):

  • To load, use a simple Download function. It automatically handles all the details for you, including downloading from network (if EnableNetwork), and returns you a TStream that contains the resource indicated by the URL.

    An example examples/tools/castle_download.lpr uses this to implement a simple command-line downloading tool (like wget) using the engine.

  • To save, use URLSaveStream function. Right now, it can only save to a local file, so it merely translates a URL to local filename and creates a TFileStream for you. Still, it's a good idea to use it, to uniformly deal with URLs throughout your application.

If you want to read or write text files from an URL, use TTextReader and TTextWriter.

2. Supported protocols

2.1. Downloading from the network: http and https

Our engine can automatically download data from the network. All you have to do is set global EnableNetwork variable (from CastleDownload unit) to true. By default it is false, because for now the downloads are not user-friendly — they are blocking (we wait for them to finish, there's no way to cancel a download). This will be improved in the future, and eventually downloading from network may be enabled by default.

Internally, we use FpHttpClient unit, which supports http and (since FPC 3.0.2) https.

In order for the https to work, make sure that OpenSSL library is available. On Windows, you will probably want to place the appropriate DLLs alongside your exe file. You can find these DLLs inside the engine tools/build-tool/data/external_libraries/ subdirectory. These DLLs are also automatically included when packaging your application using the build tool, if you include <dependency name="Https" /> in your CastleEngineManifest.xml.

Note that (almost) all of the parameters and attributes in the engine API are URLs. So you can refer to network resources (http, https) anywhere, and it should "just work".

For example, your game level data may be actually downloaded from the network when loading level. To do this, level.xml may use an http protocol when referring to a scene. Like this:

<?xml version="1.0"?>
<level
  name="pits"
  type="Level"
  scene="https://raw.githubusercontent.com/castle-engine/castle-game/master/data/levels/fountain/fountain_final.x3dv"
  title="The Pits of Azeroth"
  placeholders="blender"
/>

and the scene with all associated resources will be downloaded.

Inside 3D models (like X3D, VRML and others), you can use network resources, for example you can Inline them (add a 3D model from another file), you can make Anchor to them, you can refer to textures and sounds and scripts and everything else from the network. Relative URLs are always resolved with respect to the containing document.

2.2. Loading local files: file

To load simple files from disk, just use a file URL. Of course this works regardless of the EnableNetwork value. These are just local filenames encoded as URL. CastleURIUtils contains routines to operate on URLs (and more general URIs), including converting between regular filenames and URLs with file: protocol.

  • Use this to convert a FileName (relative or absolute) to an absolute URL.

    URL := FilenameToURISafe(FileName);
  • Use this to convert something that may be a FileName or URL to an URL. This is safer than FilenameToURISafe(...), in that it will never touch something that already is an URL. On the other hand, there are some obscure cases (when a relative filename starts with a component with colon inside) when it may think that it has URL, while in fact it has a filename that should be converted to file: URL.
    URL := AbsoluteURI(FileNameOrURL);
  • Use this to convert URL back to a FileName. When the URL is a file: protocol, it will decode back the simple filename. Right now, URL without protocol is also returned back as a simple filename. When the URL uses a different protocol (like http), returns empty string.
    FileName := URIToFilenameSafe(URL);

See reference of FilenameToURISafe, AbsoluteURI, URIToFilenameSafe. See >doc/miscellaneous_notes/uri_filename.txt for more internal comments.

If you read/write filenames from/to Lazarus classes, for example if you use Lazarus TOpenDialog.FileName or TSaveDialog.FileName, use the UTF-8 variants instead: URIToFilenameSafeUTF8 and FilenameToURISafeUTF8. That is because Lazarus uses UTF-8 for all strings (as opposed to FPC RTL that uses system encoding).

2.3. Loading data files: castle-data

(Since Castle Game Engine >= 6.5).

This protocol allows to load data files of your project. Loading from the castle-data:/images/my_image.png is equivalent to using ApplicationData in code and loading from the ApplicationData('images/my_image.png').

During development of a normal cross-plaform game, the data files are simply things inside the data subdirectory of your project. See documentation about the data directory.

The data location is usually (at least on desktop systems) just a regular directory on disk. So loading files from castle-data:/xxx protocol usually boils down to loading from a file:/xxx protocol. But you can adjust ApplicationDataOverride to host your data files wherever you want, this way data files may even be loaded from http location.

2.4. Embedded data: data

data is a special protocol that doesn't refer to an external resource. Instead, the complete data URI contains the contents. This allows to embed various resources (like textures, sounds, other 3D models) inside a parent file. For example instead of referring to the texture filename from 3D model — you can embed the actual texture contents inside 3D model file. This is sometimes a very nice feature (it makes the file easier to distribute).

See data: URI specification. Our engine includes a tool examples/tools/to_data_uri.lpr that can turn any file into a data URI, and you can use such data URI everywhere where we expect URL.

Wherever our engine, or X3D, says that it expects a URL — you can use data URI to provide the contents "right there", without using any additional file.

Demos of using data URI are inside our VRML/X3D demo models, see in particular x3d/data_uri.x3dv.

2.5. (Internal) Android assets: castle-android-assets

This protocol is called castle-android-assets or (deprecated name) assets. It is only available on Android.

Used to access data files in an Android application. "Asset" files live inside your application's .apk file, together with your compiled game. The build tool will copy the data directory of your game to Android assets. For example, file that was in data/my_texture.png in your source code can be accessed (from the Android app) using the URL assets:/my_texture.png.

You should never explicitly use this protocol name, as it does not work on other platforms than Android. Instead, use ApplicationData to refer to your data files from code. The ApplicationData will always return an absolute URL to the data file location on current platform. On Android it will start with castle-android-assets:/... but you should treat this as an internal detail.

3. Dialog windows that support URLs

If you use TCastleWindow, it gives you a ready TCastleWindowCustom.FileDialog that takes and returns URLs.

If you use Lazarus with TCastleControl, we advise to use our dialog components: TCastleOpenDialog, TCastleSaveDialog, TCastleOpen3DDialog, TCastleOpenPictureDialog, TCastleSavePictureDialog. You can also continue using standard Lazarus dialog components. Our routines (almost) always handle a filename instead of an URL, or you can explicitly convert between filenames and URLs using functions mentioned earlier.

4. Notes about terminology: URI vs URL

URI is a more general term. URI uniquely identifies a resource but does not necessarily tell us how to load (download) or save (upload) it. We have many routines in CastleURIUtils unit that process URIs (strings), they use the more general term URI. They complement standard FPC URIParser routines.

URL is a specific type of URI that also tells you how to load or save the resource. For example http and file protocols define URLs. Most of our routines that load or save use the term URL.

Things get a little more cloudy when you realize there's also data URI scheme. It's not precisely an URL (it's not an address of a resource), but you can load it (since the URI itself contains the resource). And we support it fully (our Download method loads it automatically). Admittedly, this means that our loading routines should rather use the term URL or data URI, but that's just long and (for those who don't use data URI) confusing, so for simplicity we just keep (over-)using the term URL. Also, other standards (like CSS and X3D and VRML) allow placing data URIs inside fields called url.

If you enjoy reading about Internet terminology, note that we use in our engine also URNs (another subtype of URI). They are used by X3D external prototypes, see X3D extensions introduction.