Persisting Data

(this is an experimental new feature, expect enhancements and changes)

Introduction

As defined in the glossary, to Persist is to convert data into the storage format most appropriate for the container type, and save a copy of this for rapid lookup in the future. This is of great potential benefit where the creation or transfer of the original data source takes some time.

This is not to be confused with the file Cache.

Usage

Any Data Source has a method .persist(). The only option that you will need to pick is a TTL, the number of seconds that the persisted version lasts before expiry (leave as None for no expiry). This creates a local copy in the persist directory, which may be in "~/.intake/persist, but can be configured.

Each container type (dataframe, array, …) will have its own implementation of persistence, and a particular file storage format associated. The call to .persist() may take arguments to tune how the local files are created, and in some cases may require additional optional packages to be installed.

Example:

cat = intake.load_catalog('mycat.yaml')  # load a remote cat
source = cat.csvsource()  # source pointing to remote data
source.persist()

source = cat.csvsource()  # future use now gives local intake_parquet.ParquetSource

To control whether a catalog will automatically give you the persisted version of a source in this way using the argument persist_mode, e.g., to ignore locally persisted versions, you could have done:

cat = intake.load_catalog('mycat.yaml', persist_mode='never')
or
source = cat.csvsource(persist_mode='never')

Note that if you give a TTL (in seconds), then the original source will be accessed and a new persisted version written transparently when the old persisted version has expired.

Note that after persisting, the original source will have source.has_been_persisted == True and the persisted source (i.e., the one loaded from local files) will have source.is_persisted == True.

Export

A similar concept to Persist, Export allows you to make a copy of some data source, in the format appropriate for its container, and place this data-set in whichever location suits you, including remote locations. This functionality (source.export()) does not touch the persist store; instead, it returns a YAML text representation of the output, so that you can put it into a catalog of your own. It would be this catalog that you share with other people.

Note that “exported” data-sources like this do contain the information of the original source they were made from in their metadata, so you can recreate the original source, if you want to, and read from there.

Persisting to Remote

If you are typically running your code inside of ephemoral containers, then persisting data-sets may be something that you want to do (because the original source is slow, or parsing is CPU/memory intensive), but the local storage is not useful. In some cases you may have access to some shared network storage mounted on the instance, but in other cases you will want to persist to a remote store.

The config value 'persist_path', which can also be set by the environment variable INTAKE_PERSIST_PATH can be a remote location such as s3://mybucket/intake-persist. You will need to install the appropriate package to talk to the external storage (e.g., s3fs, gcsfs, pyarrow), but otherwise everything should work as before, and you can access the persisted data from any container.

The Persist Store

You can interact directly with the class implementing persistence:

from intake.container.persist import store

This singleton instance, which acts like a catalog, allows you to query the contents of the instance store and to add and remove entries. It also allows you to find the original source for any given persisted source, and refresh the persisted version on demand.

For details on the methods of the persist store, see the API documentation: intake.container.persist.PersistStore(). Sources in the store carry a lot of information about the sources they were made from, so that they can be remade successfully. This all appears in the source metadata. The sources use the “token” of the original data source as their key in the store, a value which can be found by source._tok for the original source, or can be taken from the metadata of a persisted source.

Note that all of the information about persisted sources is held in a single YAML file in the persist directory (typically /persisted/cat.yaml within the config directory, but see intake.config.conf['persist_path']). This file can be edited by hand if you wanted to, for example, set some persisted source not to expire. This is only recommended for experts.

Future Enhancements

  • CLI functionality to investigate and alter the state of the persist store, similar to the cache commands.

  • Time check-pointing of persisted data, such that you can not only get the “most recent” but any version in the time-series.

  • (eventually) pipeline functionality, whereby a persisted data source depends on another persisted data source, and the whole train can be refreshed on a schedule or on demand.