Other Classes

Cache Types

intake.source.cache.FileCache(driver, spec)

Cache specific set of files

intake.source.cache.DirCache(driver, spec[, ...])

Cache a complete directory tree

intake.source.cache.CompressedCache(driver, spec)

Cache files extracted from downloaded compressed source

intake.source.cache.DATCache(driver, spec[, ...])

Use the DAT protocol to replicate data

intake.source.cache.CacheMetadata(*args, ...)

Utility class for managing persistent metadata stored in the Intake config directory.

class intake.source.cache.FileCache(driver, spec, catdir=None, cache_dir=None, storage_options={})

Cache specific set of files

Input is a single file URL, URL with glob characters or list of URLs. Output is a specific set of local files.

class intake.source.cache.DirCache(driver, spec, catdir=None, cache_dir=None, storage_options={})

Cache a complete directory tree

Input is a directory root URL, plus a depth parameter for how many levels of subdirectories to search. All regular files will be copied. Output is the resultant local directory tree.

class intake.source.cache.CompressedCache(driver, spec, catdir=None, cache_dir=None, storage_options={})

Cache files extracted from downloaded compressed source

For one or more remote compressed files, downloads to local temporary dir and extracts all contained files to local cache. Input is URL(s) (including globs) pointing to remote compressed files, plus optional decomp, which is “infer” by default (guess from file extension) or one of the key strings in intake.source.decompress.decomp. Optional regex_filter parameter is used to load only the extracted files that match the pattern. Output is the list of extracted files.

class intake.source.cache.DATCache(driver, spec, catdir=None, cache_dir=None, storage_options={})

Use the DAT protocol to replicate data

For details of the protocol, see https://docs.datproject.org/ The executable dat must be available.

Since in this case, it is not possible to access the remote files directly, this cache mechanism takes no parameters. The expectation is that the url passed by the driver is of the form:

dat://<dat hash>/file_pattern

where the file pattern will typically be a glob string like “*.json”.

class intake.source.cache.CacheMetadata(*args, **kwargs)

Utility class for managing persistent metadata stored in the Intake config directory.

keys() a set-like object providing a view on D's keys
pop(k[, d]) v, remove specified key and return the corresponding value.

If key is not found, d is returned if given, otherwise KeyError is raised.

update([E, ]**F) None.  Update D from mapping/iterable E and F.

If E present and has a .keys() method, does: for k in E: D[k] = E[k] If E present and lacks .keys() method, does: for (k, v) in E: D[k] = v In either case, this is followed by: for k, v in F.items(): D[k] = v

Auth

intake.auth.secret.SecretAuth(*args, **kwargs)

A very simple auth mechanism using a shared secret

intake.auth.secret.SecretClientAuth(secret)

Matching client auth plugin to SecretAuth

class intake.auth.secret.SecretAuth(*args, **kwargs)

A very simple auth mechanism using a shared secret

Parameters
secret: str

The string that must be matched in the requests. If None, a random UUID is generated and logged.

key: str

Header entry in which to seek the secret

allow_access(header, source, catalog)

Is the given HTTP header allowed to access given data source

Parameters
header: dict

The HTTP header from the incoming request

source: CatalogEntry

The data source the user wants to access.

catalog: Catalog

The catalog object containing this data source.

allow_connect(header)

Is the requests header given allowed to talk to the server

Parameters
header: dict

The HTTP header from the incoming request

class intake.auth.secret.SecretClientAuth(secret, key='intake-secret')

Matching client auth plugin to SecretAuth

Parameters
secret: str

The string that must be included requests.

key: str

HTTP Header key for the shared secret

get_headers()

Returns a dictionary of HTTP headers for the remote catalog request.

Containers

intake.container.dataframe.RemoteDataFrame(...)

Dataframe on an Intake server

intake.container.ndarray.RemoteArray(*args, ...)

nd-array on an Intake server

intake.container.semistructured.RemoteSequenceSource(...)

Sequence-of-things source on an Intake server

class intake.container.dataframe.RemoteDataFrame(*args, **kwargs)

Dataframe on an Intake server

read()

Load entire dataset into a container and return it

to_dask()

Return a dask container for this data source

class intake.container.ndarray.RemoteArray(*args, **kwargs)

nd-array on an Intake server

read()

Load entire dataset into a container and return it

read_partition(i)

Return a part of the data corresponding to i-th partition.

By default, assumes i should be an integer between zero and npartitions; override for more complex indexing schemes.

to_dask()

Return a dask container for this data source

class intake.container.semistructured.RemoteSequenceSource(*args, **kwargs)

Sequence-of-things source on an Intake server

read()

Load entire dataset into a container and return it

to_dask()

Return a dask container for this data source

Server

intake.cli.server.server.IntakeServer(catalog)

Main intake-server tornado application

intake.cli.server.server.ServerInfoHandler(...)

Basic info about the server

intake.cli.server.server.SourceCache()

Stores DataSources requested by some user

intake.cli.server.server.ServerSourceHandler(...)

Open or stream data source

class intake.cli.server.server.IntakeServer(catalog)

Main intake-server tornado application

class intake.cli.server.server.ServerInfoHandler(application: Application, request: HTTPServerRequest, **kwargs: Any)

Basic info about the server

initialize(cache, catalog, auth)
class intake.cli.server.server.SourceCache

Stores DataSources requested by some user

peek(uuid)

Get the source but do not change the last access time

class intake.cli.server.server.ServerSourceHandler(application: Application, request: HTTPServerRequest, **kwargs: Any)

Open or stream data source

The requests “action” field (open|read) specified what the request wants to do. Open caches the source and created an ID for it, read uses that ID to reference the source and read a partition.

get()

Access one source’s info.

This is for direct access to an entry by name for random access, which is useful to the client when the whole catalog has not first been listed and pulled locally (e.g., in the case of pagination).

initialize(catalog, cache, auth)

GUI

intake.interface.base.Base([visible, ...])

Base class for composable panel objects that make up the GUI.

intake.interface.base.BaseSelector([...])

Base class for capturing selector logic.

intake.interface.base.BaseView([visible, ...])

intake.interface.catalog.add.FileSelector([...])

Panel interface for picking files

intake.interface.catalog.add.URLSelector(...)

Panel interface for inputting a URL to a remote catalog

intake.interface.catalog.add.CatAdder([...])

Panel for adding new cats from local file or remote

intake.interface.catalog.search.Search(...)

Search panel for searching a list of catalogs

intake.interface.source.defined_plots.Plots([...])

Panel for displaying pre-defined plots from catalog.

class intake.interface.base.Base(visible=True, visible_callback=None, logo=False)

Base class for composable panel objects that make up the GUI.

Parameters
children: list of panel objects

children that will be used to populate the panel when visible

panel: panel layout object

instance of a panel layout (row or column) that contains children when visible

watchers: list of param watchers

watchers that are set on children - cleaned up when visible is set to false.

visible: bool

whether or not the instance should be visible. When not visible panel is empty.

logobool, opt

whether to show the intake logo in a panel to the left of the main panel. Default is False

classmethod from_state(state)

Create a new object from a serialized exising object.

setup()

Should instantiate widgets, set children, and set watchers

unwatch()

Get rid of any lingering watchers and remove from list

property visible

Whether or not the instance should be visible.

class intake.interface.base.BaseSelector(visible=True, visible_callback=None, logo=False)

Base class for capturing selector logic.

Parameters
preprocess: function

run on every input value when creating options

widget: panel widget

selector widget which this class keeps uptodate with class properties

add(items)

Add items to options

property items

Available items to select from

property labels

Labels of items in widget

property options

Options available on the widget

remove(items)

Remove items from options

property selected

Value selected on the widget

class intake.interface.base.BaseView(visible=True, visible_callback=None, logo=False)
class intake.interface.catalog.add.FileSelector(filters=['yaml', 'yml'], done_callback=None, **kwargs)

Panel interface for picking files

The current path is stored in .path and the current selection is stored in .url.

Parameters
filters: list of string

extentions that are included in the list of files - correspond to catalog extensions.

done_callback: func, opt

called when the object’s main job has completed. In this case, selecting a file.

Attributes
url: str

path to local catalog file

children: list of panel objects

children that will be used to populate the panel when visible

panel: panel layout object

instance of a panel layout (row or column) that contains children when visible

watchers: list of param watchers

watchers that are set on children - cleaned up when visible is set to false.

setup()

Should instantiate widgets, set children, and set watchers

property url

Path to local catalog file

class intake.interface.catalog.add.URLSelector(**kwargs)

Panel interface for inputting a URL to a remote catalog

The inputted URL is stored in .url.

Attributes
url: str

url to remote files (including protocol)

children: list of panel objects

children that will be used to populate the panel when visible

panel: panel layout object

instance of a panel layout (row or column) that contains children when visible

watchers: list of param watchers

watchers that are set on children - cleaned up when visible is set to false.

setup()

Should instantiate widgets, set children, and set watchers

property url

URL to remote files (including protocol)

class intake.interface.catalog.add.CatAdder(done_callback=None, **kwargs)

Panel for adding new cats from local file or remote

Parameters
done_callback: function with cat as input

function that is called when the “Add Catalog” button is clicked.

Attributes
cat_url: str

url to remote files or path to local files. Depends on active tab

cat: catalog

catalog object initialized from from cat_url

children: list of panel objects

children that will be used to populate the panel when visible

panel: panel layout object

instance of a panel layout (row or column) that contains children when visible

watchers: list of param watchers

watchers that are set on children - cleaned up when visible is set to false.

add_cat(arg=None)

Add cat and close panel

property cat

Catalog object initialized from from cat_url

property cat_url

URL to remote files or path to local files. Depends on active tab.

setup()

Should instantiate widgets, set children, and set watchers

class intake.interface.catalog.search.Search(done_callback: callable)

Search panel for searching a list of catalogs

Parameters
done_callback: function with cats as input

function that is called when new cats have been generated via the search functionality

class intake.interface.source.defined_plots.Plots(source=None, **kwargs)

Panel for displaying pre-defined plots from catalog.

Parameters
source: intake catalog entry, or list of same

source to describe in this object

edit_callback: callback to alert that plot has been edited
Attributes
has_plots: bool

whether the source has plots defined

options: list

plots options defined on the source

selected: str

name of selected plot

children: list of panel objects

children that will be used to populate the panel when visible

panel: panel layout object

instance of a panel layout (row or column) that contains children when visible

watchers: list of param watchers

watchers that are set on children - cleaned up when visible is set to false.

property has_plots

Whether the source has plots defined

property options

Plots options defined on the source

property selected

Name of selected plot

setup()

Should instantiate widgets, set children, and set watchers