Other Classes¶

Cache Types¶

`intake.source.cache.FileCache`(driver, spec)	Cache specific set of files
`intake.source.cache.DirCache`(driver, spec[, …])	Cache a complete directory tree
`intake.source.cache.CompressedCache`(driver, spec)	Cache files extracted from downloaded compressed source
`intake.source.cache.DATCache`(driver, spec[, …])	Use the DAT protocol to replicate data
`intake.source.cache.CacheMetadata`(*args, …)	Utility class for managing persistent metadata stored in the Intake config directory.

class intake.source.cache.FileCache(driver, spec, catdir=None, cache_dir=None, storage_options={})¶

Cache specific set of files

Input is a single file URL, URL with glob characters or list of URLs. Output is a specific set of local files.

class intake.source.cache.DirCache(driver, spec, catdir=None, cache_dir=None, storage_options={})¶

Cache a complete directory tree

Input is a directory root URL, plus a depth parameter for how many levels of subdirectories to search. All regular files will be copied. Output is the resultant local directory tree.

class intake.source.cache.CompressedCache(driver, spec, catdir=None, cache_dir=None, storage_options={})¶

Cache files extracted from downloaded compressed source

For one or more remote compressed files, downloads to local temporary dir and extracts all contained files to local cache. Input is URL(s) (including globs) pointing to remote compressed files, plus optional decomp, which is “infer” by default (guess from file extension) or one of the key strings in intake.source.decompress.decomp. Optional regex_filter parameter is used to load only the extracted files that match the pattern. Output is the list of extracted files.

class intake.source.cache.DATCache(driver, spec, catdir=None, cache_dir=None, storage_options={})¶

Use the DAT protocol to replicate data

For details of the protocol, see https://docs.datproject.org/ The executable dat must be available.

Since in this case, it is not possible to access the remote files directly, this cache mechanism takes no parameters. The expectation is that the url passed by the driver is of the form:

dat://<dat hash>/file_pattern

where the file pattern will typically be a glob string like “*.json”.

class intake.source.cache.CacheMetadata(*args, **kwargs)¶

Utility class for managing persistent metadata stored in the Intake config directory.

keys(self)¶

pop(self, key, default=None)¶: If key is not found, d is returned if given, otherwise KeyError is raised.

update(self, key, cache_entry)¶: If E present and has a .keys() method, does: for k in E: D[k] = E[k] If E present and lacks .keys() method, does: for (k, v) in E: D[k] = v In either case, this is followed by: for k, v in F.items(): D[k] = v

Auth¶

`intake.auth.secret.SecretAuth`([secret, key])	A very simple auth mechanism using a shared secret
`intake.auth.secret.SecretClientAuth`(secret)	Matching client auth plugin to SecretAuth

class intake.auth.secret.SecretAuth(secret=None, key='intake-secret')¶

A very simple auth mechanism using a shared secret

Parameters

secret: str: The string that must be matched in the requests. If None, a random UUID is generated and logged.
key: str: Header entry in which to seek the secret

allow_access(self, header, source, catalog)¶

Is the given HTTP header allowed to access given data source

Parameters

header: dict: The HTTP header from the incoming request
source: CatalogEntry: The data source the user wants to access.
catalog: Catalog: The catalog object containing this data source.

allow_connect(self, header)¶

Is the requests header given allowed to talk to the server

Parameters

header: dict: The HTTP header from the incoming request

class intake.auth.secret.SecretClientAuth(secret, key='intake-secret')¶

Matching client auth plugin to SecretAuth

Parameters

secret: str: The string that must be included requests.
key: str: HTTP Header key for the shared secret

get_headers(self)¶: Returns a dictionary of HTTP headers for the remote catalog request.

Containers¶

`intake.container.dataframe.RemoteDataFrame`(…)	Dataframe on an Intake server
`intake.container.ndarray.RemoteArray`(url, …)	nd-array on an Intake server
`intake.container.semistructured.RemoteSequenceSource`(…)	Sequence-of-things source on an Intake server

class intake.container.dataframe.RemoteDataFrame(url, headers, **kwargs)¶

Dataframe on an Intake server

read(self)¶: Load entire dataset into a container and return it

to_dask(self)¶: Return a dask container for this data source

class intake.container.ndarray.RemoteArray(url, headers, **kwargs)¶

nd-array on an Intake server

read(self)¶: Load entire dataset into a container and return it

read_partition(self, i)¶

Return a part of the data corresponding to i-th partition.

By default, assumes i should be an integer between zero and npartitions; override for more complex indexing schemes.

to_dask(self)¶: Return a dask container for this data source

class intake.container.semistructured.RemoteSequenceSource(url, headers, **kwargs)¶

Sequence-of-things source on an Intake server

read(self)¶: Load entire dataset into a container and return it

to_dask(self)¶: Return a dask container for this data source

Server¶

`intake.cli.server.server.IntakeServer`(catalog)	Main intake-server tornado application
`intake.cli.server.server.ServerInfoHandler`(…)	Basic info about the server
`intake.cli.server.server.SourceCache`()	Stores DataSources requested by some user
`intake.cli.server.server.ServerSourceHandler`(…)	Open or stream data source

class intake.cli.server.server.IntakeServer(catalog)¶: Main intake-server tornado application

class intake.cli.server.server.ServerInfoHandler(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)¶: Basic info about the server

class intake.cli.server.server.SourceCache¶

Stores DataSources requested by some user

peek(self, uuid)¶: Get the source but do not change the last access time

class intake.cli.server.server.ServerSourceHandler(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)¶

Open or stream data source

The requests “action” field (open|read) specified what the request wants to do. Open caches the source and created an ID for it, read uses that ID to reference the source and read a partition.

get(self)¶

Access one source’s info.

This is for direct access to an entry by name for random access, which is useful to the client when the whole catalog has not first been listed and pulled locally (e.g., in the case of pagination).

write_error(self, status_code, \*\*kwargs)¶

Override to implement custom error pages.

write_error may call write, render, set_header, etc to produce output as usual.

If this error was caused by an uncaught exception (including HTTPError), an exc_info triple will be available as kwargs["exc_info"]. Note that this exception may not be the “current” exception for purposes of methods like sys.exc_info() or traceback.format_exc.

GUI¶

`intake.gui.base.Base`([visible, …])	Base class for composable panel objects that make up the GUI.
`intake.gui.base.BaseSelector`([visible, …])	Base class for capturing selector logic.
`intake.gui.base.BaseView`([visible, …])
`intake.gui.catalog.add.FileSelector`([…])	Panel interface for picking files
`intake.gui.catalog.add.URLSelector`(**kwargs)	Panel interface for inputting a URL to a remote catalog
`intake.gui.catalog.add.CatAdder`([done_callback])	Panel for adding new cats from local file or remote
`intake.gui.catalog.gui.CatGUI`([cats, …])	Top level GUI panel that contains controls and all visible sub-panels
`intake.gui.catalog.search.Search`(cats[, …])	Search panel for searching a list of catalogs
`intake.gui.catalog.search.SearchInputs`(**kwargs)	Input areas to control search parameters
`intake.gui.catalog.select.CatSelector`([…])	The cat selector takes a variety of inputs such as a catalog instance, a path to a catalog, or a list of either of those.
`intake.gui.source.defined_plots.Plots`([source])	Panel for displaying pre-defined plots from catalog.
`intake.gui.source.gui.SourceGUI`([cats, …])	Top level GUI panel that contains controls and all visible sub-panels
`intake.gui.source.description.Description`([…])	Class for displaying a textual description of a data source.
`intake.gui.source.select.SourceSelector`([…])	The source selector takes a variety of inputs such as cats or sources and uses those to populate a select widget containing all the sources.

class intake.gui.base.Base(visible=True, visible_callback=None, logo=False)¶

Base class for composable panel objects that make up the GUI.

Parameters

children: list of panel objects: children that will be used to populate the panel when visible
panel: panel layout object: instance of a panel layout (row or column) that contains children when visible
watchers: list of param watchers: watchers that are set on children - cleaned up when visible is set to false.
visible: bool: whether or not the instance should be visible. When not visible panel is empty.
logobool, opt: whether to show the intake logo in a panel to the left of the main panel. Default is False

classmethod from_state(state)¶: Create a new object from a serialized exising object.

setup(self)¶: Should instantiate widgets, set children, and set watchers

unwatch(self)¶: Get rid of any lingering watchers and remove from list

property visible¶: Whether or not the instance should be visible.

class intake.gui.base.BaseSelector(visible=True, visible_callback=None, logo=False)¶

Base class for capturing selector logic.

Parameters

preprocess: function: run on every input value when creating options
widget: panel widget: selector widget which this class keeps uptodate with class properties

add(self, items)¶: Add items to options

property items¶: Available items to select from

property labels¶: Labels of items in widget

property options¶: Options available on the widget

remove(self, items)¶: Remove items from options

property selected¶: Value selected on the widget

class intake.gui.base.BaseView(visible=True, visible_callback=None, logo=False)¶

class intake.gui.catalog.add.FileSelector(filters=['yaml', 'yml'], done_callback=None, **kwargs)¶

Panel interface for picking files

The current path is stored in .path and the current selection is stored in .url.

Parameters

filters: list of string: extentions that are included in the list of files - correspond to catalog extensions.
done_callback: func, opt: called when the object’s main job has completed. In this case, selecting a file.

Attributes

url: str: path to local catalog file
children: list of panel objects: children that will be used to populate the panel when visible
panel: panel layout object: instance of a panel layout (row or column) that contains children when visible
watchers: list of param watchers: watchers that are set on children - cleaned up when visible is set to false.

setup(self)¶: Should instantiate widgets, set children, and set watchers

property url¶: Path to local catalog file

validate(self, arg=None)¶: Check that inputted path is valid - set validator accordingly

class intake.gui.catalog.add.URLSelector(**kwargs)¶

Panel interface for inputting a URL to a remote catalog

The inputted URL is stored in .url.

Attributes

url: str: url to remote files (including protocol)
children: list of panel objects: children that will be used to populate the panel when visible
panel: panel layout object: instance of a panel layout (row or column) that contains children when visible
watchers: list of param watchers: watchers that are set on children - cleaned up when visible is set to false.

setup(self)¶: Should instantiate widgets, set children, and set watchers

property url¶: URL to remote files (including protocol)

class intake.gui.catalog.add.CatAdder(done_callback=None, **kwargs)¶

Panel for adding new cats from local file or remote

Parameters

done_callback: function with cat as input: function that is called when the “Add Catalog” button is clicked.

Attributes

cat_url: str: url to remote files or path to local files. Depends on active tab
cat: catalog: catalog object initialized from from cat_url
children: list of panel objects: children that will be used to populate the panel when visible
panel: panel layout object: instance of a panel layout (row or column) that contains children when visible
watchers: list of param watchers: watchers that are set on children - cleaned up when visible is set to false.

add_cat(self, arg=None)¶: Add cat and close panel

property cat¶: Catalog object initialized from from cat_url

property cat_url¶: URL to remote files or path to local files. Depends on active tab.

remove_error(self, \*args)¶: Remove error from the widget

setup(self)¶: Should instantiate widgets, set children, and set watchers

tab_change(self, event)¶: When tab changes remove error, and enable widget if on url tab

class intake.gui.catalog.gui.CatGUI(cats=None, done_callback=None, **kwargs)¶

Top level GUI panel that contains controls and all visible sub-panels

This class is responsible for coordinating the inputs and outputs of various sup-panels and their effects on each other.

Parameters

cats: list of catalogs: catalogs used to initalize the select
done_callback: func, opt: called when the object’s main job has completed. In this case, selecting catalog(s).

Attributes

children: list of panel objects: children that will be used to populate the panel when visible
panel: panel layout object: instance of a panel layout (row or column) that contains children when visible
watchers: list of param watchers: watchers that are set on children - cleaned up when visible is set to false.

callback(self, cats)¶: When a catalog is selected, enable widgets that depend on that condition and do done_callback

property cats¶: Cats that have been selected from the select

classmethod from_state(state)¶: Create a new object from a serialized verion of another one.

on_click_search_widget(self, event)¶: When the search control is toggled, set visibility and hand down cats

setup(self)¶: Should instantiate widgets, set children, and set watchers

property visible¶: Whether or not the instance should be visible.

class intake.gui.catalog.search.Search(cats, done_callback=None, **kwargs)¶

Search panel for searching a list of catalogs

Parameters

cats: list of catalogs: catalogs that will be searched
done_callback: function with cats as input: function that is called when new cats have been generated via the search functionality

Attributes

children: list of panel objects: children that will be used to populate the panel when visible
panel: panel layout object: instance of a panel layout (row or column) that contains children when visible
watchers: list of param watchers: watchers that are set on children - cleaned up when visible is set to false.

do_search(self, arg=None)¶: Do search and close panel

setup(self)¶: Should instantiate widgets, set children, and set watchers

class intake.gui.catalog.search.SearchInputs(**kwargs)¶

Input areas to control search parameters

Attributes

text: str: text to use in search. Displayed in text_widget
depth: int or ‘All’: depth of nestedness to use in search. Displayed in depth_widget.
children: list of panel objects: children that will be used to populate the panel when visible
panel: panel layout object: instance of a panel layout (row or column) that contains children when visible
watchers: list of param watchers: watchers that are set on children - cleaned up when visible is set to false.

property depth¶: Depth of nestedness to use in search. Displayed in depth_widget

setup(self)¶: Should instantiate widgets, set children, and set watchers

property text¶: Text to use in search. Displayed in text_widget

class intake.gui.catalog.select.CatSelector(cats=None, done_callback=None, **kwargs)¶

The cat selector takes a variety of inputs such as a catalog instance, a path to a catalog, or a list of either of those.

Once the cat selector is populated with these options, the user can select which catalog(s) are of interest. These catalogs are stored on the selected property of the class.

Parameters

cats: list of catalogs, opt: catalogs used to initalize, can be provided as objects or urls pointing to local or remote catalogs.
done_callback: func, opt: called when the object’s main job has completed. In this case, selecting catalog(s).

Attributes

selected: list of cats: list of selected cats
items: list of cats: list of all the catalog values represented in widget
labels: list of str: list of labels for all the catalog represented in widget
options: dict: dict of widget labels and values (same as dict(zip(self.labels, self.values)))
children: list of panel objects: children that will be used to populate the panel when visible
panel: panel layout object: instance of a panel layout (row or column) that contains children when visible
watchers: list of param watchers: watchers that are set on children - cleaned up when visible is set to false.

collapse_nested(self, cats, max_nestedness=10)¶: Collapse any items that are nested under cats. max_nestedness acts as a fail-safe to prevent infinite looping.

expand_nested(self, cats)¶: Populate widget with nested catalogs

classmethod preprocess(cat)¶: Function to run on each cat input

remove_selected(self, \*args)¶: Remove the selected catalog - allow the passing of arbitrary args so that buttons work. Also remove any nested catalogs.

setup(self)¶: Should instantiate widgets, set children, and set watchers

class intake.gui.source.defined_plots.Plots(source=None, **kwargs)¶

Panel for displaying pre-defined plots from catalog.

Parameters

source: intake catalog entry, or list of same: source to describe in this object

Attributes

has_plots: bool: whether the source has plots defined
instructions_contents: str: instructions to put on the plot selector label
options: list: plots options defined on the source
selected: str: name of selected plot
children: list of panel objects: children that will be used to populate the panel when visible
panel: panel layout object: instance of a panel layout (row or column) that contains children when visible
watchers: list of param watchers: watchers that are set on children - cleaned up when visible is set to false.

property has_plots¶: Whether the source has plots defined

property instructions_contents¶: Instructions to put on the plot selector label

property options¶: Plots options defined on the source

property selected¶: Name of selected plot

setup(self)¶: Should instantiate widgets, set children, and set watchers

class intake.gui.source.gui.SourceGUI(cats=None, sources=None, done_callback=None, **kwargs)¶

Top level GUI panel that contains controls and all visible sub-panels

This class is responsible for coordinating the inputs and outputs of various sup-panels and their effects on each other.

Parameters

cats: list of catalogs, opt: catalogs used to initalize, provided as objects.
sources: list of sources, opt: sources used to initalize, provided as objects.
done_callback: func, opt: called when the object’s main job has completed. In this case, selecting source(s).

Attributes

children: list of panel objects: children that will be used to populate the panel when visible
panel: panel layout object: instance of a panel layout (row or column) that contains children when visible
watchers: list of param watchers: watchers that are set on children - cleaned up when visible is set to false.

callback(self, sources)¶: When a source is selected, enable widgets that depend on that condition and do done_callback

classmethod from_state(state)¶: Create a new object from a serialized exising object.

on_click_plot_widget(self, event)¶: When the plot control is toggled, set visibility and hand down source

setup(self)¶: Should instantiate widgets, set children, and set watchers

property source_instance¶: DataSource from the current selection using current parameters

property sources¶: Sources that have been selected from the source GUI

property visible¶: Whether or not the instance should be visible.

class intake.gui.source.description.Description(source=None, **kwargs)¶

Class for displaying a textual description of a data source.

Parameters

source: intake catalog entry, or list of same: source to describe in this object

Attributes

contents: str: string representation of the source’s description
label: str: label to display at top of panel - contains name of source
children: list of panel objects: children that will be used to populate the panel when visible
panel: panel layout object: instance of a panel layout (row or column) that contains children when visible
watchers: list of param watchers: watchers that are set on children - cleaned up when visible is set to false.

property contents¶: String representation of the source’s description

property label¶: Label to display at top of panel

setup(self)¶: Should instantiate widgets, set children, and set watchers

class intake.gui.source.select.SourceSelector(sources=None, cats=None, done_callback=None, **kwargs)¶

The source selector takes a variety of inputs such as cats or sources and uses those to populate a select widget containing all the sources.

Once the source selector is populated with these options, the user can select which source(s) are of interest. These sources are stored on the selected property of the class.

Parameters

cats: list of catalogs, opt: catalogs used to initalize, provided as objects.
sources: list of sources, opt: sources used to initalize, provided as objects.
done_callback: func, opt: called when the object’s main job has completed. In this case, selecting source(s).

Attributes

selected: list of sources: list of selected sources
items: list of sources: list of all the source values represented in widget
labels: list of str: list of labels for all the sources represented in widget
options: dict: dict of widget labels and values (same as dict(zip(self.labels, self.values)))
children: list of panel objects: children that will be used to populate the panel when visible
panel: panel layout object: instance of a panel layout (row or column) that contains children when visible
watchers: list of param watchers: watchers that are set on children - cleaned up when visible is set to false.

property cats¶: Cats represented in the sources options

setup(self)¶: Should instantiate widgets, set children, and set watchers