Other Classes
Cache Types
|
Cache specific set of files |
|
Cache a complete directory tree |
|
Cache files extracted from downloaded compressed source |
|
Use the DAT protocol to replicate data |
|
Utility class for managing persistent metadata stored in the Intake config directory. |
- class intake.source.cache.FileCache(driver, spec, catdir=None, cache_dir=None, storage_options={})
Cache specific set of files
Input is a single file URL, URL with glob characters or list of URLs. Output is a specific set of local files.
- class intake.source.cache.DirCache(driver, spec, catdir=None, cache_dir=None, storage_options={})
Cache a complete directory tree
Input is a directory root URL, plus a
depth
parameter for how many levels of subdirectories to search. All regular files will be copied. Output is the resultant local directory tree.
- class intake.source.cache.CompressedCache(driver, spec, catdir=None, cache_dir=None, storage_options={})
Cache files extracted from downloaded compressed source
For one or more remote compressed files, downloads to local temporary dir and extracts all contained files to local cache. Input is URL(s) (including globs) pointing to remote compressed files, plus optional
decomp
, which is “infer” by default (guess from file extension) or one of the key strings inintake.source.decompress.decomp
. Optionalregex_filter
parameter is used to load only the extracted files that match the pattern. Output is the list of extracted files.
- class intake.source.cache.DATCache(driver, spec, catdir=None, cache_dir=None, storage_options={})
Use the DAT protocol to replicate data
For details of the protocol, see https://docs.datproject.org/ The executable
dat
must be available.Since in this case, it is not possible to access the remote files directly, this cache mechanism takes no parameters. The expectation is that the url passed by the driver is of the form:
dat://<dat hash>/file_pattern
where the file pattern will typically be a glob string like “*.json”.
- class intake.source.cache.CacheMetadata(*args, **kwargs)
Utility class for managing persistent metadata stored in the Intake config directory.
- keys() a set-like object providing a view on D's keys
- pop(k[, d]) v, remove specified key and return the corresponding value.
If key is not found, d is returned if given, otherwise KeyError is raised.
- update([E, ]**F) None. Update D from mapping/iterable E and F.
If E present and has a .keys() method, does: for k in E: D[k] = E[k] If E present and lacks .keys() method, does: for (k, v) in E: D[k] = v In either case, this is followed by: for k, v in F.items(): D[k] = v
Auth
|
A very simple auth mechanism using a shared secret |
Matching client auth plugin to SecretAuth |
- class intake.auth.secret.SecretAuth(*args, **kwargs)
A very simple auth mechanism using a shared secret
- Parameters
- secret: str
The string that must be matched in the requests. If None, a random UUID is generated and logged.
- key: str
Header entry in which to seek the secret
- allow_access(header, source, catalog)
Is the given HTTP header allowed to access given data source
- Parameters
- header: dict
The HTTP header from the incoming request
- source: CatalogEntry
The data source the user wants to access.
- catalog: Catalog
The catalog object containing this data source.
- allow_connect(header)
Is the requests header given allowed to talk to the server
- Parameters
- header: dict
The HTTP header from the incoming request
- class intake.auth.secret.SecretClientAuth(secret, key='intake-secret')
Matching client auth plugin to SecretAuth
- Parameters
- secret: str
The string that must be included requests.
- key: str
HTTP Header key for the shared secret
- get_headers()
Returns a dictionary of HTTP headers for the remote catalog request.
Containers
Dataframe on an Intake server |
|
|
nd-array on an Intake server |
Sequence-of-things source on an Intake server |
- class intake.container.dataframe.RemoteDataFrame(*args, **kwargs)
Dataframe on an Intake server
- read()
Load entire dataset into a container and return it
- to_dask()
Return a dask container for this data source
- class intake.container.ndarray.RemoteArray(*args, **kwargs)
nd-array on an Intake server
- read()
Load entire dataset into a container and return it
- read_partition(i)
Return a part of the data corresponding to i-th partition.
By default, assumes i should be an integer between zero and npartitions; override for more complex indexing schemes.
- to_dask()
Return a dask container for this data source
Server
Main intake-server tornado application |
|
Basic info about the server |
|
Stores DataSources requested by some user |
|
Open or stream data source |
- class intake.cli.server.server.IntakeServer(catalog)
Main intake-server tornado application
- class intake.cli.server.server.ServerInfoHandler(application: Application, request: HTTPServerRequest, **kwargs: Any)
Basic info about the server
- initialize(cache, catalog, auth)
- class intake.cli.server.server.SourceCache
Stores DataSources requested by some user
- peek(uuid)
Get the source but do not change the last access time
- class intake.cli.server.server.ServerSourceHandler(application: Application, request: HTTPServerRequest, **kwargs: Any)
Open or stream data source
The requests “action” field (open|read) specified what the request wants to do. Open caches the source and created an ID for it, read uses that ID to reference the source and read a partition.
- get()
Access one source’s info.
This is for direct access to an entry by name for random access, which is useful to the client when the whole catalog has not first been listed and pulled locally (e.g., in the case of pagination).
- initialize(catalog, cache, auth)
GUI
|
Base class for composable panel objects that make up the GUI. |
Base class for capturing selector logic. |
|
|
|
Panel interface for picking files |
|
Panel interface for inputting a URL to a remote catalog |
|
Panel for adding new cats from local file or remote |
|
Search panel for searching a list of catalogs |
|
Panel for displaying pre-defined plots from catalog. |
- class intake.interface.base.Base(visible=True, visible_callback=None, logo=False)
Base class for composable panel objects that make up the GUI.
- Parameters
- children: list of panel objects
children that will be used to populate the panel when visible
- panel: panel layout object
instance of a panel layout (row or column) that contains children when visible
- watchers: list of param watchers
watchers that are set on children - cleaned up when visible is set to false.
- visible: bool
whether or not the instance should be visible. When not visible
panel
is empty.- logobool, opt
whether to show the intake logo in a panel to the left of the main panel. Default is False
- classmethod from_state(state)
Create a new object from a serialized exising object.
- setup()
Should instantiate widgets, set
children
, and set watchers
- unwatch()
Get rid of any lingering watchers and remove from list
- property visible
Whether or not the instance should be visible.
- class intake.interface.base.BaseSelector(visible=True, visible_callback=None, logo=False)
Base class for capturing selector logic.
- Parameters
- preprocess: function
run on every input value when creating options
- widget: panel widget
selector widget which this class keeps uptodate with class properties
- add(items)
Add items to options
- property items
Available items to select from
- property labels
Labels of items in widget
- property options
Options available on the widget
- remove(items)
Remove items from options
- property selected
Value selected on the widget
- class intake.interface.base.BaseView(visible=True, visible_callback=None, logo=False)
- class intake.interface.catalog.add.FileSelector(filters=['yaml', 'yml'], done_callback=None, **kwargs)
Panel interface for picking files
The current path is stored in .path and the current selection is stored in .url.
- Parameters
- filters: list of string
extentions that are included in the list of files - correspond to catalog extensions.
- done_callback: func, opt
called when the object’s main job has completed. In this case, selecting a file.
- Attributes
- url: str
path to local catalog file
- children: list of panel objects
children that will be used to populate the panel when visible
- panel: panel layout object
instance of a panel layout (row or column) that contains children when visible
- watchers: list of param watchers
watchers that are set on children - cleaned up when visible is set to false.
- setup()
Should instantiate widgets, set
children
, and set watchers
- property url
Path to local catalog file
- class intake.interface.catalog.add.URLSelector(**kwargs)
Panel interface for inputting a URL to a remote catalog
The inputted URL is stored in .url.
- Attributes
- url: str
url to remote files (including protocol)
- children: list of panel objects
children that will be used to populate the panel when visible
- panel: panel layout object
instance of a panel layout (row or column) that contains children when visible
- watchers: list of param watchers
watchers that are set on children - cleaned up when visible is set to false.
- setup()
Should instantiate widgets, set
children
, and set watchers
- property url
URL to remote files (including protocol)
- class intake.interface.catalog.add.CatAdder(done_callback=None, **kwargs)
Panel for adding new cats from local file or remote
- Parameters
- done_callback: function with cat as input
function that is called when the “Add Catalog” button is clicked.
- Attributes
- cat_url: str
url to remote files or path to local files. Depends on active tab
- cat: catalog
catalog object initialized from from cat_url
- children: list of panel objects
children that will be used to populate the panel when visible
- panel: panel layout object
instance of a panel layout (row or column) that contains children when visible
- watchers: list of param watchers
watchers that are set on children - cleaned up when visible is set to false.
- add_cat(arg=None)
Add cat and close panel
- property cat
Catalog object initialized from from cat_url
- property cat_url
URL to remote files or path to local files. Depends on active tab.
- setup()
Should instantiate widgets, set
children
, and set watchers
- class intake.interface.catalog.search.Search(done_callback: callable)
Search panel for searching a list of catalogs
- Parameters
- done_callback: function with cats as input
function that is called when new cats have been generated via the search functionality
- class intake.interface.source.defined_plots.Plots(source=None, **kwargs)
Panel for displaying pre-defined plots from catalog.
- Parameters
- source: intake catalog entry, or list of same
source to describe in this object
- edit_callback: callback to alert that plot has been edited
- Attributes
- has_plots: bool
whether the source has plots defined
- options: list
plots options defined on the source
- selected: str
name of selected plot
- children: list of panel objects
children that will be used to populate the panel when visible
- panel: panel layout object
instance of a panel layout (row or column) that contains children when visible
- watchers: list of param watchers
watchers that are set on children - cleaned up when visible is set to false.
- property has_plots
Whether the source has plots defined
- property options
Plots options defined on the source
- property selected
Name of selected plot
- setup()
Should instantiate widgets, set
children
, and set watchers