Attention
Cate has become the ESA Climate Toolbox. This documentation is being discontinued. You can find the documentation on the ESA Climate Toolbox here.
9. Detailed Design
This chapter provides the CCI Toolbox detailed design documentation. Its is generated from the docstrings that are extensively used throughout the Python code.
The documentation is generated for individual modules. Note that this modularisation reflects the effective, internal (and physical) structure of the Python code. This is not the official API, which comprises a relatively stable subset of the components, types, interfaces, and variables describes here and is described in chapter API Reference.
Each top level module documentation in the following sections provides a sub-section Description that provides the module’s purpose, contents, and possibly its usage. Module descriptions may link into Operation Specifications for further explanation and traceability of the detailed design. An optional sub-section Technical Requirements provides a mapping from URD requirements to technical requirements and software features that drove the design of a module. If available, links to verifying unit-tests are given in sub-sections called Verification. The sub-section Components lists all documented, non-private components of a module, including variables, functions, and classes.
9.1. Module cate.core.ds
9.2. Module cate.core.op
9.3. Module cate.core.workflow
9.4. Module cate.core.plugin
9.5. Package cate.conf
9.5.1. Description
This package provides Cate’s configuration API.
It is independent of other ``cate.*``packages and can therefore be used stand-alone.
9.5.2. Verification
The module’s unit-tests are located in
test/conf and may be executed using
$ py.test test/conf --cov=cate/conf
for extra code coverage information.
9.5.3. Components
9.6. Package cate.ops
9.7. Module cate.cli.main
9.7.1. Description
This module provides Cate’s CLI executable.
To use the CLI executable, invoke the module file as a script,
type python3 cate/cli/main.py [ARGS] [OPTIONS]
.
Type python3 cate/cli/main.py –help` for usage help.
The CLI operates on sub-commands.
New sub-commands can be added by inheriting from the Command
class
and extending the Command.REGISTRY
list of known command classes.
9.7.2. Technical Requirements
Extensible CLI with multiple sub-commands
- Description
The CCI Toolbox should only have a single CLI executable that comes with multiple sub-commands instead of maintaining a number of different executables for each purpose. Plugins shall be able to add new CLI sub-commands.
- URD-Source
CCIT-UR-CR0001: Extensibility.
CCIT-UR-A0002: Offer a Command Line Interface (CLI).
Run operations and workflows
- Description
Allow for executing registered operations an workflows composed of operations.
- URD-Source
CCIT-UR-CL0001: Reading and executing script files written in XML or similar
List available data, operations and extensions
- Description
Allow for listing dynamic content including available data, operations and plugin extensions.
- URD-Source
CCIT-UR-E0001: Dynamic extension by the use of plug-ins
Display information about available climate data sources
- Description
Before downloading ECV datasets to the local computer, users shall be able to display information about them, e.g. included variables, total size, spatial and temporal resolution.
- URD-Source
CCIT-UR-DM0009: Holding information of any CCI ECV type
CCIT-UR-DM0010: Attain meta-level status information per ECV type
Synchronize locally cached climate data
- Description
Allow for listing dynamic content including available data, operations and plugin extensions.
- URD-Source
CCIT-UR-DM0006: Access to and ingestion of ESA CCI datasets
9.7.3. Verification
The module’s unit-tests are located in
test/cli/test_main.py
and may be executed using $ py.test test/cli/test_main.py --cov=cate/cli/test_main.py
for extra code coverage information.
9.7.4. Components
- cate.cli.main.CLI_NAME = 'cate'
Name of the Cate CLI executable (=
cate
).
- cate.cli.main.COMMAND_REGISTRY = [<class 'cate.cli.main.DataSourceCommand'>, <class 'cate.cli.main.OperationCommand'>, <class 'cate.cli.main.WorkspaceCommand'>, <class 'cate.cli.main.ResourceCommand'>, <class 'cate.cli.main.RunCommand'>, <class 'cate.cli.main.IOCommand'>, <class 'cate.cli.main.UpdateCommand'>]
Cate plugins may extend this list by their commands during plugin initialisation.
- class cate.cli.main.DataSourceCommand[source]
The
ds
command implements various operations w.r.t. datasets.- classmethod configure_parser_and_subparsers(parser, subparsers)[source]
Configure the given parser and its sub-parsers.
- Overrides of this method must, e.g.::
list_parser = subparsers.add_parser(‘list’, …) # … configure list_parser here, and finally set its “sub_command_function” like so: list_parser.set_defaults(sub_command_function=cls._execute_list)
Sub-command functions shall raise a
CommandError
instance on failure.- Parameters
parser – The command parser to configure.
subparsers – A factory for sub-command parsers.
- classmethod parser_kwargs()[source]
Return parser keyword arguments dictionary passed to a
argparse.ArgumentParser(**parser_kwargs)
call.For the possible keywords in the returned dictionary, refer to https://docs.python.org/3.5/library/argparse.html#argparse.ArgumentParser.
- Returns
A keyword arguments dictionary.
- class cate.cli.main.IOCommand[source]
The
io
command implements various operations w.r.t. supported data and file formats.- classmethod configure_parser_and_subparsers(parser, subparsers)[source]
Configure the given parser and its sub-parsers.
- Overrides of this method must, e.g.::
list_parser = subparsers.add_parser(‘list’, …) # … configure list_parser here, and finally set its “sub_command_function” like so: list_parser.set_defaults(sub_command_function=cls._execute_list)
Sub-command functions shall raise a
CommandError
instance on failure.- Parameters
parser – The command parser to configure.
subparsers – A factory for sub-command parsers.
- classmethod parser_kwargs()[source]
Return parser keyword arguments dictionary passed to a
argparse.ArgumentParser(**parser_kwargs)
call.For the possible keywords in the returned dictionary, refer to https://docs.python.org/3.5/library/argparse.html#argparse.ArgumentParser.
- Returns
A keyword arguments dictionary.
- class cate.cli.main.OperationCommand[source]
The
op
command implements various operations w.r.t. operations.- classmethod configure_parser_and_subparsers(parser, subparsers)[source]
Configure the given parser and its sub-parsers.
- Overrides of this method must, e.g.::
list_parser = subparsers.add_parser(‘list’, …) # … configure list_parser here, and finally set its “sub_command_function” like so: list_parser.set_defaults(sub_command_function=cls._execute_list)
Sub-command functions shall raise a
CommandError
instance on failure.- Parameters
parser – The command parser to configure.
subparsers – A factory for sub-command parsers.
- classmethod parser_kwargs()[source]
Return parser keyword arguments dictionary passed to a
argparse.ArgumentParser(**parser_kwargs)
call.For the possible keywords in the returned dictionary, refer to https://docs.python.org/3.5/library/argparse.html#argparse.ArgumentParser.
- Returns
A keyword arguments dictionary.
- class cate.cli.main.PluginCommand[source]
The
pi
command lists the content of various plugin registry.- classmethod configure_parser_and_subparsers(parser, subparsers)[source]
Configure the given parser and its sub-parsers.
- Overrides of this method must, e.g.::
list_parser = subparsers.add_parser(‘list’, …) # … configure list_parser here, and finally set its “sub_command_function” like so: list_parser.set_defaults(sub_command_function=cls._execute_list)
Sub-command functions shall raise a
CommandError
instance on failure.- Parameters
parser – The command parser to configure.
subparsers – A factory for sub-command parsers.
- classmethod parser_kwargs()[source]
Return parser keyword arguments dictionary passed to a
argparse.ArgumentParser(**parser_kwargs)
call.For the possible keywords in the returned dictionary, refer to https://docs.python.org/3.5/library/argparse.html#argparse.ArgumentParser.
- Returns
A keyword arguments dictionary.
- class cate.cli.main.ResourceCommand[source]
The
res
command implements various operations w.r.t. workspaces.- classmethod configure_parser_and_subparsers(parser, subparsers)[source]
Configure the given parser and its sub-parsers.
- Overrides of this method must, e.g.::
list_parser = subparsers.add_parser(‘list’, …) # … configure list_parser here, and finally set its “sub_command_function” like so: list_parser.set_defaults(sub_command_function=cls._execute_list)
Sub-command functions shall raise a
CommandError
instance on failure.- Parameters
parser – The command parser to configure.
subparsers – A factory for sub-command parsers.
- classmethod parser_kwargs()[source]
Return parser keyword arguments dictionary passed to a
argparse.ArgumentParser(**parser_kwargs)
call.For the possible keywords in the returned dictionary, refer to https://docs.python.org/3.5/library/argparse.html#argparse.ArgumentParser.
- Returns
A keyword arguments dictionary.
- class cate.cli.main.RunCommand[source]
The
run
command is used to invoke registered operations and JSON workflows.- classmethod configure_parser(parser)[source]
Configure parser, i.e. make any required
parser.add_argument(*args, **kwargs)
calls. See https://docs.python.org/3.5/library/argparse.html#argparse.ArgumentParser.add_argument- Parameters
parser – The command parser to configure.
- execute(command_args)[source]
Execute this command.
The command’s arguments in command_args are attributes namespace returned by
argparse.ArgumentParser.parse_args()
. Also refer to to https://docs.python.org/3.5/library/argparse.html#argparse.ArgumentParser.parse_argsexecute``implementations shall raise a ``CommandError
instance on failure.- Parameters
command_args – The command’s arguments.
- classmethod parser_kwargs()[source]
Return parser keyword arguments dictionary passed to a
argparse.ArgumentParser(**parser_kwargs)
call.For the possible keywords in the returned dictionary, refer to https://docs.python.org/3.5/library/argparse.html#argparse.ArgumentParser.
- Returns
A keyword arguments dictionary.
- class cate.cli.main.UpdateCommand[source]
The
update
command is used to update an existing cate environment to a specific or the latest cate version.- classmethod configure_parser(parser)[source]
Configure parser, i.e. make any required
parser.add_argument(*args, **kwargs)
calls. See https://docs.python.org/3.5/library/argparse.html#argparse.ArgumentParser.add_argument- Parameters
parser – The command parser to configure.
- execute(command_args)[source]
Execute this command.
The command’s arguments in command_args are attributes namespace returned by
argparse.ArgumentParser.parse_args()
. Also refer to to https://docs.python.org/3.5/library/argparse.html#argparse.ArgumentParser.parse_argsexecute``implementations shall raise a ``CommandError
instance on failure.- Parameters
command_args – The command’s arguments.
- classmethod parser_kwargs()[source]
Return parser keyword arguments dictionary passed to a
argparse.ArgumentParser(**parser_kwargs)
call.For the possible keywords in the returned dictionary, refer to https://docs.python.org/3.5/library/argparse.html#argparse.ArgumentParser.
- Returns
A keyword arguments dictionary.
- class cate.cli.main.WorkspaceCommand[source]
The
ws
command implements various operations w.r.t. workspaces.- classmethod configure_parser_and_subparsers(parser, subparsers)[source]
Configure the given parser and its sub-parsers.
- Overrides of this method must, e.g.::
list_parser = subparsers.add_parser(‘list’, …) # … configure list_parser here, and finally set its “sub_command_function” like so: list_parser.set_defaults(sub_command_function=cls._execute_list)
Sub-command functions shall raise a
CommandError
instance on failure.- Parameters
parser – The command parser to configure.
subparsers – A factory for sub-command parsers.
- classmethod parser_kwargs()[source]
Return parser keyword arguments dictionary passed to a
argparse.ArgumentParser(**parser_kwargs)
call.For the possible keywords in the returned dictionary, refer to https://docs.python.org/3.5/library/argparse.html#argparse.ArgumentParser.
- Returns
A keyword arguments dictionary.
9.8. Package cate.webapi
This package implements Cate’s WebAPI service.
9.9. Package cate.util
9.9.1. Description
The cate.util
package provides application-independent utility functions.
This package is independent of other ``cate.*``packages and can therefore be used stand-alone.
9.9.2. Verification
The module’s unit-tests are located in
test/util and may be executed using
$ py.test test/util --cov=cate/util
for extra code coverage information.
9.9.3. Components
9.10. Module cate.util.cache
9.10.1. Description
This module defines the Cache
class which represents a general-purpose cache.
A cache is configured by a CacheStore
which is responsible for storing and reloading cached items.
The default cache stores are
Every cache has capacity in physical units defined by the CacheStore
. When the cache capacity is exceeded
a replacement policy for cached items is applied until the cache size falls below a given ratio of the total capacity.
The default replacement policies are
This package is independent of other ``cate.*``packages and can therefore be used stand-alone.
9.10.2. Components
- class cate.util.cache.Cache(store=<cate.util.cache.MemoryCacheStore object>, capacity=1000, threshold=0.75, policy=<function _policy_lru>, parent_cache=None)[source]
An implementation of a cache. See https://en.wikipedia.org/wiki/Cache_algorithms
- class cate.util.cache.CacheStore[source]
Represents a store to which cached values can be stored into and restored from.
- abstract can_load_from_key(key) bool [source]
Test whether a stored value representation can be loaded from the given key. :rtype:
bool
:param key: the key :return: True, if so
- abstract discard_value(key, stored_value)[source]
Discard a value from it’s storage. :param key: the key :param stored_value: the stored representation of the value
- abstract load_from_key(key)[source]
Load a stored value representation of the value and its size from the given key. :param key: the key :return: a 2-element sequence containing the stored representation of the value and it’s size
- class cate.util.cache.FileCacheStore(cache_dir: str, ext: str)[source]
Simple file store for values which can be written and read as bytes, e.g. encoded PNG images.
- can_load_from_key(key) bool [source]
Test whether a stored value representation can be loaded from the given key. :rtype:
bool
:param key: the key :return: True, if so
- discard_value(key, stored_value)[source]
Discard a value from it’s storage. :param key: the key :param stored_value: the stored representation of the value
- load_from_key(key)[source]
Load a stored value representation of the value and its size from the given key. :param key: the key :return: a 2-element sequence containing the stored representation of the value and it’s size
- class cate.util.cache.MemoryCacheStore[source]
Simple memory store.
- can_load_from_key(key) bool [source]
Test whether a stored value representation can be loaded from the given key. :rtype:
bool
:param key: the key :return: True, if so
- discard_value(key, stored_value)[source]
Clears the value in the given stored_value. :param key: the key :param stored_value: the stored representation of the value
- load_from_key(key)[source]
Load a stored value representation of the value and its size from the given key. :param key: the key :return: a 2-element sequence containing the stored representation of the value and it’s size
- cate.util.cache.POLICY_LFU(item)
Discard Least Frequently Used first
- cate.util.cache.POLICY_LRU(item)
Discard Least Recently Used items first
- cate.util.cache.POLICY_MRU(item)
Discard Most Recently Used first
- cate.util.cache.POLICY_RR(item)
Discard items by Random Replacement
9.11. Module cate.util.cli
- class cate.util.cli.Command[source]
Represents a (sub-)command of a command-line interface.
- classmethod configure_parser(parser: ArgumentParser) None [source]
Configure parser, i.e. make any required
parser.add_argument(*args, **kwargs)
calls. See https://docs.python.org/3.5/library/argparse.html#argparse.ArgumentParser.add_argument- Parameters
parser (
ArgumentParser
) – The command parser to configure.
- execute(command_args: Namespace) None [source]
Execute this command.
The command’s arguments in command_args are attributes namespace returned by
argparse.ArgumentParser.parse_args()
. Also refer to to https://docs.python.org/3.5/library/argparse.html#argparse.ArgumentParser.parse_argsexecute``implementations shall raise a ``CommandError
instance on failure.- Parameters
command_args (
Namespace
) – The command’s arguments.
- classmethod new_monitor() Monitor [source]
Create a new console progress monitor.
- Returns
a new Monitor instance.
- classmethod parser_kwargs() dict [source]
Return parser keyword arguments dictionary passed to a
argparse.ArgumentParser(**parser_kwargs)
call.For the possible keywords in the returned dictionary, refer to https://docs.python.org/3.5/library/argparse.html#argparse.ArgumentParser.
- Returns
A keyword arguments dictionary.
- exception cate.util.cli.CommandError(message)[source]
An error type signaling command-line errors.
- Parameters
message – Error message
- class cate.util.cli.NoExitArgumentParser(*args, **kwargs)[source]
Special
argparse.ArgumentParser
that never directly exits the current process. It raises anExitException
instead.
- class cate.util.cli.SubCommandCommand[source]
- classmethod configure_parser(parser: ArgumentParser) None [source]
Add a new sub-parsers to the given parser. Call
configure_parser_and_subparsers
with the new sub-parsers.- Parameters
parser (
ArgumentParser
) – The command parser to configure.
- abstract classmethod configure_parser_and_subparsers(parser, subparsers)[source]
Configure the given parser and its sub-parsers.
- Overrides of this method must, e.g.::
list_parser = subparsers.add_parser(‘list’, …) # … configure list_parser here, and finally set its “sub_command_function” like so: list_parser.set_defaults(sub_command_function=cls._execute_list)
Sub-command functions shall raise a
CommandError
instance on failure.- Parameters
parser – The command parser to configure.
subparsers – A factory for sub-command parsers.
- cate.util.cli.run_main(name: str, description: str, version: str, command_classes: Sequence[Command], license_text: Optional[str] = None, docs_url: Optional[str] = None, error_message_trimmer=None, args: Optional[Sequence[str]] = None) int [source]
A CLI’s entry point function.
To be used in your own code as follows:
>>> if __name__ == '__main__': >>> sys.exit(run_main(...))
- Return type
int
- Parameters
name (
str
) – The program’s name.description (
str
) – The program’s description.version (
str
) – The program’s version string.command_classes – The CLI commands.
license_text (
str
) – An optional license text.docs_url (
str
) – An optional documentation URL.error_message_trimmer – An optional callable (str)->str that trims error message strings.
args – list of command-line arguments. If not passed, sys.argv[1:] is used.
- Returns
An exit code where
0
stands for success.
9.12. Package cate.util.im
9.13. Package cate.util.web
9.13.1. Description
The cate.util.web
package provides application-independent utility functions for implementing
JSON-RPC web services using the
Tornado web server.
This package is independent of other cate.*``packages, but it depends on the external ``tornado
package.
9.13.2. Verification
The module’s unit-tests are located in
test/util/web and may be executed using
$ py.test test/util/web --cov=cate/util/web
for extra code coverage information.