9. Detailed Design

This chapter provides the CCI Toolbox detailed design documentation. Its is generated from the docstrings that are extensively used throughout the Python code.

The documentation is generated for individual modules. Note that this modularisation reflects the effective, internal (and physical) structure of the Python code. This is not the official API, which comprises a relatively stable subset of the components, types, interfaces, and variables describes here and is described in chapter API Reference.

Each top level module documentation in the following sections provides a sub-section Description that provides the module’s purpose, contents, and possibly its usage. Module descriptions may link into Operation Specifications for further explanation and traceability of the detailed design. An optional sub-section Technical Requirements provides a mapping from URD requirements to technical requirements and software features that drove the design of a module. If available, links to verifying unit-tests are given in sub-sections called Verification. The sub-section Components lists all documented, non-private components of a module, including variables, functions, and classes.

9.1. Module cate.core.ds

9.2. Module cate.core.op

9.3. Module cate.core.workflow

9.4. Module cate.core.plugin

9.5. Module cate.conf

9.6. Module cate.ds

9.6.1. Description

The ds package comprises all specific data source implementations.

This is a plugin package automatically imported by the installation script’s entry point cate_ds (see the projects setup.py file).

9.6.2. Verification

The module’s unit-tests are located in test/ds and may be executed using $ py.test test/ops/test_<MODULE>.py --cov=cate/ops/<MODULE>.py for extra code coverage information.

9.6.3. Components

9.7. Module cate.ops

9.8. Module cate.cli.main

9.8.1. Description

This module provides Cate’s CLI executable.

To use the CLI executable, invoke the module file as a script, type python3 cate/cli/main.py [ARGS] [OPTIONS]. Type python3 cate/cli/main.py –help` for usage help.

The CLI operates on sub-commands. New sub-commands can be added by inheriting from the Command class and extending the Command.REGISTRY list of known command classes.

9.8.2. Technical Requirements

Extensible CLI with multiple sub-commands

Description

The CCI Toolbox should only have a single CLI executable that comes with multiple sub-commands instead of maintaining a number of different executables for each purpose. Plugins shall be able to add new CLI sub-commands.

URD-Source

• CCIT-UR-CR0001: Extensibility.

• CCIT-UR-A0002: Offer a Command Line Interface (CLI).

---

Run operations and workflows

Description

Allow for executing registered operations an workflows composed of operations.

URD-Source

• CCIT-UR-CL0001: Reading and executing script files written in XML or similar

---

List available data, operations and extensions

Description

Allow for listing dynamic content including available data, operations and plugin extensions.

URD-Source

• CCIT-UR-E0001: Dynamic extension by the use of plug-ins

---

Display information about available climate data sources

Description

Before downloading ECV datasets to the local computer, users shall be able to display information about them, e.g. included variables, total size, spatial and temporal resolution.

URD-Source

• CCIT-UR-DM0009: Holding information of any CCI ECV type

• CCIT-UR-DM0010: Attain meta-level status information per ECV type

---

Synchronize locally cached climate data

Description

Allow for listing dynamic content including available data, operations and plugin extensions.

URD-Source

• CCIT-UR-DM0006: Access to and ingestion of ESA CCI datasets

---

9.8.3. Verification

The module’s unit-tests are located in test/cli/test_main.py and may be executed using $ py.test test/cli/test_main.py --cov=cate/cli/test_main.py for extra code coverage information.

9.8.4. Components

cate.cli.main.CLI_NAME = 'cate'

Name of the Cate CLI executable (= cate).

cate.cli.main.COMMAND_REGISTRY = [<class 'cate.cli.main.DataSourceCommand'>, <class 'cate.cli.main.OperationCommand'>, <class 'cate.cli.main.WorkspaceCommand'>, <class 'cate.cli.main.ResourceCommand'>, <class 'cate.cli.main.RunCommand'>, <class 'cate.cli.main.IOCommand'>, <class 'cate.cli.main.UpdateCommand'>]

Cate plugins may extend this list by their commands during plugin initialisation.

class cate.cli.main.DataSourceCommand

The ds command implements various operations w.r.t. datasets.

classmethod configure_parser_and_subparsers(parser, subparsers)

Configure the given parser and its sub-parsers.

Overrides of this method must, e.g.::

list_parser = subparsers.add_parser(‘list’, …) # … configure list_parser here, and finally set its “sub_command_function” like so: list_parser.set_defaults(sub_command_function=cls._execute_list)

Sub-command functions shall raise a CommandError instance on failure.

Parameters

• parser – The command parser to configure.

• subparsers – A factory for sub-command parsers.

classmethod name()

Returns

A unique command name

classmethod parser_kwargs()

Return parser keyword arguments dictionary passed to a argparse.ArgumentParser(**parser_kwargs) call.

For the possible keywords in the returned dictionary, refer to https://docs.python.org/3.5/library/argparse.html#argparse.ArgumentParser.

Returns

A keyword arguments dictionary.

class cate.cli.main.IOCommand

The io command implements various operations w.r.t. supported data and file formats.

classmethod configure_parser_and_subparsers(parser, subparsers)

Configure the given parser and its sub-parsers.

Overrides of this method must, e.g.::

list_parser = subparsers.add_parser(‘list’, …) # … configure list_parser here, and finally set its “sub_command_function” like so: list_parser.set_defaults(sub_command_function=cls._execute_list)

Sub-command functions shall raise a CommandError instance on failure.

Parameters

• parser – The command parser to configure.

• subparsers – A factory for sub-command parsers.

classmethod name()

Returns

A unique command name

classmethod parser_kwargs()

Return parser keyword arguments dictionary passed to a argparse.ArgumentParser(**parser_kwargs) call.

For the possible keywords in the returned dictionary, refer to https://docs.python.org/3.5/library/argparse.html#argparse.ArgumentParser.

Returns

A keyword arguments dictionary.

class cate.cli.main.OperationCommand

The op command implements various operations w.r.t. operations.

classmethod configure_parser_and_subparsers(parser, subparsers)

Configure the given parser and its sub-parsers.

Overrides of this method must, e.g.::

list_parser = subparsers.add_parser(‘list’, …) # … configure list_parser here, and finally set its “sub_command_function” like so: list_parser.set_defaults(sub_command_function=cls._execute_list)

Sub-command functions shall raise a CommandError instance on failure.

Parameters

• parser – The command parser to configure.

• subparsers – A factory for sub-command parsers.

classmethod name()

Returns

A unique command name

classmethod parser_kwargs()

Return parser keyword arguments dictionary passed to a argparse.ArgumentParser(**parser_kwargs) call.

For the possible keywords in the returned dictionary, refer to https://docs.python.org/3.5/library/argparse.html#argparse.ArgumentParser.

Returns

A keyword arguments dictionary.

class cate.cli.main.PluginCommand

The pi command lists the content of various plugin registry.

classmethod configure_parser_and_subparsers(parser, subparsers)

Configure the given parser and its sub-parsers.

Overrides of this method must, e.g.::

list_parser = subparsers.add_parser(‘list’, …) # … configure list_parser here, and finally set its “sub_command_function” like so: list_parser.set_defaults(sub_command_function=cls._execute_list)

Sub-command functions shall raise a CommandError instance on failure.

Parameters

• parser – The command parser to configure.

• subparsers – A factory for sub-command parsers.

classmethod name()

Returns

A unique command name

classmethod parser_kwargs()

Return parser keyword arguments dictionary passed to a argparse.ArgumentParser(**parser_kwargs) call.

For the possible keywords in the returned dictionary, refer to https://docs.python.org/3.5/library/argparse.html#argparse.ArgumentParser.

Returns

A keyword arguments dictionary.

class cate.cli.main.ResourceCommand

The res command implements various operations w.r.t. workspaces.

classmethod configure_parser_and_subparsers(parser, subparsers)

Configure the given parser and its sub-parsers.

Overrides of this method must, e.g.::

list_parser = subparsers.add_parser(‘list’, …) # … configure list_parser here, and finally set its “sub_command_function” like so: list_parser.set_defaults(sub_command_function=cls._execute_list)

Sub-command functions shall raise a CommandError instance on failure.

Parameters

• parser – The command parser to configure.

• subparsers – A factory for sub-command parsers.

classmethod name()

Returns

A unique command name

classmethod parser_kwargs()

Return parser keyword arguments dictionary passed to a argparse.ArgumentParser(**parser_kwargs) call.

For the possible keywords in the returned dictionary, refer to https://docs.python.org/3.5/library/argparse.html#argparse.ArgumentParser.

Returns

A keyword arguments dictionary.

class cate.cli.main.RunCommand

The run command is used to invoke registered operations and JSON workflows.

classmethod configure_parser(parser)

Configure parser, i.e. make any required parser.add_argument(*args, **kwargs) calls. See https://docs.python.org/3.5/library/argparse.html#argparse.ArgumentParser.add_argument

Parameters

parser – The command parser to configure.

execute(command_args)

Execute this command.

The command’s arguments in command_args are attributes namespace returned by argparse.ArgumentParser.parse_args(). Also refer to to https://docs.python.org/3.5/library/argparse.html#argparse.ArgumentParser.parse_args

execute``implementations shall raise a ``CommandError instance on failure.

Parameters

command_args – The command’s arguments.

classmethod name()

Returns

A unique command name

classmethod parser_kwargs()

Return parser keyword arguments dictionary passed to a argparse.ArgumentParser(**parser_kwargs) call.

For the possible keywords in the returned dictionary, refer to https://docs.python.org/3.5/library/argparse.html#argparse.ArgumentParser.

Returns

A keyword arguments dictionary.

class cate.cli.main.UpdateCommand

The update command is used to update an existing cate environment to a specific or the latest cate version.

classmethod configure_parser(parser)

Configure parser, i.e. make any required parser.add_argument(*args, **kwargs) calls. See https://docs.python.org/3.5/library/argparse.html#argparse.ArgumentParser.add_argument

Parameters

parser – The command parser to configure.

execute(command_args)

Execute this command.

The command’s arguments in command_args are attributes namespace returned by argparse.ArgumentParser.parse_args(). Also refer to to https://docs.python.org/3.5/library/argparse.html#argparse.ArgumentParser.parse_args

execute``implementations shall raise a ``CommandError instance on failure.

Parameters

command_args – The command’s arguments.

classmethod name()

Returns

A unique command name

classmethod parser_kwargs()

Return parser keyword arguments dictionary passed to a argparse.ArgumentParser(**parser_kwargs) call.

For the possible keywords in the returned dictionary, refer to https://docs.python.org/3.5/library/argparse.html#argparse.ArgumentParser.

Returns

A keyword arguments dictionary.

class cate.cli.main.WorkspaceCommand

The ws command implements various operations w.r.t. workspaces.

classmethod configure_parser_and_subparsers(parser, subparsers)

Configure the given parser and its sub-parsers.

Overrides of this method must, e.g.::

list_parser = subparsers.add_parser(‘list’, …) # … configure list_parser here, and finally set its “sub_command_function” like so: list_parser.set_defaults(sub_command_function=cls._execute_list)

Sub-command functions shall raise a CommandError instance on failure.

Parameters

• parser – The command parser to configure.

• subparsers – A factory for sub-command parsers.

classmethod name()

Returns

A unique command name

classmethod parser_kwargs()

Return parser keyword arguments dictionary passed to a argparse.ArgumentParser(**parser_kwargs) call.

For the possible keywords in the returned dictionary, refer to https://docs.python.org/3.5/library/argparse.html#argparse.ArgumentParser.

Returns

A keyword arguments dictionary.

9.9. Module cate.webapi

9.10. Module cate.util

9.10.1. Description

The cate.util package provides application-independent utility functions.

This package is independent of other ``cate.*``packages and can therefore be used stand-alone.

9.10.2. Verification

The module’s unit-tests are located in test/util and may be executed using $ py.test test/util --cov=cate/util for extra code coverage information.

9.10.3. Components

9.11. Module cate.util.cache

9.11.1. Description

This module defines the Cache class which represents a general-purpose cache. A cache is configured by a CacheStore which is responsible for storing and reloading cached items.

The default cache stores are

• MemoryCacheStore

• FileCacheStore

Every cache has capacity in physical units defined by the CacheStore. When the cache capacity is exceeded a replacement policy for cached items is applied until the cache size falls below a given ratio of the total capacity.

The default replacement policies are

• POLICY_LRU

• POLICY_MRU

• POLICY_LFU

• POLICY_RR

This package is independent of other ``cate.*``packages and can therefore be used stand-alone.

9.11.2. Components

class cate.util.cache.Cache(store=<cate.util.cache.MemoryCacheStore object>, capacity=1000, threshold=0.75, policy=<function _policy_lru>, parent_cache=None)

An implementation of a cache. See https://en.wikipedia.org/wiki/Cache_algorithms

class Item

Cache-private class representing an item in the cache.

discard(store, key)

static load_from_key(store, key)

restore(store, key)

store(store, key, value)

class cate.util.cache.CacheStore

Represents a store to which cached values can be stored into and restored from.

abstract can_load_from_key(key) → bool

Test whether a stored value representation can be loaded from the given key. :rtype: bool :param key: the key :return: True, if so

abstract discard_value(key, stored_value)

Discard a value from it’s storage. :param key: the key :param stored_value: the stored representation of the value

abstract load_from_key(key)

Load a stored value representation of the value and its size from the given key. :param key: the key :return: a 2-element sequence containing the stored representation of the value and it’s size

abstract restore_value(key, stored_value)

Restore a vale from its stored representation. :param key: the key :param stored_value: the stored representation of the value :return: the item

abstract store_value(key, value)

Store a value and return it’s stored representation and size in any unit, e.g. in bytes. :param key: the key :param value: the value :return: a 2-element sequence containing the stored representation of the value and it’s size

class cate.util.cache.FileCacheStore(cache_dir: str, ext: str)

Simple file store for values which can be written and read as bytes, e.g. encoded PNG images.

can_load_from_key(key) → bool

Test whether a stored value representation can be loaded from the given key. :rtype: bool :param key: the key :return: True, if so

discard_value(key, stored_value)

Discard a value from it’s storage. :param key: the key :param stored_value: the stored representation of the value

load_from_key(key)

Load a stored value representation of the value and its size from the given key. :param key: the key :return: a 2-element sequence containing the stored representation of the value and it’s size

restore_value(key, stored_value)

Restore a vale from its stored representation. :param key: the key :param stored_value: the stored representation of the value :return: the item

store_value(key, value)

Store a value and return it’s stored representation and size in any unit, e.g. in bytes. :param key: the key :param value: the value :return: a 2-element sequence containing the stored representation of the value and it’s size

class cate.util.cache.MemoryCacheStore

Simple memory store.

can_load_from_key(key) → bool

Test whether a stored value representation can be loaded from the given key. :rtype: bool :param key: the key :return: True, if so

discard_value(key, stored_value)

Clears the value in the given stored_value. :param key: the key :param stored_value: the stored representation of the value

load_from_key(key)

Load a stored value representation of the value and its size from the given key. :param key: the key :return: a 2-element sequence containing the stored representation of the value and it’s size

restore_value(key, stored_value)

Parameters

• key – the key

• stored_value – the stored representation of the value

Returns

the original value.

store_value(key, value)

Return (value, 1). :param key: the key :param value: the original value :return: the tuple (stored value, size) where stored value is the sequence [key, value].

cate.util.cache.POLICY_LFU(item)

Discard Least Frequently Used first

cate.util.cache.POLICY_LRU(item)

Discard Least Recently Used items first

cate.util.cache.POLICY_MRU(item)

Discard Most Recently Used first

cate.util.cache.POLICY_RR(item)

Discard items by Random Replacement

9.12. Module cate.util.cli

class cate.util.cli.Command

Represents a (sub-)command of a command-line interface.

classmethod configure_parser(parser: argparse.ArgumentParser) → None

Configure parser, i.e. make any required parser.add_argument(*args, **kwargs) calls. See https://docs.python.org/3.5/library/argparse.html#argparse.ArgumentParser.add_argument

Parameters

parser (ArgumentParser) – The command parser to configure.

execute(command_args: argparse.Namespace) → None

Execute this command.

The command’s arguments in command_args are attributes namespace returned by argparse.ArgumentParser.parse_args(). Also refer to to https://docs.python.org/3.5/library/argparse.html#argparse.ArgumentParser.parse_args

execute``implementations shall raise a ``CommandError instance on failure.

Parameters

command_args (Namespace) – The command’s arguments.

classmethod name() → str

Returns

A unique command name

classmethod new_monitor() → cate.util.monitor.Monitor

Create a new console progress monitor.

Returns

a new Monitor instance.

classmethod parser_kwargs() → dict

Return parser keyword arguments dictionary passed to a argparse.ArgumentParser(**parser_kwargs) call.

For the possible keywords in the returned dictionary, refer to https://docs.python.org/3.5/library/argparse.html#argparse.ArgumentParser.

Returns

A keyword arguments dictionary.

exception cate.util.cli.CommandError(message)

An error type signaling command-line errors.

Parameters

message – Error message

class cate.util.cli.NoExitArgumentParser(*args, **kwargs)

Special argparse.ArgumentParser that never directly exits the current process. It raises an ExitException instead.

exception ExitException(status, message)

Raises instead of exiting the current process.

exit(status=0, message=None)

Overrides the base class method in order to raise an ExitException.

class cate.util.cli.SubCommandCommand

classmethod configure_parser(parser: argparse.ArgumentParser) → None

Add a new sub-parsers to the given parser. Call configure_parser_and_subparsers with the new sub-parsers.

Parameters

parser (ArgumentParser) – The command parser to configure.

abstract classmethod configure_parser_and_subparsers(parser, subparsers)

Configure the given parser and its sub-parsers.

Overrides of this method must, e.g.::

list_parser = subparsers.add_parser(‘list’, …) # … configure list_parser here, and finally set its “sub_command_function” like so: list_parser.set_defaults(sub_command_function=cls._execute_list)

Sub-command functions shall raise a CommandError instance on failure.

Parameters

• parser – The command parser to configure.

• subparsers – A factory for sub-command parsers.

execute(command_args)

Executes the function given by the “sub_command_function” attribute of given command_args with command_args as only argument.

Parameters

command_args –

cate.util.cli.run_main(name: str, description: str, version: str, command_classes: Sequence[cate.util.cli.Command], license_text: Optional[str] = None, docs_url: Optional[str] = None, error_message_trimmer=None, args: Optional[Sequence[str]] = None) → int

A CLI’s entry point function.

To be used in your own code as follows:

>>> if __name__ == '__main__':
>>>    sys.exit(run_main(...))

Return type

int

Parameters

• name (str) – The program’s name.

• description (str) – The program’s description.

• version (str) – The program’s version string.

• command_classes – The CLI commands.

• license_text (str) – An optional license text.

• docs_url (str) – An optional documentation URL.

• error_message_trimmer – An optional callable (str)->str that trims error message strings.

• args – list of command-line arguments. If not passed, sys.argv[1:] is used.

Returns

An exit code where 0 stands for success.

9.13. Module cate.util.im

9.13.1. Description

The cate.util.im package provides application-independent utility functions for working with tiled image pyramids.

The Cate project uses this package for implementing a RESTful web service that provides image tiles from image pyramids.

This package is independent of other cate.* packages, but it depends on the following external packages

• numpy

• pillow (for PIL)

• matplotlib

9.13.2. Verification

The module’s unit-tests are located in test/util/im and may be executed using $ py.test test/util/im --cov=cate/util/im for extra code coverage information.

9.13.3. Components

9.14. Module cate.util.web