4.3. Command-Line Interface

4.3.1. Overview

The CCI Toolbox comprises a single command-line executable, which is called cate and is available after installing the CCI Toolbox on your computer. See section Local Setup for more information. The command-line interface allows for accessing local and remote datasets as well as running virtually all CCI Toolbox operations on them.

Developers only: If you build and install the CCI Toolbox from Python sources into your current Python environment, cate will be registered as an executable script. It can be found as $PYTHON_PREFIX/bin/cate.sh on Unix systems and as %PYTHON_PREFIX%\\Scripts\\cate.exe on Windows systems where PYTHON_PREFIX is the path to the current Python environment.

In the console / terminal window type:

cate -h

This should output the following usage help::

usage: cate [-h] [--version] [--traceback] [--license] [--docs] COMMAND ...

ESA CCI Toolbox (Cate) command-line interface, version 2.0

positional arguments:
  COMMAND      One of the following commands. Type "COMMAND -h" to get
               command-specific help.
    ds         Manage data sources.
    op         Manage data operations.
    ws         Manage workspaces.
    res        Manage workspace resources.
    run        Run an operation or Workflow file.
    io         Manage supported data and file formats.
    upd        Update an existing cate environment to a specific or to the
               latest cate version

optional arguments:
  -h, --help   show this help message and exit
  --version    show program's version number and exit
  --traceback  show (Python) stack traceback for the last error
  --license    show software license and exit
  --docs       show software documentation in a browser window

cate uses up to two sub-command levels. Each sub-command has its own set of options and arguments and can display help when used with the option `--help or -h. The first sub-command level comprises the following list of commands:

• cate ds - Dataset Management

• cate op - Operation Management

• cate run - Running Operations and Workflows

The following first level sub-commands are used to work interactively with datasets and operations:

• cate ws: Workspace Management

• cate res - Workspace Resources Management

When you encounter any error while using cate and you want to report the problem to the development team, we kindly ask you to rerun the command with option --traceback and include the Python stack traceback with a short description of your problem.

4.3.2. Examples

The following examples shall help you understand the basic concepts behind the various cate commands.

4.3.2.1. Manage datasets

To list all available data sources, type:

cate ds list

To query all data sources that have ozone in their name, type:

cate ds list -n ozone

To get more detailed information on a specific data source, e.g. esacci.OZONE.mon.L3..., type:

cate ds info esacci.OZONE.mon.L3.NP.multi-sensor.multi-platform.MERGED.fv0002.r1

To add a local data source from all NetCDF files in e.g. data/sst_v3 and name it e.g. SSTV3, type:

cate ds def SSTV3 data/sst_v3/*.nc

Make sure it is there:

cate ds list -n SSTV3

To make a temporal subset ECV data source locally available, i.e. avoid remote data access during its usage:

cate ds copy esacci.OZONE.mon.L3.NP.multi-sensor.multi-platform.MERGED.fv0002.r1 -t 2006-01-01,2007-12-31

The section Configuration in Local Setup describes, how to configure the location of directory in which Cate stores such synchronised data.

4.3.2.2. Inspect available operations

To list all available operations, type:

cate op list

To display more details about a particular operation, e.g. tseries_point, type:

cate op info tseries_point

4.3.2.3. Run an operation

The cate run command is used to execute single operations. The open and read options are used to ingest datasets which can then be referenced by name. A write option allows to write the operation result into a file.

To run the tseries_point operation on a dataset, e.g. the local.SSTV3 (from above), at lat=0 and lon=0, type:

cate run --open ds=local.SSTV3 --write ts2.nc tseries_point ds=ds lat=0 lon=0

To run the tseries_point operation on a netCDF file, e.g. test/ui/precip_and_temp.nc at lat=0 and lon=0, type:

cate run --read ds=test/ui/precip_and_temp.nc --write ts2.nc tseries_point ds=ds lat=0 lon=0

4.3.2.4. Interactive session

The following command sequence is a simple example for an interactive session using the Cate command-line:

cate ws new
cate res open sst local.SSTV3
cate res set sst_ts tseries_point ds=@sst lat=0 lon=0
cate res plot sst_ts
cate res write sst_ts sst_ts.nc
cate ws status

The steps above explained:

1. cate ws new is used to create a new in-memory workspace. A workspace can hold any number of named workspace resources which may refer to opened datasets or any other ingested or computed objects.

2. cate res open is used to open a dataset from the available data stores and assign the opened dataset to the workspace resource sst. Accordingly, cate res read could have been used to read from a local netCDF file.

3. cate res set assigns the result of the tseries_point operation to workspace resource sst_ts. Note the at-character “@” used as prefix for the input ds. This indicates that value for input ds of step tseries_point will be retrieved “at” the open step named sst. It establishes a connection between step open and tseries_point. In fact, this is the way processing graphs are constructed using the CLI.

4. cate res plot plots the workspace resource sst_ts.

5. cate res write writes the workspace resource sst_ts to a netCDF file ./sst_ts.nc.

6. cate ws status shows the current workspace status and lists all workspace resource assignments.

We could now save the current workspace state and close it:

cate ws save
cate ws close

cate ws save creates a hidden sub-directory .cate-workspace and herewith makes the current directory a workspace directory. cate uses this hidden directory to persist the workspace state information. At a later point in time, you could cd into any of your workspace directories, and:

cate ws open
cate ws status

in order to reopen it, display its status, and continue interactively working with its resources.

The following subsections provide detailed information about the cate commands.

4.3.3. cate ds - Dataset Management

Provides a set of sub-commands used to manage climate data sources. Data sources are used to open local and remote datasets which are input to various analysis and processing operations. Type “cate op -h” to find out more about available operations.

usage: cate ds [-h] COMMAND ...

4.3.7.1. Positional Arguments

COMMAND

Possible choices: list, info, add, del, copy

One of the following commands. Type “COMMAND -h” for help.

4.3.7.2. Sub-commands

4.3.6.2.10. list

List all available data sources

cate ds list [-h] [--name NAME] [--all] [--coverage]

4.3.7.2.1.2. Named Arguments

--name, -n

List only data sources named NAME or that have NAME in their name. The comparison is case insensitive.

--all, -a

Show also data sources that can not be opened in Cate

Default: True

--coverage, -c

Also display temporal coverage

Default: False

4.3.4.2.2. info

Display information about a data source.

cate ds info [-h] [--var] [--local] DS

4.3.7.2.1.1. Positional Arguments

DS

A data source name. Type “cate ds list” to show all possible data source names.

4.3.7.2.2.2. Named Arguments

--var, -v

Also display information about contained dataset variables.

Default: False

--local, -l

Also display temporal coverage of cached datasets.

Default: False

4.3.3.2.3. add

Add a new file data source using a file path pattern.

cate ds add [-h] DS FILE [FILE ...]

4.3.7.2.2.1. Positional Arguments

DS

A name for the data source.

FILE

A list of files comprising this data source. The files can contain the wildcard characters “*” and “?”.

4.3.7.2.6. del

Removes a data source from file data store.

cate ds del [-h] [-k] [-y] DS

4.3.7.2.3.1. Positional Arguments

DS

A name for the data source.

4.3.7.2.3.2. Named Arguments

-k, --keep_files

Do not ask for confirmation.

Default: False

-y, --yes

Do not ask for confirmation.

Default: False

4.3.3.2.5. copy

Makes a file copy of any other data source. The copy may be limited to a subset by optional constraints.

cate ds copy [-h] [--name NAME] [--time TIME] [--region REG] [--vars VARS]
             REF_DS

4.3.7.2.4.1. Positional Arguments

REF_DS

A name of origin data source.

4.3.7.2.4.2. Named Arguments

--name, -n

A name for new data source.

--time, -t

Time range constraint. Use format “YYYY-MM-DD,YYYY-MM-DD”.

--region, -r

Region constraint. Use format: “min_lon,min_lat,max_lon,max_lat”.

--vars, -v

Names of variables to be included. Use format “pattern1,pattern2,…”

4.3.4. cate op - Operation Management

Provides a set of commands to inquire the available operations used to analyse and process climate datasets.

usage: cate op [-h] COMMAND ...

4.3.7.1. Positional Arguments

COMMAND

Possible choices: list, info

One of the following commands. Type “COMMAND -h” for help.

4.3.7.2. Sub-commands

4.3.6.2.10. list

List operations.

cate op list [-h] [--name NAME] [--tag TAG] [--deprecated] [--internal]

4.3.7.2.1.2. Named Arguments

--name, -n

List only operations with name NAME or that have NAME in their name. The comparison is case insensitive.

--tag, -t

List only operations tagged by TAG or that have TAG in one of their tags. The comparison is case insensitive.

--deprecated, -d

List deprecated operations.

Default: False

--internal, -i

List operations tagged “internal”.

Default: False

4.3.4.2.2. info

Show usage information about an operation.

cate op info [-h] OP

4.3.7.2.1.1. Positional Arguments

OP

Fully qualified operation name.

4.3.5. cate run - Running Operations and Workflows

Runs the given operation or Workflow file with the specified operation arguments. Argument values may be constant values or the names of data loaded by the –open or –read options. Type “cate op list” to list all available operations. Type “cate op info” to find out which arguments are supported by a given operation.

usage: cate run [-h] [-m] [-o DS_EXPR] [-r FILE_EXPR] [-w FILE_EXPR] OP ...

4.3.7.1. Positional Arguments

OP

Fully qualified operation name or Workflow file. Type “cate op list” to list available operators.

...

Operation arguments given as KEY=VALUE. KEY is any supported input by OP. VALUE depends on the expected data type of an OP input. It can be a True, False, a string, a numeric constant, one of the names specified by the –open and –read options, or a Python expression. Type “cate op info OP” to print information about the supported OP input names to be used as KEY and their data types to be used as VALUE.

4.3.7.2.1.2. Named Arguments

-m, --monitor

Display progress information during execution.

Default: False

-o, --open

Open a dataset from DS_EXPR. The DS_EXPR syntax is NAME=DS[,START[,END]]. DS must be a valid data source name. Type “cate ds list” to show all known data source names. START and END are dates and may be used to create temporal data subsets. The dataset loaded will be assigned to the arbitrary name NAME which is used to pass the datasets or its variables as an OP argument. To pass a variable use syntax NAME.VAR_NAME.

-r, --read

Read object from FILE_EXPR. The FILE_EXPR syntax is NAME=PATH[,FORMAT]. Type “cate io list -r” to see which formats are supported.If FORMAT is not provided, file format is derived from the PATH’s filename extensions or file content. NAME may be passed as an OP argument that receives a dataset, dataset variable or any other data type. To pass a variable of a dataset use syntax NAME.VAR_NAME

-w, --write

Write result to FILE_EXPR. The FILE_EXPR syntax is [NAME=]PATH[,FORMAT]. Type “cate io list -w” to see which formats are supported.If FORMAT is not provided, file format is derived from the object type and the PATH’s filename extensions. If OP returns multiple named output values, NAME is used to identify them. Multiple -w options may be used in this case.

4.3.6. cate ws: Workspace Management

Used to create, open, save, modify, and delete workspaces. Workspaces contain named workflow resources, which can be datasets read from data stores, or any other data objects originating from applying operations to datasets and other data objects. The origin of every resource is stored in the workspace’s workflow description. Type “cate res -h” for more information about workspace resource commands.

usage: cate ws [-h] COMMAND ...

4.3.7.1. Positional Arguments

COMMAND

Possible choices: init, new, open, close, save, run, del, clean, status, list, exit

One of the following commands. Type “COMMAND -h” for help.

4.3.7.2. Sub-commands

4.3.6.2.1. init

Initialize workspace.

cate ws init [-h] [-d DIR] [--desc DESCRIPTION]

4.3.7.2.1.2. Named Arguments

-d, --dir

The workspace’s base directory. If not given, the current working directory is used.

Default: “.”

--desc

Workspace description.

4.3.6.2.2. new

Create new in-memory workspace.

cate ws new [-h] [-d DIR] [--desc DESCRIPTION]

4.3.7.2.2.2. Named Arguments

-d, --dir

The workspace’s base directory. If not given, the current working directory is used.

Default: “.”

--desc

Workspace description.

4.3.7.2.1. open

Open workspace.

cate ws open [-h] [-d DIR]

4.3.7.2.3.2. Named Arguments

-d, --dir

The workspace’s base directory. If not given, the current working directory is used.

Default: “.”

4.3.6.2.4. close

Close workspace.

cate ws close [-h] [-d DIR] [-a] [-s]

4.3.7.2.4.2. Named Arguments

-d, --dir

The workspace’s base directory. If not given, the current working directory is used.

Default: “.”

-a, --all

Close all workspaces. Ignores DIR option.

Default: False

-s, --save

Save modified workspace before closing.

Default: False

4.3.6.2.5. save

Save workspace.

cate ws save [-h] [-d DIR] [-a]

4.3.7.2.5.2. Named Arguments

-d, --dir

The workspace’s base directory. If not given, the current working directory is used.

Default: “.”

-a, --all

Save all workspaces. Ignores DIR option.

Default: False

4.3.6.2.6. run

Run operation.

cate ws run [-h] [-d DIR] OP ...

4.3.7.2.1.1. Positional Arguments

OP

Operation name or Workflow file path. Type “cate op list” to list available operations.

...

Operation arguments given as KEY=VALUE. KEY is any supported input by OP. VALUE depends on the expected data type of an OP input. It can be either a value or a reference an existing resource prefixed by the add character “@”. The latter connects to operation steps with each other. To provide a (constant) value you can use boolean literals True and False, strings, or numeric values. Type “cate op info OP” to print information about the supported OP input names to be used as KEY and their data types to be used as VALUE.

4.3.7.2.6.2. Named Arguments

-d, --dir

The workspace’s base directory. If not given, the current working directory is used.

Default: “.”

4.3.7.2.6. del

Delete workspace.

cate ws del [-h] [-d DIR] [-y]

4.3.7.2.7.2. Named Arguments

-d, --dir

The workspace’s base directory. If not given, the current working directory is used.

Default: “.”

-y, --yes

Do not ask for confirmation.

Default: False

4.3.6.2.8. clean

Clean workspace (removes all resources).

cate ws clean [-h] [-d DIR] [-y]

4.3.7.2.8.2. Named Arguments

-d, --dir

The workspace’s base directory. If not given, the current working directory is used.

Default: “.”

-y, --yes

Do not ask for confirmation.

Default: False

4.3.6.2.9. status

Print workspace information.

cate ws status [-h] [-d DIR]

4.3.6.2.9.1. Named Arguments

-d, --dir

The workspace’s base directory. If not given, the current working directory is used.

Default: “.”

4.3.6.2.10. list

List all opened workspaces.

cate ws list [-h]

4.3.6.2.11. exit

Exit interactive mode. Closes all open workspaces.

cate ws exit [-h] [-y] [-s]

4.3.6.2.11.1. Named Arguments

-y, --yes

Do not ask for confirmation.

Default: False

-s, --save

Save any modified workspaces before closing.

Default: False

4.3.7. cate res - Workspace Resources Management

Used to set, run, open, read, write, plot, etc. workspace resources. All commands expect an opened workspace. Type “cate ws -h” for more information about workspace commands.

usage: cate res [-h] COMMAND ...

4.3.7.1. Positional Arguments

COMMAND

Possible choices: open, read, write, set, rename, del, print, plot

One of the following commands. Type “COMMAND -h” for help.

4.3.7.2. Sub-commands

4.3.7.2.1. open

Open a dataset from a data source and set a resource.

cate res open [-h] [-d DIR] NAME DS [START] [END] [REGION] [VAR_NAMES]

4.3.7.2.1.1. Positional Arguments

NAME

Name of the new target resource.

DS

A data source named DS. Type “cate ds list” to list valid data source names.

START

Start date. Use format “YYYY[-MM[-DD]]”.

END

End date. Use format “YYYY[-MM[-DD]]”.

REGION

Region constraint. Use format “min_lon,min_lat,max_lon,max_lat”.

VAR_NAMES

Names of variables to be included. Use format “pattern1,pattern2,pattern3”.

4.3.7.2.1.2. Named Arguments

-d, --dir

The workspace’s base directory. If not given, the current working directory is used.

Default: “.”

4.3.7.2.2. read

Read an object from a file and set a resource.

cate res read [-h] [-d DIR] [-f FORMAT] NAME FILE

4.3.7.2.2.1. Positional Arguments

NAME

Name of the new target resource.

FILE

File path.

4.3.7.2.2.2. Named Arguments

-d, --dir

The workspace’s base directory. If not given, the current working directory is used.

Default: “.”

-f, --format

File format. Type “cate io list -r” to see which formats are supported.

4.3.7.2.3. write

Write a resource to a file.

cate res write [-h] [-d DIR] [-f FORMAT] NAME FILE

4.3.7.2.3.1. Positional Arguments

NAME

Name of an existing resource.

FILE

File path.

4.3.7.2.3.2. Named Arguments

-d, --dir

The workspace’s base directory. If not given, the current working directory is used.

Default: “.”

-f, --format

File format. Type “cate io list -w” to see which formats are supported.

4.3.7.2.4. set

Set a resource from the result of an operation.

cate res set [-h] [-d DIR] [-o] NAME OP ...

4.3.7.2.4.1. Positional Arguments

NAME

Name of the target resource to be set. Use -o to overwrite an existing NAME.

OP

Operation name. Type “cate op list” to list available operation names.

...

Operation arguments given as KEY=VALUE. KEY is any supported input by OP. VALUE depends on the expected data type of an OP input. It can be either a value or a reference an existing resource prefixed by the add character “@”. The latter connects to operation steps with each other. To provide a (constant) value you can use boolean literals True and False, strings, or numeric values. Type “cate op info OP” to print information about the supported OP input names to be used as KEY and their data types to be used as VALUE.

4.3.7.2.4.2. Named Arguments

-d, --dir

The workspace’s base directory. If not given, the current working directory is used.

Default: “.”

-o, --overwrite

Overwrite an existing workflow step / target resource with same NAME.

Default: False

4.3.7.2.5. rename

Rename a resource.

cate res rename [-h] [-d DIR] NAME NEW_NAME

4.3.7.2.5.1. Positional Arguments

NAME

Resource name.

NEW_NAME

New resource name.

4.3.7.2.5.2. Named Arguments

-d, --dir

The workspace’s base directory. If not given, the current working directory is used.

Default: “.”

4.3.7.2.6. del

Delete a resource.

cate res del [-h] [-d DIR] NAME

4.3.7.2.6.1. Positional Arguments

NAME

Resource name.

4.3.7.2.6.2. Named Arguments

-d, --dir

The workspace’s base directory. If not given, the current working directory is used.

Default: “.”

4.3.7.2.7. print

If EXPR is omitted, print value of all current resources. Otherwise, if EXPR identifies a resource, print its value. Else print the value of a (Python) expression evaluated in the context of the current workspace.

cate res print [-h] [-d DIR] [EXPR]

4.3.7.2.7.1. Positional Arguments

EXPR

Name of an existing resource or a valid (Python) expression.

4.3.7.2.7.2. Named Arguments

-d, --dir

The workspace’s base directory. If not given, the current working directory is used.

Default: “.”

4.3.7.2.8. plot

Plot a resource or the value of a (Python) expression evaluated in the context of the current workspace.

cate res plot [-h] [-d DIR] [-v [VAR]] [-o [FILE]] EXPR

4.3.7.2.8.1. Positional Arguments

EXPR

Name of an existing resource or any (Python) expression.

4.3.7.2.8.2. Named Arguments

-d, --dir

The workspace’s base directory. If not given, the current working directory is used.

Default: “.”

-v, --var

Name of a variable to plot.

-o, --out

Output file to write the plot figure to.