Matrix disk cache

from earthkit.geo import cache, config

earthkit-geo uses a dedicated directory to store files downloaded from remote sources. This includes the interpolation matrices and the related index files when using regrid() with backend="precomputed".

By default this directory serves a cache and is managed (its size is checked/limited). It means if we run regrid() again with the same input and output grid it will load the matrix from the cache instead of downloading it again. Additionally, caching offers monitoring and disk space management. When the cache is full, cached data is deleted according to the configuration (i.e. oldest data is deleted first).

In the examples below we will change the configuration multiple times. First we ensure all the changes are temporary and no options are saved into the configuration file. We also reset the configuration to the defaults.

config.autosave = False
config.reset()

User defined cache directory (default)

The primary key to control the cache in the configuration is cache-policy. The default value is “user”, which means that the cache location is defined by the user-cache-directory config option. By default it is set to "~/.cache/earthkit-regrid".

The user cache directory is not cleaned up on exit. So next time you start earthkit-regrid it will be there again unless it is deleted manually or it is set in way that on each startup a different path is assigned to it. Also, when you run multiple sessions of earthkit-regrid under the same user they will share the same cache.

The configuration tells us all the details about the cache policy and location:

config.set("cache-policy", "user")
print(config.get("cache-policy"))
print(config.get("user-cache-directory"))

user
/Users/cgr/.cache/earthkit-geo

The path to the current cache directory can also be queried through the cache object:

cache.directory()

'/Users/cgr/.cache/earthkit-geo'

We are free to change the user cache directory to another path:

config.set("user-cache-directory", "~/earthkit-regrid-demo-cache")
cache.directory()

'/Users/cgr/earthkit-regrid-demo-cache'

Temporary cache directory

When the cache-policy is “temporary” the cache will be located in a temporary directory created by tempfile.TemporaryDirectory. This directory will be unique for each earthkit-regrid session. When the directory object goes out of scope (at the latest on exit) the cache is cleaned up.

config.set("cache-policy", "temporary")
print(config.get("cache-policy"))

temporary

The path to the cache directory has to be queried through the cache object:

cache.directory()

'/var/folders/93/w0p869rx17q98wxk83gn9ys40000gn/T/tmpwrnl24xy'

We can specify the parent directory for the the temporary cache by using the temporary-cache-directory-root config option. By default it is set to None (no parent directory specified).

s = {"cache-policy": "temporary", "temporary-cache-directory-root": "~/my_demo_regrid_cache"}
config.set(s)
cache.directory()

'/Users/cgr/my_demo_regrid_cache/tmpuuu3mu5v'

No caching

When the cache-policy is “off” no disk based caching is available.

In this case all files are downloaded into an unmanaged temporary directory created by tempfile.TemporaryDirectory. Since caching is disabled all calls to regrid() for remote services and URLSs will download the data again! This temporary directory will be unique for each earthkit-geo session. When the directory object goes out of scope (at the latest on exit) the directory will be cleaned up.

config.set("cache-policy", "off")
print(config.get("cache-policy"))

off

The path to the temporary directory has to be queried through the cache object:

cache.directory()

'/var/folders/93/w0p869rx17q98wxk83gn9ys40000gn/T/tmp5bct2nsn'

We can specify the parent directory for the temporary directory by using the temporary-directory-root config option. By default it is set to None (no parent directory specified).

s = {"cache-policy": "off", "temporary-directory-root": "~/my_demo_regrid_tmp"}
config.set(s)
cache.directory()

'/var/folders/93/w0p869rx17q98wxk83gn9ys40000gn/T/tmp3py0o9ri'