Octue SDK (Python)

The python SDK for Octue data services, digital twins, and applications - get faster data groundwork so you have more time for the science!

Definition

Octue service

An Octue data service, digital twin, or application that can be asked questions, process them, and send answers. Octue services can communicate with each other with no extra setup.

Key features

Unified cloud/local file, dataset, and manifest operations

• Create and build datasets easily

• Organise them with timestamps, labels, and tags

• Filter and combine them using this metadata

• Store them locally or in the cloud (or both for low-latency reading/writing with cloud-guaranteed data availability)

• Use internet/cloud-based datasets as if they were local e.g.

  • https://example.com/important_dataset.dat

  • gs://example-bucket/important_dataset.dat

• Create manifests (a set of datasets needed for a particular analysis) to modularise your dataset input/output

Ask existing services questions from anywhere

• Send them data to process from anywhere

• Automatically have their logs, monitor messages, and any errors forwarded to you and displayed as if they were local

• Receive their output data as JSON

• Receive a manifest of any output datasets they produce for you to download or access as you wish

Create, run, and deploy your apps as services

• No need to change your app - just wrap it

• Use the octue CLI to run your service locally or deploy it to Google Cloud Run or Google Dataflow

• Create JSON-schema interfaces to explicitly define the form of configuration, input, and output data

• Ask other services questions as part of your app (i.e. build trees of services)

• Automatically display readable, colourised logs, or use your own log handler

• Avoid time-consuming and confusing devops, cloud configuration, and backend maintenance

High standards, quick responses, and good intentions

• Open-source and transparent on GitHub - anyone can see the code and raise an issue

• Automated testing, standards, releases, and deployment

• High test coverage

• Works on MacOS, Linux, and Windows

• Developed not-for-profit for the renewable energy industry

Need help, found a bug, or want to request a new feature?

We use GitHub Issues 1 to manage:

• Bug reports

• Feature requests

• Support requests

Footnotes

1

Bug reports, feature requests and support requests, may also be made directly to your Octue support contact, or via the support pages.

Installation

Pip

pip install octue==x.y.z

Poetry

Read more about Poetry here.

poetry add octue=x.y.z

Add to your dependencies

To use a specific version of the Octue SDK in your python application, simply add:

octue==x.y.z

to your requirements.txt or setup.py file, where x.y.z is your preferred version of the SDK (we recommend the latest stable version).

Datafiles, datasets, and manifests

One of the main features of octue is making using, creating, and sharing scientific datasets easy. There are three main data classes in the SDK that do this.

• Datafile - a single local or cloud file and its metadata.

• Dataset - a set of related datafiles that exist in the same location, plus metadata.

• Manifest - a set of related datasets that exist anywhere, plus metadata. Typically produced by or for one analysis.

Datafile

Definitions

Datafile

A single local or cloud file, its metadata, and helper methods.

Locality

A datafile has one of these localities:

• Cloud-based: it exists only in the cloud

• Local: it exists only on your local filesystem

• Cloud-based and local: it’s cloud-based but has been downloaded for low-latency reading/writing

Tip

Use a datafile to work with a file if you want to:

• Read/write to local and cloud files in the same way

• Include it in a dataset that can be sent to an Octue service for processing

• Add metadata to it for future sorting and filtering

Key features

Work with local and cloud data

Working with a datafile is the same whether it’s local or cloud-based. It’s also almost identical to using python built-in open function. For example, to write to a datafile:

from octue.resources import Datafile

datafile = Datafile("path/to/file.dat")

# Or:

datafile = Datafile("gs://my-bucket/path/to/file.dat")

with datafile.open("w") as f:
    f.write("Some data")
    datafile.labels.add("processed")

All the same file modes you’d use with the python built-in open context manager are available for datafiles e.g. "r" and "a".

Automatic lazy downloading

Save time and bandwidth by only downloading when necessary.

Downloading data from cloud datafiles is automatic and lazy so you get both low-latency content read/write and quick metadata reads. This makes viewing and filtering by the metadata of cloud datasets and datafiles quick and avoids unnecessary data transfer, energy usage, and costs.

Datafile content isn’t downloaded until you:

• Try to read or write its contents using the Datafile.open context manager

• Call its download method

• Use its local_path property

Read more about downloading files here.

CLI command friendly

Use any command line tool on your datafiles. Datafiles are python objects, but they represent real files that can be fed to any CLI command you like

import subprocess
output = subprocess.check_output(["openfast", datafile.local_path])

Easy and expandable custom metadata

Find the needle in the haystack by making your data searchable. You can set the following metadata on a datafile:

• Timestamp

• Labels (a set of lowercase strings)

• Tags (a dictionary of key-value pairs)

This metadata is stored locally in a .octue file for local datafiles or on the cloud objects for cloud datafiles and is used during Datafile instantiation. It can be accessed like this:

datafile.timestamp
>>> datetime.datetime(2022, 5, 4, 17, 57, 57, 136739)

datafile.labels
>>> {"processed"}

datafile.tags
>>> {"organisation": "octue", "energy": "renewable"}

You can update the metadata by setting it on the instance while inside the Datafile.open context manager.

with datafile.open("a"):
    datafile.labels.add("updated")

You can do this outside the context manager too, but you then need to call the update method:

datafile.labels.add("updated")
datafile.update_metadata()

Upload an existing local datafile

Back up and share your datafiles for collaboration. You can upload an existing local datafile to the cloud without using the Datafile.open context manager if you don’t need to modify its contents:

datafile.upload("gs://my-bucket/my_datafile.dat", update_metadata=True)

Get file and metadata hashes

Make your analysis reproducible: guarantee a datafile contains exactly the same data as before by checking its hash.

datafile.hash_value
>>> 'mnG7TA=='

You can also check that any metadata is the same.

datafile.metadata_hash_value
>>> 'DIgCHg=='

Immutable ID

Each datafile has an immutable UUID:

datafile.id
>>> '9a1f9b26-6a48-4f2d-be80-468d3270d79b'

Check a datafile’s locality

Is this datafile local or in the cloud?

datafile.exists_locally
>>> True

datafile.exists_in_cloud
>>> False

A cloud datafile that has been downloaded will return True for both of these properties.

Represent HDF5 files

Support fast I/O processing and storage.

Warning

If you want to represent HDF5 files with a Datafile, you must include the extra requirements provided by the hdf5 key at installation i.e.

pip install octue[hdf5]

or

poetry add octue -E hdf5

Usage examples

The Datafile class can be used functionally or as a context manager. When used as a context manager, it is analogous with the python built-in open function. On exiting the context (the with block), it closes the datafile locally and, if the datafile also exists in the cloud, updates the cloud object with any data or metadata changes.

Example A

Scenario: Download a cloud object, calculate Octue metadata from its contents, and add the new metadata to the cloud object

Starting point: Object in cloud with or without Octue metadata

Goal: Object in cloud with updated metadata

from octue.resources import Datafile


datafile = Datafile("gs://my-bucket/path/to/data.csv")

with datafile.open() as f:
    data = f.read()
    new_metadata = metadata_calculating_function(data)

    datafile.timestamp = new_metadata["timestamp"]
    datafile.tags = new_metadata["tags"]
    datafile.labels = new_metadata["labels"]

Example B

Scenario: Add or update Octue metadata on an existing cloud object without downloading its content

Starting point: A cloud object with or without Octue metadata

Goal: Object in cloud with updated metadata

from datetime import datetime
from octue.resources import Datafile


datafile = Datafile("gs://my-bucket/path/to/data.csv")

datafile.timestamp = datetime.now()
datafile.tags = {"manufacturer": "Vestas", "output": "1MW"}
datafile.labels = {"new"}

datafile.upload(update_metadata=True)  # Or, datafile.update_metadata()

Example C

Scenario: Read in the data and Octue metadata of an existing cloud object without intent to update it in the cloud

Starting point: A cloud object with Octue metadata

Goal: Cloud object data (contents) and metadata held locally in local variables

from octue.resources import Datafile


datafile = Datafile("gs://my-bucket/path/to/data.csv")

with datafile.open() as f:
    data = f.read()

metadata = datafile.metadata()

Example D

Scenario: Create a new cloud object from local data, adding Octue metadata

Starting point: A file-like locally (or content data in local variable) with Octue metadata stored in local variables

Goal: A new object in the cloud with data and Octue metadata

For creating new data in a new local file:

from octue.resources import Datafile


datafile = Datafile(
    "path/to/local/file.dat",
    tags={"cleaned": True, "type": "linear"},
    labels={"Vestas"}
)

with datafile.open("w") as f:
    f.write("This is some cleaned data.")

datafile.upload("gs://my-bucket/path/to/data.dat")

For existing data in an existing local file:

from octue.resources import Datafile


tags = {"cleaned": True, "type": "linear"}
labels = {"Vestas"}

datafile = Datafile(path="path/to/local/file.dat", tags=tags, labels=labels)
datafile.upload("gs://my-bucket/path/to/data.dat")

Dataset

Definitions

Dataset

A set of related datafiles that exist in the same location, dataset metadata, and helper methods.

Locality

A dataset has one of these localities:

• Cloud-based: it exists only in the cloud

• Local: it exists only on your local filesystem

Tip

Use a dataset if you want to:

• Group together a set of files that naturally relate to each other e.g. a timeseries that’s been split into multiple files.

• Add metadata to it for future sorting and filtering

• Include it in a manifest with other datasets and send them to an Octue service for processing

Key features

Work with local and cloud datasets

Working with a dataset is the same whether it’s local or cloud-based.

from octue.resources import Dataset

dataset = Dataset(path="path/to/dataset", recursive=True)

dataset = Dataset(path="gs://my-bucket/path/to/dataset", recursive=True)

Upload a dataset

Back up and share your datasets for collaboration.

dataset.upload("gs://my-bucket/path/to/upload")

Download a dataset

Use a shared or public dataset or retrieve a backup.

dataset.download("path/to/download")

Easy and expandable custom metadata

Find the needle in the haystack by making your data searchable. You can set the following metadata on a dataset:

• Name

• Labels (a set of lowercase strings)

• Tags (a dictionary of key-value pairs)

This metadata is stored locally in a .octue file in the same directory as the dataset and is used during Dataset instantiation. It can be accessed like this:

dataset.name
>>> "my-dataset"

dataset.labels
>>> {"processed"}

dataset.tags
>>> {"organisation": "octue", "energy": "renewable"}

You can update the metadata by setting it on the instance while inside the Dataset context manager.

with dataset:
    datafile.labels.add("updated")

You can do this outside the context manager too, but you then need to call the update method:

dataset.labels.add("updated")
dataset.update_metadata()

Get dataset and metadata hashes

Make your analysis reproducible: guarantee a dataset contains exactly the same data as before by checking its hash.

dataset.hash_value
>>> 'uvG7TA=='

Note

A dataset’s hash is a function of its datafiles’ hashes. Datafile and dataset metadata do not affect it.

You can also check that dataset metadata is the same.

dataset.metadata_hash_value
>>> 'DIgCHg=='

Immutable ID

Each dataset has an immutable UUID:

dataset.id
>>> '9a1f9b26-6a48-4f2d-be80-468d3270d79c'

Check a dataset’s locality

Is this dataset local or in the cloud?

dataset.exists_locally
>>> True

dataset.exists_in_cloud
>>> False

A dataset can only return True for one of these at a time.

Filter datasets

Narrow down a dataset to just the files you want to avoiding extra downloading and processing.

Datafiles in a dataset are stored in a FilterSet, meaning they can be easily filtered by any attribute of the datafiles contained e.g. name, extension, ID, timestamp, tags, labels, size. The filtering syntax is similar to Django’s i.e.

# Get datafiles that have an attribute that satisfies the filter.
dataset.files.filter(<datafile_attribute>__<filter>=<value>)

# Or, if your filter is a simple equality filter:
dataset.files.filter(<datafile_attribute>=<value>)

Here’s an example:

# Make a dataset.
dataset = Dataset(
    path="blah",
    files=[
        Datafile(path="my_file.csv", labels=["one", "a", "b" "all"]),
        Datafile(path="your_file.txt", labels=["two", "a", "b", "all"),
        Datafile(path="another_file.csv", labels=["three", "all"]),
    ]
)

# Filter it!
dataset.files.filter(name__starts_with="my")
>>> <FilterSet({<Datafile('my_file.csv')>})>

dataset.files.filter(extension="csv")
>>> <FilterSet({<Datafile('my_file.csv')>, <Datafile('another_file.csv')>})>

dataset.files.filter(labels__contains="a")
>>> <FilterSet({<Datafile('my_file.csv')>, <Datafile('your_file.txt')>})>

You can iterate through the filtered files:

for datafile in dataset.files.filter(labels__contains="a"):
    print(datafile.name)
>>> 'my_file.csv'
    'your_file.txt'

If there’s just one result, get it via the FilterSet.one method:

dataset.files.filter(name__starts_with="my").one()
>>> <Datafile('my_file.csv')>

You can also chain filters or specify them all at the same time - these two examples produce the same result:

# Chaining multiple filters.
dataset.files.filter(extension="csv").filter(labels__contains="a")
>>> <FilterSet({<Datafile('my_file.csv')>})>

# Specifying multiple filters at once.
dataset.files.filter(extension="csv", labels__contains="a")
>>> <FilterSet({<Datafile('my_file.csv')>})>

For the full list of available filters, click here.

Order datasets

A dataset can also be ordered by any of the attributes of its datafiles:

dataset.files.order_by("name")
>>> <FilterList([<Datafile('another_file.csv')>, <Datafile('my_file.csv')>, <Datafile(path="your_file.txt")>])>

The ordering can also be carried out in reverse (i.e. descending order) by passing reverse=True as a second argument to the FilterSet.order_by method.

Manifest

Definitions

Manifest

A set of related cloud and/or local datasets, metadata, and helper methods. Typically produced by or needed for processing by an Octue service.

Tip

Use a manifest to send datasets to an Octue service as a question (for processing) - the service will send an output manifest back with its answer if the answer includes output datasets.

Key features

Group related datasets together

Make a clear grouping of datasets needed for a particular analysis.

from octue.resources import Manifest

manifest = Manifest(
    datasets={
        "my_dataset_0": "gs://my-bucket/my_dataset_0",
        "my_dataset_1": "gs://my-bucket/my_dataset_1",
        "my_dataset_2": "gs://another-bucket/my_dataset_2",
    }
)

Send datasets to a service

Get an Octue service to analyse data for you as part of a larger analysis.

from octue.resources import Child

child = Child(
    id="octue/wind-speed:latest",
    backend={"name": "GCPPubSubBackend", "project_name": "my-project"},
)

answer = child.ask(input_manifest=manifest)

See here for more information.

Receive datasets from a service

Get output datasets from an Octue service from the cloud when you’re ready.

answer["output_manifest"]["an_output_dataset"].files
>>> <FilterSet({<Datafile('my_file.csv')>, <Datafile('another_file.csv')>})>

Hint

Datasets in an output manifest are stored in the cloud. You’ll need to keep a reference to where they are to access them - the output manifest is this reference. You’ll need to use it straight away or save it to make use of it.

Further information

Manifests of local datasets

You can include local datasets in your manifest if you can guarantee all services that need them can access them. A use case for this is, for example, a supercomputer cluster running several octue services locally that process and transfer large amounts of data. It is much faster to store and access the required datasets locally than upload them to the cloud and then download them again for each service (as would happen with cloud datasets).

Warning

If you want to ask a child a question that includes a manifest containing one or more local datasets, you must include the allow_local_files parameter. For example, if you have an analysis object with a child called “wind_speed”:

input_manifest = Manifest(
    datasets={
        "my_dataset_0": "gs://my-bucket/my_dataset_0",
        "my_dataset_1": "local/path/to/my_dataset_1",
    }
)

analysis.children["wind_speed"].ask(
    input_values=analysis.input_values,
    input_manifest=analysis.input_manifest,
    allow_local_files=True,
)

Asking services questions

Octue services

There’s a growing range of live services in the Octue ecosystem that you can ask questions to and get answers from. Currently, all of them are related to wind energy. Here’s a quick glossary of terms before we tell you more:

Definitions

Child

An Octue service that can be asked a question. This name reflects the tree structure of services (specifically, a DAG) formed by the service asking the question (the parent), the child it asks the question to, any children that the child asks questions to as part of forming its answer, and so on.

Parent

An Octue service that asks a question to another Octue service (a child).

Asking a question

Sending data (input values and/or an input manifest) to a child for processing/analysis.

Receiving an answer

Receiving data (output values and/or an output manifest) from a child you asked a question to.

Octue ecosystem

The set of services running the octue SDK as their backend. These services guarantee:

• Defined JSON schemas and validation for input and output data

• An easy interface for asking them questions and receiving their answers

• Logs and exceptions (and potentially monitor messages) forwarded to you

• High availability if deployed in the cloud

How to ask a question

You can ask any service a question if you have its service ID, project name, and permissions. The question is formed of input values and/or an input manifest.

from octue.resources import Child

child = Child(
    id="my-organisation/my-service:latest",
    backend={"name": "GCPPubSubBackend", "project_name": "my-project"},
)

answer = child.ask(
    input_values={"height": 32, "width": 3},
    input_manifest=manifest,
)

answer["output_values"]
>>> {"some": "data"}

answer["output_manifest"]["my_dataset"].files
>>> <FilterSet({<Datafile('my_file.csv')>, <Datafile('another_file.csv')>})>

You can also set the following options when you call Child.ask:

• children - If the child has children of its own (i.e. grandchildren of the parent), this optional argument can be used to override the child’s “default” children. This allows you to specify particular versions of grandchildren to use (see this subsection below).

• subscribe_to_logs - if true, the child will forward its logs to you

• allow_local_files - if true, local files/datasets are allowed in any input manifest you supply

• handle_monitor_message - if provided a function, it will be called on any monitor messages from the child

• record_messages_to – if given a path to a JSON file, messages received from the parent while it processes the question are saved to it

• allow_save_diagnostics_data_on_crash – if true, the input values and input manifest (including its datasets) will be saved by the child for future crash diagnostics if it fails while processing them

• question_uuid - if provided, the question will use this UUID instead of a generated one

• timeout - how long in seconds to wait for an answer (None by default - i.e. don’t time out)

If a child raises an exception while processing your question, the exception will always be forwarded and re-raised in your local service or python session. You can handle exceptions in whatever way you like.

If setting a timeout, bear in mind that the question has to reach the child, the child has to run its analysis on the inputs sent to it (this most likely corresponds to the dominant part of the wait time), and the answer has to be sent back to the parent. If you’re not sure how long a particular analysis might take, it’s best to set the timeout to None initially or ask the owner/maintainer of the child for an estimate.

Asking multiple questions in parallel

You can also ask multiple questions to a service in parallel.

child.ask_multiple(
    {"input_values": {"height": 32, "width": 3}},
    {"input_values": {"height": 12, "width": 10}},
    {"input_values": {"height": 7, "width": 32}},
)
>>> [
        {"output_values": {"some": "output"}, "output_manifest": None},
        {"output_values": {"another": "result"}, "output_manifest": None},
        {"output_values": {"different": "result"}, "output_manifest": None},
    ]

This method uses threads, allowing all the questions to be asked at once instead of one after another.

Asking a question within a service

If you have created your own Octue service and want to ask children questions, you can do this more easily than above. Children are accessible from the analysis object by the keys you give them in the app configuration file. For example, you can ask an elevation service a question like this:

answer = analysis.children["elevation"].ask(input_values={"longitude": 0, "latitude": 1})

if your app configuration file is:

{
  "children": [
    {
      "key": "wind_speed",
      "id": "template-child-services/wind-speed-service:latest",
      "backend": {
        "name": "GCPPubSubBackend",
        "project_name": "my-project"
      }
    },
    {
      "key": "elevation",
      "id": "template-child-services/elevation-service:latest",
      "backend": {
        "name": "GCPPubSubBackend",
        "project_name": "my-project"
      }
    }
  ]
}

and your twine.json file includes the child keys in its children field:

{
    "children": [
        {
            "key": "wind_speed",
            "purpose": "A service that returns the average wind speed for a given latitude and longitude.",
        },
        {
            "key": "elevation",
            "purpose": "A service that returns the elevation for a given latitude and longitude.",
        }
    ]
}

See the parent service’s app configuration and app.py file in the child-services app template to see this in action.

Overriding a child’s children

If the child you’re asking a question to has its own children (static children), you can override these by providing the IDs of the children you want it to use (dynamic children) to the Child.ask method. Questions that would have gone to the static children will instead go to the dynamic children. Note that:

• You must provide the children in the same format as they’re provided in the app configuration

• If you override one static child, you must override others, too

• The dynamic children must have the same keys as the static children (so the child knows which service to ask which questions)

• You should ensure the dynamic children you provide are compatible with and appropriate for questions from the child service

For example, if the child requires these children in its app configuration:

[
    {
        "key": "wind_speed",
        "id": "template-child-services/wind-speed-service:latest",
        "backend": {
            "name": "GCPPubSubBackend",
            "project_name": "octue-amy"
        },
    },
    {
        "key": "elevation",
        "id": "template-child-services/elevation-service:latest",
        "backend": {
            "name": "GCPPubSubBackend",
            "project_name": "octue-amy"
        },
    }
]

then you can override them like this:

answer = child.ask(
    input_values={"height": 32, "width": 3},
    children=[
        {
            "key": "wind_speed",
            "id": "my/own-service:latest",
            "backend": {
                "name": "GCPPubSubBackend",
                "project_name": "octue-amy"
            },
        },
        {
            "key": "elevation",
            "id": "organisation/another-service:latest",
            "backend": {
                "name": "GCPPubSubBackend",
                "project_name": "octue-amy"
            },
        },
    ],
)

Overriding beyond the first generation

It’s an intentional choice to only go one generation deep with overriding children. If you need to be able to specify a whole tree of children, grandchildren, and so on, please upvote this issue.

Creating services

One of the main features of the Octue SDK is to allow you to easily create services that can accept questions and return answers. They can run locally on any machine or be deployed to the cloud. Currently:

• The backend communication between twins uses Google Pub/Sub whether they’re local or deployed

• The deployment options are Google Cloud Run or Google Dataflow

• The language of the entrypoint must by python3 (you can call processes using other languages within this though)

Anatomy of an Octue service

An Octue service is defined by the following files (located in the repository root by default).

app.py

None

This is where you write your app. The app.py file can contain any valid python, including import and use of any number of external packages or your own subpackages. It has only two requirements:

1. It must contain exactly one of the octue python app interfaces that serve as an entrypoint to your code. These take a single Analysis instance:

   • Option 1: A function named run with the following signature:

     def run(analysis):
         """A function that uses input and configuration from an ``Analysis`` instance and stores any
         output values and output manifests on it.
     
         :param octue.resources.Analysis analysis:
         :return None:
         """
         ...
     

   • Option 2: A class named App with the following signature:

     class App:
         """A class that takes an ``Analysis`` instance and anything else you like. It can contain any
         methods you like but it must also have a ``run`` method.
     
         :param octue.resources.Analysis analysis:
         :return None:
         """
     
         def __init__(self, analysis, *args, **kwargs):
             self.analysis = analysis
             ...
     
         def run(self):
             """A method that that uses input and configuration from an ``Analysis`` instance and stores
             any output values and output manifests on it.
     
             :return None:
             """
             ...
     
         ...
     

2. It must access configuration/input data from and store output data on the analysis parameter/attribute:

• Configuration values: analysis.configuration_values

• Configuration manifest: analysis.configuration_manifest

• Input values: analysis.input_values

• Input manifest: analysis.input_manifest

• Output values: analysis.output_values

• Output manifest: analysis.output_manifest

This allows standardised configuration/input/output for services while allowing you to do anything you like with the input data to produce the output data.

twine.json

None

This file defines your schema for configuration, input, and output values and manifests. Read more here and see an example here.

Dependencies file

None

This is a setup.py file (read more here) or requirements.txt file (read more here) listing all the python packages your app depends on and the version ranges that will work with your app.

octue.yaml

None

This file defines the basic structure of your service. It must contain at least:

services:
  - namespace: my-organisation
    name: my-app

It may also need the following key-value pairs:

• app_source_path: <path> - if your app.py file is not in the repository root

• app_configuration_path: <path> - if your app needs an app configuration file that isn’t in the repository root

• dockerfile_path: <path> - if your app needs a Dockerfile that isn’t in the repository root

All paths should be relative to the repository root. Other valid entries can be found in the ServiceConfiguration constructor.

Warning

Currently, only one service can be defined per repository, but it must still appear as a list item of the “services” key. At some point, it will be possible to define multiple services in one repository.

App configuration file (optional)

None

If your app needs any configuration, asks questions to any other Octue services, or produces output datafiles/datasets, you will need to provide an app configuration. Currently, this must take the form of a JSON file. It can contain the following keys:

• configuration_values

• configuration_manifest

• children

• output_location

If an app configuration file is provided, its path must be specified in octue.yaml under the “app_configuration_path” key.

See the AppConfiguration constructor for more information.

Dockerfile (optional)

None

Octue services run in a Docker container if they are deployed. They can also run this way locally. The SDK provides a default Dockerfile for these purposes that will work for most cases:

• For deploying to Google Cloud Run

• For deploying to Google Dataflow

However, you may need to write and provide your own Dockerfile if your app requires:

• Non-python or system dependencies (e.g. openfast, wget)

• Python dependencies that aren’t installable via pip

• Private python packages

Here are two examples of a custom Dockerfile that use different base images:

• A TurbSim service

• An OpenFAST service

If you do provide one, you must specify its path in octue.yaml under the dockerfile_path key.

As always, if you need help with this, feel free to drop us a message or raise an issue!

Naming services

Definitions

Service revision

A specific instance of an Octue service that can be individually addressed. The revision could correspond to a version of the service, a dynamic development branch for it, or a deliberate duplication or variation of it.

Service revision unique identifier (SRUID)

The combination of a service revisions’s namespace, name, and revision tag that uniquely identifies it. For example, octue/my-service:1.3.0 where the namespace is octue, the name is my-service, and the revision tag is 1.3.0.

Service namespace

The group to which the service belongs e.g. your name or your organisation’s name. If in doubt, use the GitHub handle of the user or organisation publishing the services.

Namespaces must be lower kebab case (i.e. they may contain the letters [a-z], numbers [0-9], and hyphens [-]). They may not begin or end with hyphens.

Service name

A name to uniquely identify the service within its namespace. This usually corresponds to the name of the GitHub repository for the service. Names must be lower kebab case (i.e. they may contain the letters [a-z], numbers [0-9] and hyphens [-]). They may not begin or end with hyphens.

Service revision tag

A tag that uniquely identifies a particular revision of a service. The revision tag could correspond to a commit hash like a3eb45, a release number like 0.12.4, a branch name (e.g. development), a particular environment the service is deployed in (e.g. production), or a combination like 0.12.4-production. Tags may contain lowercase and uppercase letters, numbers, underscores, periods, and hyphens, but can’t start with a period or a dash. They can contain a maximum of 128 characters. These requirements are the same as the Docker tag format.

Service ID

The SRUID is a special case of the service ID. A service ID can be an SRUID or just the service namespace and name. It can be used to ask a question to a service without specifying a specific revision of it. This enables asking questions to, for example, the service octue/my-service and automatically having them routed to its latest revision. Note that this will be a future feature; currently, you will still be required to provide a revision tag (i.e. a full SRUID).

Where to specify the namespace, name, and revision tag

Namespace

• Required: yes

• Set in:

  • octue.yaml

  • OCTUE_SERVICE_NAMESPACE environment variable (takes priority)

Name

• Required: yes

• Set in:

  • octue.yaml

  • OCTUE_SERVICE_NAME environment variable (takes priority)

Revision tag

• Required: no

• Default: a random “coolname” (e.g. hungry-hippo)

• Set in:

  • OCTUE_SERVICE_REVISION_TAG environment variable

  • If using octue start command, the --revision-tag option (takes priority)

Template apps

We’ve created some template apps for you to look at and play around with. We recommend going through them in this order:

1. The fractal app template - introduces a basic Octue service that returns output values to its parent.

2. The using-manifests app template - introduces using a manifest of output datasets to return output files to its parent.

3. The child-services app template - introduces asking questions to child services and using their answers to form an output to return to its parent.

Deploying services automatically

Automated deployment with Octue means:

• Your service runs in Google Cloud, ready to accept questions from and return answers to other services.

• You don’t need to do anything to update your deployed service with new code changes - the service simply gets rebuilt and re-deployed each time you push a commit to your main branch, or merge a pull request into it (other branches and deployment strategies are available, but this is the default).

• Serverless is the default - your service only runs when questions from other services are sent to it, meaning there is no cost to having it deployed but not in use.

To enable automated deployments, contact us so we can create a Google Cloud Build trigger linked to your git repository. This requires no work from you apart from authorising the connection to GitHub (or another git provider).

If you want to deploy services yourself, see here.

Running services locally

Services can be operated locally (e.g. for testing or ad-hoc data processing). You can:

• Run your service once (i.e. run one analysis):

  • Via the CLI

  • By using the octue library in a python script

• Start your service as a child, allowing it to answer any number of questions from any other Octue service:

  • Via the CLI

Running a service once

Via the CLI

1. Ensure you’ve created a valid octue.yaml file for your service

2. If your service requires inputs, create an input directory with the following structure

   input_directory
   |---  values.json    (if input values are required)
   |---  manifest.json  (if an input manifest is required)
   

3. Run:

   octue run --input-dir=my_input_directory
   

Any output values will be printed to stdout and any output datasets will be referenced in an output manifest file named output_manifest_<analysis_id>.json.

Via a python script

Imagine we have a simple app that calculates the area of a square. It could be run locally on a given height and width like this:

from octue import Runner

runner = Runner(app_src="path/to/app.py", twine="path/to/twine.json")
analysis = runner.run(input_values={"height": 5, "width": 10})

analysis.output_values
>>> {"area": 50}

analysis.output_manifest
>>> None

See the Runner API documentation for more advanced usage including providing configuration, children, and an input manifest.

Starting a service as a child

Via the CLI

1. Ensure you’ve created a valid octue.yaml file for your service

2. Run:

   octue start
   

This will run the service as a child waiting for questions until you press Ctrl + C or an error is encountered. The service will be available to be questioned by other services at the service ID organisation/name as specified in the octue.yaml file.

Tip

You can use the --timeout option to stop the service after a given number of seconds.

Deploying services (developer’s guide)

This is a guide for developers that want to deploy Octue services themselves - it is not needed if Octue manages your services for you or if you are only asking questions to existing Octue services.

Attention

The octue deploy CLI command can be used to deploy services automatically, but it:

• Is in alpha so may not work as intended

• Requires the gcloud CLI tool with Google Cloud SDK 367.0.0 and beta 2021.12.10 to be available

• Requires the correct permissions via the gcloud tool logged into a Google user account and/or with an appropriate service account available

For now, we recommend contacting us to help set up deployments for you.

What deployment enables

Deploying an Octue service to Google Cloud Run means it:

• Is deployed as a docker container

• Is ready to be asked questions by any other Octue service that has the correct permissions (you can control this)

• Can ask questions to any other Octue service for which it has the correct permissions

• Will automatically build and redeploy upon the conditions you provide (e.g. pushes or merges into main)

• Will automatically start and run when Pub/Sub messages are received from the topic you created. The Pub/Sub messages can be sent from anywhere in the world, but the container will only run in the region you chose (you can create multiple Cloud Run services in different regions for the same repository if this is a problem).

• Will automatically stop shortly after finishing the analyses asked for in the Pub/Sub message (although you can set a minimum container count so one is always running to minimise cold starts).

How to deploy

1. Ensuring you are in the desired project, go to the Google Cloud Run page and create a new service

2. Give your service a unique name

3. Choose a low-carbon region that supports Eventarc triggers and is in a convenient geographic location for you (e.g. physically close to you for low latency or in a region compatible with your data protection requirements).

3. Click “Next”. When changes are made to the source code, we want them to be deployed automatically. So, we need to connect the repository to GCP to enable this. Select “Continuously deploy new revisions from a source repository” and then “Set up with cloud build”.

4. Choose your source code repository provider and the repository containing the code you’d like to deploy. You’ll have to give the provider permission to access the repository. If your provider isn’t GitHub, BitBucket, or Google Cloud Source Repositories (GCSR), you’ll need to mirror the repository to GCSR before completing this step as Google Cloud Build only supports these three providers currently.

5. Click “Next”, enter a regular expression for the branches you want to automatically deploy from (main by default). As the service will run in a docker container, select “Dockerfile” and click “Save”.

6. Click “Next”. If you want your service to be private, select “Allow internal traffic only” and “Require authentication”. This stops anyone without permission from using the service.

7. The service needs a trigger to start up and respond to. We’ll be using Google Pub/Sub. Click “Add eventarc trigger”, choose “Cloud Pub/Sub topic” as the trigger event, click on the menu called “Select a Cloud Pub/Sub topic”, then click “Create a topic”. Any services that want to ask your service a question will publish their question to this topic.

8. The topic ID should be in the form octue.services.my-organisation.my-service. Click “Create topic”.

9. Under “Invocation settings”, click on the “Service account” menu and then “Create new service account”.

10. Make a new service account with a related name e.g. “my-service”, then click “Create”. Add the “octue-service-user” and “Cloud Run Invoker” roles to the service account. Contact us if the “octue-service-user” role is not available.

11. Click “Save” and then “Create”.

12. You can now view your service in the list of Cloud Run services and view its build trigger in the list of Cloud Build triggers.

Testing services

We recommend writing automated tests for your service so anyone who wants to use it can have confidence in its quality and reliability at a glance. Here’s an example test for our example service.

Emulating children

If your app has children, you should emulate them in your tests instead of communicating with the real ones. This makes your tests:

• Independent of anything external to your app code - i.e. independent of the remote child, your internet connection, and communication between your app and the child (Google Pub/Sub).

• Much faster - the emulation will complete in a few milliseconds as opposed to the time it takes the real child to actually run an analysis, which could be minutes, hours, or days. Tests for our child services template app run around 900 times faster when the children are emulated.

The Child Emulator

We’ve written a child emulator that takes a list of messages and returns them to the parent for handling in the order given - without contacting the real child or using Pub/Sub. Any messages a real child can produce are supported. Child instances can be mocked like-for-like by ChildEmulator instances without the parent knowing. You can provide the emulated messages in python or via a JSON file.

Message types

You can emulate any message type that your app (the parent) can handle. The table below shows what these are.

Message type	Number of messages supported	Example
log_record	Any number	{“type”: “log_record”: “log_record”: {“msg”: “Starting analysis.”}}
monitor_message	Any number	{“type”: “monitor_message”: “data”: ‘{“progress”: “35%”}’}
exception	One	{“type”: “exception”, “exception_type”: “ValueError”, “exception_message”: “x cannot be less than 10.”}
result	One	{“type”: “result”, “output_values”: {“my”: “results”}, “output_manifest”: None}

Notes

• Message formats and contents are validated by ChildEmulator

• The log_record key of a log_record message is any dictionary that the logging.makeLogRecord function can convert into a log record.

• The data key of a monitor_message message must be a JSON-serialised string

• Any messages after a result or exception message won’t be passed to the parent because execution of the child emulator will have ended.

Instantiating a child emulator in python

messages = [
    {
        "type": "log_record",
        "log_record": {"msg": "Starting analysis."},
    },
    {
        "type": "monitor_message",
        "data": '{"progress": "35%"}',
    },
    {
        "type": "log_record",
        "log_record": {"msg": "Finished analysis."},
    },
    {
        "type": "result",
        "output_values": [1, 2, 3, 4, 5],
        "output_manifest": None,
    },
]

child_emulator = ChildEmulator(
    backend={"name": "GCPPubSubBackend", "project_name": "my-project"},
    messages=messages
)

def handle_monitor_message(message):
    ...

result = child_emulator.ask(
    input_values={"hello": "world"},
    handle_monitor_message=handle_monitor_message,
)
>>> {"output_values": [1, 2, 3, 4, 5], "output_manifest": None}

Instantiating a child emulator from a JSON file

You can provide a JSON file with either just messages in or with messages and some or all of the ChildEmulator constructor parameters. Here’s an example JSON file with just the messages:

{
    "messages": [
        {
            "type": "log_record",
            "log_record": {"msg": "Starting analysis."}
        },
        {
            "type": "log_record",
            "log_record": {"msg": "Finished analysis."}
        },
        {
            "type": "monitor_message",
            "data": "{\"progress\": \"35%\"}"
        },
        {
            "type": "result",
            "output_values": [1, 2, 3, 4, 5],
            "output_manifest": null
        }
    ]
}

You can then instantiate a child emulator from this in python:

child_emulator = ChildEmulator.from_file("path/to/emulated_child.json")

def handle_monitor_message(message):
    ...

result = child_emulator.ask(
    input_values={"hello": "world"},
    handle_monitor_message=handle_monitor_message,
)
>>> {"output_values": [1, 2, 3, 4, 5], "output_manifest": None}

Using the child emulator

To emulate your children in tests, patch the Child class with the ChildEmulator class.

from unittest.mock import patch

from octue import Runner
from octue.cloud.emulators import ChildEmulator


app_directory_path = "path/to/directory_containing_app"

# You can explicitly specify your children here as shown or
# read the same information in from your app configuration file.
children = [
    {
        "key": "my_child",
        "id": "octue/my-child-service:latest",
        "backend": {
            "name": "GCPPubSubBackend",
            "project_name": "my-project"
        }
    },
]

runner = Runner(
    app_src=app_directory_path,
    twine=os.path.join(app_directory_path, "twine.json"),
    children=children,
    service_id="you/your-service:latest",
)

emulated_children = [
    ChildEmulator(
        id="octue/my-child-service:latest",
        internal_service_name="you/your-service:latest",
        messages=[
            {
                "type": "result",
                "output_values": [300],
                "output_manifest": None,
            },
        ]
    )
]

with patch("octue.runner.Child", side_effect=emulated_children):
    analysis = runner.run(input_values={"some": "input"})

Notes

• If your app uses more than one child, provide more child emulators in the emulated_children list in the order they’re asked questions in your app.

• If a given child is asked more than one question, provide a child emulator for each question asked in the same order the questions are asked.

Creating a test fixture

Since the child is emulated, it doesn’t actually do any calculation - if you change the inputs, the outputs won’t change correspondingly (or at all). So, it’s up to you to define a set of realistic inputs and corresponding outputs (the list of emulated messages) to test your service. These are called test fixtures.

Note

Unlike a real child, the inputs given to the emulator and the outputs returned aren’t validated against the schema in the child’s twine - this is because the twine is only available to the real child. This is ok - you’re testing your service, not the child.

You can create test fixtures manually or by recording messages from a real child to a JSON file. To record messages:

import json
from octue.resources import Child


child = Child(
    id="octue/my-child:latest",
    backend={"name": "GCPPubSubBackend", "project_name": "my-project"},
)

result = child.ask(
    input_values=[1, 2, 3, 4],
    record_messages_to="child_messages.json",
)

with open("child_messages.json") as f:
    child_messages = json.load(f)

child_messages
>>> [
        {
            'type': 'delivery_acknowledgement',
            'delivery_time': '2022-08-16 11:49:57.244263',
            'message_number': 0
        },
        {
            'type': 'log_record',
            'log_record': {
                'msg': 'Finished analysis.',
                'args': None,
                'levelname': 'INFO',
                ...
            },
            'analysis_id': '0ce8386d-564d-47fa-9d11-3b728f557bfe',
            'message_number': 1
        },
        {
            'type': 'result',
            'output_values': {"some": "results"},
            'output_manifest': None,
            'message_number': 2
        }
    ]

You can then feed these into a child emulator to emulate one possible response of the child:

from octue.cloud.emulators import ChildEmulator


child_emulator = ChildEmulator(messages=child_messages)

child_emulator.ask(input_values=[1, 2, 3, 4])
>>> {"some": "results"}

Troubleshooting services

Allowing crash diagnostics

A parent can give a child permission to save the following data to the cloud in the event the child fails while processing a question:

• Input values

• Input manifest and datasets

• Child configuration values

• Child configuration manifest and datasets

• Messages sent from the child to the parent

The parent can give permission on a question-by-question basis by setting allow_save_diagnostics_data_on_crash=True in Child.ask. For example:

child = Child(
    id="my-organisation/my-service:latest",
    backend={"name": "GCPPubSubBackend", "project_name": "my-project"},
)

answer = child.ask(
    input_values={"height": 32, "width": 3},
    allow_save_diagnostics_data_on_crash=True,
)

For crash diagnostics to be saved, the child must have the crash_diagnostics_cloud_path field in its service configuration (octue.yaml file) set to a Google Cloud Storage path.

Accessing crash diagnostics

In the event of a child crash, the child will upload the crash diagnostics and send the cloud path to them to the parent as a log message. A user with credentials to access this path can use the octue CLI to retrieve the crash diagnostics data:

octue get-crash-diagnostics <cloud-path>

More information on the command:

>>> octue get-crash-diagnostics -h

Usage: octue get-crash-diagnostics [OPTIONS] CLOUD_PATH

  Download crash diagnostics for an analysis from the given directory in
  Google Cloud Storage. The cloud path should end in the analysis ID.

  CLOUD_PATH: The path to the directory in Google Cloud Storage containing the
  diagnostics data.

Options:
  --local-path DIRECTORY  The path to a directory to store the directory of
                          diagnostics data in. Defaults to the current working
                          directory.
  -h, --help              Show this message and exit.

Logging

By default, octue leaves handling of log messages raised by your app to you. However, if you just want a simple, readable logging arrangement, you can let octue format and stream your logs to stderr. If this is for you, simply set USE_OCTUE_LOG_HANDLER=1 in the environment running your app. This will attach the Octue log handler to the root python logger.

Readable logs

Some advantages of the Octue log handler are:

• Its readable format

• Its clear separation of log context from log message.

Below, the context is on the left and includes:

• The time

• Log level

• Module producing the log

• Octue analysis ID

This is followed by the actual log message on the right:

[2021-07-10 20:03:12,713 | INFO | octue.runner | analysis-102ee7d5-4b94-4f8a-9dcd-36dbd00662ec] Hello! The child services template app is running!

Colourised services

Another advantage to using the Octue log handler is that each Octue service is coloured according to its position in the tree, making it much easier to read log messages from multiple levels of children.

In this example:

• The log context is in blue

• Anything running in the root parent service’s app is labeled with the analysis ID in green

• Anything running in the immediate child services (elevation and wind_speed) are labelled with the analysis ID in yellow

• Any children further down the tree (i.e. children of the child services and so on) will have their own labels in other colours consistent to their level

Add extra information

You can add certain log record attributes to the logging context by also providing the following environment variables:

• INCLUDE_LINE_NUMBER_IN_LOGS=1 - include the line number

• INCLUDE_PROCESS_NAME_IN_LOGS=1 - include the process name

• INCLUDE_THREAD_NAME_IN_LOGS=1 - include the thread name

Authentication

You need authentication while using octue to:

• Access data from Google Cloud Storage

• Use, run, or deploy Octue services

Authentication can be provided by using one of:

• A service account

• Application Default Credentials

Creating a service account

1. Create a service account (see Google’s getting started guide)

2. Make sure your service account has access to any buckets you need, Google Pub/Sub, and Google Cloud Run if your service is deployed on it (see here)

Using a service account

Locally

1. Create and download a key for your service account - it will be called your-project-XXXXX.json.

Danger

It’s best not to store this in your project to prevent accidentally committing it or building it into a docker image layer. Instead, bind mount it into your docker image from somewhere else on your local system.

If you must keep within your project, it’s good practice to name the file gha-greds-<whatever>.json and make sure that gha-creds-* is in your .gitignore and .dockerignore files.

2. If you’re developing in a container (like a VSCode .devcontainer), mount the file into the container. You can make gcloud available too - check out this tutorial.

3. Set the GOOGLE_APPLICATION_CREDENTIALS environment variable to the path of the key file.

On GCP infrastructure

• Credentials are provided when running code on GCP infrastructure (e.g. Google Cloud Run)

• octue uses these when when running on these platforms

• You should ensure the correct service account is being used by the deployed instance

Inter-service compatibility

Parents and children running nearly all versions of octue can communicate with each other compatibly, although a small number can’t. The table below shows which parent SDK versions (rows) send questions that can be processed by each child SDK version (columns).

Key

• 0 = incompatible

• 1 = compatible

	0.40.1	0.40.0	0.39.0	0.38.1	0.38.0	0.37.0	0.36.0	0.35.0	0.34.1	0.34.0	0.33.0	0.32.0	0.31.0	0.30.0	0.29.9	0.29.8	0.29.7	0.29.6	0.29.5	0.29.4	0.29.3	0.29.2	0.29.11	0.29.10	0.29.1	0.29.0	0.28.2	0.28.1	0.28.0	0.27.3	0.27.2	0.27.1	0.27.0	0.26.2	0.26.1	0.26.0	0.25.0	0.24.1	0.24.0	0.23.6	0.23.5	0.23.4	0.23.3	0.23.2	0.23.1	0.23.0	0.22.1	0.22.0	0.21.0	0.20.0	0.19.0	0.18.2	0.18.1	0.18.0	0.17.0	0.16.0
0.40.1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	0
0.40.0	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	0
0.39.0	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	0
0.38.1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	0
0.38.0	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	0
0.37.0	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	0
0.36.0	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	0
0.35.0	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	0
0.34.1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	0
0.34.0	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	0
0.33.0	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	0
0.32.0	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	0
0.31.0	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	0
0.30.0	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	0
0.29.9	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	0
0.29.8	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	0
0.29.7	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	0
0.29.6	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	0
0.29.5	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	0
0.29.4	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	0
0.29.3	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	0
0.29.2	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	0
0.29.11	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	0
0.29.10	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	0
0.29.1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	0
0.29.0	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	0
0.28.2	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	0
0.28.1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	0
0.28.0	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	0
0.27.3	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	0
0.27.2	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	0
0.27.1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	0
0.27.0	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	0
0.26.2	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	0
0.26.1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	0
0.26.0	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	0
0.25.0	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	0
0.24.1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	0
0.24.0	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	0
0.23.6	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	0
0.23.5	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	0
0.23.4	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	0
0.23.3	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	0
0.23.2	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	0
0.23.1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	0
0.23.0	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	0
0.22.1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	0
0.22.0	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	0
0.21.0	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	0
0.20.0	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	0
0.19.0	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	0
0.18.2	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	0
0.18.1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	0
0.18.0	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	0
0.17.0	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	0
0.16.0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1

API

Datafile

class octue.resources.datafile.Datafile(path, local_path=None, cloud_path=None, timestamp=None, mode='r', update_metadata=True, ignore_stored_metadata=False, id=None, tags=None, labels=None, **kwargs)

A representation of a data file with metadata.

Metadata consists of id, timestamp, tags, and labels, available as attributes on the instance. On instantiation, metadata for the file is obtained from its stored location (the corresponding cloud object metadata or a local .octue metadata file) if present. Metadata values can alternatively be passed as arguments at instantiation but will only be used if stored metadata cannot be found - i.e. stored metadata always takes precedence (use the ignore_stored_metadata parameter to override this behaviour). Stored metadata can be updated after instantiation using the update_metadata method.

Parameters

• path (str|None) – The path of this file locally or in the cloud, which may include folders or subfolders, within the dataset

• local_path (str|None) – If a cloud path is given as the path parameter, this is the path to an existing local file that is known to be in sync with the cloud object

• cloud_path (str|None) – If a local path is given for the path parameter, this is a cloud path to keep in sync with the local file

• timestamp (datetime.datetime|int|float|None) – A posix timestamp associated with the file, in seconds since epoch, typically when it was created but could relate to a relevant time point for the data

• mode (str) – if using as a context manager, open the datafile for reading/editing in this mode (the mode options are the same as for the builtin open function)

• update_metadata (bool) – if using as a context manager and this is True, update the stored metadata of the datafile when the context is exited

• ignore_stored_metadata (bool) – if True, ignore any metadata stored for this datafile locally or in the cloud and use whatever is given at instantiation

• id (str) – The Universally Unique ID of this file (checked to be valid if not None, generated if None)

• tags (dict|octue.resources.tag.TagDict|None) – key-value pairs with string keys conforming to the Octue tag format (see TagDict)

• labels (iter(str)|octue.resources.label.LabelSet|None) – Space-separated string of labels relevant to this file

Return None

classmethod deserialise(serialised_datafile, from_string=False)

Deserialise a Datafile from a dictionary or JSON string.

Parameters

• serialised_datafile (dict|str) –

• from_string (bool) –

Return Datafile

property name

Get the name of the datafile.

Return str

property extension

Get the extension of the datafile.

Return str

property cloud_path

Get the cloud path of the datafile.

Return str|None

property cloud_hash_value

Get the hash value of the datafile according to its cloud file.

Return str|None

None if no cloud metadata is available

property timestamp

Get the timestamp of the datafile.

Return float

property posix_timestamp

Get the timestamp of the datafile in posix format.

Return float

property size_bytes

Get the size of the datafile in bytes.

Return float|None

property exists_locally

Return True if the file exists locally.

Return bool

property path

Alias to the local_path property.

Return str

property local_path

Get the local path for the datafile, downloading it from the cloud to a temporary file if necessary.

Return str

The local path of the datafile.

property open

Open the datafile for reading/writing. Usage is the same as the python built-in open context manager but it can only be used as a context manager e.g.

with datafile.open("w") as f:
    f.write("some data")

upload(cloud_path=None, update_cloud_metadata=True)

Upload a datafile to Google Cloud Storage.

Parameters

• cloud_path (str|None) – full path to cloud storage location to store datafile at (e.g. gs://bucket_name/path/to/file.csv)

• update_cloud_metadata (bool) – if True, update the metadata of the datafile in the cloud at upload time

Return str

gs:// path for datafile

download(local_path=None)

Download the file from the cloud to the given local path or a temporary path if none is given.

Parameters

local_path (str|None) – The local path to download the datafile to. A temporary path is used if none is given.

Raises

octue.exceptions.CloudLocationNotSpecified – If the datafile does not exist in the cloud

Return str

The path to the local file

metadata(include_id=True, include_sdk_version=True, use_octue_namespace=True)

Get the datafile’s metadata in a serialised form (i.e. the attributes id, timestamp, labels, tags, and sdk_version).

Parameters

• include_id (bool) – if True, include the ID of the datafile

• include_sdk_version (bool) – if True, include the octue version that instantiated the datafile in the metadata

• use_octue_namespace (bool) – if True, prefix metadata names with “octue__”

Return dict

update_metadata()

Using the datafile instance’s in-memory metadata, update its cloud metadata (if the datafile is cloud-based) or its local metadata file (if the datafile is local).

Return None

update_cloud_metadata()

Update the cloud metadata for the datafile.

Return None

update_local_metadata()

Create or update the local octue metadata file with the datafile’s metadata.

Return None

generate_signed_url(expiration=datetime.timedelta(days=7))

Generate a signed URL for the datafile.

Parameters

expiration (datetime.datetime|datetime.timedelta) – the amount of time or date after which the URL should expire

Return str

the signed URL for the datafile

add_labels(*args)

Add one or more new labels to the object. New labels will be cleaned and validated.

add_tags(tags=None, **kwargs)

Add one or more new tags to the object. New tags will be cleaned and validated.

property hash_value

Get the hash of the instance.

Return str

property id

Get the ID of the identifiable instance.

Return str

property labels

Get the labels of the labelled object.

Return iter

reset_hash()

Reset the hash value to the calculated hash (rather than whatever value has been set).

Return None

serialise(**kwargs)

Serialise the instance to a JSON string of primitives. See the Serialisable constructor for more information.

Return str

a JSON string containing the instance as a serialised python primitive

property tags

Get the tags of the taggable instance.

Return iter

to_file(filename, **kwargs)

Write the instance to a JSON file.

Parameters

filename (str) – path of file to write to, including relative or absolute path and .json extension

Return None

to_primitive()

Convert the instance into a JSON-compatible python dictionary of its attributes as primitives. See the Serialisable constructor for more information.

Return dict

Dataset

class octue.resources.dataset.Dataset(path=None, files=None, recursive=False, ignore_stored_metadata=False, id=None, name=None, tags=None, labels=None)

A representation of a dataset with metadata.

The default usage is to provide the path to a local or cloud directory and create the dataset from the files it contains. Alternatively, the files parameter can be provided and only those files are included. Either way, the path parameter should be explicitly set to something meaningful.

Metadata consists of id, name, tags, and labels, available as attributes on the instance. On instantiation, metadata for the dataset is obtained from its stored location (the corresponding cloud object metadata or a local .octue metadata file) if present. Metadata values can alternatively be passed as arguments at instantiation but will only be used if stored metadata cannot be found - i.e. stored metadata always takes precedence (use the ignore_stored_metadata parameter to override this behaviour). Stored metadata can be updated after instantiation using the update_metadata method.

Parameters

• path (str|None) – the path to the dataset (defaults to the current working directory if none is given)

• files (iter(str|dict|octue.resources.datafile.Datafile)|None) – the files belonging to the dataset

• recursive (bool) – if True, include in the dataset all files in the subdirectories recursively contained within the dataset directory

• ignore_stored_metadata (bool) – if True, ignore any metadata stored for this dataset locally or in the cloud and use whatever is given at instantiation

• id (str|None) – an optional UUID to assign to the dataset (defaults to a random UUID if none is given)

• name (str|None) – an optional name to give to the dataset (defaults to the dataset directory name)

• tags (dict|octue.resources.tag.TagDict|None) – key-value pairs with string keys conforming to the Octue tag format (see TagDict)

• labels (iter(str)|octue.resources.label.LabelSet|None) – space-separated string of labels relevant to the dataset

Return None

property name

Get the name of the dataset

Return str

property exists_locally

Return True if the dataset exists locally.

Return bool

property all_files_are_in_cloud

Do all the files of the dataset exist in the cloud?

Return bool

upload(cloud_path=None, update_cloud_metadata=True)

Upload a dataset to the given cloud path.

Parameters

• cloud_path (str|None) – cloud path to store dataset at (e.g. gs://bucket_name/path/to/dataset)

• update_cloud_metadata (bool) – if True, update the metadata of the dataset in the cloud at upload time

Return str

cloud path for dataset

update_metadata()

Using the dataset instance’s in-memory metadata, update its cloud metadata (if the dataset is cloud-based) or its local metadata file (if the dataset is local).

Return None

update_cloud_metadata()

Create or update the cloud metadata file for the dataset.

Return None

update_local_metadata()

Create or update the local octue metadata file with the dataset’s metadata.

Return None

generate_signed_url(expiration=datetime.timedelta(days=7))

Generate a signed URL for the dataset. This is done by uploading a uniquely named metadata file containing signed URLs to the datasets’ files and returning a signed URL to that metadata file.

Parameters

expiration (datetime.datetime|datetime.timedelta) – the amount of time or date after which the URL should expire

Return str

the signed URL for the dataset

add(datafile, path_in_dataset=None)

Add a datafile to the dataset. If the datafile’s location is outside the dataset, it is copied to the dataset root or to the path_in_dataset if provided.

Parameters

• datafile (octue.resources.datafile.Datafile) – the datafile to add to the dataset

• path_in_dataset (str|None) – if provided, set the datafile’s local path to this path within the dataset

Raises

octue.exceptions.InvalidInputException – if the datafile is not a Datafile instance

Return None

get_file_by_label(label)

Get a single datafile from a dataset by filtering for files with the provided label.

Parameters

label (str) – the label to filter for

Raises

octue.exceptions.UnexpectedNumberOfResultsException – if zero or more than one results satisfy the filters

Return octue.resources.datafile.DataFile

download(local_directory=None)

Download all files in the dataset into the given local directory. If no path to a local directory is given, the files will be downloaded to temporary locations.

Parameters

local_directory (str|None) –

Return None

to_primitive(include_files=True)

Convert the dataset to a dictionary of primitives, converting its files into their paths for a lightweight serialisation.

Parameters

include_files (bool) – if True, include the files parameter in the dictionary

Return dict

Manifest

class octue.resources.manifest.Manifest(datasets=None, id=None, name=None)

A representation of a manifest, which can contain multiple datasets This is used to manage all files coming into (or leaving), a data service for an analysis at the configuration, input or output stage.

Parameters

• datasets (dict(str, octue.resources.dataset.Dataset|dict|str)|None) – a mapping of dataset names to Dataset instances, serialised datasets, or paths to datasets

• id (str|None) – the UUID of the manifest (a UUID is generated if one isn’t given)

• name (str|None) – an optional name to give to the manifest

Return None

classmethod from_cloud(cloud_path)

Instantiate a Manifest from Google Cloud storage.

Parameters

cloud_path (str) – full path to manifest in cloud storage (e.g. gs://bucket_name/path/to/manifest.json)

Return Dataset

property all_datasets_are_in_cloud

Do all the files of all the datasets of the manifest exist in the cloud?

Return bool

use_signed_urls_for_datasets()

Generate signed URLs for any cloud datasets in the manifest and use these as their paths instead of regular cloud paths. URLs will not be generated for any local datasets in the manifest.

Return None

to_cloud(cloud_path)

Upload a manifest to a cloud location, optionally uploading its datasets into the same directory.

Parameters

cloud_path (str) – full path to cloud storage location to store manifest at (e.g. gs://bucket_name/path/to/manifest.json)

Return None

get_dataset(key)

Get a dataset by its key (as defined in the twine).

Parameters

key (str) –

Return octue.resources.dataset.Dataset

prepare(data)

Prepare new manifest from a manifest_spec.

Parameters

data (dict) –

Return Manifest

to_primitive()

Convert the manifest to a dictionary of primitives, converting its datasets into their paths for a lightweight serialisation.

Return dict

classmethod deserialise(serialised_object, from_string=False)

Deserialise the given JSON-serialised object into an instance of the class.

Parameters

• serialised_object (str|dict) – the string or dictionary of python primitives to deserialise into an instance

• from_string (bool) – if True, deserialise from a JSON string; otherwise, deserialise from a dictionary

Return any

classmethod hash_non_class_object(object_)

Use the Hashable class to hash an arbitrary object that isn’t an attribute of a class instance.

Parameters

object (any) –

Return str

property hash_value

Get the hash of the instance.

Return str

property id

Get the ID of the identifiable instance.

Return str

metadata(include_id=True, include_sdk_version=True, **kwargs)

Get the instance’s metadata in primitive form. The metadata is the set of attributes included in the class variable self._METADATA_ATTRIBUTES.

Parameters

• include_id (bool) – if True, include the ID of the instance if it is included in self._METADATA_ATTRIBUTES

• include_sdk_version (bool) – if True, include the octue version that instantiated the instance

• kwargs – any kwargs to use in an overridden self.metadata method

Return dict

property metadata_hash_value

Get the hash of the instance’s metadata, not including its ID.

Return str

property name

Get the name of the identifiable instance.

Return str

reset_hash()

Reset the hash value to the calculated hash (rather than whatever value has been set).

Return None

serialise(**kwargs)

Serialise the instance to a JSON string of primitives. See the Serialisable constructor for more information.

Return str

a JSON string containing the instance as a serialised python primitive

to_file(filename, **kwargs)

Write the instance to a JSON file.

Parameters

filename (str) – path of file to write to, including relative or absolute path and .json extension

Return None

Analysis

class octue.resources.analysis.Analysis(twine, handle_monitor_message=None, **kwargs)

A class representing a scientific or computational analysis. It holds references to all configuration, input, and output data, logs, connections to child services, credentials, etc. It’s essentially the “Internal API” for your service - a single point of contact where you can get or update anything you need.

An Analysis instance is automatically provided to the app in an Octue service when a question is received. Its attributes include every strand that can be added to a Twine, although only the strands specified in the service’s twine will be non-None. Incoming data is validated before it’s added to the analysis.

All input and configuration attributes are hashed using a BLAKE3 hash so the inputs and configuration that produced a given output in your app can always be verified. These hashes exist on the following attributes:

• input_values_hash

• input_manifest_hash

• configuration_values_hash

• configuration_manifest_hash

If a strand is None, so will its corresponding hash attribute be. The hash of a datafile is the hash of its file, while the hash of a manifest or dataset is the cumulative hash of the files it refers to.

Parameters

• twine (twined.Twine|dict|str) – the twine, dictionary defining a twine, or path to “twine.json” file defining the service’s data interface

• handle_monitor_message (callable|None) – an optional function for sending monitor messages to the parent that requested the analysis

• configuration_values (any) – the configuration values for the analysis - this can be expressed as a python primitive (e.g. dict), a path to a JSON file, or a JSON string.

• configuration_manifest (octue.resources.manifest.Manifest) – a manifest of configuration datasets for the analysis if required

• input_values (any) – the input values for the analysis - this can be expressed as a python primitive (e.g. dict), a path to a JSON file, or a JSON string.

• input_manifest (octue.resources.manifest.Manifest) – a manifest of input datasets for the analysis if required

• output_values (any) – any output values the analysis produces

• output_manifest (octue.resources.manifest.Manifest) – a manifest of output dataset from the analysis if it produces any

• children (dict) – a mapping of string key to Child instance for all the children used by the service

• id (str) – Optional UUID for the analysis

Return None

property finalised

Check whether the analysis has been finalised (i.e. whether its outputs have been validated and, if an output manifest is produced, its datasets uploaded).

Return bool

send_monitor_message(data)

Send a monitor message to the parent that requested the analysis.

Parameters

data (any) – any JSON-compatible data structure

Return None

finalise(upload_output_datasets_to=None)

Validate the output values and output manifest and, if the analysis produced an output manifest, upload its output datasets to a unique subdirectory within the analysis’s output location. This output location can be overridden by providing a different cloud path via the upload_output_datasets_to parameter. Either way, the dataset paths in the output manifest are replaced with signed URLs for easier, expiring access.

Parameters

upload_output_datasets_to (str|None) – If not provided but an output location was provided at instantiation, upload any output datasets into a unique subdirectory within this output location; if provided, upload into this location instead. The output manifest is updated with the upload locations.

Return None

add_labels(*args)

Add one or more new labels to the object. New labels will be cleaned and validated.

add_tags(tags=None, **kwargs)

Add one or more new tags to the object. New tags will be cleaned and validated.

classmethod deserialise(serialised_object, from_string=False)

Deserialise the given JSON-serialised object into an instance of the class.

Parameters

• serialised_object (str|dict) – the string or dictionary of python primitives to deserialise into an instance

• from_string (bool) – if True, deserialise from a JSON string; otherwise, deserialise from a dictionary

Return any

property id

Get the ID of the identifiable instance.

Return str

property labels

Get the labels of the labelled object.

Return iter

property name

Get the name of the identifiable instance.

Return str

serialise(**kwargs)

Serialise the instance to a JSON string of primitives. See the Serialisable constructor for more information.

Return str

a JSON string containing the instance as a serialised python primitive

property tags

Get the tags of the taggable instance.

Return iter

to_file(filename, **kwargs)

Write the instance to a JSON file.

Parameters

filename (str) – path of file to write to, including relative or absolute path and .json extension

Return None

to_primitive()

Convert the instance into a JSON-compatible python dictionary of its attributes as primitives. See the Serialisable constructor for more information.

Return dict

Child

class octue.resources.child.Child(id, backend, internal_service_name=None)

A class representing an Octue child service that can be asked questions. It is a convenience wrapper for Service that makes question asking more intuitive and allows easier selection of backends.

Parameters

• id (str) – the ID of the child

• backend (dict) – must include the key “name” with a value of the name of the type of backend e.g. “GCPPubSubBackend” and key-value pairs for any other parameters the chosen backend expects

• internal_service_name (str|None) – the name to give to the internal service used to ask questions to the child

Return None

ask(input_values=None, input_manifest=None, children=None, subscribe_to_logs=True, allow_local_files=False, handle_monitor_message=None, record_messages_to=None, allow_save_diagnostics_data_on_crash=True, question_uuid=None, timeout=86400)

Ask the child a question and wait for its answer - i.e. send it input values and/or an input manifest and wait for it to analyse them and return output values and/or an output manifest. The input values and manifest must conform to the schema in the child’s twine.

Parameters

• input_values (any|None) – any input values for the question

• input_manifest (octue.resources.manifest.Manifest|None) – an input manifest of any datasets needed for the question

• children (list(dict)|None) – a list of children for the child to use instead of its default children (if it uses children). These should be in the same format as in an app’s app configuration file and have the same keys.

• subscribe_to_logs (bool) – if True, subscribe to logs from the child and handle them with the local log handlers

• allow_local_files (bool) – if True, allow the input manifest to contain references to local files - this should only be set to True if the child will have access to these local files

• handle_monitor_message (callable|None) – a function to handle monitor messages (e.g. send them to an endpoint for plotting or displaying) - this function should take a single JSON-compatible python primitive as an argument (note that this could be an array or object)

• record_messages_to (str|None) – if given a path to a JSON file, messages received in response to the question are saved to it

• allow_save_diagnostics_data_on_crash (bool) – if True, allow the input values and manifest (and its datasets) to be saved by the child if it fails while processing them

• question_uuid (str|None) – the UUID to use for the question if a specific one is needed; a UUID is generated if not

• timeout (float) – time in seconds to wait for an answer before raising a timeout error

Raises

TimeoutError – if the timeout is exceeded while waiting for an answer

Return dict

a dictionary containing the keys “output_values” and “output_manifest”

ask_multiple(*questions)

Ask the child multiple questions in parallel and wait for the answers. Each question should be provided as a dictionary of Child.ask keyword arguments.

Parameters

questions – any number of questions provided as dictionaries of arguments to the Child.ask method

Return list

the answers to the questions in the same order as the questions

Child emulator

class octue.cloud.emulators.child.ChildEmulator(id=None, backend=None, internal_service_name=None, messages=None)

An emulator for the octue.resources.child.Child class that sends the given messages to the parent for handling without contacting the real child or using Pub/Sub. Any messages a real child could produce are supported. Child instances can be replaced/mocked like-for-like by ChildEmulator without the parent knowing.

Parameters

• id (str|None) – the ID of the child; a UUID is generated if none is provided

• backend (dict|None) – a dictionary including the key “name” with a value of the name of the type of backend (e.g. “GCPPubSubBackend”) and key-value pairs for any other parameters the chosen backend expects; a mock backend is used if none is provided

• internal_service_name (str|None) – the name to give to the internal service used to ask questions to the child; defaults to “<id>-parent”

• messages (list(dict)|None) – the list of messages to send to the parent

Return None

classmethod from_file(path)

Instantiate a child emulator from a JSON file at the given path. All/any/none of the instantiation arguments can be given in the file.

Parameters

path (str) – the path to a JSON file representing a child emulator

Return ChildEmulator

ask(input_values=None, input_manifest=None, subscribe_to_logs=True, allow_local_files=False, handle_monitor_message=None, record_messages_to=None, question_uuid=None, timeout=86400)

Ask the child emulator a question and receive its emulated response messages. Unlike a real child, the input

values and manifest are not validated against the schema in the child’s twine as it is only available to the real child. Hence, the input values and manifest do not affect the messages returned by the emulator.

Parameters

• input_values (any|None) – any input values for the question

• input_manifest (octue.resources.manifest.Manifest|None) – an input manifest of any datasets needed for the question

• subscribe_to_logs (bool) – if True, subscribe to logs from the child and handle them with the local log handlers

• allow_local_files (bool) – if True, allow the input manifest to contain references to local files - this should only be set to True if the child will have access to these local files

• handle_monitor_message (callable|None) – a function to handle monitor messages (e.g. send them to an endpoint for plotting or displaying) - this function should take a single JSON-compatible python primitive as an argument (note that this could be an array or object)

• record_messages_to (str|None) – if given a path to a JSON file, messages received in response to the question are saved to it

• question_uuid (str|None) – the UUID to use for the question if a specific one is needed; a UUID is generated if not

• timeout (float) – time in seconds to wait for an answer before raising a timeout error

Raises

TimeoutError – if the timeout is exceeded while waiting for an answer

Return dict

a dictionary containing the keys “output_values” and “output_manifest”

Filter containers

FilterSet

class octue.resources.filter_containers.FilterSet

filter(ignore_items_without_attribute=True, **kwargs)

Return a new instance containing only the Filterable`s to which the given filter criteria are `True.

Parameters

• ignore_items_without_attribute (bool) – if True, just ignore any members of the container without a filtered-for attribute rather than raising an error

• {str – any} kwargs: keyword arguments whose keys are the name of the filter and whose values are the values to filter for

Return octue.resources.filter_containers.FilterContainer

one(**kwargs)

If a single result exists for the given filters, return it. Otherwise, raise an error.

Parameters

{str – any} kwargs: keyword arguments whose keys are the name of the filter and whose values are the values to filter for

Raises

octue.exceptions.UnexpectedNumberOfResultsException – if zero or more than one results satisfy the filters

Return octue.resources.mixins.filterable.Filterable

order_by(attribute_name, check_start_value=None, check_constant_increment=None, reverse=False)

Order the Filterable`s in the container by an attribute with the given name, returning them as a new `FilterList regardless of the type of filter container begun with (`FilterSet`s and `FilterDict`s are inherently orderless).

Parameters

• attribute_name (str) – name of attribute (optionally nested) to order by e.g. “a”, “a.b”, “a.b.c”

• check_start_value (any) – if provided, check that the first item in the ordered container has the given start value for the attribute ordered by

• check_constant_increment (int|float|None) – if given, check that the ordered-by attribute of each of the items in the ordered container increases by the given value when progressing along the sequence

• reverse (bool) – if True, reverse the ordering

Raises

octue.exceptions.InvalidInputException – if an attribute with the given name doesn’t exist on any of the container’s members

Return FilterList

FilterList

class octue.resources.filter_containers.FilterList(iterable=(), /)

filter(ignore_items_without_attribute=True, **kwargs)

Return a new instance containing only the Filterable`s to which the given filter criteria are `True.

Parameters

• ignore_items_without_attribute (bool) – if True, just ignore any members of the container without a filtered-for attribute rather than raising an error

• {str – any} kwargs: keyword arguments whose keys are the name of the filter and whose values are the values to filter for

Return octue.resources.filter_containers.FilterContainer

one(**kwargs)

If a single result exists for the given filters, return it. Otherwise, raise an error.

Parameters

{str – any} kwargs: keyword arguments whose keys are the name of the filter and whose values are the values to filter for

Raises

octue.exceptions.UnexpectedNumberOfResultsException – if zero or more than one results satisfy the filters

Return octue.resources.mixins.filterable.Filterable

order_by(attribute_name, check_start_value=None, check_constant_increment=None, reverse=False)

Order the Filterable`s in the container by an attribute with the given name, returning them as a new `FilterList regardless of the type of filter container begun with (`FilterSet`s and `FilterDict`s are inherently orderless).

Parameters

• attribute_name (str) – name of attribute (optionally nested) to order by e.g. “a”, “a.b”, “a.b.c”

• check_start_value (any) – if provided, check that the first item in the ordered container has the given start value for the attribute ordered by

• check_constant_increment (int|float|None) – if given, check that the ordered-by attribute of each of the items in the ordered container increases by the given value when progressing along the sequence

• reverse (bool) – if True, reverse the ordering

Raises

octue.exceptions.InvalidInputException – if an attribute with the given name doesn’t exist on any of the container’s members

Return FilterList

FilterDict

class octue.resources.filter_containers.FilterDict(**kwargs)

A dictionary that is filterable by its values’ attributes. Each key can be anything, but each value must be an octue.mixins.filterable.Filterable instance.

filter(ignore_items_without_attribute=True, **kwargs)

Return a new instance containing only the Filterables for which the given filter criteria apply are satisfied.

Parameters

• ignore_items_without_attribute (bool) – if True, just ignore any members of the container without a filtered-for attribute rather than raising an error

• {str – any} kwargs: keyword arguments whose keys are the name of the filter and whose values are the values to filter for

Return FilterDict

order_by(attribute_name, reverse=False)

Order the instance by the given attribute_name, returning the instance’s elements as a new FilterList.

Parameters

• attribute_name (str) – name of attribute (optionally nested) to order by e.g. “a”, “a.b”, “a.b.c”

• reverse (bool) – if True, reverse the ordering

Raises

octue.exceptions.InvalidInputException – if an attribute with the given name doesn’t exist on any of the FilterDict’s values

Return FilterList

one(**kwargs)

If a single item exists for the given filters, return it. Otherwise, raise an error.

Parameters

{str – any} kwargs: keyword arguments whose keys are the name of the filter and whose values are the values to filter for

Raises

octue.exceptions.UnexpectedNumberOfResultsException – if zero or more than one results satisfy the filters

Return (any, octue.resources.mixins.filterable.Filterable)

Configuration

octue.configuration.load_service_and_app_configuration(service_configuration_path)

Load the service configuration from the given YAML file and the app configuration referenced in it. If no app configuration is referenced, an empty one is returned.

Parameters

service_configuration_path (str) – path to service configuration file

Return (octue.configuration.ServiceConfiguration, octue.configuration.AppConfiguration)

Service configuration

class octue.configuration.ServiceConfiguration(name, namespace, app_source_path='.', twine_path='twine.json', app_configuration_path=None, crash_diagnostics_cloud_path=None, repository_name=None, repository_owner=None, project_name=None, region=None, dockerfile_path=None, cloud_build_configuration_path=None, maximum_instances=10, branch_pattern='^main$', environment_variables=None, secrets=None, concurrency=10, memory='128Mi', cpus=1, minimum_instances=0, temporary_files_location=None, setup_file_path=None, service_account_email=None, machine_type=None, **kwargs)

A class containing the details needed to configure a service.

Parameters

• name (str) – the name to give the service

• namespace (str) – the namespace for grouping the service with others (e.g. the name of an organisation or individual)

• app_source_path (str) – the path to the directory containing the app’s source code

• twine_path (str) – the path to the twine file defining the schema for input, output, and configuration data for the service

• app_configuration_path (str|None) – the path to the app configuration file containing configuration data for the service; if this is None, the default application configuration is used

• crash_diagnostics_cloud_path (str|None) – the path to a cloud directory to store crash diagnostics in the event that the service fails while processing a question (this includes the configuration, input values and manifest, and logs)

Return None

classmethod from_file(path)

Load a service configuration from a file.

Parameters

path (str) –

Return ServiceConfiguration

App configuration

class octue.configuration.AppConfiguration(configuration_values=None, configuration_manifest=None, children=None, output_location=None, **kwargs)

A class containing the configuration data needed to start an app as a service. The configuration data should conform to the service’s twine schema.

Parameters

• configuration_values (str|dict|list|None) – values to configure the app

• configuration_manifest (str|dict|octue.resources.Manifest|None) – a manifest of datasets to configure the app

• children (str|list(dict)|None) – details of the children the app requires

• output_location (str|None) – the path to a cloud directory to save output datasets at

Return None

classmethod from_file(path)

Load an app configuration from a file.

Parameters

path (str) –

Return AppConfiguration

Runner

class octue.runner.Runner(app_src, twine='twine.json', configuration_values=None, configuration_manifest=None, children=None, output_location=None, crash_diagnostics_cloud_path=None, project_name=None, service_id=None)

A runner of analyses for a given service.

The Runner class provides a set of configuration parameters for use by your application, together with a range of methods for managing input and output file parsing as well as controlling logging.

Parameters

• app_src (callable|type|module|str) – either a function that accepts an Octue analysis, a class with a run method that accepts an Octue analysis, or a path to a directory containing an app.py file containing one of these

• twine (str|dict|twined.Twine) – path to the twine file, a string containing valid twine json, or a Twine instance

• configuration_values (str|dict|None) – The strand data. Can be expressed as a string path of a *.json file (relative or absolute), as an open file-like object (containing json data), as a string of json data or as an already-parsed dict.

• configuration_manifest (str|dict|None) – The strand data. Can be expressed as a string path of a *.json file (relative or absolute), as an open file-like object (containing json data), as a string of json data or as an already-parsed dict.

• children (str|list(dict)|None) – The children strand data. Can be expressed as a string path of a *.json file (relative or absolute), as an open file-like object (containing json data), as a string of json data or as an already-parsed dict.

• output_location (str|None) – the path to a cloud directory to save output datasets at

• crash_diagnostics_cloud_path (str|None) – the path to a cloud directory to store crash diagnostics in the event that the service fails while processing a question (this includes the configuration, input values and manifest, and logs)

• project_name (str|None) – name of Google Cloud project to get credentials from

• service_id (str|None) – the ID of the service being run

Return None

run(analysis_id=None, input_values=None, input_manifest=None, children=None, analysis_log_level=20, analysis_log_handler=None, handle_monitor_message=None, allow_save_diagnostics_data_on_crash=True, sent_messages=None)

Run an analysis.

Parameters

• analysis_id (str|None) – UUID of analysis

• input_values (str|dict|None) – the input_values strand data. Can be expressed as a string path of a *.json file (relative or absolute), as an open file-like object (containing json data), as a string of json data or as an already-parsed dict.

• input_manifest (str|dict|octue.resources.manifest.Manifest|None) – The input_manifest strand data. Can be expressed as a string path of a *.json file (relative or absolute), as an open file-like object (containing json data), as a string of json data or as an already-parsed dict.

• children (list(dict)|None) – a list of children to use instead of the children provided at instantiation. These should be in the same format as in an app’s app configuration file and have the same keys.

• analysis_log_level (str) – the level below which to ignore log messages

• analysis_log_handler (logging.Handler|None) – the logging.Handler instance which will be used to handle logs for this analysis run. Handlers can be created as per the logging cookbook https://docs.python.org/3/howto/logging-cookbook.html but should use the format defined above in LOG_FORMAT.

• handle_monitor_message (callable|None) – a function that sends monitor messages to the parent that requested the analysis

• allow_save_diagnostics_data_on_crash (bool) – if True, allow the input values and manifest (and its datasets) to be saved if the analysis fails

• sent_messages (list|None) – the list of messages sent by the service running this runner (this should update in real time) to save if crash diagnostics are enabled

Return octue.resources.analysis.Analysis

Octue essential monitor messages

A module containing helper functions for sending monitor messages that conform to the Octue essential monitor message schema https://refs.schema.octue.com/octue/essential-monitors/0.0.2.json

octue.essentials.monitor_messages.send_status_text(analysis, text, service_name)

Send a status-type monitor message and additionally log it to the info level.

Parameters

• analysis (octue.resources.analysis.Analysis) – the analysis from which to send the status text

• text (str) – the text of the status message

• service_name (str) – the name of the service/child running the analysis

Return None

octue.essentials.monitor_messages.send_estimated_seconds_remaining(analysis, estimated_seconds_remaining, service_name)

Send an estimated-seconds-remaining monitor message.

Parameters

• analysis (octue.resources.analysis.Analysis) – the analysis from which to send the estimate

• estimated_seconds_remaining (float) –

• service_name (str) – the name of the service/child running the analysis

Octue log handler

octue.log_handlers.apply_log_handler(logger_name=None, logger=None, handler=None, log_level=20, formatter=None, include_line_number=False, include_process_name=False, include_thread_name=False)

Apply a log handler with the given formatter to the logger with the given name. By default, the default Octue log handler is used on the root logger.

Parameters

• logger_name (str|None) – the name of the logger to apply the handler to; if this and logger are None, the root logger is used

• logger (logging.Logger|None) – the logger instance to apply the handler to (takes precedence over a logger name)

• handler (logging.Handler|None) – The handler to use. If None, the default StreamHandler will be attached.

• log_level (int|str) – ignore log messages below this level

• formatter (logging.Formatter|None) – if provided, this formatter is used and the other formatting options are ignored

• include_line_number (bool) – if True, include the line number in the log context

• include_process_name (bool) – if True, include the process name in the log context

• include_thread_name (bool) – if True, include the thread name in the log context

Return logging.Handler

License

The Boring Bit

See the octue-sdk-python license.

Third Party Libraries

octue-sdk-python includes or is linked against code from third party libraries - see our attributions page.

Version History

See our releases on GitHub.

Semantic versioning

We use semantic versioning so you can see when new releases make breaking changes or just add new features or bug fixes. Breaking changes are highlighted in our pull request descriptions and release notes.

Important

Note that octue is still in beta, so its major version number remains at 0 (i.e. 0.y.z). This means that, for now, both breaking changes and new features are denoted by an increase in the minor version number (y in x.y.z). When we come out of beta, breaking changes will be denoted by an increase in the major version number (x in x.y.z).

Deprecated code

When code is deprecated, it will still work but a deprecation warning will be issued with a suggestion on how to update it. After an adjustment period, deprecations will be removed from the codebase according to the code removal schedule. This constitutes a breaking change.

Bibliography

Agarwal

S. Agarwal, N. Snavely, S. M. Seitz and R. Szeliski, Bundle Adjustment in the Large, Proceedings of the European Conference on Computer Vision, pp. 29–42, 2010.

Bjorck

A. Bjorck, Numerical Methods for Least Squares Problems, SIAM, 1996

Brown

D. C. Brown, A solution to the general problem of multiple station analytical stereo triangulation, Technical Report 43, Patrick Airforce Base, Florida, 1958.

ByrdNocedal

R. H. Byrd, J. Nocedal, R. B. Schanbel, Representations of Quasi-Newton Matrices and their use in Limited Memory Methods, Mathematical Programming 63(4):129–-156, 1994.

ByrdSchnabel

R.H. Byrd, R.B. Schnabel, and G.A. Shultz, Approximate solution of the trust region problem by minimization over two dimensional subspaces, Mathematical programming, 40(1):247–263, 1988.

Chen

Y. Chen, T. A. Davis, W. W. Hager, and S. Rajamanickam, Algorithm 887: CHOLMOD, Supernodal Sparse Cholesky Factorization and Update/Downdate, TOMS, 35(3), 2008.

Conn

A.R. Conn, N.I.M. Gould, and P.L. Toint, Trust region methods, Society for Industrial Mathematics, 2000.

GolubPereyra

G.H. Golub and V. Pereyra, The differentiation of pseudo-inverses and nonlinear least squares problems whose variables separate, SIAM Journal on numerical analysis, 10(2):413–432, 1973.

HartleyZisserman

R.I. Hartley & A. Zisserman, Multiview Geometry in Computer Vision, Cambridge University Press, 2004.

KanataniMorris

K. Kanatani and D. D. Morris, Gauges and gauge transformations for uncertainty description of geometric structure with indeterminacy, IEEE Transactions on Information Theory 47(5):2017-2028, 2001.

Keys

R. G. Keys, Cubic convolution interpolation for digital image processing, IEEE Trans. on Acoustics, Speech, and Signal Processing, 29(6), 1981.

KushalAgarwal

A. Kushal and S. Agarwal, Visibility based preconditioning for bundle adjustment, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2012.

Kanzow

C. Kanzow, N. Yamashita and M. Fukushima, Levenberg–Marquardt methods with strong local convergence properties for solving nonlinear equations with convex constraints, Journal of Computational and Applied Mathematics, 177(2):375–397, 2005.

Levenberg

K. Levenberg, A method for the solution of certain nonlinear problems in least squares, Quart. Appl. Math, 2(2):164–168, 1944.

LiSaad

Na Li and Y. Saad, MIQR: A multilevel incomplete qr preconditioner for large sparse least squares problems, SIAM Journal on Matrix Analysis and Applications, 28(2):524–550, 2007.

Madsen

K. Madsen, H.B. Nielsen, and O. Tingleff, Methods for nonlinear least squares problems, 2004.

Mandel

J. Mandel, On block diagonal and Schur complement preconditioning, Numer. Math., 58(1):79–93, 1990.

Marquardt

D.W. Marquardt, An algorithm for least squares estimation of nonlinear parameters, J. SIAM, 11(2):431–441, 1963.

Mathew

T.P.A. Mathew, Domain decomposition methods for the numerical solution of partial differential equations, Springer Verlag, 2008.

NashSofer

S.G. Nash and A. Sofer, Assessing a search direction within a truncated newton method, Operations Research Letters, 9(4):219–221, 1990.

Nocedal

J. Nocedal, Updating Quasi-Newton Matrices with Limited Storage, Mathematics of Computation, 35(151): 773–782, 1980.

NocedalWright

J. Nocedal & S. Wright, Numerical Optimization, Springer, 2004.

Oren

S. S. Oren, Self-scaling Variable Metric (SSVM) Algorithms Part II: Implementation and Experiments, Management Science, 20(5), 863-874, 1974.

Ridders

C. J. F. Ridders, Accurate computation of F’(x) and F’(x) F”(x), Advances in Engineering Software 4(2), 75-76, 1978.

RuheWedin

A. Ruhe and P.Å. Wedin, Algorithms for separable nonlinear least squares problems, Siam Review, 22(3):318–337, 1980.

Saad

Y. Saad, Iterative methods for sparse linear systems, SIAM, 2003.

Stigler

S. M. Stigler, Gauss and the invention of least squares, The Annals of Statistics, 9(3):465-474, 1981.

TenenbaumDirector

J. Tenenbaum & B. Director, How Gauss Determined the Orbit of Ceres.

TrefethenBau

L.N. Trefethen and D. Bau, Numerical Linear Algebra, SIAM, 1997.

Triggs

B. Triggs, P. F. Mclauchlan, R. I. Hartley & A. W. Fitzgibbon, Bundle Adjustment: A Modern Synthesis, Proceedings of the International Workshop on Vision Algorithms: Theory and Practice, pp. 298-372, 1999.

Wiberg

T. Wiberg, Computation of principal components when data are missing, In Proc. Second Symp. Computational Statistics, pages 229–236, 1976.

WrightHolt

S. J. Wright and J. N. Holt, An Inexact Levenberg Marquardt Method for Large Sparse Nonlinear Least Squares, Journal of the Australian Mathematical Society Series B, 26(4):387–403, 1985.