Testing services
We recommend writing automated tests for your service so anyone who wants to use it can have confidence in its quality and reliability at a glance. Here’s an example test for our example service.
Emulating children
If your app has children, you should emulate them in your tests instead of communicating with the real ones. This makes your tests:
Independent of anything external to your app code - i.e. independent of the remote child, your internet connection, and communication between your app and the child (Google Pub/Sub).
Much faster - the emulation will complete in a few milliseconds as opposed to the time it takes the real child to actually run an analysis, which could be minutes, hours, or days. Tests for our child services template app run around 900 times faster when the children are emulated.
The Child Emulator
We’ve written a child emulator that takes a list of events and returns them to the parent for handling in the order
given - without contacting the real child or using Pub/Sub. Any events a real child can produce are supported.
Child
instances can be mocked like-for-like by
ChildEmulator
instances without the parent knowing.
Event kinds
You can emulate any event that your app (the parent) can handle. The table below shows what these are.
Event kind |
Number of events supported |
Example |
---|---|---|
|
One |
{“event”: {“kind”: “delivery_acknowledgement”}, “attributes”: {“question_uuid”: 79192e90-9022-4797-b6c7-82dc097dacdb, …}} |
|
Any number |
{“event”: {“kind”: “heartbeat”}, “attributes”: {“question_uuid”: 79192e90-9022-4797-b6c7-82dc097dacdb, …} |
|
Any number |
{“event”: {“kind”: “log_record”: “log_record”: {“msg”: “Starting analysis.”}}, “attributes”: {“question_uuid”: 79192e90-9022-4797-b6c7-82dc097dacdb, …} |
|
Any number |
{“event”: {“kind”: “monitor_message”: “data”: ‘{“progress”: “35%”}’}, “attributes”: {“question_uuid”: 79192e90-9022-4797-b6c7-82dc097dacdb, …} |
|
One |
{“event”: {“kind”: “exception”, “exception_type”: “ValueError”, “exception_message”: “x cannot be less than 10.”}, “attributes”: {“question_uuid”: 79192e90-9022-4797-b6c7-82dc097dacdb, …} |
|
One |
{“event”: {“kind”: “result”, “output_values”: {“my”: “results”}, “output_manifest”: None}, “attributes”: {“question_uuid”: 79192e90-9022-4797-b6c7-82dc097dacdb, …} |
Notes
Event formats and contents must conform with the service communication schema.
Every event must be accompanied with the required event attributes
The
log_record
key of alog_record
event is any dictionary that thelogging.makeLogRecord
function can convert into a log record.The
data
key of amonitor_message
event must be a JSON-serialised stringAny events after a
result
orexception
event won’t be passed to the parent because execution of the child emulator will have ended.
Instantiating a child emulator
events = [
{
{
"event": {
"kind": "log_record",
"log_record": {"msg": "Starting analysis."},
... # Left out for brevity.
},
"attributes": {
"datetime": "2024-04-11T10:46:48.236064",
"uuid": "a9de11b1-e88f-43fa-b3a4-40a590c3443f",
"retry_count": 0,
"question_uuid": "d45c7e99-d610-413b-8130-dd6eef46dda6",
"parent_question_uuid": "5776ad74-52a6-46f7-a526-90421d91b8b2",
"originator_question_uuid": "86dc55b2-4282-42bd-92d0-bd4991ae7356",
"parent": "octue/test-service:1.0.0",
"originator": "octue/test-service:1.0.0",
"sender": "octue/test-service:1.0.0",
"sender_type": "CHILD",
"sender_sdk_version": "0.51.0",
"recipient": "octue/another-service:3.2.1"
},
},
},
{
"event": {
"kind": "monitor_message",
"data": '{"progress": "35%"}',
},
"attributes": {
... # Left out for brevity.
},
},
{
"event": {
"kind": "log_record",
"log_record": {"msg": "Finished analysis."},
... # Left out for brevity.
},
"attributes": {
... # Left out for brevity.
},
},
{
"event": {
"kind": "result",
"output_values": [1, 2, 3, 4, 5],
},
"attributes": {
... # Left out for brevity.
},
},
]
child_emulator = ChildEmulator(events)
def handle_monitor_message(message):
...
result, question_uuid = child_emulator.ask(
input_values={"hello": "world"},
handle_monitor_message=handle_monitor_message,
)
>>> {"output_values": [1, 2, 3, 4, 5], "output_manifest": None}
Using the child emulator
To emulate your children in tests, patch the Child
class with the
ChildEmulator
class.
from unittest.mock import patch
from octue import Runner
from octue.cloud.emulators import ChildEmulator
app_directory_path = "path/to/directory_containing_app"
# You can explicitly specify your children here as shown or
# read the same information in from your app configuration file.
children = [
{
"key": "my_child",
"id": "octue/my-child-service:2.1.0",
"backend": {
"name": "GCPPubSubBackend",
"project_name": "my-project"
}
},
]
runner = Runner(
app_src=app_directory_path,
twine=os.path.join(app_directory_path, "twine.json"),
children=children,
service_id="your-org/your-service:2.1.0",
)
emulated_children = [
ChildEmulator(
events=[
{
"event": {
"kind": "result",
"output_values": [300],
},
"attributes": {
"datetime": "2024-04-11T10:46:48.236064",
"uuid": "a9de11b1-e88f-43fa-b3a4-40a590c3443f",
"retry_count": 0,
"question_uuid": "d45c7e99-d610-413b-8130-dd6eef46dda6",
"parent_question_uuid": "5776ad74-52a6-46f7-a526-90421d91b8b2",
"originator_question_uuid": "86dc55b2-4282-42bd-92d0-bd4991ae7356",
"parent": "you/your-service:2.1.0",
"originator": "you/your-service:2.1.0",
"sender": "octue/my-child-service:2.1.0",
"sender_type": "CHILD",
"sender_sdk_version": "0.56.0",
"recipient": "you/your-service:2.1.0"
},
},
],
)
]
with patch("octue.runner.Child", side_effect=emulated_children):
analysis = runner.run(input_values={"some": "input"})
analysis.output_values
>>> [300]
analysis.output_manifest
>>> None
Notes
If your app uses more than one child, provide more child emulators in the
emulated_children
list in the order they’re asked questions in your app.If a given child is asked more than one question, provide a child emulator for each question asked in the same order the questions are asked.
Creating a test fixture
Since the child is emulated, it doesn’t actually do any calculation - if you change the inputs, the outputs won’t change correspondingly (or at all). So, it’s up to you to define a set of realistic inputs and corresponding outputs (the list of emulated events) to test your service. These are called test fixtures.
Note
Unlike a real child, the inputs given to the emulator aren’t validated against the schema in the child’s twine - this is because the twine is only available to the real child. This is ok - you’re testing your service, not the child your service contacts. The events given to the emulator are still validated against the service communication schema, though.
You can create test fixtures manually or by using the Child.received_events
property after questioning a real
child.
import json
from octue.resources import Child
child = Child(
id="octue/my-child:2.1.0",
backend={"name": "GCPPubSubBackend", "project_name": "my-project"},
)
result, question_uuid = child.ask(input_values=[1, 2, 3, 4])
child.received_events
>>> [
{
"event": {
'kind': 'delivery_acknowledgement',
},
"attributes": {
... # Left out for brevity.
},
},
{
"event": {
'kind': 'log_record',
'log_record': {
'msg': 'Finished analysis.',
'args': None,
'levelname': 'INFO',
... # Left out for brevity.
},
},
"attributes": {
... # Left out for brevity.
},
},
{
"event": {
'kind': 'result',
'output_values': {"some": "results"},
},
"attributes": {
... # Left out for brevity.
},
},
]
You can then feed these into a child emulator to emulate one possible response of the child:
from octue.cloud.emulators import ChildEmulator
child_emulator = ChildEmulator(events=child.received_events)
result, question_uuid = child_emulator.ask(input_values=[1, 2, 3, 4])
result
>>> {"some": "results"}
You can also create test fixtures from downloaded service crash diagnostics.