Subscriptions for Event Notifications

This tutorial explains how to subscribe for event notifications in the Form3 API and receive the notifications on a webhook or Amazon SQS queue. It also outlines the recommended setup for receiving inbound transactions.

A step-by-step guide walks you through setting up a subscription and receiving a notification on it, as well as providing information on how to manage subscriptions and fine tune them using filters.

Overview

Form3 offers a notification mechanism to inform you of events in the system. This can be the arrival of an inbound payment, the successful submission of an outbound payment or any other event observable via the Form3 API.

To receive event notifications, you create subscriptions to specific kinds of events. These are defined by two characteristics:

• The record type, or type of API resource that is changed when the event occurs. This can be any resource type that exists in the Form3 API.
• The type of change. This defines what happens to the resource when the event occurs. It can be either created, updated, deleted.

Additional filters can be applied to only receive notifications if a certain condition is met, for example if a specific field of the resource is updated. This is useful to subscribe only to updates for specific types of payments, for example.

Notifications can be delivered to either a webhook or an Amazon SQS queue that you manage. The notification mechanism uses advanced time-out and retry logic to ensure notifications are delivered even if your webhook or queue should become temporarily unavailable.

Callback Endpoints

The notification mechanism allows you to use an Amazon SQS queue or a webhook URL as a callback endpoint to where notifications are delivered.

We generally recommend the SQS option over a webhook as this is a managed, highly-available, high-throughput and persistent message bus that eliminates any delays or issues you may face if your webhook endpoint is not reachable.

Amazon SQS Queue

For SQS queues, we support setups with and without Server Side Encryption (SSE). See the step-by-step guide for how to set up both versions.

Webhooks

For webhook URLs, the notification service uses uses mutual TLS authentication to ensure that the notifications you receive are from Form3.

When hosting your own webhook service for us to call, we require that you use a valid certificate signed by a reputable certification authority.

Form3 validates the certificate based on the following criteria:

• The certificate is valid and not expired
• It is issued by a reputable certification authority (see below)
• It must not be self-signed
• The certificate chain to the trusted root certificate cannot be more than 4 levels deep
• It has to be for the domain name of your webhook. We do accept wildcards for subdomains.

Reputable certificate providers include:

• NameCheap
• DigiCert
• Let's Encrypt

See here for a full list of certification authorities that we trust.

Delivery Errors

To acknowledge the successful delivery of a notification, a webhook or queue must respond with an HTTP success status (2XX). In case the endpoint responds with a different code or not at all, Form3 will retry the delivery up to 8 times using an exponential backoff algorithm that increases the delay between retries with every failed delivery. The maximum delay time before the 8th attempt is 15 minutes (see table below). A successful delivery will reset the failure counter.

After the 8th unsuccessful attempt, the endpoint is considered permanently unreachable and the subscription is deactivated. This is indicated by setting the deactivated field of the Subscription resource to true.

When this happens, event notifications for the deactivated subscription are queued at Form3. You can reactivate a subscription by updating the Subscription resource and setting the deactivated field to false again. Upon reactivation, the queued notifications are sent to the endpoint and new notifications are delivered to the endpoint again without queueing.

 Exponential backoff

Notification delivery is retried 8 times. The table below lists the time delays before each retry:

After the 8th unsuccessful retry, the subscription is deactivated.

Filters

You can apply filters to subscriptions to only send a notification for the given resource and event type if the filter condition is met. This is done by setting the filter field in the Subscription resource.

If the filter field is set, the notification is only sent if the attribute-value pair defined in the filter is present in the notification.

Filters can be set for any attribute of the resource the notification message is sent for. This includes nested lists and linked resources listed in the relationships section of a resource.

For more information on filters and how to use them, see the Filters section of the API documentation. For common use cases, see below.

Common Use Cases

Using multiple subscriptions for the same resource and event type in combination with filters is a useful way to route the flow of notifications into topic-based channels.

Common subscription routings include:

• Filter by payment scheme: Send admission or submission notifications for different schemes to different endpoints. When subscribing to Admission or Submission resource evens, this field is located in the relationships section of the notification: "filter": "data.relationships.payment.data.attributes.payment_scheme: FPS"
• Filter by status: Only receive notifications for failed submissions or admissions or send notifications for successful transactions to a different endpoint. Example to filter to failed admissions: "filter": "data.attributes.status: failed"

In the Form3 API

Subscriptions and notifications are grouped in the Notification API. The Subscription resource is used to create and manage subscriptions. Notifications don't have their own resource type, instead they use the type of the resource they refer to. Their syntax is very simple, see the Notification section below for a quick overview.

Subscriptions

The Subscription resource is used to manage if and how you receive notifications for a specific type of event. To subscribe to an event, create a Subscription resource and specify the event by filling in the record_type and event_type fields. The callback_uri field allows you to define where a notification should be sent.

To unsubscribe from an event, simply delete the Subscription resource. You can also modify the parameters of a subscription by patching the resource.

Below is a list of resource fields in the Subscription resource:

For more information on the Subscription resource, see the API documentation.

Notifications

Notifications are the messages we send to your webhook or queue when an event occurs. They are essentially the resource that is being created, updated or deleted with a thin wrapper around it. Below is an example of a notification for the creation of a Payment Admission resource:

HTTP/1.1 200 OK
Content-Type: application/vnd.api+json

{
  "id": "9c2e4b09-0405-483d-92d6-6a37af9a5e4b",
  "organisation_id": "a95c9c69-a4ab-49bc-bfc4-ad78d3a3d178",
  "event_type": "created",
  "record_type": "payment_admissions",
  "data_record_type": "PaymentAdmission",
  "version": 0,
  "data": {
    "data": {
      "type": "payment_admissions",
      "id": "14048b5a-88f5-4586-aec7-603703ada3ae",
      "version": 0,
      "organisation_id": "a95c9c69-a4ab-49bc-bfc4-ad78d3a3d178",
      "attributes": {
        "status": "confirmed",
        "status_reason": "accepted",
        "admission_datetime": "2019-07-30T16:35:57.421Z",
        "settlement_date": "2019-07-30",
        "settlement_cycle": 1
      },
      "relationships": {
        "payment": {
          "data": [
            {
              "type": "payments",
              "id": "107812e8-955e-4eec-9f1f-dfbe4e391482",
              "version": 0,
              "organisation_id": "a95c9c69-a4ab-49bc-bfc4-ad78d3a3d178",
              "created_on": "2019-07-30T16:35:57.380Z",
              "modified_on": "2019-07-30T16:35:57.380Z",
              "attributes": {
                "amount": "600.00",
                "beneficiary_party": {
                  "account_name": "Mr Sending Test",
                  "account_number": "20621803",
                  "account_number_code": "BBAN",
                  "account_type": 1,
                  "account_with": {
                    "bank_id": "112233",
                    "bank_id_code": "GBDSC"
                  }
                },
                "charges_information": {
                  "bearer_code": "SHAR"
                },
                "currency": "GBP",
                "debtor_party": {
                  "account_name": "Mrs Receiving Test",
                  "account_number": "71268996",
                  "account_number_code": "BBAN",
                  "account_with": {
                    "bank_id": "400302",
                    "bank_id_code": "GBDSC"
                  }
                },
                "end_to_end_reference": "NOREF",
                "numeric_reference": "0001",
                "scheme_transaction_id": "00865465238980327",
                "unique_scheme_id": "00865465238980327 1020190730826400302",
                "payment_purpose": "X",
                "payment_scheme": "FPS",
                "payment_type": "Credit",
                "processing_date": "2019-07-30",
                "reference": "1112223330",
                "scheme_payment_sub_type": "TelephoneBanking",
                "scheme_payment_type": "ImmediatePayment"
              }
            }
          ]
        }
      }
    }
  }
}

The Payment Admission resource is located in the data object. The event_type and record_type fields define the type of notification just like in the Subscription resource. See the API documentation for a full description of all fields.

Note that the relationships sections of resources in notifications always contain the full related resource instead of only their ID. This makes it easier to get all relevant information of an event from the notification without having to make another GET call to fetch additional resources.

Recommended Use Cases

We recommend subscribing to events as the main mechanism to receive asynchronous messages and status updates through the Form3 API. The sections below outline the two main use case classes for receiving notifications, inbound transactions and status updates of outbound transactions.

Inbound Transactions

When an inbound transaction arrives, you want to know so you can process it. The standard way to do this in the Form3 API is subscribing to the created event of the relevant Admission resource type. This way, you receive a notification to your webhook or queue whenever a new inbound transaction arrives.

The notification will contain the Admissions resource as well as the full transaction resource (Payment, Direct Debit, Return, etc.) in the relationships section of the Admission resource, providing you with all information you need.

You can apply filters to only receive notifications about certain transactions. This is useful if you want to receive notifications for failed transactions or different payment types (e.g. SIP, SOP, etc. for UK Faster Payments) at different endpoints.

See the step by step guide below for an example of a notification for an inbound payment.

Status Updates on Outbound Transactions

When sending outbound payments, you want to make sure they are delivered successfully and automatically react in case they aren't. You can do this by subscribing to the updated event of the relevant Submission resource types. You will receive a notification every time a field in the Submission resource changes, most importantly the status field.

You can even use filters to send all updates where the status field has changed to delivery_confirmed to one endpoint and all updates where the status changes to delivery_failed to another.

See the step by step guide below for an example of receiving notifications on status updates.

Test New or Updated Subscription Setups

It is recommended to test the functionality of any subscription setup before use. This includes new setups as well as changes made to an exisiting subscription setup. Test should be carried out in the staging environment.

To test a new subscription setup:

• Ensure all callback endpoints are reachable and ready to receive notifications
• Create the Subscription resources
• Confirm notification flow for all subscriptions

To test updates to an existing subscription setup:

• Replicate the existing setup in the staging environment if necessary
• Ensure all callback endpoints used before and after the change are reachable and ready to receive notifications

When updating callback URIs:

• Create the new Subscription resource for the new callback endpoint
• Confirm notification flow to both the new and old callback endpoints
• Deactivate the Subscription resource pointing to the old callback endpoint
• Confirm notification flow to the new callback endpoint
• Delete the Subscription resource pointing to the old callback endpoint

When adding filter parameters:

• Confirm unfiltered notification flow for the existing subscription
• Update the Subscription resource to include the filter parameters
• Confirm filtered notification flow for the subscription

Note that updating existing filter parameters is not recommended. Instead, create new subscriptions and add the updated filter rules to them. Migrate traffic to the new set of subscriptions following the procedure outlined for updating callback URIs above.

If you have to update an existing filter, e.g. in an incident situation, follow the steps below. Note that it is currently not possible to remove an existing filter.

When updating existing filter parameters:

• Confirm filtered notification flow with the existing filter parameters
• Update the filter field of the Subscription resource
• Confirm filtered notification flow with the new filter parameters

Step by Step Guide

This section guides you through the setup of an example subscription and the receipt of notifications at your callback endpoint. It uses the Transaction Simulator in our Staging Environment to trigger an inbound transaction to receive notifications on.

Prerequisites

Before getting started, make sure you have the following things ready to follow along:

• Access to the Form3 Staging Environment. For OAuth-based authentication (deprecated), this is your client_id and client_secret. For request signing, you need to have completed the setup and be able to sign your HTTP requests to the Form3 API.
• An Organisation ID that is valid on the Staging Environment. Contact Form3 to obtain it.
• At least one account number and bank ID registered with Form3.
• Access to the Transaction Simulator.
• Access to a whitelisted webhook endpoint or Amazon SQS queue (see information about whitelisting below).

Setup

For security reasons, we require you to whitelist the domain of your webhook URLs and the AWS account ID used for your SQS queues with us. This can be done via the Form3 Portal. Speak to your implementation manager for more information on how to whitelist a specific domain or AWS account ID.

Webhook

If you're using a webhook, make sure that:

• The webhook is configured to receive and acknowledge POST requests from the public internet.
• It uses a valid certificate signed by a reputable certification authority.

We sign the notification messages with our own certificate. See the API documentation for more information on our certification details. Note that we use different certificates for the Staging and Production environments.

Amazon SQS Queue

Follow the setup steps below to set up your SQS queue. In order to use Server Side Encryption, also follow the additional steps outlined in the section below.

SQS

You can either use the AWS console or the awscli tool.

1) Using your AWS account, create a SQS queue:

aws sqs create-queue --queue-name acme-co --region eu-west-1

Note that the command above sets the message size limit to the default maximum value. We do not recommend using a smaller message size limit than the maximum.

2) Add the required permission to the queue:

For a test environment

aws sqs add-permission --aws-account-ids 288840537196 --actions SendMessage --queue-url https://eu-west-1.queue.amazonaws.com/288840537196/acme-co --label foo --region eu-west-1

For a production environment

aws sqs add-permission --aws-account-ids 124201431238 --actions SendMessage --queue-url https://eu-west-1.queue.amazonaws.com/288840537196/acme-co --label foo --region eu-west-1

For more information about AWS permissions, see here.

SQS with SSE (Server Side Encryption)

1) Additionally to the steps above, when you create the SQS queue set it to use an AWS KMS Customer Master Key (CMK)

2) Log into the AWS console, navigate to KMS, select Customer Managed Keys, select the encryption key that your notification queue is using, go to the Key Policy tab and select edit. Append the following statement block (based on the Form3 environment) into existing policy AWS created for you:

For the staging environment:

{ "Sid": "Enable Form3", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::288840537196:root" }, "Action": [ "kms:Decrypt", "kms:GenerateDataKey*" ], "Resource": "*" }

For the production environment:

{ "Sid": "Enable Form3", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::124201431238:root" }, "Action": [ "kms:Decrypt", "kms:GenerateDataKey*" ], "Resource": "*" }

This enables the Form3 notification service to send messages to the encrypted SQS queue.

3) Your service that consumes the SQS queue will also need the same KMS permissions to read the messages.

Create Subscriptions

Once your callback webhook or queue is set up, you can create the subscription. You do this by creating a Subscription resource: POST /v1/notification/subscriptions

In this example, we are going create two Subscription resources to track both inbound and outbound payments. To keep it simple, notifications for all events will be sent to the same callback endpoint.

The first Subscription resource is for inbound payments. Notifications will be sent when a new Payment Admission resource is created.

Request for creating a new Subscription for the "created" event for Payment Admission resources

{
  "data": {
    "type": "subscriptions",
    "id": "fe4b377b-5a96-4f71-8d2c-4592e6aa4baa",
    "organisation_id": "cbe58206-cfc6-4848-ae4b-32f7ddeee695",
    "attributes": {
      "callback_uri": "https://your-webhook-url.com",
      "callback_transport": "http",
      "event_type": "created",
      "record_type": "payment_admissions"
    }
  }
}

Response for creating a new Subscription for the "created" event for Payment Admission resources

{
    "data": {
        "type": "subscriptions",
        "id": "fe4b377b-5a96-4f71-8d2c-4592e6aa4baa",
        "version": 0,
        "organisation_id": "cbe58206-cfc6-4848-ae4b-32f7ddeee695",
        "created_on": "2020-06-17T13:03:38.952Z",
        "modified_on": "2020-06-17T13:03:38.952Z",
        "attributes": {
            "callback_uri": "https://your-webhook-url.com",
            "callback_transport": "http",
            "user_id": "e27d040e-7df8-42ff-9b29-5f67c7aed1f2",
            "record_type": "payment_admissions",
            "event_type": "created",
            "deactivated": false
        }
    },
    "links": {
        "self": "/v1/notification/subscriptions/fe4b377b-5a96-4f71-8d2c-4592e6aa4baa"
    }
}

The seccond Subscription resource is for outbound payments. Notifications will be sent when a Payment Submission resource is updated, e.g. when its status value changes.

Request for creating a new Subscription for the "updated" event for Payment Submission resources

{
  "data": {
    "type": "subscriptions",
    "id": "1b2d0f9c-15ef-41ea-ae1c-c4d9718e15c4",
    "organisation_id": "cbe58206-cfc6-4848-ae4b-32f7ddeee695",
    "attributes": {
      "callback_uri": "https://your-webhook-url.com",
      "callback_transport": "http",
      "event_type": "updated",
      "record_type": "payment_submissions"
    }
  }
}

Response for creating a new Subscription for the "updated" event for Payment Submission resources

{
    "data": {
        "type": "subscriptions",
        "id": "1b2d0f9c-15ef-41ea-ae1c-c4d9718e15c4",
        "version": 0,
        "organisation_id": "cbe58206-cfc6-4848-ae4b-32f7ddeee695",
        "created_on": "2020-06-17T13:06:26.715Z",
        "modified_on": "2020-06-17T13:06:26.715Z",
        "attributes": {
            "callback_uri": "https://your-webhook-url.com",
            "callback_transport": "http",
            "user_id": "e27d040e-7df8-42ff-9b29-5f67c7aed1f2",
            "record_type": "payment_submissions",
            "event_type": "updated",
            "deactivated": false
        }
    },
    "links": {
        "self": "/v1/notification/subscriptions/1b2d0f9c-15ef-41ea-ae1c-c4d9718e15c4"
    }
}

Receive a Notification

Now it's time to put the new subscriptions to the test. To do this, we use the transaction simulator in the staging environment. We will send an outbound payment with the amount 600.00 which will trigger the arrival of a separate inbound payment.

Note that we use FPS payments for this example. However, the steps shown also work with every other payment scheme supported by Form3.

Send and Receive a Payment

To send a outbound payment and trigger an inbound payment, first create a Payment resource. Note the following key attributes:

• amount is set to 600.00 to trigger an inbound payment
• currency is set to GBP
• payment_scheme is FPS
• scheme_payment_type is ImmediatePayment
• The debtor_party attributes need to be for a account you have registered via the Account API

Take a look at this tutorial and example to learn how to create a Payment resource.

Finally, to send the trigger payment, create a Payment Submission resource. See here to learn how.

The payment is now submitted and will be processed by Form3 and the FPS simulator. Once the payment is delivered successfully, an identical inbound payment will be created, sent from the beneficiary of the outbound payment to its debtor.

Receive Submission Notifications

As soon as the Payment Submission for the outbound payment is created, the callback endpoint will receive notifications for each update of the Submission resource.

The status field of the Submission resource is updated as the payment is processed by Form3 and then submitted to FPS (or the FPS simulator in the staging environment).

Here is an example of a notification for the final status change of the Submission resource to delivery_confirmed:

Payment Submission "updated" event notification

{
  "id": "67a75b5a-f964-4f72-bdf4-7fadcedabcde",
  "organisation_id": "cbe58206-cfc6-4848-ae4b-32f7ddeee695",
  "event_type": "updated",
  "record_type": "payment_submissions",
  "data_record_type": "PaymentSubmission",
  "data": {
    "data": {
      "type": "payment_submissions",
      "id": "d6f8284e-127d-44a9-a404-482acc2c0d13",
      "version": 6,
      "organisation_id": "cbe58206-cfc6-4848-ae4b-32f7ddeee695",
      "created_on": "2020-06-17T13:28:24.983Z",
      "modified_on": "2020-06-17T13:28:26.147Z",
      "attributes": {
        "status": "delivery_confirmed",
        "status_reason": "Accept",
        "submission_datetime": "2020-06-17T13:28:24.981Z",
        "settlement_date": "2020-06-17",
        "settlement_cycle": 2,
        "transaction_start_datetime": "2020-06-17T13:28:24.974Z"
      },
      "relationships": {
        "payment": {
          "data": [
            {
              "type": "payments",
              "id": "07b3ed70-5e5d-4243-a511-d194c63c535f",
              "version": 0,
              "organisation_id": "cbe58206-cfc6-4848-ae4b-32f7ddeee695",
              "created_on": "2020-06-17T13:28:08.474Z",
              "modified_on": "2020-06-17T13:28:08.474Z",
              "attributes": {
                "amount": "600.00",
                "beneficiary_party": {
                  "account_name": "Mrs Receiving Test",
                  "account_number": "71268996",
                  "account_number_code": "BBAN",
                  "account_type": 1,
                  "account_with": {
                    "bank_id": "400302",
                    "bank_id_code": "GBDSC"
                  },
                  "country": "GB"
                },
                "currency": "GBP",
                "debtor_party": {
                  "account_name": "Mr Sending Test",
                  "account_number": "62281465",
                  "account_number_code": "BBAN",
                  "account_with": {
                    "bank_id": "040013",
                    "bank_id_code": "GBDSC"
                  },
                  "country": "GB"
                },
                "numeric_reference": "0001",
                "scheme_transaction_id": "16659413033400758",
                "payment_scheme": "FPS",
                "processing_date": "2020-06-17",
                "reference": "1112223330",
                "scheme_payment_type": "ImmediatePayment"
              }
            }
          ]
        }
      }
    }
  }
}

Receive Admission Notifications

When the outbound payment is delivered successfully, the FPS simulator triggers a new inbound payment with the same amount and reversed beneficiary and debtor details.

When this payment arrives, new Payment and Payment Admission resources are created. Since you have subscribed to the creation of Payment Admission resource, a notification is sent to your callback endpoint:

Payment Admission "created" notification

{
  "id": "db7f32f4-6599-4d19-9092-816f93a08b8f",
  "organisation_id": "cbe58206-cfc6-4848-ae4b-32f7ddeee695",
  "event_type": "created",
  "record_type": "payment_admissions",
  "data_record_type": "PaymentAdmission",
  "data": {
    "data": {
      "type": "payment_admissions",
      "id": "a4a2b8e4-01db-4d66-8bc3-6208365812ae",
      "version": 0,
      "organisation_id": "cbe58206-cfc6-4848-ae4b-32f7ddeee695",
      "created_on": "2020-06-17T13:39:25.751Z",
      "modified_on": "2020-06-17T13:39:25.751Z",
      "attributes": {
        "status": "confirmed",
        "status_reason": "accepted",
        "admission_datetime": "2020-06-17T13:39:25.732Z",
        "settlement_date": "2020-06-17",
        "settlement_cycle": 1
      },
      "relationships": {
        "payment": {
          "data": [
            {
              "type": "payments",
              "id": "344bf30d-e544-4944-bb4d-becb8ab7e0d6",
              "version": 0,
              "organisation_id": "cbe58206-cfc6-4848-ae4b-32f7ddeee695",
              "created_on": "2020-06-17T13:39:25.693Z",
              "modified_on": "2020-06-17T13:39:25.693Z",
              "attributes": {
                "amount": "600.00",
                "beneficiary_party": {
                  "account_name": "Mr Sending Test",
                  "account_number": "20621803",
                  "account_number_code": "BBAN",
                  "account_type": 1,
                  "account_with": {
                    "bank_id": "112233",
                    "bank_id_code": "GBDSC"
                  }
                },
                "currency": "GBP",
                "debtor_party": {
                  "account_name": "Mrs Receiving Test",
                  "account_number": "71268996",
                  "account_number_code": "BBAN",
                  "account_with": {
                    "bank_id": "400302",
                    "bank_id_code": "GBDSC"
                  }
                },
                "end_to_end_reference": "NOREF",
                "numeric_reference": "0001",
                "scheme_transaction_id": "00000000134356670",
                "unique_scheme_id": "00000000134356670 1020200617826400302",
                "payment_purpose": "X",
                "payment_scheme": "FPS",
                "payment_type": "Credit",
                "processing_date": "2020-06-17",
                "reference": "1112223330",
                "scheme_payment_sub_type": "TelephoneBanking",
                "scheme_payment_type": "ImmediatePayment"
              }
            }
          ]
        }
      }
    }
  }
}

The notification contains the Payment Admission resource and the related Payment resource. All relevant information about the inbound payment are thus available in the notification, there is no need to make another GET call to fetch the Payment or Payment Admission resource.

Further Reading

This tutorial explained how to create subscriptions and receive event notifications. For more information on using subscriptions, see our API documentation.

For advice on which events and resources to subscribe to, consult the User Guide for the product you are implementing (available from the Form3 Portal) or ask your Implementation Manager.