Chapter 4 System Build and Operations Setup Procedures (Data as a Product)

This chapter presents the procedures for building a minimum configuration system capable of performing data exchange between participants, using the deploy definition files provided by ODS SDK for Onboarding (hereinafter "SDK").

System Build Procedures (4.1–4.3)

4.1 System Architecture

The system architecture to be built in this chapter is as shown in Figure 3. Rectangles represent components or services, and arrows represent dependencies between them.

Figure 3　System Architecture Provided by the SDK

This SDK uses PostgreSQL as the RDBMS, Keycloak as the authentication system, and OpenFGA as the ReBAC authorization system. In the following sections, the Identity Component and Web API Transfer Module may be referred to simply as L3 and L2, respectively.

4.2 Initial Configuration of Each Component

The SDK provides files that aggregate the deploy definition files for each service, which can be used to start and stop all services at once. The usage instructions for the Docker Compose version are shown below.

Clone the SDK repository locally.

$ git clone https://github.com/open-dataspaces/SDK-docker-compose.git

Navigate into the cloned directory and clone the repositories for L2, L3, and the clearing and payment service.

$ cd SDK-docker-compose
$ git clone --depth 1 https://github.com/open-dataspaces/L2-dp-webapi.git
$ git clone --depth 1 https://github.com/open-dataspaces/L3-identity-component.git
$ git clone --depth 1 https://github.com/open-dataspaces/DCS-Payment.git

Run Docker Compose using the docker-compose.yml file located at the top level of the repository to start all services.

$ docker compose up -d

The setup is complete when all services have started, as shown below. For procedures on starting and stopping components individually, refer to "4.5 Starting and Stopping."

[+] Running 17/17
 ✔ gateway                      Built                                                            0.0s
 ✔ payment-app                  Built                                                            0.0s
 ✔ l3-app                       Built                                                            0.0s
 ✔ Volume "ods_pgdata"          Created                                                          0.0s
 ✔ Volume "ods_postgres_data"   Created                                                          0.0s
 ✔ Volume "ods_pgdata_openfga"  Created                                                          0.0s
 ✔ Container minio              Started                                                          1.0s
 ✔ Container postgres           Started                                                          1.1s
 ✔ Container fluentd            Started                                                          1.1s
 ✔ Container l3-app             Started                                                          1.0s
 ✔ Container payment-db         Healthy                                                         11.5s
 ✔ Container postgres-openfga   Started                                                          1.0s
 ✔ Container payment-app        Started                                                         12.0s
 ✔ Container keycloak           Started                                                          1.7s
 ✔ Container openfga            Started                                                          1.6s
 ✔ Container ods-minio-init-1   Started                                                          1.4s
 ✔ Container gateway            Started                                                          2.2s

Next, perform the initial configuration for L3, OpenFGA, and L2 by running the scripts provided by the SDK. For details, refer to the SDK documentation.

Operations Setup Procedures (4.3–4.8)

4.3 Data Configuration for Starting Operations

Before starting operations, it is necessary to register participant business information in L3. For details, refer to the relevant section of the SDK documentation.

4.4 Environment Configuration Between Components

It is necessary to configure the L3 URL in L2's configuration file so that L2 can communicate with L3. For details, refer to the relevant section of the SDK documentation.

4.5 Starting and Stopping

The procedures for starting and stopping each component individually are as follows.

L3: Identity Component

Start

$ docker compose -f l3/docker-compose.yml up -d

Stop

$ docker compose -f l3/docker-compose.yml down

Logging Service

Start

$ docker compose -f logging/docker-compose.yml up -d

Stop

$ docker compose -f logging/docker-compose.yml down

L2: Web API Transfer Module

Start (L3 and Logging must be started beforehand)

$ docker compose up -d gateway 

Stop

$ docker compose -f l2/docker-compose.yml down

Clearing and Payment Service

Start (L3 must be started beforehand)

$ docker compose -f payment/docker-compose.yml up -d

Stop

$ docker compose -f payment/docker-compose.yml down

4.6 Application Integration

Data providers must configure authorization settings for the APIs published by the industry service, for the business entity created in "4.3 Data Configuration for Starting Operations." For details, refer to the relevant section of the SDK documentation.

Data users must set the headers shown in Table 2 in their HTTP requests.

Table 2: Required Header Information

Header Name	Description
API-Key	API key issued by this service
Authorization	Access token issued by L3 (Identity Component) in JWT format
X-TrackingId	Log output item for provenance management (UUID format)
X-ODS-xxx	Item subject to logging. Specify for xxx the string designated by the service provider (e.g., X-ODS-UserId)

4.7 Executing and Verifying Data Exchange

Data users acquire data from providers using the following procedure.

1. Obtain an access token: Follow L3 Reference Implementation Tutorial 5-3. User Authentication (Authorization Code Flow) to obtain an access token.

2. Access data: Use the obtained access token to access data. A sample command is shown below.

   $ curl -X POST "http://localhost:8090/test " \
     -H 'api-key: 2dfd3409-ce01-4451-96fa-7e10c9681422y' \
     -H "Authorization: bearer $ACCESS_TOKEN" \
     -H 'X-ODS-UserId: 112233' \
     -H "Content-Type: application/json" \
     -H "Prefer: return=representation" \
     -d '{"userid":112233}' | jq .

For details, refer to the relevant section of the SDK documentation.

4.8 Monitoring (Log Management)

The types of logs output by each component are as follows.

L2: Web API Transfer Module

Logs output by L2 serve as the basis for billing records and are therefore collected by the logging service and stored as files in object storage. For information on output destinations, rotation intervals, and other details, refer to the relevant section of the SDK documentation.

L3: Identity Component

L3 outputs logs to standard output and standard error. When running in a container, logs can be viewed with the following command.

$ docker logs l3-app

Clearing and Payment Service

The clearing and payment service outputs logs to standard output and standard error. When running in a container, logs can be viewed with the following command.

$ docker logs payment-app