Protocol
Overview
This protocol is a fundamental protocol that corresponds to ODS-RAM L4 Discovery and Search and is responsible for the advancement of discovery and search based on metadata, and performs resource discovery and endpoint resolution using information published on the Metadata Exchange.
Abstract Normative Specification
Concepts and Roles
Business Domain
A specific business domain. Common services such as a Discovery Service may be constructed per business domain.
Provider
Provides data or services related to the business domain and may manage multiple resources.
Consumer
Uses data or services provided by a Provider.
Metadata Server
Publishes metadata as RDF. Accepts external retrieval requests at a Metadata Endpoint. Retrieves static data from a Metadata Store and dynamic data from an Application Server as needed.
Metadata Store
Stores metadata as RDF and primarily handles static data.
Metadata Client
Retrieves metadata, for example through a JavaScript library in a browser.
Metadata Endpoint
RDF retrieval endpoint in a Metadata Server, including a SPARQL endpoint.
Discovery Finder
A service that determines which Discovery Service to use. It provides metadata including the endpoint of a Discovery Service.
Discovery Service
Searches metadata based on resource attributes (e.g., latitude/longitude, strings). The returned metadata includes not only the target resource but also the semantic definitions and endpoints of the related data/service provider interfaces. Endpoints include the Metadata Endpoint as well as service-specific endpoints (e.g., reservation APIs).
Note that the "Discovery Service" described in "Why Open Dataspaces: Design Philosophy and the Architectural Paradigm" is a concept that encompasses both the "Discovery Finder" (discovery of Discovery Services) and the "Discovery Service" (discovery of destination endpoints) as defined in this protocol specification.
Scope
Discovery Service resolution
Resource search
Endpoint retrieval
Normative Requirements
Discovery Service SHALL provide an API (register) to register searchable resources.
Rationale: Discovery assumes that service providers or data holders actively register their data or services to be searchable. Unlike passive indexing such as web search engines, the Discovery Service performs active indexing initiated by the site.
Discovery Service SHALL provide an API (find) that returns metadata including endpoints, using attributes as keys, in order for clients to resolve endpoints necessary to access data or services.
Rationale: In distributed data management, data and services may exist at arbitrary global locations, enabling data usage control. However, discovery becomes a technical challenge for clients; therefore, it is necessary to enable access to endpoints based on searchable attributes.
Discovery Service SHALL return metadata including resource endpoints as search results.
Rationale: Since the Discovery Service is used to identify the location of data or services, it must return metadata containing endpoints.
Discovery Service SHOULD support keyword search. It SHOULD provide an API (find) that returns metadata including endpoints using string-based keys.
Rationale: An index that returns metadata for data or services associated with keyword matches is broadly useful.
Metadata registration in the Discovery Service SHOULD become invalid after a certain period (TTL). To remain searchable, a keep-alive instruction is required.
Rationale: To minimize the existence of stale metadata when the underlying data/service no longer exists (cf. soft-state management in distributed systems).
Discovery Service MAY support geospatial search. It provide an API (find) that returns metadata including endpoints using geospatial attributes as keys.
Rationale: An index that returns metadata for data or services associated with real-world latitude and longitude can be useful depending on the use case.
Consistency requirements for metadata management in the Discovery Service MAY be eventual consistency.
Rationale: Scalability and fault tolerance are important in distributed systems. Consistency between metadata stored in the Discovery Service and that of the original provider can typically be verified when accessing the actual data/service.
Metadata of the Discovery Service itself MAY be managed using the same format and method (e.g., service definitions using SAMM).
Rationale: Enables uniform access from clients.
Construction of a Discovery Service in each domain MAY be facilitated by defining indexable attributes in metadata (e.g., via SAMM service definitions), allowing necessary components to be generated.
Rationale: Building a Discovery Service requires database setup and message handling, much of which is common and can be automated. Some domain-specific logic remains for developers, so generation is provided up to a template level.
Discovery Finder SHALL provide an API that returns metadata including the endpoint of a Discovery Service, based on the Discovery Service name or domain name, in order to resolve the endpoint required for clients to access a Discovery Service.
Rationale: Since multiple Discovery Services may exist per domain, endpoint resolution itself is required.
Clients SHOULD statically maintain metadata such as the endpoint and API information of the Discovery Finder.
Rationale: It is assumed that only one or a limited number of Discovery Finders exist within Open Dataspaces, and the frequency of retrieving domain-specific Discovery Services is relatively low.
Message Types
findDiscoveryService
Metadata Client ⇒ Discovery Finder
Search for a Discovery Service
findResource
Metadata Client ⇒ Discovery Service
Search for a resource
Protocol Flow
State Machine
States
When information about a data/service provider is registered in the Discovery Service, its lifecycle is managed according to the following state transitions.
Initial
State before being registered in the Discovery Service
Active
Registered in the Discovery Service and searchable
Terminated
Not registered in the Discovery Service or invalidated
create
Registration of provider information in the Discovery Service
delete
Deletion of provider information from the Discovery Service
timeout
Invalidation after a certain period following creation/update
keep alive
Instruction to maintain searchable status
State Machine Diagram
Error Handling
This section describes the error handling used in this protocol.
Invalid Query
Invalid conditions
Correct input
Not Found
No matching result
Notification only
System Error
Internal failure
Notify administrator
Last updated