Innovation That Matters

or: How I Learned to Stop Worrying and Love the API

TL;DR — In this article I review common patterns for exposing business objects over HTTP and discuss some of the challenges of query federation in a microservice architecture.

Do you miss the Good Old Days, when every programmer worth their salt would open a socket and read arbitrary binary data, implementing an optimized Remote Procedure Call protocol from scratch? 

But how to describe the format of the data so that other programmers can create cross-platform clients to call the RPC API? An Interface Description Language was required, and like busses, soon we had more than enough of those!

Feeling RESTed?

Fast-forward a couple of decades, and Roy Fielding mapped proprietary RPC operations for Create, Read, Update and Delete (CRUD) onto HTTP and defined the Representational State Transfer architecture pattern, using HTTP POST, GET, PUT and DELETE for the respective CRUD verbs. REST proposes a “less is more” architecture style, fundamentally polyglot, web-scale, based on semantics of well-defined transformations to resources, and open to a host of intermediaries (current and future), such as firewalls, caches, proxies and routers.

Adopting REST architecture means that when an HTTP cache sees:

POST https://example.com/api/books/book/

Body:

{
 "author" : "Daniel Selman",
 "isbn" : "9781930110359",
 "title" : "Java 3D Programming",
 "publisher" : "Manning Publications"
}

It knows that a new book resource has been created, while when it sees:

DELETE https://example.com/api/books/book/9781930110359

It knows that the resource /books/book/9781930110359 can be safely removed from the cache. 

Data Models

How does the programmer know what payload to send to POST https://example.com/api/books/book/? In the early days of REST APIs this would typically involve reading documentation — as programmers rejected prior attempts at XML standardization, such as SOAP and WSDL, for describing their JSON processing web APIs.

More recently JSON Schema has slowly gained popularity, and with it, Swagger aka Open API as a machine-readable description of HTTP endpoints as well as the expected request and response types.

Limitations of REST

Surely there’s more to this cruddy world than creating, reading, updating and deleting resources!? Some common questions people run into as they design REST APIs are how to:

1. Model batch operations, such as, “Delete all books more than 5 years out of print”
2. Organize the HTTP request arguments for simple resource queries, such as, “Find all books where the author is `Daniel Selman` and the publisher is `Manning Publications`
3. Perform complex resource queries (joins across resource types), such as, “Find all books where the book is still in print and we have not paid the author royalties for more than 2 years”
4. Specify that only a limited number of responses should be returned?
5. Specify that only records 100-200 (or 100 records starting at a cursor) from a set of responses should be returned?
6. How to specify a trigger (active-query), such as “Notify me using a Web Socket when someone creates a new book where the publisher is `Manning Publications`”

REST is (intentionally?) silent on these topics, while some loosely adhered to design-patterns have emerged in some areas. 

I want you to remember one thing, the folks back home is a countin’ on ya, and by golly we ain’t about to let ’em down. Tell you somethin’ else. If this thing turns out to be half as important is I figure it just might be, I’d say that you’re all in line for some important promotions and personal citations when this thing’s over with. 

Maj. ‘King’ Kong, Dr Strangelove

What About gRPC et al?

HTTP is not the only game in town, and therefore you may want to also evaluate gRPC, Thrift, and other similar RPC protocols. They may have uses in specialized (high performance?) domains, but here we are primarily interested in a protocol to expose microservices via a public API to a constellation of diverse clients. HTTP has proved itself extremely useful in this regard, and a well-designed HTTP API has many desirable properties for observability, documentation, cross-platform adoption and, yes, even performance.

A Word About JSON Schema

Open API uses a superset of JSON Schema to describe the expected shape of request and response payloads. JSON Schema is a rather odd format (clearly inspired by elements of XML Schema) in that it has some very document oriented validation constructs that don’t map naturally to type systems used in mainstream programming languages, making it difficult to generate useful code from JSON Schema definitions.

For example, the JSON Schema metamodel includes:

allOf - data valid against all of the sub-schemas
oneOf - data valid against one of the sub-schemas
anyOf - data valid against any of the sub-schemas
not - data valid if not valid by a sub-schema
If / then / else - if valid against sub-schema, then apply testA else testB
prefixItems - data valid if first N elements in array valid against sub-schemas
items - data valid is items in array valid against sub-schemas
contains - data valid is array contains any item that is valid against sub-schema
properties
patternProperties
additionalProperties
propertyNames
unevaluatedItems
unevaluatedProperties

Open API

Open API (fka Swagger) deserves a longer mention, because it has the most traction of all open source tools and is the defacto-standard for defining “simple” endpoint-oriented API. Rather than being resource oriented (like GraphQL) it is endpoint oriented, modelling the HTTP endpoints exposed by a service, along with their expected payloads and responses using JSON Schema.

Open API is evolving fairly quickly and in newer versions has better alignment with JSON Schema as well as adding support for callbacks (aka subscriptions). And, yes — it is slowly, but surely, morphing into WSDL with a different syntax!

AsyncAPI is an effort (heavily influenced by Open API) to define the structure of events for async (message passing) services and event-driven architectures.

Salesforce

Arguably the most successful REST API for business is the Salesforce API. It exposes out-of-the-box Salesforce objects, with custom properties, as well as custom objects over a standards compliant REST API.

As you will see in some of the examples below, they have also had to overcome some of the limitations inherent in REST.

Basic access to resources is via the /sobjects/ path, which uses the familiar REST verbs for basic CRUD operations.

DELET/GET/POST/PUT https://yourInstance.salesforce.com/services/data/v52.0/sobjects/Account/

Note that all Salesforce APIs are capped to return a maximum of 2,000 records, and responses may include a URL to retrieve additional records, if they are available (an Hypermedia REST API).

Metadata API

Salesforce customers can use the Metadata API to dynamically retrieve information about all the objects, and their customizations, in their account. The Metadata API is significantly less easy to use than the basic sObject REST API as it is based on SOAP and WSDL, delivering XML over HTTP.

Pseudo Resources

Salesforce uses “pseudo resources” to represent complex/common operations. For example, the /updated/ pseudo-resource allows a developer to retrieve all the objects updated between two time periods.

/services/data/v52.0/sobjects/Merchandise__c/updated/​​​?start=2013-05-06T00%3A00%3A00%2B00%3A00&end=2013-05-10T00%3A00%3A00%2B00%3A00

Batching

Multiple requests can be batched server-side (for efficiency) using the /batch/ pseudo-resource. The body of the request specifies the individual requests to be batched.

curl https://yourInstance.salesforce.com/services/data/v52.0/composite/batch/ -H "Authorization: Bearer token -H "Content-Type: application/json" -d "@batch.json"
{
"batchRequests" : [
   {
   "method" : "PATCH",
   "url" : "v52.0/sobjects/account/001D000000K0fXOIAZ",
   "richInput" : {"Name" : "NewName"}
   },{
   "method" : "GET",
   "url" : "v52.0/sobjects/account/001D000000K0fXOIAZ?fields=Name,BillingPostalCode"
   }]
}

Composite

The /composite/ pseudo-resource allows a set of operations to be conditionally performed to a resource – for example updating an account with a new contact in a single HTTP request.

curl https://yourInstance.salesforce.com/services/data/v52.0/composite/ -H "Authorization: Bearer token -H "Content-Type: application/json" -d "@composite.json"
{
   "allOrNone" : true,
   "compositeRequest" : [{
       "method" : "PATCH",
       "url" : "/services/data/v52.0/sobjects/Account/001xx000003DIpcAAG",
       "referenceId" : "UpdatedAccount",
       "body" : { 
           "Name" : "Salesforce",
           "BillingStreet" : "Landmark @ 1 Market Street",
           "BillingCity" : "San Francisco",
           "BillingState" : "California",
           "Industry" : "Technology"
       }
   },{
       "method" : "POST",
       "referenceId" : "NewContact",
       "url" : "/services/data/v52.0/sobjects/Contact/",
       "body" : { 
           "lastname" : "John Doe",
           "Phone" : "1234567890"
       }
   },{
       "method" : "POST",
       "referenceId" : "JunctionRecord",
       "url" : "/services/data/v52.0/sobjects/AccountContactJunction__c",
       "body" : { 
           "accountId__c" : "001xx000003DIpcAAG",
           "contactId__c" : "@{NewContact.id}"
       }
   }]
}

Query

Complex queries are executed via the /query/ resource with a query string specified using SOQL or SOSL syntax. SOQL is an SQL inspired query language for Salesforce objects, while SOSL allows Salesforce objects to be queried using text search.

curl https://yourInstance.salesforce.com/services/data/v52.0/query/?q=SELECT+name+from+Account -H "Authorization: Bearer token"

Let’s Get GraphiQL

Over at Facebook, where they apparently have over ten thousand types of resources!, they were busy subverting REST for their own purposes, creating GraphQL as a very non-RESTful alternative to describe and interact with resources via HTTP.

GraphQL has now escaped the confines of Facebook and is a major Open Source project in its own right, boasting many users, including Shopify, GitHub, Twitter and Yelp.

GraphQL exposes a single HTTP endpoint for querying data, mutating data and for defining triggers (called subscriptions in GraphQL). It also exposes a metadata endpoint to allow tools to discover the names of resources, their properties, named queries and named mutations.

A Word About the GraphQL Type System

The type system for GraphQL is far simpler than JSON Schema and appears to be influenced by recent work in mainstream programming languages on type systems and type inference, such as Typescript. It includes objects, interfaces, unions, enums, directives (aka decorators aka annotations). It does not directly support inheritance, though it does allow types to be extended. It does also have some more esoteric composition features, such as fragments.

GraphQL Servers (and Clients)

One interesting, fairly recent, development is the emergence of GraphQL servers — whereby developers can model their resources using GraphQL and then the entire implementation of persistence, query, CRUD operations and subscriptions is provided by a GraphQL server, all without writing any server-side code. There are a growing number of vendors and open source projects in this space — one of the easiest to get started with is Hasura, which exposes Postgres tables as GraphQL API and provides a server.

GraphQL servers blur the line between proprietary tools in the space, such as Airtable and other low-code or no-code platforms, and traditional developer-created APIs and services.

Frontend developers can also use a GraphQL document to generate a mock implementation of the backend, which will return sample data for all of the routes.

GraphQL can’t benefit from the caching that is baked into HTTP, however GraphQL clients can offer richer caching support based on the semantics of GraphQL (queries/mutations) as well as a deep understanding of the GraphQL model for an application.

It should come as no surprise that GraphQL integrates very naturally with React (another Facebook project) — allowing client-side developers to easily create lazily-loaded dynamic web experiences.

Falcor

GraphQL is not the only game in town, and Falcor (from Netflix) simultaneously set out to solve similar challenges — though without the emphasis on modelling resources.

The Falcor programming model presents a virtualized JSON Graph to client developers, accessible via an async JS library. The client library performs sophisticated caching by exploiting the hierarchical nature of the graph. A server side request router is used to determine how segments of the graph are resolved by backend Data Sources.

I think it is fair to say that Falcor has not had the traction of GraphQL, clearly visible when comparing the commits to the GraphQL JavaScript implementation vs Falcor.

GitHub commit activity for GraphQL JavaScript implementation

From Monoliths to Microservices (via SOA)

Over the past couple of decades we’ve also seen a move away from siloed applications (monoliths) to architectures that decompose applications into a suite of resilient and loosely collaborating microservices (at least in theory!).

In a microservice architecture small teams each create many discrete, and largely independent, services based-around business capabilities. Each service will publish an API to consumers (and ideally SLAs!) and will in turn rely on other published APIs and infrastructure services, for persistence, messaging, authentication etc.

An interesting challenge with a microservice approach is how to present a uniform “front-door” API for all the discrete API published by the suite of microservices. You probably don’t want team Alpha publishing services that use HTTP Basic-Auth for authentication and processes XML payloads, while team Bravo uses JWT tokens and processes JSON data! Most of these issues have to be handled via cross-team communication and definition of standards for Good Services, however there is some technical work to aggregate the APIs into a uniform developer experience — which could involve merging multiple Open API documents, or building a GraphQL API that is implemented, via delegation, by a suite of microservices. 

Note that there is also the opportunity to add versioning, authentication and observability logic to the “front-door” gateway service, allowing it more control over how the backend (origin) services are called.

restQL

The Open Source restQL project provides a query language and middleware to generate HTTP calls to origin services. It attempts to stick closer to RESTful semantics for resources, whilst layering on a query language. The query language is used to build HTTP requests while a server (written in GoLang) resolves the queries (concurrently when possible) and returns the results. restQL is a relatively new project with a single major contributor.

restQL is configured to map the names of resources appearing in queries to physical URLs.

Federated Queries

With the move to microservices we now have multiple teams building origin services and exposing them via HTTP — calling a single service should be relatively easy, based on a human, or machine-readable, specification for the API. The service will expose queries and use a set of conventions for query parameters:

GET https://example.com/api/books?author=”Dan%”&publisher=”Manning Publications”&offset=0&limit=50

Here the service supports querying books based on their attributes, returning a limited number of results. Note the embedded string like (%) operator as well as the offset/limit arguments used to limit results. These sorts of conventions are very loose, and will often expose details of how the backend has been implemented. Typically, they require the developers of the API to read API documentation to understand.

Things get even more complicated once the API needs to allow developers to query across resources (a join). For example, the query below, to find all books that have a Manning Publications author that has been paid:

GET https://example.com/api/query?book.author=”Dan%”&book.publisher=”Manning Publications”&invoice.author=[0]&invoice.status=PAID&offset=0&limit=50

Note that this omits which types/fields should be returned, so in reality the example would be even more complex! Also note that the `query` endpoint is not a RESTful resource, so we are starting to stray away from the benefits and semantics of REST.

One of the benefits of GraphQL is that it defines the data model, and the query language over that model, using a single (unified) specification.

There are other small projects in this space, such as https://github.com/a8m/rql and https://rapidql.com. 
GraphQL Mesh is an interesting project that uses the GraphQL metamodel to federate a wide variety of origin services, exposing them all via a uniform GraphQL interface. It is based on the open source library open-api-to-graphql which is itself based on a paper from IBM in converting REST-like APIs to GraphQL.

Executing queries optimally across a collection of origin services is a Hard Problem — especially if latency varies widely between the origin services, and in the face of network failures, as it requires building a robust query plan based on service metadata.

Recommendations

Given the current state of the art it is hard to ignore either Open API, or GraphQL. Open API is fundamentally better aligned with HTTP, while GraphQL provides much richer support for complex (non RESTful?) interaction patterns for resources, as well as queries. 

In an ideal world Open API would be supplemented with standardized query and complex mutation endpoints from GraphQL, however that is likely to take time to emerge. My current recommendations are to provide both an Open API and GraphQL interface for new APIs — providing choice for customers based on their usage patterns and workloads. To make this feasible the data models must be compatible with both JSON Schema and GraphQL.

1. Resources MUST be exposed as RESTful APIs. This is the pattern most developers have come to expect and is well-aligned to the architecture of HTTP
2. Resources MUST have a data model, allowing for document generation (developer UX) and runtime payload verification. The metamodel for resources is critical in ensuring that origin services can be conveniently created in a variety of programming languages, is compatible with Open API and GraphQL, and that simple client SDKs can be generated where useful.
3. Origin services SHOULD expose simple resource queries using a uniform set of conventions for property filtering, pagination, authentication etc.
4. Developers MUST be able to introspect APIs using standard tools, discovering the available resources, their properties and HTTP endpoints
5. Resources MAY expose subscriptions/callback/webhook using a uniform convention
6. Origin services SHOULD expose resources via GraphQL, allowing developers to specify the fields to retrieve, and to write complex queries and mutations.

Red Flags

Origin services performing any of the following should be considered a “red flag” that merits additional analysis and discussion.

1. Creating custom metadata retrieval endpoints
2. Custom mechanisms to define subscriptions/webhooks
3. Data model descriptions that are neither JSON Schema nor GraphQL Schema compatible
4. Custom query languages
5. Complex/custom languages to specify the fields to retrieve
6. Batch/bulk specification languages