| title | Distributed Systems |
|---|
Distributed systems offer vast capabilities, but they also come with inherent complexities. In The Graph, these complexities are amplified at a global scale.
This document provides an explanation of why requests can appear inconsistent, how block reorganization (re-org) events affect data delivery, and why certain solutions exist to maintain consistency.
Whenever different computers are working together across the globe, certain problems become unavoidable. Connections fail, computers get restarted, and clocks fall out of sync. In the context of The Graph, multiple Indexers ingest blocks at slightly different times, and clients can make requests to any of these Indexers. The result is that requests may arrive out of order or be answered based on different block states.
A perfect example of this is the so-called "block wobble" phenomenon, where a client sees block data appear to jump forward and backward in unexpected ways. This phenomenon becomes especially noticeable during a block re-org — a situation where a previously ingested block is replaced by a different one under consensus.
To understand the impact, consider a scenario where a client continuously fetches the latest block from an Indexer:
- Indexer ingests block 8
- Request served to the client for block 8
- Indexer ingests block 9
- Indexer ingests block 10A
- Request served to the client for block 10A
- Indexer detects re-org to 10B and rolls back 10A
- Request served to the client for block 9
- Indexer ingests block 10B
- Indexer ingests block 11
- Request served to the client for block 11
From the Indexer's viewpoint, it sees a forward-moving progression with a brief need to roll back an invalid block. But from the client's viewpoint, responses seem to arrive in a puzzling order: block 8, block 10, then suddenly block 9, and finally block 11.
This disruption can cause data to appear contradictory. The challenge is magnified when clients are routed to multiple Indexers, each of which may be at different block heights.
In a distributed protocol like The Graph, it is the responsibility of both client and server to coordinate on a consistency strategy. Systems often require different approaches based on their tolerance for receiving out-of-date data versus their need for up-to-the-moment accuracy.
Reasoning through these distributed-system implications is difficult, but solutions exist to reduce confusion. For instance, certain patterns and APIs ensure either forward-only progress or consistent snapshot views of data. Below, two notable methods to maintain a clearer sense of order are explored.
Sometimes, your system only needs to avoid "going backward" in time. The Graph provides the block: { number_gte: $minBlock } argument to help support this by guaranteeing that returned data will always be from a block number equal to or greater than a specified minimum:
/// Updates the protocol.paused variable by always querying
/// for a block at or beyond the last known block.
async function updateProtocolPaused() {
let minBlock = 0
for (;;) {
// Wait until the next block is likely available.
const nextBlock = new Promise((f) => {
setTimeout(f, 14000)
})
const query = `
query GetProtocol($minBlock: Int!) {
protocol(block: { number_gte: $minBlock }, id: "0") {
paused
}
_meta {
block {
number
}
}
}`
const variables = { minBlock }
const response = await graphql(query, variables)
minBlock = response._meta.block.number
// This ensures time never travels backward.
console.log(response.protocol.paused)
// Wait before fetching the next block.
await nextBlock
}
}In an environment with multiple Indexers, the Gateway can filter out Indexers not yet at minBlock, ensuring you consistently move forward in time.
In other scenarios, you need to retrieve multiple related data points or perform pagination without risking differences in blocks. This is where pinning a query to a single block hash becomes critical:
/// Gets a list of domain names from a single block using pagination.
async function getDomainNames() {
let pages = 5
const perPage = 1000
const listDomainsQuery = `
query ListDomains($perPage: Int!) {
domains(first: $perPage) {
name
id
}
_meta {
block {
hash
}
}
}`
let data = await graphql(listDomainsQuery, { perPage })
let result = data.domains.map((d) => d.name)
let blockHash = data._meta.block.hash
while (data.domains.length == perPage && --pages) {
let lastID = data.domains[data.domains.length - 1].id
let query = `
query ListDomains($perPage: Int!, $lastID: ID!, $blockHash: Bytes!) {
domains(
first: $perPage,
where: { id_gt: $lastID },
block: { hash: $blockHash }
) {
name
id
}
}`
data = await graphql(query, { perPage, lastID, blockHash })
for (domain of data.domains) {
result.push(domain.name)
}
}
return result
}By pinning queries to a single block hash, all responses relate to the same point in time, eliminating any variation caused by a re-org that might occur if different blocks were referenced. However, if a re-org does affect that chosen block, the client must repeat the process with a new canonical block hash.
Distributed systems can seem unpredictable, but understanding the root causes of events like out-of-order requests and block reorganizations helps clarify why the data may appear contradictory. By using the patterns described above, both crossing block boundaries forward in time and maintaining a single consistent block snapshot become possible strategies in managing this inherent complexity.