Skip to main content

Preloading

Preloaders play a crucial role in fetching low-level data, such as block details and transaction receipts. This ensures that subsequent base snapshot building can proceed without unnecessary redundant queries against the data source smart contracts.

Each project type within the project configuration, found in config/projects.json, can specify the preloaders that their base snapshot builds depend on. Once the dependent preloaders have completed their fetches, the Processor Distributor subsequently triggers the base snapshot builders for each project type.

config/projects.example.json
loading...

Preloading

Interface​

The preloaders implement one of the following two generic interfaces:

GenericPreloader​

snapshotter/utils/callback_helpers.py
loading...

GenericDelegatorPreloader​

These preloaders are tasked with fetching large volumes of data and utilize delegated workers. They submit large workloads over a request queue and wait for the results to be returned over a response queue.

snapshotter/utils/callback_helpers.py
loading...

Shipped preloaders​

Currently, the snapshotter template includes three generic preloaders:

  • Block details - Prefetches block details for all blocks in an epoch and caches them in Redis.
  • Eth Price - Prefetches ETH price for blocks in an epoch and caches it in Redis.
  • Transaction receipts - Prefetches all transaction details present in each epoch and caches the data in Redis. Since fetching all block transactions is a substantial workload, it utilizes the delegated workers architecture to parallelize and fetch data quickly and reliably.