Technical Roundup #6
The last couple of weeks it’s been quiet here on the surface as we were all busy working out the details of integrations with the first partners; there’s still a lot to do, and this will apparently take some time. Meanwhile we started a new stream of work, now elaborating on the so far missing component of the Infrastructure Layer — the Event Log Storage. While in the initial prototype we focused on running the dRTB transaction between the participants, and logs could be simply written locally on the SSP/DSP gateways — the full-fledged log storage is a critical infrastructure pillar of the fully transparent advertising ecosystem, so now it is our next major milestone. We are currently in the midst of the design, focusing on the major requirements to the log storage solution:
- Handle immense volumes of operational logs written in parallel by multiple participants;
- Provide data immutability and availability guarantees;
- Encrypt data in a granular manner so that its owners could later grant any new participants (appointed auditors) access to the selected ranges of logs;
- Efficiently handle data retrieval to support reporting for all the interested parties.
There are multiple options under examination for the key decisions on arranging individual log entries into large data blocks (the choice of grouping keys), data indices organization (to support data retrieval in the context of ad campaign, for example), dissemination of the data across storage nodes and how those nodes are incentivized.
For the first implementation we are focusing on the target data storing logic (encryption, pre-aggregation to facilitate subsequent reporting, data addressing and indexing), with only very basic nodes management services to orchestrate data replication, replicas monitoring and healing. Initially this may use some pieces of centralized services run on the Papyrus infrastructure, and subsequently will be elaborated to a fully decentralized mechanism (built by Papyrus or based on existing decentralized solutions, this is still an open question).
More details to follow — stay with us!