Technical Roundup #16: First pilot!

In the last few weeks, our development team primarily dedicated their efforts to the preparation of the production environment. Now we are glad to announce our first pilot project, a chance to test our whole system! The first integration will start with SmartyAds to generate live traffic in our system. A pipeline will be set up to receive impression events from Smarty SSP and Smarty DSP. This will allow us to run an SSP- DSP impression settlement. If the discrepancy between the events from SSP and DSP is within the discrepancy threshold (channel configuration), billing information will be generated automatically, otherwise, if the discrepancy is beyond the threshold, a manual resolution will be required.

Let’s go through the pilot technical pipeline step by step:

1. During the process of impression registering on the SSP/DSP side, SmartyAds calls Papyrus Channel Node using REST API to log information about the impression;
2. Impressions are conducted through the Channel Layer and then stored in IPFS. Once stored, the impressions become immutable since data-hashes are put in the Papyrus Blockchain;
3. Every hour a Block is created to keep raw information about the events in the channel;
4. The information about every channel block is then written into a blockchain;

And here we feature our brand new tool — Papyrus Channel Monitor! It queries the blocks and the channels related to certain settlement directly from blockchain and display the lists of channels, blocks and billing information in the UI.

Papyrus Channel Monitor aims at market participants who use channels for their billing settlement or validation. It is an effective user interface which allows simple monitoring of the traffic going through the channels, brings visibility for the billing processes and provides functionality to resolve the data discrepancies from different parties. Papyrus Channel Monitor tool provides such information to our participants (SSPs, DSPs, Verificators etc.) as:

• Channel list. A Participant is now able to see the list of channels and the detailed information about any channel. A channel is a data flow, which is set to a particular agreement between specific participants. The data represents the events like bid request, bid response, impression event, etc.
• Block list. Each channel contains aggregated data, called blocks. All data going through the channels is signed by the participants and encrypted with the master-key and stored in IPFS. Once per hour the data (number of events, monetary amounts calculated, discrepancies identified) is aggregated and stored in an entity, which we call a block. A block contains the collected values as well as the link to the raw data stored in IPFS.

Channels, Blocks, and Participants are stored in Papyrus Blockchain-Registry. Channel contracts are stored in contract ChannelManager, blocks are stored in channel contracts and participants are stored in ParticipantManager.

The Papyrus Channel Monitor processes data from the Registry and presents it for the user in a convenient way.

As we mentioned in our previous blog post, all the data in Papyrus is encrypted, however, since the Channel Monitor is a native tool for the Papyrus Ecosystem, it can vastly simplify decrypting process for the user. Channel Monitor automatically gets public keys from the Channel and transforms them into Block keys using its own Private keys. As complicated it might seem, we at Papyrus believe that there can be no excess in security.

The Papyrus Dashboard, made by our team creates a detailed visual analysis of the entire chain of delivery of advertising. To create aggregated views and insights based on the raw events, our backend should provide a powerful solution to run complex queries, aggregations with filters and conditions on multiple entities.

Considering such requirements as scalability, running OLAP requests and real-time views, we have chosen a column-oriented database management system ClickHouse!

ClickHouse has some specific advantages, which we found useful for our environment:

• Primary focus on the online analytical processing of queries (OLAP)
• Data is stored in columns, which allows storing compact indices in memory and reduces hard-disk operations due to compression of data.
• Partitioning and replication out of the box
• Designed for batch data (>1000 rows at a time) insertion

All these new tools and technologies serve the same goal — successful integration and meeting the needs of our partners.

Follow the news to find out how the integration of our platform with the first real advertising platform will pass. And we will do our best to turn this story into a big success!

Stay turned!