The experience is natively integrated into ClickHouse Cloud through ClickPipes, the integration engine designed to simplify moving massive volumes of data to ClickHouse. This eliminates the need for external ETL tools, which are often expensive, slow, and don’t scale for Postgres.

👉You can sign up to the private preview by following this link.

Just a reminder, ClickHouse joined forces with PeerDB, a leading Change Data Capture (CDC) provider for Postgres, a few months ago. PeerDB already supports multiple enterprise-grade workloads and has helped replicate petabytes of data from Postgres to ClickHouse. Over the past few months, the team has worked hard to natively integrate PeerDB into ClickHouse Cloud. This announcement marks the first release of this integration, enabling users to seamlessly move data from Postgres to ClickHouse.

The Postgres CDC connector was built in close collaboration with several customers and design partners who are already running production-grade workloads. Here are a few customer testimonials:

“PeerDB has been a game-changer for us, effortlessly migrating tens of terabytes from our Postgres warehouse into ClickHouse and keeping millions of daily orders synced with just seconds of latency. We're really excited about PeerDB's native integration into ClickHouse Cloud via ClickPipes and all of the opportunities it opens up for us.” - SpotOn

“We already reduced our Postgres to ClickHouse snapshot times from 10+ hours down to 15 minutes with PeerDB. Combining ClickHouse’s powerful analytics natively with PeerDB’s real-time data capture capabilities will greatly simplify our data processing workflows. This integration will enable us to build analytical applications faster, giving us a competitive edge in the market.” - Vueling

Without further ado, here is the demo of Postgres CDC connector in ClickPipes:

Postgres + ClickHouse, a powerful data stack

Using ClickHouse and PostgreSQL through a seamless CDC integration creates a powerful data stack by combining PostgreSQL's robust transactional capabilities with ClickHouse's high-performance analytics. CDC ensures real-time synchronization, allowing ClickHouse to handle fast queries on massive datasets without burdening PostgreSQL. This integration delivers real-time insights and scalable analytics, making it an ideal solution for modern, data-driven workflows. Below are a few main advantages of this architecture:

1. Full workload isolation: You can continue building your O LTP application on Postgres and your OLAP application on ClickHouse, with complete workload isolation—analytics will not affect your transactional workload.

2. No compromises on features: It also allows you to build your applications using the full capabilities and features (e.g., SQL coverage, performance, etc.) of both Postgres and ClickHouse, each optimized for a specific workload.

We believe customers derive the most value in solving real-world data problems by leveraging purpose-built databases like Postgres and ClickHouse as they were designed, with full flexibility, rather than relying on alternatives that retrofit one database engine into another, compromising the full feature set of each. We are observing a clear trend towards the Postgres + ClickHouse architecture among real-world customers.

Key Benefits

The Postgres CDC connector in ClickPipes is purpose-built for Postgres and ClickHouse, ensuring a fast, simple and a cost effective replication experience. Here are some key benefits for customers:

Blazing Fast Performance

With features like parallel snapshotting, you can achieve 10x faster initial loads, transferring terabytes of data in hours instead of days, and experience replication latency as low as a few seconds for continuous replication (CDC).

Super Simple

You can start replicating your Postgres databases to ClickHouse in just a few clicks and minutes. Simply add your Postgres database as a source, select the specific tables/columns you want to replicate, and you're ready to go.

Postgres and ClickHouse native features

This connector supports native Postgres features such as replication of schema changes, partitioned tables, built-in monitoring and alerting for replication slot size, and support for complex data types such as JSONB and ARRAYs, among others.

On the ClickHouse side, it supports features such as selecting specialized table engines, configuring custom order keys, choosing nullable columns, and so on during the replication process.

Enterprise-grade security

At ClickHouse, security is a top priority, even before performance and features. We’ve extended the same level of security to the Postgres CDC connector in ClickPipes. It includes features such as SSH tunneling and Private Link to securely connect to your Postgres databases. Data in transit is fully encrypted using SSL.

No vendor lock-in

The Postgres CDC connector is powered by PeerDB, which is fully open source https://github.com/PeerDB-io/peerdb/. With the exception of the UI, we have ensured that all components are directly extended from the PeerDB open-source project. This underscores our commitment to open-source and ensures there is no vendor lock-in for our customers.

How to sign up for Private Preview?

You can sign up for the private preview by filling out the form on this page. Our team will reach out to you within a day and closely collaborate with you to provide early access. The Private Preview entails no cost and is fully free. This is a great opportunity for you to get firsthand experience with the native Postgres integration in ClickHouse Cloud and directly influence the roadmap. Looking forward to having you onboard!

This question arises because ClickHouse and Postgres differ in data modeling, as each is a purpose-built database highly optimized for its specific workload -Postgres is a transactional (OLTP) database, while ClickHouse is an analytical (OLAP) columnar database. This guide walks you through essential data modeling concepts in ClickHouse for users coming from the Postgres world. Note that this is part 1 of a blog series, with more to come in the future.

ReplacingMergeTree table engine

PeerDB maps PostgreSQL tables to ClickHouse using the ReplacingMergeTree engine. ClickHouse performs best with append-only workloads and does not recommend frequent UPDATEs. This is where ReplacingMergeTree is particularly powerful.

ReplacingMergeTree supports workloads that involve both data ingestion and modifications. Each table is append-only, with user updates ingested as versioned INSERTs. The ReplacingMergeTree engine manages deduplication (merging) of rows in the background. This is one of the key factors that enables ClickHouse to deliver exceptional real-time ingestion performance.

In PeerDB, both INSERTs and UPDATEs from Postgres are captured as new rows with different versions (using _peerdb_version) in ClickHouse. The ReplacingMergeTree table engine periodically handles deduplication in the background using the Ordering Key (ORDER BY columns), retaining only the row with the latest _peerdb_version. DELETEs from PostgreSQL are propagated as new rows marked as deleted (using the _peerdb_is_deleted column). The snippet below shows the target table definition for the public_goals table in ClickHouse.

clickhouse-cloud :) SHOW CREATE TABLE public_goals;
CREATE TABLE peerdb.public_goals
(
    `id` Int64,
    `owned_user_id` String,
    `goal_title` String,
    `goal_data` String,
    `enabled` Bool,
    `ts` DateTime64(6),
    `_peerdb_synced_at` DateTime64(9) DEFAULT now(),
    `_peerdb_is_deleted` Int8,
    `_peerdb_version` Int64
)
ENGINE = SharedReplacingMergeTree
('/clickhouse/tables/{uuid}/{shard}', '{replica}', _peerdb_version)
PRIMARY KEY id
ORDER BY id
SETTINGS index_granularity = 8192

You might still see duplicates for rows—how should you handle them?

ReplacingMergeTree clears out duplicates asynchronously in the background but doesn't guarantee the absence of duplicates. So, when you query the data, you might still see duplicates for the same row or primary key but with different versions. This is expected. To remove duplicates, you have a couple of approaches:

Use FINAL in your queries

ClickHouse has a unique modifier called FINAL, which performs de-duplication (merging of rows) at query time. This de-duplication occurs after filtering (WHERE clause) but before aggregations (GROUP BY).

A historical concern has been that FINAL can slow down query performance. While it does impact query performance to some extent, recent releases of ClickHouse have introduced significant improvements to enhance FINAL query performance. So, don’t hesitate to use the FINAL clause and evaluate how your queries perform. Below is an example of how to use the FINAL clause:

SELECT owner_user_id, COUNT(*) FROM goals FINAL 
WHERE enabled = true GROUP BY owner_user_id;

Use argMax to deduplicate rows at query time

In ClickHouse, argMax is a powerful function for deduplicating rows dynamically during query execution. This is particularly useful when you need to retain the most recent or relevant record based on a versioning or timestamp column.

For instance, if you're working with a table like peerdb.public_goals, where id is the primary key and _peerdb_version tracks versions, you can use argMax to select the row with the highest _peerdb_version for each id. This approach allows you to efficiently remove duplicates without altering the underlying data. You can then run your aggregations as a subquery over this deduplicated result set for further analysis. Below query is an example of using argMax

SELECT
    owned_user_id,
    COUNT(*) AS active_goals_count,
    MAX(ts) AS latest_goal_time
FROM
(
    SELECT
        id,
        argMax(owned_user_id, _peerdb_version) AS owned_user_id,
        argMax(goal_title, _peerdb_version) AS goal_title,
        argMax(goal_data, _peerdb_version) AS goal_data,
        argMax(enabled, _peerdb_version) AS enabled,
        argMax(ts, _peerdb_version) AS ts,
        argMax(_peerdb_synced_at, _peerdb_version) AS _peerdb_synced_at,
        argMax(_peerdb_is_deleted, _peerdb_version) AS _peerdb_is_deleted,
        max(_peerdb_version) AS _peerdb_version
    FROM peerdb.public_goals
    WHERE enabled = true
    GROUP BY id
) AS deduplicated_goals
GROUP BY owned_user_id;

Use WINDOW FUNCTIONS

You can use ClickHouse's window functions to achieve similar deduplication by selecting the row with the highest _peerdb_version within each id partition. Here's an example:

SELECT
    owned_user_id,
    COUNT(*) AS active_goals_count,
    MAX(ts) AS latest_goal_time
FROM
(
    SELECT
        *,
        ROW_NUMBER() OVER (PARTITION BY id ORDER BY _peerdb_version DESC) AS rn
    FROM peerdb.public_goals
    WHERE enabled = true
) AS ranked_goals
WHERE rn = 1
GROUP BY owned_user_id;

Use Views to simplify deduplication

Encapsulate deduplication in a view to make it simple for BI tools to query the most up-to-date data. For example, use a window function in the view to keep only the latest version of each row:

CREATE VIEW goals AS
SELECT * FROM
(
    SELECT
        *,
        ROW_NUMBER() OVER (PARTITION BY id ORDER BY _peerdb_version DESC) AS rn
    FROM peerdb.public_goals
    WHERE enabled = true
) WHERE rn = 1;

SELECT
    owned_user_id,
    COUNT(*) AS active_goals_count,
    MAX(ts) AS latest_goal_time
FROM goals
GROUP BY owned_user_id;

Nullable Columns

If you're coming from the Postgres world, one surprising aspect of ClickHouse is that it doesn’t store NULL values for columns unless you explicitly wrap the column types in Nullable. For example, instead of storing NULL for dates, ClickHouse stores 1970-01-01 as the default value, which might be unexpected. This behavior is due to the fact that storing NULLs can impact query performance in ClickHouse, as it’s a columnar database. Hence, ClickHouse requires users to explicitly define Nullable types.

In PeerDB, we’ve introduced a setting called PEERDB_NULLABLE, which, when set to true, automatically detects nullable columns in Postgres and marks them as Nullable in ClickHouse during the replication process. This means you don’t need to manually define Nullable types during replication. You can read more about this feature in the following PR.

Data Types

ClickHouse offers a wide variety of data types, ranging from numbers, text, timestamps, dates, and arrays to the recently introduced JSON type. Many of the data types in Postgres can be natively stored in ClickHouse without much modification.

As a reference, here goes the data type matrix we use at PeerDB when replicating data from Postgres to ClickHouse.

The Ordering Key

What is an Ordering Key?

Choosing the right ordering key is crucial for query performance in ClickHouse. Defined by the ORDER BY clause when creating a table, the ordering key functions similarly to a index in Postgres but is optimized for analytics. Unlike Postgres, which uses a B-tree index with entries pointing to each row, ClickHouse uses Sparse Indexing:

1. Data is sorted based on Ordering Key: The ordering key ensures that data on disk is sorted according to the specified columns. This allows for better compression, as correlated values are stored together.

2. Ordering Key also creates a sparse index: The ordering key also creates a sparse index, storing only ranges of columns, with each entry pointing to a group of sorted rows. This keeps the index small, allowing ClickHouse to quickly identify relevant groups of rows using a binary search and execute queries efficiently. You can read more about this here.

You can think of ordering keys as similar to BRIN indexes in Postgres, but in ClickHouse, the data is automatically sorted based on the ordering key via asynchronous merging of parts, so you don’t need to handle sorting during data ingestion.

Choosing an appropriate Ordering Key

When selecting an ordering key, base your choice on the columns most frequently used in your query filters. Prioritize columns that are commonly used in WHERE clauses, and order them in ascending sequence of cardinality—starting with columns that have the fewest distinct values. This approach optimizes data compression and query performance. For a deeper understanding of this topic, refer to the detailed guide here.

PRIMARY KEY vs Ordering Key

If you observe the table definition of public_goals, it has a PRIMARY KEY. You might be wondering how the PRIMARY KEY differs from the Ordering Key. Let us understand how they differ:

1. PRIMARY KEY, if specified, defines the columns in the sparse index, while the columns in the ORDER BY clause determine how the data is sorted on disk. They are also used for deduplicating data by the ReplacingMergeTree.

2. If the PRIMARY KEY isn't specified, the Ordering Key automatically becomes the PRIMARY KEY and defines the columns in the sparse index.

NOTE: Columns in the PRIMARY KEY should always be prefixed in the Ordering Key. This ensures that the index aligns with the physical data order, maximizing query performance by minimizing unnecessary data scans.

An example where PRIMARY KEY could differ from Ordering Key

An example where you might have different PRIMARY KEY and ORDER BY columns is when your queries are primarily filtered on customer_id rather than id. In this case, you can define the PRIMARY KEY on just customer_id and the ORDER BY on customer_id, id. This approach ensures a smaller, more efficient sparse index for querying, while data deduplication occurs on id, ensuring no data is lost.

NOTE: Unlike in Postgres, where the PRIMARY KEY is a B-tree index that guarantees uniqueness, in ClickHouse, it does not ensure uniqueness. Instead, it defines the columns that should be part of the sparse index.

Modifying the Ordering Key

Choosing the right ordering key is crucial for query performance in ClickHouse, as it acts as an index when querying data. By default, PeerDB uses the PostgreSQL PRIMARY KEY to define the ordering key in ClickHouse tables, but you can change it using the following methods:

Use materialized views

You can use materialized views to create a new table with a different ordering key suitable for your workload. Include the primary key columns at the end of the ordering key to ensure proper deduplication, as ReplacingMergeTree uses the ORDER BY clause for deduplication, and including the primary key ensures that no data is lost.

CREATE MATERIALIZED VIEW goals_mv
ENGINE = ReplacingMergeTree(_peerdb_version)
ORDER BY (enabled, ts, id)  POPULATE AS
SELECT * FROM peerdb.public_goals;

NOTE: After creating the materialized view, be sure to follow the steps described in the previous section on handling duplicates to ensure proper deduplication during query time.

Predefine target tables with the desired Ordering Key

To change the ordering key, you can predefine new tables with your desired Ordering Key and then swap them with the existing tables. Here's how you can do it:

1. Create a Dummy Mirror: Create a dummy mirror in PeerDB to generate the default tables with the correct metadata columns and data types.

2. Create a New Table with the Desired Ordering Key: Use the table created by PeerDB to define a new table with your desired ordering key. Include the primary key columns at the end of the ordering key to ensure proper deduplication. Here is an example:

CREATE TABLE public_events_new AS public_events
ENGINE = ReplacingMergeTree(_peerdb_version)
ORDER BY (user_id,id);

3. Drop the Old Table:

DROP TABLE public_events;

4. Rename the New Table: Rename the new table to the actual table

RENAME TABLE public_events_new TO public_events;

5. Start MIRROR to Point to the New Table: Configure the mirror to point to the actual table. PeerDB uses CREATE TABLE IF NOT EXISTS behind the scenes and continues to ingest data into the new table.

Handling DELETEs

As mentioned, DELETEs from PostgreSQL are propagated as new rows marked as deleted (using the _peerdb_is_deleted column). To exclude deleted rows from your queries, you can create row-level policies in ClickHouse based on the _peerdb_is_deleted column. Here’s an example:

CREATE ROW POLICY policy_name ON table_name
FOR SELECT USING _peerdb_is_deleted = 0;

This policy ensures that only rows where _peerdb_is_deleted is 0 are visible when querying the table.

Conclusion

I hope you enjoyed reading the blog. I aimed to cover the most common data-modeling challenges you might encounter when migrating from PostgreSQL to ClickHouse. In the next blog, I plan to dive into more advanced topics, such as joins, writing efficient SQL queries, and so on. If you want to give PeerDB and ClickHouse a try to start replicating data from Postgres to ClickHouse, please check out the links below or reach out to us directly!

1. Try ClickHouse Cloud for Free

2. Try PeerDB Cloud for Free

3. Docs on Postgres to ClickHouse Replication

4. Talk to the PeerDB team directly

In the past few months, the PeerDB team had the opportunity to work with multiple ClickHouse customers, helping them replicate billions of rows and terabytes of data from Postgres to ClickHouse. In this blog, we will take a deep dive into some of the top features that were released recently to make the replication experience rock-solid. These features focus on enhancing the speed, stability, and security of replication from Postgres to ClickHouse.

Efficiently flush the replication slot

PeerDB uses Postgres Logical Replication Slots to implement Change Data Capture (CDC). Logical Replication Slots provide a stream of INSERTs, UPDATEs, and DELETEs occurring in the Postgres database. It is recommended to always consume the replication slot. If the replication slot isn't consumed continuously, WAL files can accumulate, posing a risk of crashing the Postgres database.

To ensure that the logical replication slot is always consumed, we implemented a feature to always read the replication slot and flush the changes to an internal stage (S3). An asynchronous process then consumes the changes from S3 and applies them to ClickHouse. Flushing the changes to the internal stage ensures S3 also ensures that replication slot is consumed even when the target (ClickHouse) is down.

Better memory handling on ClickHouse

While replicating data from Postgres to ClickHouse, customers occasionally ran into memory-related issues on ClickHouse. This was more common when customers were on a free trial of ClickHouse and provisioned an instance with fewer resources (RAM and compute). PeerDB writes rows in batches to ClickHouse via INSERT queries and INSERT SELECT queries. We were seeing 2 types of issues:

1. Some queries were failing because they were consuming more memory than allocated on the ClickHouse server.

2. Some queries would be killed by ClickHouse's overcommit tracker.

We attempted to thoroughly understand the various database settings that ClickHouse provides, which influence memory utilization. Based on this, we modified the following settings:

1. max_block_size: This is useful for our INSERT SELECT queries, where this setting determines how many blocks are loaded by the SELECT and inserted. We reduced this with the hope that more blocks would reduce memory spikes when our queries are executed.

2. max_insert_block_size: Similar to max_block_size except this applies to our INSERT queries. We reduced this for the same reason as above.

3. max_threads: This setting controls the number of threads used for processing queries on ClickHouse. According to the documentation, the lower this number, the less memory is consumed. Therefore, we reduced this parameter.

4. memory_overcommit_ratio_denominator: This is related to the overcommit tracker mentioned earlier. We disabled this setting for our queries by setting it to 0.

5. dial_timeout: Sometimes queries were taking longer than 1 minute, so we increased the dial_timeout to a higher value.

These changes drastically reduced memory-related issues on smaller ClickHouse clusters. We are actively working with the core team to further fine-tune ClickHouse-specific settings. Additionally, we are working on a feature that improves the handling of large datasets by breaking them into manageable parts for more efficient processing and storage.

Row-level transformations

A few months ago, PeerDB shipped Lua-based row-level transformations while replicating data from Postgres to Queues such as Kafka. We have now extended this feature to ClickHouse. With this feature, customers can write simple Lua scripts to perform row-level transformations, enabling use cases such as masking PII data, generating columns, and more. Below is a quick demo of this feature to mask PII columns while replicating data from Postgres to ClickHouse:

Improved security on PeerDB Cloud

At PeerDB, safeguarding data replication from Postgres to ClickHouse is crucial. To enhance security, we have implemented several key measures around AWS S3, which we use for internally staging data before pushing it to ClickHouse.

Temporary credentials with IAM roles

One significant enhancement is the use of AWS S3 buckets with strict access controls. Instead of traditional, long-lived user-generated access keys, which pose a higher risk of compromise, we use IAM roles to generate temporary credentials. These credentials are automatically rotated by AWS, ensuring they are always up-to-date and valid for only short periods, thus minimizing the risk of unauthorized access.

Additionally, with the introduction of the AWS_SESSION_TOKEN parameter in ClickHouse version 24.3.1, our security practices have been further strengthened. This update allows the use of short-lived credentials, aligning with our approach to secure data replication.

Attribute Based Access Control (ABAC)

In a multi-tenant environment, managing access to S3 buckets poses several challenges, such as ensuring tenant isolation, preventing unauthorized access, and minimizing role proliferation. To address these issues, we employ Attribute Based Access Control (ABAC). ABAC allows us to define dynamic, fine-grained access policies based on user roles, resource tags, and environmental variables. This method not only provides enhanced security but also improves scalability by eliminating the need for creating numerous roles. By using ABAC, we ensure that only authorized components can access sensitive data, maintaining a secure and manageable system.

Conclusion

Hope you enjoyed reading the blog. PeerDB has spent multiple cycles hardening Postgres CDC experience to ClickHouse and is now supporting multiple customers in replicating billions of records in real-time from Postgres to ClickHouse. If you want to give PeerDB and ClickHouse a try, please check out the links below or reach out to us directly!

1. Try PeerDB Cloud for Free

2. Try ClickHouse Cloud for Free

3. Docs on Postgres to ClickHouse Replication

4. Talk to the PeerDB team directly

References

This blog is a replica of the original blog, which can be found here.

We released the ClickHouse target connector for Postgres Change Data Capture (CDC) earlier this year. Since then, ClickHouse has become the fastest-growing connector in terms of usage, surpassing other targets such as Snowflake and BigQuery. With this acquisition, we will be powering the Postgres CDC connector for ClickPipes, the native integration engine that helps customers move data into ClickHouse.

Our thesis behind the acquisition

A few months ago, when the prospect of an acquisition came up, we debated whether it was the right move for PeerDB. After much consideration, we decided that it was the best move. It all came down to 3 main reasons:

1. Amplifying Customer Value - ClickHouse was the fastest-growing target connector for Postgres CDC. We observed this firsthand at PeerDB, where we helped multiple customers move billions of rows from Postgres to ClickHouse. This aligns with the ClickHouse community's feedback on Postgres CDC. This acquisition will accelerate PeerDB's reach and make it accessible to thousands of ClickHouse customers, generating significant value.

2. Postgres and ClickHouse: A Match Made in Heaven - Postgres is becoming the de facto operational (OLTP) database, while ClickHouse is the fastest analytical database on the planet. Both originate from the same ethos of open source and have a strong presence in the community. We believe that for customers using Postgres as their default OLTP database, ClickHouse is the natural OLAP counterpart. This is already reflected by prominent users like GitLab, LangChain, Cloudflare and Instacart, who run both databases in conjunction. This acquisition would bridge this gap further, making it easier for customers to use Postgres and ClickHouse together.

3. We love the ClickHouse team - We have been closely collaborating with the ClickHouse team for the past several months. Their customer-first approach, strong emphasis on product quality, and growth mindset align with what PeerDB stands for. We felt that if we both joined forces, we could build something magical for customers.

What does this mean for existing and future customers?

As part of this acquisition, a few important product decisions were made to ensure that the PeerDB community continues to thrive and that existing customers are not affected.

1. PeerDB will remain free and open under the same ELv2 license.

2. PeerDB Enterprise offering that comes with production-grade Helm charts is being made free and open under the same ELv2 license. This means anyone can now run production-grade PeerDB workloads in a self-managed way for free!

3. The end-of-life (EOL) of PeerDB for existing paid customers using non-ClickHouse Cloud connectors will be one year from now, i.e. July 30th, 2025. Customers will receive the same support and SLAs as promised in their contracts, and we will assist them with the transition plan.

4. Until PeerDB is fully integrated into ClickPipes, we will be supporting PeerDB Cloud, the fully managed offering of PeerDB. For new customers from now on, PeerDB Cloud will only support ClickHouse Cloud as the target connector for Postgres CDC. If customers want to use other target connectors, they can use the free and open self-managed PeerDB options.

PeerDB Cloud for blazing fast Postgres CDC to ClickHouse Cloud

Until PeerDB is fully integrated into ClickPipes, customers can use PeerDB Cloud to replicate data from Postgres to ClickHouse Cloud. Over the last few months, we have dedicated multiple cycles to enhancing speed, tightening security, and adding features to provide an enterprise-grade Postgres CDC experience for ClickHouse. You can follow the links below to get started:

1. Start Trial of PeerDB Cloud

2. Postgres to ClickHouse CDC

Thank you!

We would like to thank all of our customers and community for your constant support from our early days. Your trust and feedback have been instrumental in helping PeerDB get to where it is today. We are grateful for everything you've done, and the above product decisions are a small token of our gratitude. If you have any questions, please don't hesitate to reach out to us at founders@peerdb.io or send a direct Slack message. Thank you again!

References

For more information on the acquisition, you can follow the below links:

1. Blog post by ClickHouse

2. Official Press Release

What is SOC 2?

SOC 2, or System and Organization Controls 2, is a framework governed by the American Institute of Certified Public Accountants (AICPA). It is designed to assess the controls and processes involved in storing, processing, and protecting customer data. SOC 2 reports focus on five Trust Services Criteria (TSC): Security, Availability, Processing Integrity, Confidentiality, and Privacy. Every SOC 2 report must cover Security, but organizations can choose to include additional criteria relevant to their operations.

SOC 2 Type II

A SOC 2 Type II report goes beyond evaluating the design and implementation of controls at a single point in time. It assesses the operating effectiveness of these controls over a defined period, typically three months to a year. Achieving SOC 2 Type II compliance means that PeerDB has not only designed appropriate security controls but also maintained their effectiveness over time.

The Journey to SOC 2 Compliance

Our path to SOC 2 compliance was meticulous and comprehensive. Here’s a look at the steps we took:

1. Policy Crafting: Documenting all policies, procedures, and operational controls.

2. Risk Assessment: Conducting thorough evaluations of our systems to identify and mitigate potential vulnerabilities.

3. Vendor Management: Conducting thorough evaluations of our third-party vendors to ensure they meet our stringent security standards. We partnered with Advantage Partners, who served as our auditor, to ensure that all vendors were compliant with our security requirements.

4. Evidence Gathering: Collecting extensive evidence to demonstrate compliance with required controls. We partnered with Vanta to streamline this process, leveraging their platform to automate evidence collection and monitoring.

Why SOC 2 Compliance Matters?

Achieving SOC 2 compliance is about more than just meeting regulatory requirements; it’s about building trust with our clients and partners. It underscores our commitment to maintaining the highest level of security and reliability.

Benefits for Our Customers

• Enhanced Security: SOC 2 compliance guarantees robust protection for your data, including advanced encryption and strict access controls.

• Transparency and Control: Our Trust Center provides detailed information about our security practices, giving you the assurance and control you need over your data.

• Ongoing Improvement: Our dedication to security doesn’t stop here. We continuously evaluate and enhance our measures to stay ahead of emerging threats.

Looking Ahead

While we celebrate this achievement, we are also focused on the future. We will continue to pursue additional certifications and audits to further validate our commitment to security excellence.

Empowering Your Business with PeerDB Cloud

PeerDB Cloud offers a secure and scalable platform for all your Postgres Data Movement needs. With SOC 2 compliance at its core, PeerDB Cloud ensures that your data is protected in a robust and reliable environment. Additionally, we are already GDPR compliant, representing cementing our dedication to data protection and privacy and creating a secure and trusted environment for all our clients.
Our docs also summarize our Cloud Security posture and provide a high level overview of what PeerDB Cloud offers in terms of data protection, compliance measures, and security best practices.

Trust and Assurance at PeerDB

At PeerDB, we are dedicated to being a reliable partner in your digital journey. For more information on our SOC 2 compliance efforts, or any other security-related inquiries, please visit our Trust Center.

Thank you for being part of this journey with us. We look forward to continuing to provide secure and trusted solutions for all your data movement needs.

Logical Decoding has evolved quite a bit in the past few years in Postgres. However, there are a few quirks that users need to overcome. In this blog, we will summarize common issues and learnings from over 20 customers replicating more than 300 TB of data per month with logical decoding.

Beware of replication slot growth – how to monitor it?

A logical replication slot captures changes in the Postgres Write-Ahead Log (WAL) and streams them in a human-readable format to the client. A common issue with logical decoding is unexpected replication slot growth, which can risk filling up storage and causing server crashes. Slot growth mostly occurs when the consumer application (a.k.a. client) that reads changes from a replication slot lags or halts. The consumer application can lag for various reasons, including not consuming the slot appropriately and the high throughput on the Postgres database, combined with logical decoding being single-threaded. More on this topic in the next section.

You can monitor replication slot growth using the below queries:

/* the below query should always return "true"
indicating slot is always getting consumed. */
SELECT slot_name,active FROM pg_replication_slots ;

/* monitor the size of the slot using the below query */
SELECT slot_name, 
 pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(),restart_lsn)) 
 AS replication_lag_bytes 
 FROM pg_replication_slots;

NOTE: Just for reference, we open-sourced the PG Slot Notify bot, which helps you monitor replication slot size. pgslot-notify-bot helps monitor PostgreSQL replication slots by sending alerts once the size threshold is reached.

Tips for keeping replication slot growth in check

Always consume the replication slot

Logical decoding is a single-threaded process, whereas Postgres allows multiple concurrent connections/threads to ingest data. This means that if the client doesn't consume the replication slot fast enough, the slot can quickly grow.

The first step toward efficiency is to ensure that the client always consumes the replication slot and maximizes resource utilization. Intermittent reading of the slot with constant disconnections can slow down logical decoding. Periodic reconnections can lead to other inefficiencies, as logical decoding may need to restart from the beginning of the WAL instead of continuing the stream.

At PeerDB, we implemented this optimization. We ensure that the replication slot is always consumed and flushed to internal stages such as S3.

Beware of long-running transactions

Long-running transactions can lead to WAL buildup. Since WAL is sequential, Postgres cannot flush the WAL until the long transaction completes, even as other transactions are being consumed. This can result in an increased slot size and slow down logical decoding. For each transaction being decoded, changes from long-running transactions that overlap with the current transaction must also be decoded again and again.

Configure statement_timeout and idle_in_transaction_session_timeout to avoid long running transactions

Long-running transactions can occur either due to active queries running for a long time or stale transactions that were never committed. To avoid these scenarios, you should configure statement_timeout, which terminates queries that run longer than expected, and idle_in_transaction_session_timeout, which terminates stale transactions.

Use logical replication protocols

The SQL API (START_REPLICATION) supports different versions of streaming, controlled via the proto_version parameter. The default proto_version (v1) allows clients to consume changes only from committed transactions. proto_version v2 allows clients to consume changes from in-flight transactions, improving performance by letting them process changes immediately without waiting for the COMMIT. However, it is the client's responsibility to handle transaction semantics. At PeerDB, we are working on a feature that supports proto_version (v2).

This changes decoding from an O(N^2) operation to an O(N) operation and also helps address scenarios with long-running transactions. This blog provides a deep dive into how replication slot growth is affected with v2.

Reference: https://blog.peerdb.io/exploring-versions-of-the-postgres-logical-replication-protocol

No activity can lead to replication slot growth.

It is common to see replication slots grow in size in dev/test/QA databases during periods of inactivity. In such scenarios, the WAL continues to grow due to maintenance processes like VACUUMs. To avoid this and ensure that the slot is consistently consumed by the client, you can follow one of the approaches below:

1. Include a heartbeat table that continuously gets updated in your replication pipeline. This ensures that the slot keeps moving. More details on this approach can be found here.

2. Use pg_logical_emit_message to periodically emit a message in the WAL and ensure that this message is consumed by the client by confirming the LSN (Log Sequence Number) of the message.

Use table filtering when creating PUBLICATIONs

If you are replicating changes from only a few tables, ensure that you create a PUBLICATION that includes just those tables. Postgres efficiently persists changes for only those tables in the replication slot. This helps reduce the size of the replication slot and improves logical decoding performance.

Some useful Postgres configs

max_slot_wal_keep_size

To keep your logical replication slots from consuming excessive disk space, set the max_slot_wal_keep_size parameter. This config limits the amount of WAL (Write-Ahead Log) data a replication slot can retain. Choose a size that suits your environment to ensure old WAL files are removed when the limit is reached, preventing disk space issues.

logical_decoding_work_mem

To control memory usage during logical decoding, adjust the logical_decoding_work_mem parameter. This setting allocates a specific amount of memory for the decoding process of each replication slot. Set a value that balances memory use and performance according to your system's capacity and workload. You can consider increasing logical_decoding_work_mem if you observe IO as the wait_event for the START_REPLICATION process. More details on tuning logical_decoding_work_mem can be found here.

Supporting DDL Changes

One of the most common and well-known issues with logical decoding is that it doesn't capture schema changes such as adding or dropping columns, changing data types, adding new tables, and so on.

An approach that clients could follow is to leverage Relation and Type messages that logical decoding provides. Whenever columns are added or dropped, Postgres sends a Relation ('R') message with the new schema, preceding the new row. Clients can perform a diff with the old schema to identify the new or dropped columns. A similar approach can be followed for supporting changing data types, where Postgres sends a Type ('T') message.

Within PeerDB, we implemented the above intricate approach to support automatic schema changes, such as adding or dropping columns.

TOAST columns need REPLICA IDENTITY FULL

Logical decoding doesn't capture TOAST (large sized >8KB) columns that haven't been changed in an update operation. You need to enable REPLICA IDENTITY FULL for a table to capture values of unchanged TOAST columns.

Here is a useful blog that talks about the impact of setting REPLICA IDENTITY FULL on your Postgres database. TL;DR:REPLICA IDENTITY FULL might be fine for tables with primary keys or from Postgres 16, where indexes can be used on the subscriber side for searching the rows.

In PeerDB, for certain targets, we implemented a method to replicate unchanged TOAST columns without requiring REPLICA IDENTITY FULL.

Logical decoding doesn’t support generated columns

One limitation of Postgres logical decoding is that it doesn’t support generated columns. Generated columns are columns whose values are automatically computed from other columns in the table using a specified expression. Values of these columns appear as NULL for clients consuming the logical replication slots. A few workarounds incl:

1. In-flight transformations: Compute the value of generated columns while the data is in transit by performing transformations. PeerDB supports row-level transformations out-of-the-box to enable such generated column use cases.

2. Extract, Load, and Transform (ELT): Another approach we've seen customers follow is to perform transformations once the data reaches the target. Customers often use transformation tools such as DBT or Coalesce.

Logical Replication Slots Don't Persist on Postgres Upgrades

Upgrading PostgreSQL versions presents a challenge because logical replication slots do not persist through upgrades. However, it's possible to manage upgrades without full resyncs by recreating the replication slot during maintenance.

The process is as follows:

1. Enter Maintenance Mode: Place the database in maintenance mode to prevent data changes.

2. Upgrade PostgreSQL: Perform the PostgreSQL version upgrade.

3. Recreate Replication Slot: Recreate the logical replication slot.

4. Exit Maintenance Mode: Resume normal operations.

This method ensures minimal disruption and avoids the need for a complete resync of the data.

Starting Postgres 17 replication slots are persisted on upgrades

Starting with PostgreSQL 17, logical replication slots will persist through version upgrades. This improvement will apply to future upgrades, such as from version 17 to 18, simplifying the upgrade process significantly.

Logical Replication Slot don't persist Failovers

Another issue with logical decoding is that replication slots don't persist during a failover, i.e., when a standby becomes primary. One potential solution is to implement retry logic in your clients to recreate the replication slot post-failover. However, this approach is not fully reliable and can incur data loss, as creating the slot right after failover without data ingestion is not trivial.

Postgres 17 will support failover slots

The good news is that PostgreSQL 17 will support failover slots, allowing replication slots to persist automatically through failovers. This enhancement simplifies the failover process, ensures data reliability, and reduces manual intervention, resulting in more robust and resilient replication handling.

Conclusion

At PeerDB, we are building a replication tool with a laser focus on Postgres. To provide the fastest and most reliable Postgres replication experience, we delve deeply into understanding Postgres logical decoding. This blog summarizes our efforts over the past year. Many of the challenges discussed above have already been addressed in our product, and for those that haven't, we work closely with our customers to find and implement workarounds. We hope you enjoyed reading the blog! If you want to give PeerDB a try, these links should prove useful: :)

1. PeerDB's Github repo

2. Quickstart

3. Directly reach out to us!

4. Join PeerDB's Slack community

5. PeerDB docs

This post explains some of the use cases that are enabled by Postgres to Elasticsearch replication, followed by a quick demo showcasing the high performance and low latency of Postgres to Elasticsearch replication using PeerDB. Finally, we go through a high level overview of the architecture of the connector.

Postgres to Elasticsearch Replication Use cases

Some common use cases for Postgres to Elasticsearch replication via CDC or query replication are:

1. Efficient search for large ingest volumes: Elasticsearch's bread and butter use case is as a search engine operating efficiently even on humongous volumes of data. From full-text and weighted search to even complex semantic searches using built-in NLP models, Elasticsearch is very flexible and tunable. It is commonly used for ingesting and indexing large volumes of logs, and even as a backing engine for searching large websites and internal knowledge bases.

2. Denormalizing data to documents: Data models are often stored in Postgres in a highly normalized form, which is great for transactional integrity but bad for complex queries where may have to use joins or CTEs. Elasticsearch being a document database prefers storing data in a denormalized form. Using PeerDB's query replication capabilities, you are able to periodically transform your data into a denormalized form which makes it more efficient for querying by downstream consumers. Some processing can also be done using an Elasticsearch ingest pipeline.

Low latency replication from Postgres to Elasticsearch using PeerDB

In this section, I'll walk through a quick demonstration of Postgres to Elasticsearch replication using PeerDB in Change Data Capture (CDC) mode. Using PeerDB for replication from Postgres to Elasticsearch offers a few benefits, primary ones being blazing fast initial loads, and sub minute latencies by constantly reading the slot, PeerDB is able to offer these by being laser focused around Postgres replication.

Postgres Setup

You can use any Postgres database in the cloud or on-prem. For simplicity, I'm using a Postgres cluster running locally in a Docker container for this demo. We create a table named oss1 with a continuous ingest of 1000 rows per second using a multi-valued insert statement.

postgres=# CREATE TABLE oss1 (
           id INT PRIMARY KEY GENERATED BY DEFAULT AS IDENTITY,
           c1 INT,
           c2 INT,
           t TEXT,
           updated_at TIMESTAMP WITH TIME ZONE DEFAULT now()
         );
CREATE TABLE
postgres=# INSERT INTO oss1 (c1, c2, t)
SELECT
    generate_series AS c1,
    generate_series * 2 AS c2,
    'text_' || generate_series AS t
FROM 
    generate_series(1, 1000); 
# to run the INSERT once per second    
postgres=# \watch 1
INSERT 0 1000
INSERT 0 1000
INSERT 0 1000

Elasticsearch Setup

You can setup an Elasticsearch instance using its Docker compose setup on-prem or on a cloud VM. Alternatively you can also use Elasticsearch Cloud. For this demo, I am using the Docker compose setup running locally.

PeerDB Setup

You can use PeerDB Open Source or PeerDB Cloud to deploy a PeerDB instance. For the scope of this demo, I'm deploying PeerDB open source locally via Docker compose.

Create Peers and Mirror for Postgres to Elasticsearch Replication

In the PeerDB world, peers refer to either source or target data stores. You can use PeerDB's UI to create the Postgres and the Elasticsearch peers. A mirror is then created between a source peer and a destination peer for data replication. You can use PeerDB's UI to create a MIRROR for replicating data from Postgres to Elasticsearch.

I have created a Change Data Capture (CDC) based MIRROR that replicates data using Postgres' Write-Ahead Log (WAL) and Logical Decoding. It involves two steps:

1. An initial load that takes a fully consistent snapshot of existing data in Postgres and copies it to Elasticsearch; Through PeerDB's parallel snapshotting, you can expect significantly faster initial loads. We've seen terabytes of data moved in hours vs days.

2. Change Data Capture (CDC): Once the initial load is completed, PeerDB constantly reads changes in Postgres through the logical replication slot and replicates those changes to Elasticsearch. Thanks to our streaming architecture, expect data latency in the range of seconds for a continuously running mirror to Elasticsearch.

The initial load should complete pretty quickly, and rows should be present in the created Elasticsearch index. After entering continuous CDC mode, new rows should show up as and when they are inserted. Attached below is a quick video showing a Postgres to Elasticsearch CDC mirror.

Architecture and Design Choices

We've talked about PeerDB's streaming architecture in detail before, but in summary PeerDB utilizes Go's goroutines and channels to efficiently read data from PostgreSQL using logical replication, and then pushes it to Elasticsearch in batches through the Bulk API. This approach enhances the execution time by enabling parallel processing.

Our data warehouse connectors store the data in a staging table before pushing it to the final table for cost and performance reasons. Due to Elasticsearch's architecture and query language, we are also able to avoid this intermediate step directly send the stream of processed records to Elasticsearch indices via the bulk API.

Handling Updates and Deletes in Elasticsearch

PeerDB supports Elasticsearch as a target for both CDC and query replication. In most cases we recommend using CDC because of its ease of use, increased reliability and its ability to replicate DELETEs to Elasticsearch. However, this limits the scope of transformations that can be done before loading to Elasticsearch.

To support deduplication on the Elasticsearch side, we need a unique ID for each document that remains consistent so we can update or delete it as per the source. For tables with one column in the primary key, the value of the column itself can be used. For tables with multiple columns in the primary key, we instead choose to hash the values of the columns together, giving a small unique identifier irrespective of the width of the row.

// simplified Go code
func primaryKeyColsHash(record []any, colIndices []int) string {
    hasher := sha256.New()

    for _, colIndex := range colIndices {
        // write the value to the hasher
        _, _ = fmt.Fprint(hasher, record[colIndex])
    }
    hashBytes := hasher.Sum(nil)
    return base64.RawURLEncoding.EncodeToString(hashBytes)
}

# Sample document uploaded by PeerDB to Elasticsearch.
# Note how the _id field is a (base64 encoded) hash of the
# primary key columns id and c1.
{
  "_index": "public.oss2",
  "_id": "SAgdSqEaQyGYWxOo8Dj2s0DbXsQXLTC_CWlds8-c4kY",
  "_version": 1,
  "_seq_no": 0,
  "_primary_term": 1,
  "found": true,
  "_source": {
    "c1": 434017,
    "c2": 922856,
    "id": 8,
    "t": "pgbenchinsertc4b998821cc6b161e65489b3",
    "updated_at": "2024-05-08T18:33:39.031107Z"
  }
}

Query replication can be done in append mode, where any change creates a fresh document in Elasticsearch or in upsert mode where some columns are designated as key columns which are deduplicated in a way similar to CDC.

Dynamic Mapping for Data Types

By default, PeerDB currently uses Elasticsearch's dynamic mapping to automatically infer a data type mapping based on the contents of the documents in an index. In practice, numeric types are mapped to either long or float, timestamp types are mapped to date and most other types map to text. A more detailed mapping is available here. This works for many use cases. If needed an explicit mapping can be provided by the user during manual index creation and PeerDB will load documents to this index.

Conclusion

Elasticsearch connector is in beta -- we already have customers who have moved billions of rows from Postgres to Elasticsearch using PeerDB. If you're an Elasticsearch user and wish to replicate data from Postgres to Elasticsearch using PeerDB, do give PeerDB a shot! We would love to help you out or get feedback:

1. Try PeerDB Cloud for free.

2. Visit PeerDB's GitHub repository to Get Started.

3. Join our Slack and say hi!

Thanks for reading!

Today, we are announcing one of the flagship features of this offering — support for row-level transformations as part of Postgres Change Data Capture (CDC). You can write simple Lua scripts to define a transformation and add it as part of the replication (MIRROR). With this feature, users will be able to seamlessly perform in-flight row-level transformations to Postgres data before it is streamed to the target.

In this blog, we will cover various use cases that require row-level transformations and how they can be accomplished using PeerDB. We will also walk through example use cases using sample Lua scripts. Toward the end, we will delve a bit deeper into why we chose Lua as the scripting language and how we implemented this feature.

Row-Level Transformation in Postgres CDC: Use Cases

There are multiple use cases that require row-level transformations during Postgres CDC. A few of the common scenarios include:

1. Masking PII Data: Replace sensitive PII with tokens or pseudonyms before data enters Kafka, obfuscating it from other micro-services in transactional outbox scenarios, thus enhancing privacy and compliance.

2. Changing Data Format: Transform data into required formats like Protobuf, JSON, MsgPack, Avro and so on for seamless integration and optimized handling across systems.

3. Generated Columns: Calculate new column values based on transformations of existing data, such as aggregations or derived metrics, and stream these new columns to enhance real-time data analysis and reporting.

4. Unnesting JSONs: Extract elements from JSON objects and flatten them into separate fields within Kafka messages, improving the accessibility and usability of data across different consumer applications.

5. Topic Routing: Distribute Change Data Capture (CDC) events to specific Kafka topics based on rules or conditions, facilitating targeted data streaming and processing.

6. Data Encryption: Apply encryption to sensitive data before it is written to Kafka, enhancing security and preventing unauthorized access as data moves between systems.

Now, let's see how some of the above use cases can be accomplished using PeerDB, through examples and sample Lua scripts.

Sample Schema

I will be using the users table shown below to demonstrate the above use cases in PeerDB. This table includes various fields relevant to our testing scenarios."

CREATE TABLE users (
    id SERIAL PRIMARY KEY,
    first_name VARCHAR(255),
    last_name VARCHAR(255),
    ssn CHAR(11),
    payload JSONB,
    salary_in_usd NUMERIC(10, 2)
);

Masking the SSN Column

You can create a simple Lua script to mask the SSN column in the users table and add it as part of the MIRROR. See below:

local json = require 'json'

local function maskSSN(ssn)
    if not ssn then
        return nil
    end
    -- Replace all but the last four digits of the SSN with "XXX-XX-"
    return string.gsub(ssn, "^(.-)(%d%d%d%d)$", "XXX-XX-%2")
end

local function RowToMap(row)
    local map = peerdb.RowTable(row)
    for col, val in pairs(map) do
        local kind = peerdb.RowColumnKind(row, col)
        if col == 'ssn' then
            -- Apply the maskSSN function to the SSN column
            map[col] = maskSSN(val)
        elseif kind == 'bytes' or kind == 'bit' then
            map[col] = json.bin(val)
        end
    end
    return map
end

local RKINDMAP = {
    insert = string.byte('i', 1),
    update = string.byte('u', 1),
    delete = string.byte('d', 1),
}

function onRecord(r)
    local kind = RKINDMAP[r.kind]
    if not kind then
        return
    end
    local record = {
        action = kind,
        lsn = r.checkpoint,
        time = r.commit_time,
        source = r.source,
    }
    if r.old then
        record.old = RowToMap(r.old)
    end
    if r.new then
        record.new = RowToMap(r.new)
    end
    return json.encode(record)
end

PeerDB offers a straightforward script editor for creating the Lua script to define the transformation. After that, you can add this transformation through the UI while creating the MIRROR. See below demo for reference:

You can try this yourself in just 10 minutes by following this Quickstart guide.

Changing Data Format to MsgPack

For the users table mentioned above, to change the data format to MsgPack, you can use this example Lua script. We've seen a few of our customers use MsgPack because MsgPack is more efficient than JSON because it uses a compact binary format, which reduces data size and speeds up both data transmission and parsing.

See the 2-minute demo below, which shows how this is done with PeerDB.

Generated Additional Columns

For the users table, let's say I want to add a new column, salary_in_cad, as part of the replication, which converts the salary from dollars to Canadian dollars. You can create a script as shown in this example (see below) and add it as part of the MIRROR. Below is a snippet of the Lua script that does

local json = require "json"

local function RowToMap(row)
    if not row then
        return
    end
    local map = peerdb.RowTable(row)
    map.salary_in_cad = map.salary_in_usd * 1.4
    return map
end

local OPMAP = {
    insert = "c",
    update = "u",
    delete = "d",
}

function onRecord(record)
    local op = OPMAP[record.kind]
    if not op then
        return
    end
    return json.encode {
        op = op,
        before = RowToMap(record.old),
        after = RowToMap(record.new),
        commitms = record.commit_time.unix_milli,
        table = record.source,
        lsn = record.checkpoint,
    }
end

Unnesting the Payload JSONB column

For the users table, let's say you want to unnest the payload JSONB column to separate fields in Kafka. You can create the script as shown in this example and add it as a part of the MIRROR.

Distribute Load of the Users Table Across Topics

For the users table, let's say you want to distribute the load across two Kafka topics, with odd IDs going to one topic and even IDs going to the other. You can create a script as shown in this example (see below) and add it as part of the MIRROR.

local bit32 = require "bit32"
local json = require "json"

local OPMAP = {
    insert = "c",
    update = "u",
    delete = "d",
}

function onRecord(record)
    local op = OPMAP[record.kind]
    if not op then
        return
    end
    local topic
    if bit32.btest(record.row.id, 1) then
        topic = "odd"
    else
        topic = "even"
    end
    return {
        topic = topic,
        value = json.encode {
            op = op,
            before = record.old,
            after = record.new,
            commitms = record.commit_time.unix_milli,
            table = record.source,
            lsn = record.checkpoint,
        }
    }
end

Why we chose Lua?

We had multiple options, including WASM, but we chose Lua as it provides a fine balance with respect to engineering velocity, integration with our Go-based platform, and end-user usability. Here are the key reasons for our decision:

• In-process scripting: Lua supports in-process scripting, avoiding the serialization and deserialization steps required by external plugin systems.

• Simplicity and flexibility: Lua's straightforward design as a glue language makes it easy to embed in various projects, with multiple robust implementations available.

• Compatibility with Go: Lua works well with Go’s garbage collector, simplifying memory management compared to using alternatives like WASM, which would necessitate complex integration with Go's memory management.

• Ease of use for end-users: Lua is an embedded language that allows for on-the-fly scripting without the need for compilation or additional setup steps, unlike systems like Debezium that use Java.

• Long-standing presence and resources: Although Lua is a verbose language, its long-standing presence has resulted in a wealth of resources. This also enables LLM-based coding assistants to be quite accurate, helping users to easily script out row-level transformations.

Conclusion

We hope you enjoyed reading the blog. We think custom transformations offer a lot of added flexibility and enables many use cases. If you use Kafka, Pub-Sub, Redpanda or any other queues and wish to replicate data from Postgres to these using PeerDB, please check out the links below or reach out to us directly!

1. PeerDB Streams Quickstart

2. Docs on Postgres to Kafka Replication

3. Try PeerDB Cloud for free

4. Visit PeerDB's GitHub repository to Get Started

We've been operating PeerDB Cloud in Private Beta for the past three months. As the system has matured and we've had the privilege of serving a growing number of customers, we're thrilled to now launch it into Public Beta.

What is PeerDB Cloud?

PeerDB Cloud is the fully managed offering of PeerDB. It is the easiest way to get started with PeerDB — in just a couple of clicks, you can have a production-ready PeerDB instance provisioned and a worry-free approach to Postgres replication. PeerDB Cloud comes bundled with PeerDB's core features like:

PeerDB Cloud comes with all PeerDB features

1. Postgres Change Data Capture with latencies of less than 1 minute for Data Warehouses and single-digit milliseconds for queues.

2. High-quality target connectors, for Data Warehouses such as Snowflake, BigQuery, Postgres, ClickHouse, etc., and Queues such as Kafka, Redpanda, Google PubSub, Azure Event Hubs, and so on.

3. Blazing Fast Parallel Initial Loads and Re-syncs: PeerDB is 10x faster compared to other tools. You can move Terabytes of data in few hours vs days.

4. Streaming Query Replication for production-ready replication based on watermark columns.

5. Web UI and Unique SQL Interface for ETL: Easily manage your data with our intuitive Web UI and SQL interface.

6. And many more

PeerDB Cloud is fully managed - 0 CAPEX and OPEX costs

In addition, PeerDB Cloud provides a fully-managed production-ready experience supporting enterprise-grade features such as:

1. High Availability (HA): Every PeerDB Cloud instance comes with HA. Under the hood, we have replica instances across Availability Zones and mechanisms for auto failover as needed.

2. Horizontal Autoscaling: As your replication load increases, we have mechanisms to auto-scale compute resources as needed.

3. In-Place/Transparent Upgrades: Enjoy hassle-free rolling upgrades with no downtime. This helps keep you up-to-date with all the latest features and ensures that you stay current with no extra effort.

4. Advanced Logs and Metrics: Monitor your system effectively with detailed logs and metrics. OpenTelemetry endpoint support is coming soon.

5. Privacy and Security: Privacy and security are our top priorities at PeerDB Cloud, surpassing even performance and functionality. We offer SSH tunneling for secure connections and ensure encryption at rest and in transit. Our platform is GDPR compliant and is currently undergoing a SOC2 audit (2 months in), with compliance expected by mid-June. Here is our trust report.

6. Dedicated Slack Channel & Support SLAs: Every PeerDB Cloud customer gets a dedicated slack channel for expert guidance during implementation, migration and post-prod support. PeerDB Cloud also comes with below Support SLAs:

   1. 

Save up to 5x costs and Predictable Pricing

Being laser-focused on Postgres replication, we have implemented multiple Postgres-native and infrastructural features to optimize costs. Here is our white paper that provides a detailed summary of all these optimizations. With this, we are able to save up to 5x in costs for our customers compared to other tools. The graph below shows how PeerDB compares to other data-movement tools (reference)

In addition to this, PeerDB Cloud provides a predictable pricing model. Instead of charging based on the number of rows or the amount of data moved, you just pay for the vCPUs provisioned. This ensures that as your workload scales, your costs don't skyrocket.

Current Metrics

These behind-the-scenes metrics for PeerDB Cloud showcase our progress and reinforce our confidence in launching it to Public Beta.

Volume of data moved in PeerDB cloud: PeerDB Cloud already serves 10+ production customers and is replicating 20TB of data from Postgres every week, amounting to approximately 100TB per month. The graph below shows the day-over-day growth over the past week. Note that the graph below captures Avro compressed data; if uncompressed, the volume would be significantly higher, ranging anywhere from 200 to 400TB of uncompressed data moved per month.

Sign-ups for PeerDB Cloud: Signups have grown at a consistent pace over the past few months, increasing by 100% month over month.

Our Customers

In just three months, we have over 10 production customers using PeerDB Cloud for production-grade Postgres replication. Our customers are spread across various verticals including Fintech, IoT, Retail, Sales Marketing Automation, and more. Below is a snapshot of a few of our publicly referenceable customers.

Here is a link to our customer stories. They demonstrate how our customers were able to achieve 10 times faster and up to 5 times cheaper Postgres replication experiences with PeerDB. Here are a few testimonials:

"Our decision to choose PeerDB was reaffirmed by their comprehensive online resources, which instilled confidence in their expertise. Not only did they help us cut costs effectively, but their unparalleled customer service provided immediate and insightful assistance, making us feel supported and empowered in managing our PostgreSQL database."**- Sang Mercado, Head of Engineering, Harmonic AI

“We’re using this connector already for our Postgres to ClickHouse ETL and it’s insanely fast and accurate! Can’t believe how well this works. The PeerDB team has been super helpful in getting us set up, helping us debug, and advising us on everything related to ClickHouse and Postgres. Great work guys!!”**- Neel Mehta, CTO of Fiber AI

Growing with Postgres 📈

Postgres has solidified itself as one of the most popular developer databases ever created. While MySQL lost some appeal after being acquired by Oracle, Postgres continues to earn developer trust. In 2023, it topped the Stack Overflow Developer Survey, and was named DBMS of the Year by DB-Engines.

At PeerDB, we believe that Postgres is going to become the database of the world. We are dedicated to contributing to this vision by making it effortless for any Postgres user to implement use cases involving data movement and ETL for Postgres. Providing a fully managed Postgres replication experience through PeerDB Cloud is a step in that direction.

Future Roadmap

We're committed to continuous improvement and have exciting new features in development:

• OpenTelemetry endpoint to integrate with your own monitoring tools such as DataDog, PagerDuty, OpsGenie, and more.

• WebHooks and REST API integration to create and manage PEERs and MIRRORs.

• Expanding from AWS to other clouds incl. GCP and Azure

• Support for private link to securely connect to your VPC. This feature is currently in private preview.

• PeerDB Cost Analysis - Fully visibility into your Data Warehouse costs and how to optimize them

Join the PeerDB Cloud Community!

💡 Ready to see what PeerDB Cloud can do for your data?

• Sign-Up for PeerDB Cloud's public beta today.

• Book a Chat with PeerDB founders to discuss how PeerDB can transform your data strategy.

• Star our GitHub repo

• Join our Slack Channel to connect with the PeerDB community.

PeerDB Cloud is poised to reshape how you manage and replicate your data. Start your journey with us now!

Now, we want to expand and bring a similar experience for Queues. With that spirit, we are excited to announce PeerDB Streams. PeerDB Streams provides a simple and native way to replicate changes as they happen in Postgres to Queues / message brokers such as Kafka, Redpanda, Google PubSub, Azure Event Hubs, and so on. Under the hood, PeerDB Streams uses Postgres logical decoding to enable Postgres Change Data Capture (CDC).

The Problem

We selected Queues as our next target because we've heard from multiple Postgres users that existing CDC tools are complex and have a significant learning curve. Debezium is the most common technology for this use-case. It is proven and has large production usage. However, a common pain point among our users is that Debezium has a significant learning curve and requires institutional knowledge to set up and manage in production. It takes a few months to fully deploy Debezium in production. A few common issues from users include -

1. Interacting through a command line interface or configuration files, understanding the various options / settings, and learning best practices for running Debezium in production requires a significant learning curve. Debezium UI, released to address usability concerns, is still in an incubating state. Additionally, reading Debezium docs/resources to get started can be overwhelming and not the most approachable.

2. Supporting data formats (ex: MsgPack) and transformations is not trivial and incurs an additional learning curve. You need to write a Java project, build JAR packages and set up a runtime path on the kafka connect plugin. It isn’t as simple as plugging in a premade template or writing a few lines of code.

3. Debezium is not as native as Kafka for other types of message brokers and does not offer the same level of configurability. For example, with Event Hubs, it is difficult to define custom partitioning schemes and stream to topics spread across namespaces and subscriptions.

TL;DR We believe that Debezium aims to provide a comprehensive experience for engineers to implement CDC rather than making it dead simple for them. So you can do a lot with Debezium but need to know a lot about it.

PeerDB Streams - Simple, Native Postgres Change Data Capture (CDC)

This is what we want to address with PeerDB. We are building a Simple, yet Comprehensive experience for Postgres Change Data Capture (CDC). The goal is to enable engineers to implement production-grade Postgres CDC with a minimal learning curve, within a few days.

PeerDB’s feature-set isn't at Debezium's level yet, and as PeerDB evolves, we might face similar usability challenges. However, we're putting Simplicity/Usability at the forefront and we believe that we can achieve the above goal. Here is how we are doing it –

Simple Postgres CDC Using PeerDB UI

First and foremost, PeerDB offers a simple UI to set up source and target data sources (such as Postgres and Kafka) by creating PEERs and initiating CDC by creating a MIRROR.

Through the UI, users can monitor the progress of CDC, including throughput (per table) and latency; search through logs; set up alerts to Slack or Email based on replication slot growth; investigate Postgres-specific metrics, including slot size, wait events for replication, and more. The UI also offers advanced features, including tuning MIRRORs, pausing MIRRORs, adding tables to MIRRORs, and more. We have strived to make these features as intuitive as possible for users, for example, by using information toolbars and simple language. Below is a demo showing of PeerDB UI in action. Here a link to the quick start for you to try PeerDB Streams in just a few minutes.

Enhanced CLI Experience: Intuitive SQL Layer for Managing Postgres CDC

Second, for users who prefer a CLI over the UI, we provide a Postgres-compatible SQL layer to initiate and manage CDC. This SQL layer offers the same level of comprehensiveness as the UI and we believe that it is far more intuitive and user-friendly compared to bash scripts and configuration files.

Simple Lua Scripts for Row-Level Transformations

Third, users can perform row-level transformations before streaming CDC changes to Kafka. They can write Lua scripts to execute these transformations. This enables powerful features such as encrypting/masking personally identifiable information (PII), supporting various data formats (JSON, MsgPack, Flatbuffers, Protobuf, etc.), and more. To make it very simple for users, we offer a script editor along with a bunch of useful templates. Additionally, applying a transformation is optional, with the default data format being JSON.

Native Connectors to non-Kafka targets

Fourth, we offer native connectors to non-Kafka targets, including Google Pub/Sub and Azure Event Hubs. Behind the scenes, we utilize the native Go APIs/libraries provided by these services to build our connectors, instead of relying on the less developed Kafka-compatible layer over these queues. We support advanced features specific to these services. For example, with Azure Event Hubs, users can perform CDC to topics distributed across different namespaces and subscriptions.

PeerDB Streams is Postgres Native

Finally, we are laser-focused on Postgres and, as of now, don't support any other databases. This allows us to implement many Postgres-native optimizations. For example, we provide Postgres-native metrics and alerts, including replication slot growth, wait events for logical decoding, number of connections and so on. Features such as parallel snapshotting for 10x faster initial loads and decoding in-flight transactions are in private beta.

Try PeerDB Streams

Checkout this 10-minute quickstart to try PeerDB for Postgres CDC to Kafka.

Separately, you can try PeerDB through one of three offerings: Open Source offering, PeerDB Cloud, our fully managed service, and a self-hosted enterprise offering that includes production-grade Helm charts.

Our vision is to provide the world’s best data-movement experience for Postgres. PeerDB Streams is another step in that direction. We built PeerDB Streams in close design partnership with a few Fintech and IoT customers implementing Postgres CDC for their transactional outbox use cases. The product has been battle-tested at scale and is constantly evolving. We would love to get your feedback on product experience, our thesis and anything else that comes to your mind. It would be super useful for us. Thank you!

The answer wasn't one feature, but an entire week bursting with launches! 🚀

Even with our small team, we knew this was ambitious. But after months of relentless work, we're thrilled to announce...

PeerDB Launch Week begins on Monday, May 6th! ✨

What to expect?

We don't want to spoil the surprise, so for now we are giving you a teaser of what to expect. Follow us on twitter / X to keep up to date: @PeerDBInc to keep up.

A few months ago, we added a ClickHouse connector for Postgres Change Data Capture (CDC). Surprisingly, this connector gained substantial traction and adoption within our community. This applies to both our fully managed service (PeerDB Cloud) and our Open Source offerings. Here is a customer story from one of our customers who uses the ClickHouse connector.

The Problem

However, there was one common piece of feedback from many of our Open Source users. The ClickHouse connector required an S3 bucket as a prerequisite, which added additional overhead for users. Non-AWS users and those without immediate access to S3 could not use the ClickHouse connector. This wasn't a problem in our fully managed offering (PeerDB Cloud), as we abstracted away the S3 bucket creation from our customers.

This blog describes how we solved this problem and made it extremely easy for our users replicating data from Postgres to ClickHouse. We used MinIO, the open source S3 alternative, to stage the intermediary Avro files as part of the Change Data Capture (CDC) from Postgres to ClickHouse.

Why does the ClickHouse connector need S3?

Under the hood, PeerDB uses the Avro format for data in transit while replicating data from Postgres to Data Warehouses. Loading Avro files through Go wasn't trivial as the clickhouse-go driver didn't support Avro ingestion. Additionally, ClickHouse has native integration for loading data from S3 and is very efficient at it, as it attempts to parallelize as much work as possible, processing files in a streaming fashion. Therefore, we chose to use S3 as an intermediary storage for Avro files before importing them into ClickHouse.

This method has proven effective, allowing users to efficiently replicate data from Postgres to ClickHouse with latencies under 30 seconds and high throughput rates.

MinIO helps make the PeerDB's ClickHouse Connector Seamless

By integrating MinIO container services into our Docker Compose files for our Open Source offering, we've enabled an in-house S3-compatible storage solution that launches seamlessly with PeerDB. PeerDB uses environment variables to manage S3 bucket credentials, allowing for easy integration. Users can set these variables to match the MinIO bucket parameters, or they can plug in their own S3 bucket details. These parameters default to the packaged MinIO bucket parameters, as a result, users no longer need to provide a separate bucket for PeerDB’s ClickHouse integration, simplifying the setup process significantly.

A huge shoutout to MinIO for building a solid product that serves as an open source alternative to S3. Integrating MinIO's Docker container within PeerDB's Docker file was a one-week project. MinIO's APIs, being fully compatible with S3, allowed for seamless integration with PeerDB and ClickHouse.

Result: Even simpler Postgres to ClickHouse replication with PeerDB.

Simplifying ClickHouse Peer Creation with Optional S3 Configuration

Integrating the MinIO Docker Container in our Open Source offering eliminates the need for users to specify S3 buckets to use our ClickHouse connector. While creating the ClickHouse Peer, adding S3 information is optional, as shown in the screenshot below.

Set Up a Postgres to ClickHouse Mirror in Under a Minute

Once the Postgres and ClickHouse Peers are created, users can create MIRRORs to replicate data from Postgres to ClickHouse within a minute. See below video:

Use the MinIO Console for complete visibility into internal staging

MinIO also comes with a sleek UI that helps you manage the internal Avro files PeerDB creates as part of the replication process.

We hope you enjoyed reading the blog. If you're a ClickHouse user and wish to replicate data from Postgres to ClickHouse using PeerDB, please check out the links below or reach out to us directly!

1. Docs on Postgres to ClickHouse Replication.

2. Try PeerDB Cloud for free.

3. Visit PeerDB's GitHub repository to Get Started.

However, they are single-threaded at the table level. This significantly slows down the dump and restore of databases with a star schema common in real-world applications such as Time series and IoT. For databases over 1 TB, pg_dump and pg_restore can take days, increasing downtime during migrations and RTOs in disaster recovery scenarios.

In this blog, we'll discuss an idea called "Parallel Snapshotting". This idea could be integrated into Postgres upstream in the future to make pg_dump and pg_restore parallelizable at a single table level. Parallel Snapshotting has already been implemented in PeerDB, an open-source Postgres replication tool. We will also cover a few interesting benchmarks of migrating a large table of 1.5TB from one Postgres Database to another with and without Parallel Snapshotting.

A quick primer on pg_dump and pg_restore

pg_dump is the most reliable way to back up a PostgreSQL database. It enables the backup of a database at a consistent snapshot; that is, the backup guarantees a state that existed previously. The backup generated by pg_dump is a logical representation of the data in PostgreSQL, not a copy of the PostgreSQL data directory. It captures objects as they appear in PostgreSQL.

pg_restore is the most reliable way to restore a backup generated by pg_dump from one PostgreSQL database to another.

Both pg_dump and pg_restore are Postgres-native; that is, they come packaged with community Postgres and can be used as command-line utilities, similar to psql.

pg_dump and pg_restore offer fine grain control

They provide fine-grained control to manage the backup and restore processes. Below are a few flags that are commonly used:

1. You have the -f flag, which lets you decide on data formats such as plain text or compressed gzip. Compressed dumps are quite helpful when you have limited network bandwidth or want to save on network costs.

2. To speed up the backup and restore process, you can use the -j flag to dump and restore multiple tables in parallel.

3. You can pick and choose specific database objects you want to backup and restore, including tables and schemas.

4. You can also choose to dump only the schema or only the data using the schema-only and data-only flags.

5. There are many more flags that they provide that can be found in community docs.

pg_dump and pg_restore can be very slow for large tables

pg_dump and pg_restore are single threaded at a table level

There is a painful issue that users often run into with pg_dump and pg_restore. pg_dump and pg_restore can be very slow for large tables. This is because they are single threaded at table level. They can dump and restore multiple tables in parallel but for a single table they are single threaded.

This means that in use cases where you have a single fact table and multiple dimension tables, pg_dump and pg_restore can get bottlenecked on the large fact table. This is very common in the star schema data-model which is used by multiple real-world use-cases such as IoT, Timeseries, Data Warehousing and so on.

Migrating a 1.5TB table can take 1.5 days

The impact of the problem described above can be significant. Using pg_dump and pg_restore to migrate a 1.5 TB pgbench_accounts table from one Postgres database to another took 1.5 days. The benchmark was conducted under optimal conditions, i.e., using the correct flags and region collocating the source, target, and the VM on which pg_dump and pg_restore were running, among other factors. This 1.5-day downtime is substantial when migrating or recovering mission-critical databases.

Parallel Snapshotting to make pg_dump & pg_restore multi-threaded per table

Now, let's explore a concept called Parallel Snapshotting, which could make pg_dump and pg_restore multi-threaded at the single table level. Note that Parallel Snapshotting is not currently implemented in the upstream versions of pg_dump and pg_restore. It represents an idea/design that could enhance pg_dump and pg_restore in the future.

Below video captures migrating 5GB of data from Postgres to Postgres within a min using the Parallel Snapshotting feature in PeerDB.

CTID forms the basis of Parallel Snapshotting

CTID forms the basis of Parallel Snapshotting. Every row in a Postgres table has an internal column called CTID, also known as the tuple identifier. CTID is unique for each row of the table. It represents the exact location of the row on disk—it is the combination of the page/block number and the page offset. You can also query the CTID column for a table through a simple SELECT as you are seeing in the below image.

Parallel Snapshotting - Logically Partition the Table by CTID and COPY Multiple Partitions Simultaneously

Let's dive into the design of Parallel Snapshotting:

1. First, create a Postgres Snapshot using the function pg_export_snapshot(). This ensures that the dump and restore operate on a consistent snapshot of the database.

2. Second, using that snapshot, logically partition the large table based on CTIDs, i.e., create CTID ranges that encapsulate the table.

3. Once that is done, copy multiple such logical partitions in parallel from the source to the target.

   1. Essentially, you are running SELECT statements with these CTID ranges to read data from the source and write it to the target.

   2. The SELECT statements with CTID ranges are very efficient because they use tid range scans, which are similar to index lookups on the CTID column.

   3. Also, note that you are reading data in the order of how it is stored on the disk.

4. We are using COPY WITH BINARY to STDOUT and from STDIN, which makes the dump and restore simultaneous.

5. We are also using cursors to ensure that the dump doesn’t exhaust memory.

Migrating a 1.5TB table 5 times faster with Parallel Snapshotting

At PeerDB, we are building a Postgres replication tool to provide a fast and cost-effective way to move data from Postgres to data warehouses such as Snowflake, BigQuery, ClickHouse, PostgreSQL, and queues like Kafka, Redpanda, Google PubSub, Azure Event Hubs, etc.

To enable faster migrations from one Postgres database to another, we have implemented Parallel Snapshotting within our product. Through this feature, our customers are able to move terabytes of data in a few hours versus days.

We did the same above benchmark to move a 1.5 TB pgbench_accounts table from one Postgres database to another, and it took just 7 hours with PeerDB. This was 5x faster than using pg_dump and pg_restore. This speedup was possible through the Parallel Snapshotting feature. The performance can be further improved by increasing the number of parallel threads for the migration and by using more beefier Postgres source and target databases.

Conclusion and References

The intent of this blog is to share the design principles we followed to enable faster database migrations and discuss how they can be extended to enhance pg_dump and pg_restore in the future. Hope you enjoyed reading the blog. Sharing a few relevant links for reference:

1. Speeding up Postgres restores

2. Faster Data Migrations in Postgres

3. Faster Postgres Migrations using PeerDB

4. Podcast on Logical replication common issues

5. Try PeerDB Open Source for fast Postgres migration and replication

6. Try PeerDB Cloud, the fully managed offering of PeerDB

SAN FRANCISCO — April 11, 2024 —PeerDB, the leading data movement platform for PostgreSQL, today announced it received $3.6 million in seed round funding. Investors in the round include lead investor 8VC, Y Combinator, Wayfinder Ventures, Webb Investment Network, Flex Capital, Rogue Capital, Pioneer Fund, Orange Collective and several angel investors.

PeerDB will use the funds to continue building its engineering team, propelling its go-to-market and client acquisition initiatives and supporting its growth. PeerDB revenue is doubling every two months.

"Postgres is becoming the database of the world and the de facto primary database for both enterprises and SMBs. Existing data movement and ETL tools are not built for Postgres – they often fail at scale due to painfully slow syncs, lack of reliability and lack of native features. The time has come for someone to give enough care to the world's most adopted open source database. Thanks to all our investors and customers for believing in us and sharing our vision," said PeerDB CEO and co-founder Sai Krishna Srirampur.

"At PeerDB, we're tackling inefficiencies surrounding Postgres data movement. With fundamental optimizations like parallel snapshotting and handling the nuances around replication slots that are absent in the existing ETL tools, we’re focused on building a system specialized for Postgres at terabyte scale – our approach diverges from traditional methods that falter at scale and resort to resyncs. Our team of Postgres experts not only provides data movement services but also becomes an essential part of client teams, offering advice on database tuning and query optimization. As we brace for the influx of data driven by the era of LLMs, we're committed to ensuring Postgres data movement remains efficient and scalable," said PeerDB CTO and co-founder Kaushik Iska.

The Problem

In the current data and AI landscape, existing data movement and ETL (extract, transform, and load) tools prioritize the breadth over the quality of connectors and are not optimized for Postgres. Users often face issues such as painfully slow syncs — syncing hundreds of gigabytes of data can take days; unreliability, characterized by frequent crashes and loss of data precision; and feature limitations, including a lack of configurability and support for native data types.

PeerDB distinguishes itself by prioritizing the quality over the breadth of connectors and tailoring its design specifically for Postgres. Through this, PeerDB offers 10 times faster data movement and at one-fifth the cost.

Why Now?

"8VC's investment in PeerDB is driven by our conviction in the exponential growth trajectory of Postgres. We see an exceptional team in Sai and Kaushik, whose deep expertise in Postgres positions them uniquely in the marketplace. We believe that a solid narrative around data movement and ETL will play a pivotal role in the success and widespread adoption of Postgres," said Bhaskar Ghosh, partner at 8VC.

The time is ripe to develop a first-class data movement tool for Postgres, as it is becoming the world's most popular database. Currently ranked fourth in the DB-Engines Ranking of database management systems (DBMS) based on popularity, Postgres is the only DBMS in the top four experiencing growth. It also received DB-Engines' DBMS of the Year 2023 award for gaining more popularity than any other of the 417 monitored systems in 2023. Enterprise adoption of Postgres is on the rise, with 50% of enterprise companies already using it. This trend is set to significantly increase the volume of data stored and moved through Postgres, requiring an ETL tool that can handle this scale.

The introduction and funding of PeerDB is also timely due to the widespread focus on artificial intelligence (AI) and hyperscale data analytics, which often require movement of massive datasets from primary database platforms such as Postgres to data warehouses for AI-based analytics to help provide insights and inform business decisions.

PeerDB's Vision and Use cases

PeerDB's vision is to become the de facto standard for data movement and ETL for companies that run their businesses on Postgres, encompassing use cases such as:

• Fast and cost-effective replication to data warehouses: Replicate data from Postgres to analytical stores (OLAP) such as Snowflake, BigQuery and ClickHouse for AI-based analytics informing business decisions in use cases like fraud or anomaly detection. PeerDB has already made its mark here with a rapidly growing customer base that strongly challenges incumbents like Fivetran.

• Real-time streaming and change data capture: Low-latency replication from Postgres to queues, such as Kafka, enabling use cases like real-time alerting and micro services-based architectures. PeerDB already supports Kafka, Azure Event Hubs, and Google PubSub as targets, serving as an enterprise-grade alternative to Debezium.

• Database migrations: Migrating data from legacy databases like Oracle and SQL to Postgres to support modernization and digital transformation initiatives.

• Enterprise-grade Postgres high availability (HA) and backups: As enterprises modernize their database stack by migrating from Oracle and SQL Server to Postgres, managing HA and backups across regions and hybrid on-premises environments becomes critical. The infrastructure that PeerDB is developing can be extended to support such mission-critical use cases in the future.

• Vector ETL: Extracting unstructured data at scale, transforming it into vector embeddings with LLMs, and loading these into Postgres using pgvector. This enables semantic searches for advanced AI applications.

PeerDB is built for Postgres

Below are a few product differentiators of PeerDB:

• Faster Postgres data movement: PeerDB implements parallel snapshotting, the fastest way for moving Postgres data. This can dramatically reduce the time required to move massive datasets, often from days to hours, while ensuring consistency.

• Native Postgres data type support and replication: PeerDB specializes in natively replicating advanced data types like JSONB and geospatial, crucial for IoT apps and geospatial applications. With data available in native formats, users save the time and effort required for data transformation, since the data is already in the format necessary for their AI-based analytics and other applications.

• Cost optimizations: PeerDB can reduce data movement costs by up to five times compared to incumbent ETL tools. PeerDB published a white paper detailing the data modeling and infrastructure optimizations employed to save costs for customers.

Customers

PeerDB customers include:

• Harmonic AI replaced an incumbent ETL solution with PeerDB, saving $80,000 yearly and reducing their yearly data-movement costs by five times.

• Expedock uses PeerDB to deliver real-time AI-driven supply chain automation, replicating 700 million rows monthly from Postgres to Snowflake with under one minute latency and five times cost savings compared to their previous ETL tool.

• Fiber AI uses PeerDB to replicate terabytes of data from Postgres to ClickHouse in real time, powering their real-time search use case.

• Flatiron Health used PeerDB to migrate 35,000 tables with terabytes of data from Postgres to Snowflake within a week.

"We’re using PeerDB already for our Postgres to ClickHouse ETL and it’s insanely fast and accurate! We can’t believe how well it works. The PeerDB team has been super helpful in getting us set up, helping us debug, and advising us on everything related to ClickHouse and Postgres. Great work, guys!"**said Neel Mehta, CTO, Fiber AI.**

How to try PeerDB?

You can try PeerDB through one of three offerings: Open source, a fully managed cloud service and a self-hosted enterprise offering.

Founders

The co-founders of PeerDB, CEO Sai Krishna Srirampur and CTO Kaushik Iska, have been friends since high school and were roommates in college at the International Institute of Information Technology Hyderabad, where they both studied computer science. Sai Krishna Srirampur was an early engineer at Citus Data, which was acquired by Microsoft. There, he led solutions engineering for all Postgres services on Microsoft Azure. Kaushik Iska built operating systems and led data teams at Google, SafeGraph, and Palantir Technologies. He also represented India in the International Collegiate Programming Contest (ACM ICPC) World Finals. They have been building Postgres products for a decade now and have closely worked with Postgres customers running into issues with existing ETL tools. To fill this gap, they founded PeerDB. Below is the image of the PeerDB Founding Team.

Our Ongoing Dedication to Data Privacy and Security

At PeerDB, we understand the critical importance of data privacy and security. Our team has worked diligently to ensure that our practices align with the stringent requirements of the GDPR. For a detailed overview of our security measures, we invite you to explore our updated Trust Center.

Benefits of GDPR Compliance for Our Clients

• Enhanced Data Security: With GDPR compliance in place, we guarantee the highest level of security and confidentiality for all data entrusted to us. From data encryption to strict access controls, we have implemented comprehensive measures to safeguard your information.

• Transparency and Control: Our Trust Center serves as a central hub for information on our security infrastructure, organizational practices, and third-party engagements. We believe in transparency and empower our clients with the control they need over their data.

• Continual Improvement: Our commitment to data security doesn't end with GDPR compliance. We are dedicated to continuously evaluating and enhancing our security measures to not only meet current regulations but also stay ahead of emerging threats.

Looking Ahead: Our Next Steps in Security

While we celebrate our GDPR compliance, we are also looking towards the future. As part of our ongoing commitment to security excellence, we are currently in the process of preparing for the SOC 2 Type 2 certification and are undergoing the audit for the same. This additional certification will further validate our dedication to maintaining high security standards for our services.

Empowering Your Business with PeerDB Cloud

PeerDB Cloud provides a secure and scalable platform for businesses to fulfil all their Postgres Data Movement needs. With GDPR compliance at its core, PeerDB Cloud offers peace of mind, knowing that your data is protected in a robust and reliable environment.

Why GDPR Compliance Matters

Achieving GDPR compliance goes beyond meeting legal requirements—it's about building trust with our clients and exceeding industry standards. It demonstrates our commitment to ensuring the privacy and security of the data we handle.

Your Trust, Our Priority

At PeerDB, we are dedicated to being a reliable partner in your digital journey. We are here to assist you in leveraging our services with confidence, knowing that your data is protected. For more information on our GDPR compliance efforts, PeerDB Cloud, or any other security-related inquiries, please visit our Trust Center.

Thank you for being part of this journey with us. We look forward to continuing to provide secure and trusted solutions for all your data movement needs.

Logical Replication is one of the many ways a Postgres database can replicate data to other Postgres database (a.k.a standby). Logical replication directly reads from the write-ahead log (WAL), recording every database change, avoiding the need to intercept queries or periodically read the table. These changes are filtered, serialized and then sent to the standby servers where they can be applied. While logical replication is intended to be used by Postgres databases to send and receive changes, it also allows ETL tools like PeerDB to get a reliable stream of changes that can be processed as needed.

Logical replication started by only allowing streaming of committed transactions. It then evolved to support in-flight transactions followed by two-phase commits and then parallel apply of in-flight transactions. This blog will dive into this evolution, its impact on performance, and present some useful benchmarks. This blog is useful for anyone who uses Postgres Logical Replication in practice!

Components of logical replication

For a quick rundown, a full logical replication setup involves several crucial components. Please skip this section if you are already familiar with the concepts of logical replication.

1. Replication Slot: A replication slot on the primary server is what reads changes from the WAL and passes it to the output plugin to be serialized and sent to the standby server (or ETL tool) to be applied. Periodically, the standby server sends a message to the primary to confirm that it has read the WAL to a certain point, at which point the slot can advance.

2. Publication: A publication is essentially a filter on the WAL changes. Publications are very powerful and can filter out schemas, tables and even particular columns of tables. You can also choose to publish inserts and not updates and also apply custom logic to filter out certain rows. When a standby starts reading from a replication slot, a set of publications are passed as input.

3. Subscriptions: A subscription is basically the Postgres syntax for creating a logical replication connection to a primary server for replicating changes from a slot and a set of publications. The standby then reads data from the primary and replicates it as long as the subscription is active. While this is Postgres specific, other tools end up behaving like subscribed standbys and get the same output from the primary server.

4. Output plugins: The replication slot passes raw WAL change data to an output plugin which serializes it to a stream of messages. This helps with the interoperability of logical replication as the message format is independent of the underlying database version or configuration. The de-facto output plugin is a Postgres project called pgoutput but other plugins like wal2json and decoderbufs enjoy support among the community.

Wait, logical replication has versions?

When starting logical replication (START_REPLICATION), there is a parameter called proto_version that allows users to opt in to newer semantics of the logical replication protocol. Starting with Postgres 14 in September 2021, three new proto_versions of logical replication have been added in consecutive releases. Looking at the docs for proto_version right now, we see this:

proto_version
    Protocol version. Currently versions 1, 2, 3, and 4 are supported.

    Version 2 is supported only for server version 14 and above, 
    and it allows streaming of large in-progress transactions.

    Version 3 is supported only for server version 15 and above, 
    and it allows streaming of two-phase commits.

    Version 4 is supported only for server version 16 and above, and it 
    allows streams of large in-progress transactions to be applied 
    in parallel.

While these all sound like good things, it's not clear for the average reader what they mean or what problems are being tackled. And for the informed reader who knows what these changes mean, it'd still be nice to understand how they are implemented and their impact on real-world workloads.

v1 - the status quo

To analyze the messages and semantics of the various protocol versions, we've written a small Go application called polorex. If you want to check out the code or try things out for yourself, check out the code in this repo.

To simulate a workload, we are running 2 transactions concurrently, inserting rows into the same table. The transactions insert rows in 100 batches of 250,000, totalling 50 million rows. The workload is simulated by a subcommand of the polorex application. The transactions are read and analyzed by another subcommand called txnreader which connects to the database and continuously reads the replication slot.

./polorex txnreader -port 7132
[in a different terminal]
./polorex txngen -port 7132 -iterations 100 -batchsize 250000 -parallelism 2

The transactions start at the green line and end at the red line. We can see how the transactions are being read only after they commit. It takes 3-4 minutes to decode both our transactions. Since we just committed 2 large transactions, the pgoutput plugin has to read a lot of WAL at once and then serialize it into 50 million INSERT messages to be sent over. While the graph shows that we are reading almost 250K inserts per second, but one can see how this could quickly go out of hand for larger transactions with wider schemas. We could quickly fall behind the primary server purely due to this decoding overhead.

Another issue which follows from this but is less obvious is with regards to the size of the replication slot. This is basically the amount of WAL being retained for the slot to decode changes without losing any data. Looking at the graph, it quickly rises as the transactions progress, but also stays high until both transactions are read at which point it falls dramatically. This can be an issue in workloads with high throughput and large transactions - the WAL being retained can reach hundreds of gigabytes within a matter of hours, thereby consuming the entire disk space and crashing the Postgres server.

With this insight in mind, we can see how version 2's promise of allowing streaming of large in-progress transactions sound enticing. But there is also a simplicity in version 1 of only sending changes over when they are committed. We read a BeginMessage and everything from there onward is fair game to be replicated immediately. In contrast, an "in-progress" transaction could be rolled back at any point, and therefore all the changes read so far need to be staged somehow before being replicated.

v2 - rows down the stream

To begin with, we restart txnreader with a flag to ask it to use protocol version 2 while connecting to the slot. We then rerun the same txngen workload.

./polorex txnreader -port 7132 -protocol 2
[in a different terminal]
./polorex txngen -port 7132 -iterations 100 -batchsize 250000 -parallelism 2

We are seeing a completely different story in terms of how the transactions are being processed here. It's clear that we're getting rows way before the transaction even commits. We're actually seeing streaming of in-progress transactions! Rows for a particular transaction come to us between a StreamStartMessage and StreamStopMessage, and we get several of these streams while rows are still being sent over. We are getting streams for both of our transactions before any of them commit, but we are still only reading 1 transaction at a time.

A transaction being streamed now commits using a StreamCommitMessage, but unlike the Commit message from earlier, we need to wait for this since the fate of the transaction is not known yet. Alternatively, we could receive a StreamAbortMessage which implies transaction rollback and so all our changes for said transaction should not be applied.

The improvements from streaming are nothing short of dramatic, we can see how transactions are fully read seconds after the rows finish inserting, approximately 4 minutes earlier compared to version 1. As a result, the slot size also decreases much more quickly.

Results - v2 enables faster decoding and shorter peak slot size duration

To reiterate, there is no magical improvement in transaction reading performance or peak slot size. The transactions themselves take about the same time to process and generate the same amount of WAL, but since the replication happens in parallel with the transaction, we see better performance.

In version 2, transactions are fully decoded, and the slot size decreases immediately after the transactions are completed, compared to version 1, which requires an additional 4 minutes. This can have drastic impact on workloads with high throughput and sizable transactions - version 2 can be very helpful in enhancing logical decoding performance and ensuring the slot size is kept in check!

v3 and v4 - 2PC and parallel apply

Version 3 introduces new message types to manage two-phase commit transactions. While significant in certain scenarios, the concept of two-phase commit remains relatively niche from an ELT standpoint.

Version 4 is less clear in its description, and even the documentation doesn't venture much farther than this. As it turns out, it doesn't refer to applying multiple transactions in parallel, but spreading out the load of applying a single large transaction over multiple processes in the standby. For this, new fields have been added to some existing messages. This is again a great feature in some workloads, but not very useful from the standpoint of something else pretending to be a Postgres standby.

Conclusion

Postgres logical replication is a powerful feature central to the distributed/HA Postgres ecosystem. By using version 2 of the logical replication protocol to stream in-flight transactions, we can efficiently manage WAL spikes during sizable transactions, enhancing logical decoding performance and mitigating disk full issues caused by replication slot growth. Additionally, this approach reduces the lag between the Postgres source and its readers.

At PeerDB, we're developing a feature that utilizes version 2 of the logical replication protocol to consume changes from a Postgres database before they are committed. We believe this feature will significantly benefit Postgres users grappling with issues related to replication slot growth. Overall, version 2 of the logical replication protocol presents a promising solution for optimizing Postgres replication processes and improving overall reliability and performance.

In this blog, we delve into existing approaches to replicate Postgres to Event Hubs and their challenges, as well as how PeerDB addresses these challenges to provide an Enterprise-grade experience!

Status Quo

Debezium is hard to use and is not built for Azure Event Hubs

A common ways to replicate data from Postgres to Event Hubs is to use Open Source tools such as Debezium. Below are a few challenges that we've heard from customers trying Debezium with Azure Event Hubs.

1. Limited Configurability: Debezium offers limited customization for Azure Event Hubs, including the inability to perform advanced mapping between tables and topics, lack of support for custom partitioning schemes per topic, and inability to flatten nested JSONs, among other limitations.

2. High Setup and Maintenance Costs: One of the common concerns we hear from customers is that setting up and managing Debezium at a production-grade level is challenging. It often requires several months of work by a data engineering team to fully implement.

3. Not Native to Azure Event Hubs: Debezium leverages the Kafka protocol over Event Hubs to support the Event Hubs connector. The Kafka protocol is not as developed as the native APIs provided by Event Hubs.

PeerDB for Change Data Capture (CDC) from Postgres to Azure Event Hubs

In the past 6 months, we have invested heavily to make replication from Postgres to Azure Event Hubs as robust as possible. We have implemented multiple usability, security, and performance-related features required for enterprise customers. Below are a few highlights.

Simple to Use - SQL Layer that makes life very easy!

Along with a simple UI, PeerDB provides a Postgres-compatible SQL layer to manage replication from Postgres to Azure Event Hubs. You just need to run a couple of SQL commands to setup a highly reliable CDC pipeline: CREATE PEER to make PeerDB aware of the Postgres and Event Hubs peers; CREATE MIRROR to kick off the replication job.

The Postgres-compatible SQL layer comes in very handy for managing replication from a fleet of Postgres databases across different tenants or micro services to Azure Event Hubs. You can script out your pipelines using Python or any other language and use any CI tool to manage your data pipelines.

The following demo showcases PeerDB in action, replicating data from Postgres, running a multi-tenant SaaS app, to Azure Event Hubs.

Blazing fast performance with Sub-Second latency

Use cases requiring replication from Postgres to Azure Event Hubs are highly latency-sensitive. For instance, consider an IoT app publishing raw changes to Event Hubs. PeerDB implements multiple optimizations to provide sub-second latency at high throughputs (10K+ TPS). A few of the optimizations include:

1. Streaming instead of batching

2. Always consuming the logical replication slot

3. Parallel apply for Azure Event Hubs

4. Using native APIs (not the Kafka layer) to ingest into Azure Event Hubs

Highly Configurable - do almost anything you want!

PeerDB provides many nuts and bolts to manage the behavior of CDC. You can control data formats/transformations, security/isolation, and performance while replicating data from Postgres to Azure Event Hubs. A few of them include:

1. Topics can be spread Namespaces and Subscriptions: You can replicate data from multiple Postgres tables to Event Hubs spread across namespaces and even subscriptions. This ensures guaranteed isolation across topics, which could be critical in multi-tenant SaaS apps.

2. Define custom partition keys and partition counts across topics: To configure performance across topics, you can define custom partition keys and partition counts per topic.

3. Flatten JSON and JSONB column: PeerDB allows you to deep flatten JSON and JSONB columns in Postgres into separate key<>value pairs on Azure Event Hubs.

Enterprise grade Security and Isolation

We designed the Azure Event Hubs connector specifically for Enterprise customers. Below are a few security features/items that PeerDB provides.

1. Guaranteed isolation across Azure Event Hubs topics: PeerDB provides the ability to replicate data from multiple tables in Postgres to separate topics spread across different namespaces and Azure subscriptions. This ensures guaranteed isolation across topics, which could be critical in multi-tenant SaaS apps, where you are providing raw DB feed to your customers.

2. PeerDB Enterprise Offering: For enterprise customers, PeerDB provides the self-hosted offering, which comes with production-ready Helm charts and Enterprise-grade support. This enables you to provision PeerDB in Azure Kubernetes Services (AKS) within your own VNET.

Production ready Observability

1. PeerDB UI: PeerDB comes with a comprehensive UI to monitor the replication jobs. You can monitor performance (throughput and latency), logs, and Postgres native metrics such as replication slot size. Additionally, you can create alerts for these metrics and send them to various channels such as Email and Slack.

2. Integration with Azure Monitor: PeerDB Enterprise can run on Azure Kubernetes Services (AKS). AKS has out-of-the-box integration with Azure Monitor to manage metrics, logs and alerts.

Conclusion

Hope you enjoyed reading this blog! The Azure Event Hubs connector is being used in production by a few large-scale Postgres Azure customers. If you are interested in trying this out, please reach out to us through the Contact Us form on our website.

We are actively working to extend similar support to other queues including Kafka and Google Pub Sub. If you are interested in previewing PeerDB for these queues, reach out to us through the Contact Us form. We also offer a 30 day free trial for PeerDB Cloud.

We often get asked about the preferred managed service for PostgreSQL. In that spirit, we are writing this blog to compare four popular options incl. AWS RDS Postgres, Azure Flexible Server Postgres, GCP Cloud SQL for Postgres, and Supabase Postgres, across Performance, Costs and Features. We also acknowledge other providers like Tembo, Crunchy Bridge, Neon and TimescaleDB which we'll cover in a future post.

Note that this this comparison aims to serve as a helpful "first" checklist for developers choosing a managed service. There may be something we missed, and we apologize for those oversights. We are happy to adjust our analysis based on feedback.

Setup

To ensure an apples-to-apples comparison, we aimed to match the four options as closely as possible in terms of RAM, vCores, disk space, PostgreSQL version, region, etc. The table below captures the details of the initial setup.

Performance

Benchmark Setup

All the performance tests were conducted using a VM (client) with the same compute capacity and collocated in the same region as the PostgreSQL database. We did 3 main performance tests:

1. pgbench representing a typical Transactional (OLTP) workload

2. COPY command to Batch Insert (Upload) data to Postgres

3. SELECT command to Batch download data from Postgres

pgbench

Across all the 4 managed PostgreSQL providers, pgbench was run for 24 hours with 8 parallel connections and 4 jobs pgbench -c 8 -j 4 -P 30. The graphs below capture a comparison of average throughput i.e. transactions per second (TPS), average latency and average CPU utilization for all the services.

AWS RDS PostgreSQL led the pack with an average of 2.7K TPS and 2.884 ms average latency. Azure Flexible Server PostgreSQL ranked second, closely trailing AWS RDS by just ~12%. It recorded an average of 2.4K TPS and an average latency of 3.260 ms. Supabase and GCP Cloud SQL PostgreSQL followed. Average CPU utilization across all the services was almost the same i.e. around 80%, except for Supabase. This could be because Supabase uses ARM processors compared to others who use x86.

Batch Upload and Download

For batch uploads, we used the COPY command to insert 1GB and 5GB files from the client to PostgreSQL. For batch downloads, we executed a SELECT query that retrieved 1GB and 5GB of data from a table in PostgreSQL to the client. The graphs below illustrate how each service performed in these tests:

In terms of batch upload with COPY command, AWS RDS was again the leader taking around ~105s to ingest 5GB of data. GCP Cloud SQL was second with 113s. Azure Flexible Server and Supabase followed.

In terms of batch download using SELECT, the numbers were close across AWS, GCP, and Azure, with GCP slightly ahead, taking 51 seconds to download 5GB data. It was interesting to note that Supabase took longer than the others, requiring 160 seconds to download 5GB of data.

CPU utilization peaks during the COPY command were almost consistent across AWS and GCP, at around ~45-50%. Supabase was at approximately 57%. However Azure peaked at 85%.

Costs

Below table captures costs across all the 4 managed services for a Postgres Database with 2vCPU, 8GB RAM and 100GB disk. More details regarding the infra can be found in this sheet.

If you notice, Supabase is the most cost-effective compared to other managed services, at $113. This could be because Supabase uses machines with ARM processors, which are more cost-effective compared to x64. GCP Cloud SQL comes in second at $116 per month. AWS RDS and Azure Flexible Server are tied at $141.44 per month.

Database Features

Postgres Managed Services typically support various important features for running production and enterprise-grade Postgres deployments. A few important features include:

Availability and Reliability:

1. High Availability (HA) to minimize downtime during DB failures/crashes.

2. Backups / Point-In-Time-Recovery to handle Disaster Recovery (DR) scenarios

3. Cross region read replicas for enterprise-grade DR

Performance and functionality:

1. Out of the box feature to help performance tuning of queries.

2. Read-replicas to segregate and scale read workloads

3. Out of the box connection pooling

4. Extension to enhance Postgres functionality

Security and Compliance:

1. SOC2 and HIPAA

2. Private Access

The table below compares each of the four managed services based on the above features:

Conclusion

Below is a summary of the results from the analyses conducted across the four managed services.

1. AWS RDS Postgres was the most mature Postgres offering of all the other managed services.

   1. Performance-wise, it surpassed Azure by just 12% and exceeded the others by over 45% in pgbench throughput and latency.

   2. Feature-wise, it supports almost all of them in the Availability and Reliability, Performance, and Security and Compliance categories.

   3. It supports the highest number of extensions, i.e., 92 of them.

2. Azure Flexible Server takes second place in performance. It was very close to AWS, being only about 12% lower in performance. It matches AWS RDS Postgres in terms of features.

3. Managed services across all three clouds offer robust support for features related to Availability & Reliability and Security & Compliance, which are important for enterprise-grade workloads.

4. Supabase and GCP Cloud SQL Postgres are the most cost-effective of all the managed services.

5. Special mention to Supabase for supporting features that make the lives of app developers incredibly easy.

Hope you enjoyed reading this blog. In future blogs we will add a few other managed services to this comparison and aim to go deeper in a few categories such as Performance.

References

Excel sheet capturing all our raw analysis to come up with this blog

NOTE: The blog was updated on April 17, 2023. The primary modification involved changing the Azure Flexible Server VM type from AMD (Standard_D2ads_v5) to Intel (Standard_D2s_v5). This change can be easily configured through radio buttons while provisioning and is set as the default across various regions. Therefore, we deemed it a fair modification in the comparison.

Central to this task is the selection of tools and techniques. We will discuss the use of open-source tools, customized scripts, ways to read data from Postgres, and Snowflake’s data loading capabilities. Key aspects like parallel processing, efficiently reading Postgres’ WAL, data compression and incremental batch loading on Snowflake will be highlighted.

I will list and discuss some of the optimizations that are implemented to minimize compute, network, and warehouse costs. Additionally, I will highlight some of the trade-offs made as part of this process. Given that most of the approaches covered in this blog stem from my explorations at PeerDB aimed at enhancing our product – The task was accomplished primarily through PeerDB.

I want to make it clear that there are some feature gaps in comparison to a mature system, and it might not be practical for all use cases. However, it does handle the most common use cases effectively while significantly reducing costs. I also want to caveat that there might be some ways in which the estimations may be off and I’d be happy to adjust based on feedback.

Setup

• Initial data load: We will consider that there are 300M rows already in the table at the start of the task, and our system should handle the initial load of all the rows.

• Inserts, Updates and Deletes (Change Data Capture): The rest of the 700M rows will be a combination of inserts, updates and deletes. Including support for toast columns.

  • 1024 rows changed per second for ~8 days.

• Recoverability: We will reboot the system every 30 mins to ensure that it's robust and can recover from disasters.

Now let us walk through an engineering design that optimally handles the above workload with the objective of minimizing costs and improving performance, one step at a time.

Initial Load from Postgres to Snowflake

Let’s start with the first operation any data sync job has to do: load the initial set of data from the source to destination. There are a few challenges that come with this:

1. How to efficiently retrieve large amounts of data from Postgres?

2. How to process the data in a way where we have minimal cost foot-print?

3. How to efficiently load this data to Snowflake?

Optimal Data retrieval from Postgres

Reading a table sequentially from Postgres is slow. It would take a long time to read 300M rows from Postgres. To make this process more efficient, we have to parallelize. We've got a clever way to quickly read parts of a table in Postgres using something called the TID Scan, which is a bit of a hidden gem. Basically, it lets us pick out specific chunks of data as stored on disk, identified by their Tuple IDs (CTIDs), which look like (page, tuple). This optimizes IO utilization and is super handy for reading big tables efficiently.

Here's how we do it: we divide the table into partitions based on the pages of the database, and each partition gets its own scan task. Each task handles about 500K rows. So, we partition the table into CTID ranges, with each partition having ~500K rows, and we process each partition parallelly (16 partitions at a time).

SELECT count(*) FROM public.challenge_1br; -- find the count

-- num_partitions = (count // rows_per_partition)

SELECT bucket, MIN(ctid) AS start, MAX(ctid) AS end
FROM (
    SELECT NTILE(1000) OVER (ORDER BY ctid) AS bucket, ctid 
  FROM public.challenge_1br
) subquery
GROUP BY bucket ORDER BY start;

Data in Transit

It is important to process the data in a way where we don’t overload the system. As we are operating under budget constraints, we need to use techniques that use the hardware effectively. We are going to be using the “your dataset fits in RAM'' paradigm of systems design. 300M rows for initial load does sound like a lot, but let's see how we can make it fit in our RAM. We need to process the data to ensure data-types are mapped correctly to the destination. We are going to convert the query results to Avro for faster loading into warehouses, and also for its logical type support.

How big is the data?

Let us take a little detour to explore how big the data is. This is a good chance to look at some real world examples to estimate things. Based on interacting with a lot of production customers, and talking to some experts, it’s safe to say that on an average we see ~15 columns per table. In our table, let’s say each row is ~512 bytes.

# for initial load
num_rows = 300_000_000
bytes_per_row = 512
total_num_bytes = num_rows * bytes_per_row
total_size_gb = total_num_bytes / 1_000_000_000
# total initial load size 153.6 GB

# memory required during initial load

num_rows_per_partition = 500_000
mb_per_partition = num_rows_per_partition * bytes_per_row / 1_000_000 # 256 MB
num_partitions_in_parallel = 16
required_memory = num_partitions_in_parallel * mb_per_partition # 4096 MB

Required Memory

Based on the above napkin math, we can see that with 4GB of RAM we should be able to do the initial load. We will allocate 8GB of RAM to account for other components.

Efficiently loading data into Snowflake

As mentioned earlier we are going to store the query results into Avro on-disk. We are further going to compress the Avro files using zstd to further reduce the disk footprint and also to save on network costs. We will take a slight deviation from the topic to talk about Bandwidth costs.

Bandwidth costs: They can break the bank!

Let's look at the network costs, you can see the variance in numbers.

It’s interesting to see the variance in the costs, so it’s best to have Postgres, our System and Snowflake in the same cloud provider and the same region. Let’s now calculate the networks costs needed for this workload.

Calculating Network Costs

Another thing to be wary of is the Warehouse configuration.

bytes_per_row = 512
num_rows = 1_000_000_000
total_data_size = 512GB
compressed_data_size_GB = 256 #avro+zstd gives atleast 2x compression
bandwidth_cost_per_10GB = $0.1

# total nework costs
# data_size_GB * bandwidth_cost_per_10GB / 10
network_costs_egress_from_postgres = $5
# compressed_data_size_GB * bandwidth_cost_per_10GB / 10
network_costs_egress_from_system_to_snowflake = $2.56 

network_costs = $7.56

Snowflake Warehouse Configuration

In many organizations, a significant portion of Snowflake expenses comes from compute usage, particularly when warehouses run idle between tasks. Snowflake's compute costs are accrued based on warehouse operational time, starting from activation to suspension. Often, idle warehouse time can contribute to 10%-25% of the total Snowflake compute costs. The Baselit team wrote an excellent blog about it: read more about it here.

The two things we will be doing is to set AUTO_SUSPEND to be 60 seconds, a warehouse idles for up to a minute after the last query before pausing, and make sure that we keep the warehouse active for the least amount of time. This is the default configuration you get if you follow the PeerDB Snowflake setup guide.

Inserts, Updates and Deletes

The next challenge for us after the initial load would be to read the change data from Postgres and replaying that to Snowflake. We are going to be doing that using Postgres’ Logical Replication. At the start of the replication, we will create a replication slot and use pgoutput plugin. This is the recommended way to read changes from the slot. Once we read the changes from the slot, we will batch them and then load them to Snowflake.

As we discussed earlier, it is important to keep the Snowflake warehouse idle for as long as we can, and batching helps with that. We store records in batches of 1M to Avro like before, and load them to an internal stage in Snowflake. Once the data is loaded into the stage, we will MERGE the records from the stage into the destination table. This way most of the heavy-lifting of the resolution is left to the warehouse and it simplifies our system.

Tools

At PeerDB, we are building a specialized data-movement tool for Postgres with laser focus on Postgres to Data Warehouse replication. Most of the above optimizations incl. parallel initial load, reducing Data Warehouse costs, native data-type mapping, support of TOAST columns, fault-tolerance and auto recovery etc. are already baked into the product. PeerDB is also Free and Open. So we chose PeerDB to implement the above workload.

Hardware

Now that we have landed on 8GB RAM, let us move onto picking the instance type.

Since ARM uses lower energy compared to x64 (due to being RISC), they are around 25% cheaper as compared to x64 machines. The tradeoff here is that x64 machines run at around 2.9GHz with a 3.5GHz Turbo (M6i instances) as compared to ARM machines at about 2.5GHz (Graviton2 - M6g) but M6i instances are about 30% more expensive as compared to M6g instances.

Effective cost is $0.0409/GHz for x64 vs $0.03616/GHz for ARM, so cost is about 13% more per GHz on x64 But cost per GHz is not the determining factor for reading in a single thread from Postres during CDC as replication slots can be read from a single process at once.

For this current experiment, I went with m6gd.large as it offers a good balance of speed and disk.

Optional read: In this blog we will use AWS for our analysis. However, here are some other learnings we had on this topic. OVH Cloud currently does not support ARM Instances and has a similar $0.118/hour c2-7 instance (in limited regions) which has a very low network speed (250MBps) with 50GB of SSD. Hetzner has a CCX13 $0.0292/hour instance (including a 118GB SSD) but no dedicated ARM instances.

Conclusion

One question that I'm often posed with: “Is this practical?”. Yes, one machine can die, but systems where there is only one machine have a remarkable amount of uptime, especially when the state is stored in a durable way.

Back to the topic at hand. If we look at the total cost of the system we built (assuming us-west-2 as the region. Over a month time this is the breakdown:

If we were to look at various ETL tools and how much they charge for moving 1 billion rows, this is what it comes out to:

I am part of a company building a software for moving data specifically from Postgres to Data warehouses. It's my job to figure out how to provide the best experience to our customers. Doing this project forced me to figure out a way to provide the best bang for buck, and to include a lot of the explored features into PeerDB. I hope it conveys some appreciation for what modern hardware is capable of, and how much you can get out of it.

When building PeerDB UI, we wanted it to be minimal but effective. Features were driven by what the customers really needed, while keeping the bloat low. For this article, I've asked the team to share their favorite part about the UI.

Replication Slot Growth Chart

This chart shows the size of slot in GB over time.

Activity Monitor

This view captures all the activity and connections open for the database.

Slack Alerts

Alerting configuration for Slack, back to our minimal roots, where we show the configured alerts channel.

Mirror Rows Over Time

A simple histogram view over time where we see the number of rows synced.

Timezone Selector

Simple touch to select the timezones.

Conclusion

These are just a glimpse of PeerDB UI. We tried to cover some of the features that make PeerDB UI unique and helpful. I'm hoping that we add a lot more of these useful features.

We hope you enjoyed reading the blog. If you're a Postgres user and wish to replicate data from Postgres to Snowflake/BigQuery/ClickHouse using PeerDB, please check out the links below or reach out to us directly!

1. Try PeerDB Cloud for free.

2. Visit PeerDB's GitHub repository to Get Started.