This way, we start with a consistent view of all of the data, yet continue reading without having lost any of the changes made while the snapshot was taking place. After the connector completes the snapshot, it continues streaming changes from the exact point at which the snapshot was made. Therefore, when the PostgreSQL connector first connects to a particular PostgreSQL database, it starts by performing a consistent snapshot of each of the database schemas. This means that the connector does not have the complete history of all changes that have been made to the database. PostgreSQL normally purges WAL segments after some period of time.
Your client applications read the Kafka topics that correspond to the database tables they’re interested in following, and react to every row-level event it sees in those topics.
The connector then produces a change event for every row-level insert, update, and delete operation that was received, recording all the change events for each table in a separate Kafka topic. Java code (the actual Kafka Connect connector) which reads the changes produced by the chosen plug-in, using PostgreSQL’s streaming replication protocol, via the PostgreSQL JDBC driver Pgoutput, the standard logical decoding plug-in in PostgreSQL 10+ (maintained by the Postgres community, used by Postgres itself for logical replication) this plug-in is always present, meaning that no additional libraries must be installed, and the Debezium connector will interpret the raw replication event stream into change events directly. Wal2json (maintained by the wal2json community, based on JSON) PostgreSQL connector contains two different parts which work together in order to be able to read and process server changes:Ī logical decoding output plug-in which has to be installed and configured in the PostgreSQL server, one ofĭecoderbufs (maintained by the Debezium community, based on ProtoBuf) This output plug-in must be installed prior to running the PostgreSQL server and enabled together with a replication slot in order for clients to be able to consume the changes. PostgreSQL’s logical decoding feature was first introduced in version 9.4 and is a mechanism which allows the extraction of the changes which were committed to the transaction log and the processing of these changes in a user-friendly manner via the help of an output plug-in.
0 kommentar(er)
