Question 1

What is real-time data infrastructure?

Accepted Answer

The pipeline layer that moves data from where it's produced (an application database, a sensor, an event) to where it's consumed within seconds instead of hours: a message broker like Kafka, change data capture off your operational database, schema contracts, and the monitoring and recovery tooling that keeps it trustworthy. The broker alone isn't the infrastructure — the reliability layer around it is.

Question 2

What's the difference between real-time, near-real-time, and batch?

Accepted Answer

Batch processes data on a schedule — hourly, nightly — and is the cheapest and easiest to debug when nobody's waiting on the result within minutes. Near-real-time (seconds to low minutes of latency) covers most dashboards and operational reporting. Real-time or streaming means sub-second to low-second latency, reserved for decisions that have to happen as the event occurs: fraud checks, inventory sync, live pricing. Most teams asking for "real-time" actually need near-real-time, which is materially cheaper to build and run.

Question 3

What's CDC and why does it matter here?

Accepted Answer

Change data capture streams row-level changes out of your operational database as they happen, instead of a nightly batch job querying the whole table. It's how you feed a real-time pipeline from a database that wasn't built for streaming, without adding load to production with polling queries.

Question 4

Do we actually need streaming, or would batch be fine?

Accepted Answer

Depends on what decision the data feeds. If a human reads a report once a day, micro-batch or hourly batch is cheaper to run and easier to debug. Streaming earns its complexity when a decision (fraud check, inventory sync, live pricing) needs to happen within seconds of the event. We'll tell you directly if batch solves your actual problem before we scope a streaming build.

Question 5

We already run Kafka. What's left for you to do?

Accepted Answer

Running Kafka and running reliable streaming infrastructure aren't the same thing. Most Kafka deployments we take over are missing schema enforcement, consumer lag alerting, exactly-once processing where it matters, and a tested recovery path for a stuck or poisoned topic. We audit what's there and build the reliability layer around it.

Question 6

How do you handle exactly-once vs at-least-once delivery?

Accepted Answer

At-least-once, with idempotent consumers that safely handle a duplicate, is the default we reach for — it's simpler to build and reason about, and a well-designed consumer makes duplicates harmless. Exactly-once processing (Kafka's idempotent producer plus transactional API, or an equivalent in your streaming platform) is worth the added complexity specifically where a duplicate has a real cost: double-charging a payment, double-decrementing inventory. We scope which boundary needs it rather than applying exactly-once everywhere by default.

Question 7

How do you handle schema evolution in a streaming pipeline?

Accepted Answer

Schema contracts enforced at the producer, with a schema registry checking backward or forward compatibility before a change ships, so a breaking field rename or type change fails in CI for the team that made it, not as a silent parse error three consumers downstream. Consumers are built to tolerate additive changes (new optional fields) without a deploy, and breaking changes go through a versioned topic or a coordinated migration instead of a surprise.

Question 8

How do you monitor for data loss or duplication in a streaming pipeline?

Accepted Answer

Consumer lag alerting, so a stalled consumer group is caught in minutes instead of when someone notices stale data, dead-letter queues for messages that fail processing instead of silently dropping them, and reconciliation checks that compare source row counts or checksums against what landed downstream. The recovery runbook for each failure mode is tested before it's needed, not written after the first incident.

Question 9

What does the assessment deliver, and how long does it take?

Accepted Answer

We map which use cases actually need sub-minute latency, audit your existing streaming stack if you have one, and identify the reliability gaps (consumer lag, schema drift, exactly-once boundaries) most likely to cause an incident during the assessment. You get a prioritized architecture and reliability plan, not a generic streaming pitch.

Question 10

Self-managed Kafka vs a managed streaming service — which should you use?

Accepted Answer

Managed (Confluent Cloud, AWS MSK, or similar) is the right default for most teams — it removes cluster operations, broker upgrades, and capacity planning, at a premium over running it yourself. Self-managed makes sense at high, steady throughput where the operational overhead is justified by the cost savings, or where data residency or networking constraints rule out a managed option. We size this against your actual throughput and team's operational capacity, not a default recommendation either way.

Question 11

How does this compare to hiring a streaming/Kafka engineer?

Accepted Answer

A dedicated streaming or Kafka engineer is a significant full-time salary commitment, and most teams only need that depth for the initial architecture and reliability build, not every week after. We deliver the assessment and pipeline build in weeks, document tested recovery runbooks, and hand off to your existing engineers, with a lighter retainer available instead of a full-time hire.

Question 12

How much does real-time data infrastructure consulting cost?

Accepted Answer

The streaming readiness assessment is priced after we know how many use cases and how much of your existing Kafka/Flink stack is in scope. Pipeline builds and reliability implementation are priced per use case once the assessment identifies what's worth streaming. Ongoing reliability support runs on a monthly retainer.

Question 13

What is Debezium, and how do you use it for CDC?

Accepted Answer

Debezium is an open-source CDC engine that reads a database's transaction log (the WAL in PostgreSQL, the binlog in MySQL) and publishes each row-level insert, update, and delete to Kafka as an event. It runs on Kafka Connect, so the source database needs no application changes and no polling queries. We deploy and tune the connectors, set snapshot and offset behavior, enforce schemas on the change events, and add lag and connector-failure alerting so a stalled connector is caught in minutes.

Question 14

Can you set up CDC on PostgreSQL, MySQL, SQL Server, or Oracle?

Accepted Answer

Yes. Debezium covers all four, and each one needs different source-side preparation: logical replication and replication slots on PostgreSQL, row-based binlog on MySQL, SQL Server's built-in change tracking or CDC feature, and LogMiner or XStream on Oracle. We handle the source configuration, connector setup, and the downstream pipeline into Kafka, so the change stream is correct from the first event, not just the first demo.

Question 15

How do you keep CDC from overloading the source database?

Accepted Answer

Log-based CDC reads the transaction log instead of querying tables, so the load on production is small compared with a polling job. The real risks are elsewhere: an unconsumed PostgreSQL replication slot keeps WAL on disk until the volume fills, and a large initial snapshot can hold locks or saturate I/O. We monitor slot lag and retained WAL, run snapshots against a replica or in controlled chunks, and set alert thresholds before the disk becomes the incident.

Question 16

How do you handle the initial snapshot when CDC starts on a large table?

Accepted Answer

The snapshot loads the table's current rows into Kafka, then the connector switches to streaming changes from the log position where the snapshot began, so nothing is missed or doubled. For multi-hundred-gigabyte tables we use incremental snapshots that read in chunks alongside live streaming, instead of one long blocking read. We also plan the downstream consumers to handle the snapshot burst without falling behind.

Question 17

Can you migrate us from self-managed Kafka to Amazon MSK or Confluent Cloud?

Accepted Answer

Yes. We map your topics, partition counts, ACLs, and connectors, then move producers and consumers in stages using MirrorMaker 2 or Confluent Cluster Linking so both clusters stay in sync during cutover. Consumer offsets are translated so consumers resume where they stopped, and each cutover step has a tested rollback. The result is a managed cluster with no downtime window and no replayed or lost messages.

Question 18

What are the alternatives to Confluent, and how do you choose?

Accepted Answer

The practical options are Amazon MSK, Redpanda, Aiven for Apache Kafka, and self-managed Kafka on Kubernetes with Strimzi. The choice comes down to four things: throughput and retention cost at your volume, which connectors and schema registry features you depend on, where your data has to live, and how much cluster operation your team can absorb. We compare them against your workload and give you a recommendation with the cost and operational trade-offs stated.

Question 19

What does a data streaming platform include?

Accepted Answer

A message broker (Kafka or a compatible service), a schema registry, Kafka Connect or an equivalent for moving data in and out, a stream processor such as Flink or Kafka Streams for transformations, and the monitoring layer: consumer lag, throughput, dead-letter queues, and connector health. Access control and data retention policy sit across all of it. We design the platform around your sources and consumers, so you only build the parts your use cases need.

Question 20

Can you stream CDC data into Snowflake, BigQuery, or another warehouse?

Accepted Answer

Yes. The pattern is CDC into Kafka, then a sink connector or stream job that writes to Snowflake, BigQuery, Redshift, or a lakehouse table, so the warehouse stays minutes behind the source instead of a day behind. We handle the parts that break in practice: applying updates and deletes correctly instead of appending duplicates, schema changes from the source, and reconciliation checks that compare row counts between source and warehouse.

Situation	Likely Solution
A dashboard the control room or ops team relies on shows data that's minutes or hours stale	CDC Implementation — change data capture feeding the streaming platform in real time
A batch job that used to finish overnight now runs into business hours and delays a decision	Streaming Architecture Design — an event-driven pipeline replacing the nightly batch run
Kafka (or a similar broker) is already running, but consumer lag or a poisoned message caused an incident	Kafka & Flink Reliability — consumer lag alerting and dead-letter handling for what's already deployed
A new use case (fraud check, inventory sync, live pricing) needs a decision within seconds and the current pipeline can't get there	Streaming Architecture Design — topic strategy and consumer topology built for the latency target

Before	After
Sensor and event data lands minutes behind, sometimes hours during peak load	Sub-minute ingestion, with consumer lag alerted on before it becomes an incident
Operational reports and dashboards time out or run against stale batch tables	Reports and dashboards read from a pipeline built for the query pattern, not a nightly dump
No tested recovery path for a stuck consumer group or a schema-breaking producer change	Documented, tested recovery runbooks for the failure modes most likely to happen
Uptime and latency are assumed, not measured	Uptime and latency are monitored against a stated target

Approach	Tradeoff
DIY, existing app team	Cheapest up front. Streaming reliability (lag alerting, exactly-once boundaries, recovery runbooks) is usually the first thing cut under deadline pressure.
Full-time streaming/Kafka engineer	Right if you need that depth every week going forward. A significant full-time salary commitment for work that's often front-loaded into the initial architecture and reliability build.
Generic cloud consultancy	Can stand up a cluster. Streaming correctness (schema contracts, exactly-once, poisoned-message recovery) is a narrower specialty most generalist shops don't carry deep experience in.
DharmOps	Assessment plus pipeline build in weeks, documented recovery runbooks, and a handoff to your team, with a lighter retainer available instead of a full-time hire.

Real-Time Data Engineering

What Problems Trigger This Engagement?

Match Your Situation to the Fix

What Is Real-Time Data Engineering?

Why Streaming Architecture, Not Just a Broker

Architecture, Not a Tool Install

CDC Without Load on Production

Recovery for the Failure Modes That Matter

What DharmOps Builds

Streaming Architecture Design

CDC Implementation

Kafka & Flink Reliability

Event & Data Contracts

How a Streaming Engagement Works

Streaming Readiness Assessment

Architecture Design

Pipeline Build

Recovery Runbooks & Handoff

Ongoing Reliability Support

Specific Use Cases

Before → After

How This Differs From Alternatives

Frequently Asked Questions

Case Study

US Traffic Management: PostgreSQL & SQL Server Optimisation