Building a Customer Journey Analytics Dashboard: Notes on Product Analytics and Innovation

Patrick Donahue · Levelbrook Consulting

The Domain Problem: Beyond the Page View

Product analytics has evolved far beyond simple page view counters. The interesting engineering problem isn't just counting events; it's reconstructing user intent from a sparse, often chaotic stream of those events. A "customer journey" is an abstraction we impose on this data. A user doesn't think in terms of funnels; they click, type, hesitate, and navigate. Our job is to build a system that can reliably translate this firehose of low-level interactions (`button_clicked`, `form_submitted`, `page_viewed`) into high-level narratives: "user struggled with checkout," "user compared three products before adding one to cart," or "user dropped off during onboarding."

This is technically compelling because it's a stateful problem at its core. To understand one event, you need the context of the events that preceded it. This immediately complicates architectures designed for stateless, horizontally-scalable services. It requires robust sessionization, handling of out-of-order and late-arriving data, and a data model that can answer complex sequence-based queries efficiently. At scale, this becomes a significant distributed systems challenge where throughput, latency, and correctness are in constant tension.

An Architectural Sketch

To tackle this, we need a pipeline that separates concerns: ingestion, processing, storage for different query patterns, and visualization. Here’s a pragmatic stack choice for a high-throughput system, orchestrated on Google Cloud Platform (GCP) with Kubernetes.

Logical Flow: Event Emitters (Web/Mobile) → Kafka → Real-time Processor → MongoDB (Raw Events/Profiles) & ClickHouse (Aggregates) → Grafana / Custom UI

• Ingestion: Apache Kafka. This is the front door. We use Kafka not just as a message queue but as a durable, replayable log of all events. It decouples our front-end event emitters from the backend processing. If a downstream consumer fails, Kafka retains the data until it comes back online. This backpressure management is non-negotiable at scale. Events are structured with a clear schema (e.g., user ID, session ID, timestamp, event name, payload) from the client, though we can't always trust the client-generated timestamp.
• Processing: A Custom Go/Rust/JVM service on Kubernetes. A set of consumers read from Kafka topics. This is where the core logic lives: sessionization (grouping events by user and time window), data enrichment (e.g., adding geo-location from an IP), and filtering bot traffic. Running this on Kubernetes (GKE on GCP) managed with Helm charts allows for easy scaling and deployment. We can scale the number of consumer pods up or down based on the lag in the Kafka topic.
• Storage: A Dual-Database Approach.
  • MongoDB: Serves as our "system of record" for raw, unprocessed events and for user profiles. Its document model is flexible for evolving event payloads and is excellent for point lookups like "fetch all data for user X."
  • ClickHouse: This is the analytical engine. As events are processed, they are cleaned, structured, and pushed into ClickHouse. As a columnar database, it's exceptionally fast for the types of large-scale aggregations and time-series queries needed for a dashboard: "count users who performed event A then event B within 5 minutes," "show the daily conversion rate for funnel X."
• Deployment: Docker, Kubernetes, Helm, and GitHub Actions. Every component (processor, API) is containerized with Docker. Kubernetes manifests are templated with Helm for configurable deployments across environments (staging, production). A GitHub Actions pipeline automates the entire CI/CD flow: on a push to `main`, it runs tests, builds Docker images, pushes them to Google Artifact Registry, and triggers a `helm upgrade` to deploy the new version to our GKE cluster. Simple Bash scripts often act as the necessary glue within these workflows.
• Visualization: Grafana. For internal dashboards and quick analysis, Grafana is hard to beat. Its ClickHouse data source plugin allows us to build powerful, query-backed visualizations with minimal effort. For a customer-facing dashboard, I'd likely build a dedicated React frontend that hits a thin API layer, providing more control over the UX and data presentation.

Data Models, Edge Cases, and Scaling Pains

The core analytical table in ClickHouse might look something like: (user_id, session_id, event_timestamp, event_name, properties, ...), partitioned by day and ordered by user and timestamp. This structure is critical for performance.

Things inevitably break at scale:

• Late-Arriving Data: A mobile client that was offline syncs a batch of events from 12 hours ago. Our real-time sessionization logic might have already closed that session. The system must be able to handle this, either by re-processing windows or by using event time (the time the event occurred on the client) rather than processing time for all calculations. This adds significant complexity.
• Identity Resolution: A user browses anonymously on their laptop, then logs in. Later, they use the mobile app. Stitching these disparate event streams into a single user journey is a difficult problem, often requiring specialized identity graphs and probabilistic matching.
• Cardinality Explosions: If you allow arbitrary strings in event properties (e.g., `url_path`) and try to group by them, you can quickly overwhelm your database. Careful schema design and sanitization are paramount.
• Thundering Herds: A marketing campaign drives a massive, sudden spike in traffic. Can Kafka absorb it without dropping events? Do the consumer pods autoscale quickly enough? Load testing these scenarios is not optional.

Pragmatic Tradeoffs and the Human in the Loop

A senior engineer’s role is to make pragmatic decisions. "Real-time" is a spectrum. Does the product team truly need sub-second latency, or is a 5-minute micro-batch that's 10x cheaper and 100x more reliable the better choice? The answer is almost always the latter.

Similarly, choosing managed services on GCP (like Pub/Sub over Kafka, or BigQuery over ClickHouse) trades control and potential cost savings for reduced operational overhead. For a small team, this is often the right call. For a large-scale system where performance tuning is critical, self-hosting on GCE instances might be justified.

Most importantly, no automated system is perfect. Data will be corrupted. A bug in a new app release might send malformed events for hours. The architecture must include a "human-in-the-loop" component. This means building administrative tools for data stewards to inspect raw event streams, manually correct user journeys, flag anomalous sessions, and trigger re-processing of data for a specific time range. Correctness is not just an algorithmic property; it's an operational one. The system must be debuggable and repairable by people.

Closing Reflection

Building a system for customer journey analytics is a fascinating microcosm of modern data engineering. It forces a synthesis of distributed systems principles for scale, meticulous data modeling for performance, and a deep, empathetic understanding of the product domain to ensure the final output is not just a collection of metrics, but a source of genuine insight. The ultimate goal isn't a perfect, hands-off machine, but a powerful tool that augments human intuition, helping us understand the narrative hidden within the noise.