Personal project — architecture & requirements
MarketStream
A distributed market-data platform design for streaming price events, historical queries, watchlists, and alerts through a Node.js backend and Next.js client.
Problem
Market data arrives continuously and can include duplicates, out-of-order events, and traffic spikes. A reliable platform needs to preserve history while delivering timely updates to clients.
Contribution
The project specification defines service boundaries, event contracts, storage responsibilities, and a phased implementation plan for an event-driven market-data pipeline.
Outcome
A product requirements document covering architecture, API contracts, event processing, failure testing, and a staged delivery roadmap.
Technical decisions
- Use Kafka-compatible Redpanda with symbol-based partitioning to support parallel processing and per-symbol ordering.
- Plan for at-least-once delivery with idempotent processing and event IDs to prevent duplicate historical records.
- Separate PostgreSQL history from Redis latest-price caching, with TimescaleDB planned for time-series storage.
- Keep ingestion separate from persistence, and expose REST queries alongside WebSocket subscriptions.
- Include retries, dead-letter events, backpressure handling, and observability in the reliability requirements.
Planned event pipeline
A configurable market simulator feeds an ingestion service that validates and normalizes events before publishing to Redpanda. Processor workers persist history, refresh Redis, and publish processed events for API, WebSocket, and alert services.
Client experience
The planned Next.js client supports instrument search, live prices, historical OHLCV queries, personal watchlists, and in-app price alerts. WebSocket requirements include authenticated subscriptions, heartbeats, disconnect cleanup, and slow-client handling.
Reliability and observability
The design calls for idempotent consumers, exponential retries, dead-letter events, graceful shutdown, and health/readiness endpoints. Pino logs and Prometheus/Grafana metrics would track processing latency, consumer lag, cache behavior, and WebSocket connections.
Validation plan
Planned validation covers unit, integration, and end-to-end tests, plus processor crashes, database outages, duplicate events, and traffic spikes. Load tests would compare one, two, four, and eight processors while measuring throughput and latency percentiles; no results are claimed yet.
Scope & limitations
This case study describes the planned system. Implementation status, deployment, and measured performance are not established by the requirements document. Event rates are benchmark targets, not achieved throughput.