Circadia Methodology.

A description of how Circadia captures and analyzes free-running sleep data, written for circadian-rhythm researchers evaluating whether the app's outputs can be cited or used in secondary analysis.

For the patient-facing version of this content, see HOW_IT_WORKS.md. For the patient-clinician handoff, see the Doctor's Report PDF exported from the app.

Last revised: 2026-06-17 (per-row drift label unified on the gap-space series everywhere it is shown; split-sleep episode span caps and anchor rule documented in §7; refreshed cohort state). Prior major revision 2026-06-09 (gap-space Smart estimator with robust Huber location; timezone-correct epoch parsing).

1. What Circadia measures

Circadia is a self-tracking app for sighted Non-24-Hour Sleep-Wake Disorder (N24SWD) and adjacent free-running phenotypes. Each user logs sleep onsets and wakes (and optionally sleepless gaps). The app derives:

• Daily drift — the shift in sleep onset from one cycle to the next, in hours.
• τ (tau) — the estimated period of the user's observed sleep-wake rhythm (a behavioral period), in hours. Computed as 24 + mean_drift. Under the limitations described in §13 (no DLMO, self-report, behaviorally derived), this approximates — but is not a direct measurement of — the intrinsic circadian period reported in lab forced-desynchrony protocols (Czeisler 1999; Duffy 2011).
• σ_obs — observed variability of the daily drift.
• σ_τ — standard error on τ.
• |drift| and |drift all| — unsigned drift magnitudes (cohort-side diagnostic, see §10).
• Cumulative debt and Process S — homeostatic measures, included for completeness; not the primary contribution.

The primary research contribution is this behavioral τ estimate and its uncertainty, derived from continuous at-home logging over multi-week windows. It is intended as an ambulatory approximation of the intrinsic period measured under controlled lab conditions (Czeisler 1999; Duffy 2011), with the limitations of self-reported onset timing that that implies. See §13 for the full caveat list.

2. Drift between two sessions

For two consecutive sleep onsets onset_prev and onset_curr, the gap-space drift used by the Smart estimator is:

gap_h   = (onset_curr − onset_prev) in hours
drift_h = gap_h − 24

drift_h is the per-cycle shift in onset relative to a 24-hour day. A positive drift indicates phase delay; negative indicates phase advance.

Gap-space drift is unwrapped: it retains the integer-day content of the gap instead of reducing the shift to a clock-face difference. This removes the +14 h-vs-−10 h aliasing that a modular representation suffers (a real +15 h forward shift is +15 h in gap-space, not −9 h), at the cost of requiring explicit gating of gaps that are not single body-cycles (§3, step 3).

The older wrapped-clock drift — d_clock = hour_of_day(onset_curr) − hour_of_day(onset_prev) wrapped to (−12, +12] (calcDrift() in src/lib/utils.js) — now survives only as the basis of the legacy Clean and Raw comparison estimators (§4). As of 2026-06-17 every per-row drift label shown to users is the gap-space value, routed through one entryDriftChip() helper (src/lib/driftLabel.js) so the log, the chart, and the calendar's monthly list cannot disagree — the calendar's list previously still rendered the wrapped value, which read with the opposite sign on multi-cycle transitions. It is not the basis of the headline τ, and drift summaries shown to users or in the Doctor's Report (median, IQR, magnitude) read the gap-space series, not the wrapped one, specifically because wrap artifacts (fake negatives) cancel real positives in summary statistics.

Timezone handling (2026-06-09)

Onsets and wakes are stored as timezone-naive local strings plus a separate per-entry IANA tz column. At the data boundary, each entry is stamped with absolute epochs onsetMs / wakeMs via localToMs(naive_local, tz) (src/lib/utils.js), which resolves the named zone with DST awareness. All gap and drift math reads these absolute epochs. Entries without a stored tz (legacy rows) fall back to parsing in the viewer's browser timezone — byte-identical to the pre-2026-06-09 behavior.

Before this change, gap math implicitly parsed naive local strings in the viewer's timezone, so the same dataset produced different drift numbers depending on who was looking (an admin in one timezone saw different values than the user in another). The fix is a no-op for a user viewing their own data in their home timezone (verified across the cohort: 0 of 44 users with entries moved) and corrects cross-timezone, admin, and cohort views.

3. τ estimator (Smart) — gap-space, robust

The production τ/drift estimator is a gap-space estimator with τ-relative cycle gating, adjacent anchoring, an EWMA-weighted fading prior, and a robust (Huber M-estimator) location. It is implemented as estimatePeriodGapSpace() in src/lib/gapSpaceTau.js and is called in production with adjacent anchoring ON and soft phase-cleanliness weighting OFF.

Two dated transitions matter for longitudinal comparisons of app outputs:

• 2026-05-29 — gap-space estimator promoted to the headline, replacing the legacy wrapped-clock EWMA regression (estimatePeriod() in utils.js; now a comparison surface, §4).
• 2026-06-09 — reported location changed from the weighted mean to a weighted Huber M-estimate, and all gap math moved to timezone-correct absolute epochs (§2).

Inputs and exclusion flags

The estimator receives the full entry list with flags and handles exclusion internally:

• xDrift — excluded from drift (nap auto-flag or user exclusion).
• fragmented — auto-flagged continuation of a prior episode (markFragmented(), §7).
• tooShort — sleep span below the user's nap threshold.
• fragManual — user explicitly marked this as a fragment of the previous sleep; never a phase marker of its own.
• xDriftManual — the user explicitly toggled this entry. Explicit intent wins: the include/exclude decision is then governed purely by xDrift, overriding the auto-flags, and a manually included entry also bypasses both cycle gates (step 3) in either direction.

Step 1 — adjacent anchor timeline

Gaps are measured against the immediately preceding entry in time (adjacent anchoring). Excluded entries still anchor the timeline — they contribute no drift of their own, but the next entry measures its gap to them rather than bridging across them. Bridging was the source of a cascade bug in which one excluded sleep manufactured a phantom multi-day "cycle" and nulled the drift of an otherwise-clean neighbor.

Two kinds of entries are dropped from the anchor timeline entirely: user-marked fragments (fragManual), and leading false-starts — auto-flagged fragments whose episode's main sleep comes after them (e.g. a 45-minute doze right before the real sleep), which would otherwise make the real sleep measure a tiny gap to its own false-start.

Step 2 — candidate transitions

Each surviving adjacent pair contributes a gap-space drift drift_i = gap_h − 24 (§2) with an EWMA recency weight

w_i = exp(−ln(2) × (t_now − t_onset_i) / halfLife),  halfLife = 28 days

Recent data is weighted more heavily than old data; the half-life is comparable to the timescale on which self-reported τ has been observed to shift in N24 patients (Emens 2013).

Step 3 — two-pass τ-relative cycle gating

A gap-space drift is only meaningful if the gap is approximately one body-cycle. Two gates decide that, both scaled to the user's own τ:

lower_gate = max(18 h,  0.7 × τ_est)
upper_gate = max(34 h, mult × τ_est),   mult = 1.3 (1.2 when τ_est > 28)

• gap < lower_gate → "snapback" — too short to be a real single cycle (a nap, fragment, or short recovery sleep that survived upstream filtering). Excluded from the point estimate.
• gap > upper_gate → "crash" — a sleep-debt collapse after a long awake stretch, not a valid phase marker. Excluded from the point estimate entirely; it contributes zero rather than dragging τ.

The estimator runs two passes: pass 1 gates with the prior τ (24.7 h) to obtain a first τ estimate, pass 2 re-gates with that estimate. The τ-relative cap matters for fast drifters: a flat 30 h gate (the legacy behavior) systematically truncated their genuine long cycles and biased τ downward.

Excluded transitions are individually labeled (snapback / crash) and surfaced in the user's log as "not counted toward drift," with the manual-include override available per entry (step 0 above).

Step 4 — fading prior and weighted mean (seed)

A weak prior centers cold-start estimates on the published N24 population mean: priorPseudoCount = 3 pseudo-observations at drift = priorTau − 24 = 0.7 h, variance 1.0 (Hayakawa 2005 N=57; Kitamura 2013 N=28), fading linearly —

effective_prior = max(0, priorPseudoCount − N_included / 3)

— so it is gone entirely by ~9 included cycles. The prior-anchored weighted mean of included drifts is still computed, but since 2026-06-09 it is the seed and fallback for the robust location below, not the reported number. It is returned as meanDrift for diagnostics.

Step 5 — robust location (weighted Huber M-estimator)

The reported drift is a weighted Huber M-estimate of location over the included cycles, using the same per-cycle EWMA weights:

scale     s = 1.4826 × MAD(drift_i)        (classic, unweighted MAD)
location  μ = IRLS fixed point of Σ w_i ψ_c((drift_i − μ)/s),  c = 2.0
τ           = 24 + μ

Residuals within c·s of the location get full weight; beyond that, influence grows only sub-linearly instead of one-for-one. The tuning constant c = 2.0 is deliberately larger than the classic 1.345: a cycle must sit more than two robust standard deviations out before it is down-weighted at all, which keeps the estimator a practical no-op for clean directional drifters while reining in the heavy two-sided tails that fragmented, bidirectional, or relative-coordination sleepers produce (a handful of huge or sign-flipped cycles can no longer drag the center away from where the bulk of the rhythm sits).

No-op guarantees: with fewer than 2 included cycles, or a degenerate (≈0) MAD, the estimator returns the plain weighted mean — byte-identical to the pre-Huber behavior. At deployment (2026-06-09) the change was validated against the live cohort: near no-op for clean drifters (one fast directional drifter moved +3.97 → +3.85 h/day), with material correction for heavy-tailed loggers (one bidirectional logger moved from a washed-out −0.33 mean to +0.51, matching her median cycle).

Step 6 — uncertainty

σ_obs²  = weighted variance of included drifts, centered on μ (the
          robust location), prior included
σ_floor = max(1.0, 0.5 × √(max(τ − 24, 0.5)))
σ_obs   = max(σ_obs, σ_floor)
N_eff   = (Σ w_i)² / Σ w_i²            # Kish, prior pseudo-count included
σ_τ     = σ_obs / √N_eff

The estimator also returns the median of included-cycle drifts (driftMedian, tauMedian) as a second robust summary; the display layer uses it (step 8).

Step 7 — trust guards and structural signals

The estimate carries explicit flags that decide whether the headline number should be shown at all:

• insufficientData — fewer than 3 included cycles.
• implausibleTau — τ < 21 h, a non-physiological period regardless of sleep structure. There is deliberately no symmetric upper bound: genuine N24 free-runners legitimately reach τ ≈ 27–29 h+, so a normal-population ceiling would wrongly gate exactly the users the app exists for.
• polyphasic — a structural multi-sleep-per-cycle signal, independent of the gates. It trips on either (a) a manual-include tell — ≥ 4 included cycles of which > 40% are sub-18 h gaps the user forced in, asserting multiple sleeps per cycle — or (b) a density tell — ≥ 6 adjacent gaps with > 1.5 logged sleeps/day or a median adjacent onset-gap < 18 h. Both are vetoed by a consolidated-main-sleep guard: if the median non-fragment sleep is ≥ 4 h, the user has one main sleep per cycle (an occasional second-sleep logger, not a polyphasic schedule) and is not flagged.

trustworthy is true only when none of the three trips; consumers withhold the number otherwise.

Step 8 — what the user actually sees

Two distinct surfaces consume the estimate:

• The headline drift number ("Smart") is the robust location (step 5) computed over the user's selected window (7d/30d/90d/custom/ all). The windowed run uses identical options to the full-history run, so a windowed number can never silently diverge in method.
• The τ badge and its confidence tier come from classifyTau() (same module), which assigns one of four display tiers: tier 1 established (σ_τ ≤ 1.0 — show τ with a tight band), tier 2 preliminary (σ_τ ≤ 1.3, or very few cycles — show τ with a wide "still refining" band), tier 3 varies (no number — implausible τ, or ≥ 30 cycles that still won't resolve), tier 4 onboarding (no number — 0–1 cycles, "log a few more nights"). Logs flagged polyphasic are first collapsed to one main sleep per cycle (onsets gap-clustered at 14 h breaks; the longest sleep in each cluster becomes that cycle's marker) and tiered on the collapsed series. The τ value displayed is the robust median (tauMedian) of the base estimate's included cycles, which tracks the median cycle exactly and is a no-op for clean single-sleepers.

4. Alternative drift estimators (for comparison)

Two simpler estimators are also exposed in the UI as user-selectable alternatives. Both operate on the legacy wrapped-clock drift (§2), not gap-space:

• Clean — unweighted arithmetic mean of wrapped drift_h over pairs where gap_h ≤ postSleeplessThreshold (default 30 h, per-user adjustable). No recency weighting, no prior, no gates, no robustness.
• Raw — unweighted mean over ALL transitions, including post-sleepless ones. Post-sleepless transitions are computed as gap_h mod 24 (forced positive). Included for transparency; biases τ upward when users have many skipped sleeps.

Because they wrap to (−12, +12], both alias true per-cycle shifts beyond 12 h (a real +15 h shift records as −9 h) and suffer sign cancellation in summary statistics. They are retained for transparency and historical comparability, not inference.

A third comparator exists in source: the legacy Smart estimator (estimatePeriod() in utils.js) — an EWMA-weighted wrapped-clock regression with the same fading prior and σ floor, plus a wraparound-unfolding pass and a bidirectional-drift median fallback. It was the headline estimator until 2026-05-29 and still backs some secondary σ surfaces and the Clean/Raw cohort columns. Its full description is preserved in this file's git history (revisions before 2026-06-09).

When evaluating cohort data, the gap-space Smart estimator (§3) is the recommended reference. Raw remains useful as a divergence diagnostic — when Raw and Smart diverge by > 3 h on the same user, most of that user's transitions are being filtered (low cycle rate) and any single summary number undersells their real shift magnitude. See §10 for the cohort-side diagnostic.

5. Adaptive forecast (Kalman IMM)

The Smart estimator above returns a single best-fit τ and its uncertainty. The Adaptive prediction engine is a separate sequential state-space model that maintains a full phase/drift distribution for each user and projects it forward. It is an Interacting Multiple Model (IMM) Kalman filter, implemented as kalmanIMMPredict() in src/lib/kalman-dlm.js. It superseded the May-2026 particle filter (particleFilterPredict() in model-lab.js, retained as a comparison surface in the Model Lab) when the IMM class was first promoted on 2026-06-03, and the live configuration has continued to evolve since — most recently a tag-aware v3 that wires self-report context and homeostatic sleep pressure into the forecast (held-out results in §6). An always-on trainer re-fits against fresh consented data and gates every candidate against the current live model on held-out nights before it can promote. The live parameter set is resolved at runtime from Circadia's own backend (/api/model-config), so a promotion is a reversible database write the app picks up on next load — no redeploy.

Architecture

The state is a local linear-trend Kalman filter: a level (the user's current onset phase) and a slope (their current per-cycle drift), with the slope free to evolve so the tracker follows acceleration as well as steady drift. Each historical entry is folded in by the standard Kalman predict → update step (levelNoiseH, slopeNoiseH, and observation noise obsSigmaH govern how fast level and slope are allowed to move and how much each logged onset is trusted).

The "IMM" part runs this filter as a bank of modes that share everything except the slope (drift) process noise:

Each step the modes are Markov-mixed: a transition matrix (stickCalm = P(stay calm), stickVol = P(stay volatile), initModeProbVol = initial mass on the volatile mode) blends their estimates by how well each has been predicting. The data routes mass into the volatile mode exactly when a transition begins, and back to calm once the user re-stabilizes. (The implementation supports a longer geometric volatility ladder and an optional shared harmonic-curvature bank; production runs the 2-mode calm/volatile IMM with curvature off.)

The band is the posterior covariance

The forecast uncertainty is not a tuned multiplier — it is the model's own posterior covariance, i.e. the mixture of the modes' Gaussians. So the band is wide (and can be multimodal) mid-transition when the modes disagree, tight when they agree, and it widens automatically as process noise accumulates across long gaps. The reported unc / uncH80 / uncH95 / conf are read straight off that mixture (mixtureQuantile()), which is why coverage is calibrated by construction rather than hand-fit.

Robustness and adaptation

• Outlier gate (outlierGate) — an observation that lands too many σ from the prediction is downweighted instead of being allowed to yank the state, so one mis-logged night doesn't whip the forecast.
• Online adaptation (adaptGain / adaptThreshold / adaptDecay) — sustained surprise (not a single outlier) lets the filter open up faster, the calibrated analogue of the old particle filter's change-point boost.
• Homeostatic sleep pressure (Process S) — a "battery" that charges over time awake and discharges over sleep; its deviation from baseline nudges the predicted onset earlier when you're over-tired. The pure Kalman discarded awake gaps entirely, so this restores the one major circadian mechanism it lacked. Active in the live model.
• Per-user observation band — the personalization framework (below) can fit each user's own band width, but in the live model the band is global (per-user fitting is built but gated off).

Context inputs (tags and self-report)

As of the tag-aware v3 promotion (June 2026), self-reported context does move the forecast. Past tagged nights inform the state, and a declared "right now" context (e.g. illness, caffeine) shifts the upcoming sleep, decaying over the next couple of cycles. Two honest guardrails:

• MNAR-gated and per-user-scaled. A tag you never log carries no weight, and tagAdaptGain scales how much your tags move your forecast from your own evidence — a tag that doesn't predict your sleep shrinks toward zero for you (bounded by a per-user prior of ≈ 12 observations).
• The effect is real but small (~10 min). The data says tags weakly predict onset even for diligent loggers, so the model deliberately does not inflate it. (A planned UX step is to state the shift explicitly — "illness → onset ~10 min later, less certain" — so a small-but-real effect is legible rather than something to squint for.)

The per-covariate weights live in KALMAN_IMM_ALPHA_PARAMS (e.g. illness, stress, medication, screens, light timing, mood / cognition / quality). Several legacy-harvest degrees of freedom (a circadian phase-response curve, periodic/calendar bases, a scallop-reset drift nudge) are implemented but currently gated off — they did not beat the live model on held-out scoring (§6), and a global PRC in particular hurt the hardest free-runners, so it is held for a per-user version.

Personalization framework (built, gated off)

A per-user track is built and held-out-verified but not yet live: it fits each user's own base parameters (notably their observation-band width — tight for clean drifters, wide for chaotic ones), shrunk to the global prior, and serves a skill-weighted blend of global vs tuned that is gated per user, online — the tuned model only earns weight where it out-predicts the live model on that user's own held-out nights (evidence-gated, never time-gated). On a 25-user enable test: 23/25 no regression, 10 clear wins, worst regression −0.068. Turning it on is a gated config promotion pending fit-caching and cohort sign-off.

The full IMM parameter set is exposed in source at KALMAN_IMM_ALPHA_PARAMS and is inspectable in the public Model Lab (#model-lab route — non-admins can browse the catalog and re-score any candidate model, including the legacy particle filter, against their own log).

6. Tuning and validation

The adaptive model is gated, not hand-waved: an always-on trainer re-fits candidates against fresh consented data and a candidate only promotes if it beats the current live model on held-out nights. The IMM class was first promoted on 2026-06-03; the lineage to it is written up in research/2026-06-03-from-regimes-to-kalman.md, and the most recent crop (legacy-feature harvest, the data-gap fix, and live tag-awareness) in research/2026-06-14-personalization-and-harvest.md. The current live promotion's held-out numbers are under "Held-out results" below.

Training corpus

The production parameter set was fit via offline hyperparameter optimization against voluntarily-shared sleep histories from Circadia alpha users as captured in the tuning corpus on 2026-05-17 (n ≈ 18 active sharers at that snapshot, ranging from a few weeks to ~13 months of logged sleeps) plus two longer-form externally shared datasets. These training-corpus figures are deliberately frozen to that snapshot — they describe what the model was actually fit on, not the current cohort (see §12 for current state). Run artifacts at research/runs/ (soft-hypothesis-* for the prior particle line, kalman-imm-* for the current adaptive model). Model registry: research/model_registry.json; the live adaptive params are KALMAN_IMM_ALPHA_PARAMS in src/lib/kalman-dlm.js.

Held-out validation

1,600+ sleep records reserved across multiple users for held-out scoring. The model never sees these during fitting.

No-time-travel rule

At each prediction step the model has access only to data preceding the predicted entry. For multi-day forecast tests, the model is frozen at a split point and asked what it would have predicted over the next 7 or 14 days without learning from those future sleeps.

Disruption-slice testing

Average scores over a whole month can hide failure modes users actually feel — a forecast that becomes useless right after one skipped night. We separately score predictions on the rows immediately following a disruption (defined as a residual > disruptionThresholdH ≈ 4 from the prior fit).

Held-out results (current live: tag-aware v3 vs the previous adaptive)

Scores are a skill number where lower is better — held-out forecast error normalized against a fixed reference, so the comparison between models matters more than the absolute value. The reliable held-out gate that authorized shipping the current live model:

The deliberate, signed-off call: the new model wins the aggregate by about a fifth and fixes a real miscalibration — the previous bands had gone overconfident on the grown cohort (cov80 0.69 means they missed more often than an 80 % band should). The cost is small regressions on the hard free-running tail, and those are in the safe direction (wider, more honest bands, not falsely-confident ones). Rollback is a single config command. The hardest held-out tail case is a sparse, chaotic free-runner — Dayah's own log (τ ≈ 27 h) — kept as a standing "don't ship if it breaks this person" gate.

Honest qualifiers

The win is broad but not unconditional. On calm, very-steady patterns the simple non-adaptive sigma·√i forecast (the default when adaptive is off) already has little to fix, so the adaptive model's edge there is small. Tag effects are real but small (~10 min), not a headline mover. And the training-corpus shape matters — a few long histories shape the global fit disproportionately, which is exactly why the per-user personalization track (§5) is being built: a global knob tuned to the cohort average can quietly hurt an atypical user.

7. Exclusion rules

A session is excluded from drift math (but still counted toward sleep totals) under these conditions:

1. Fragmentation — a session starts within fragmentationThreshold hours (default 6 h) of the previous wake. Consecutive sessions that each begin inside that window are grouped into one episode (a single circadian cycle), bounded by a 14 h span cap measured from the episode's first sleep — a run of short wake-ups stays one night, but a genuinely long gap opens a fresh cycle. If the prior part ended in a forced wake ("interrupted, not finished"), the cap stretches to 30 h, so an alarm-broken night with a later catch-up still reads as one cycle. Within an episode the earliest real sleep anchors the cycle's phase (a part that ends in a forced wake can anchor even below the nap threshold); shorter leading parts become fragments. A non-anchor part that is both ≥ the nap threshold and ≥ 0.6× the episode's longest sleep is kept as its own cycle rather than a fragment — two genuine polyphasic sleeps a few hours apart should not collapse into one — unless the user marked it a fragment or the part immediately before it ended in a forced wake. Threshold is per-user configurable; polyphasic / ME-CFS / split-sleep users typically lower to 3–5 h, while clean monophasic N24 users can raise to 8 h. Reference: markFragmented() in utils.js.
2. Manual fragment (fragManual) — the user explicitly marked the session as a fragment of the previous sleep. Never a phase marker; also removed from the Smart estimator's anchor timeline (§3 step 1) so the next real sleep measures its cycle against the previous real sleep.
3. Nap auto-flag — a session shorter than napThreshold hours (default 4 h) is auto-flagged as a crash nap. User can override per-entry; manual choice wins.
4. Cycle gates (Smart estimator only, per transition rather than per session) — a gap below max(18 h, 0.7 × τ) ("snapback") or above max(34 h, ~1.3 × τ) ("crash") is not a valid single-cycle phase marker and is excluded from the τ point estimate (§3 step 3). The legacy Clean estimator still uses the user's flat postSleeplessThresholdH (default 30 h) instead; Raw includes everything with modular wrap.

Manual exclusions via the per-entry xDrift toggle override fragmentation and nap auto-flag in either direction, and a manual include additionally bypasses both cycle gates. xDriftManual is preserved separately so threshold changes after the fact don't clobber the user's explicit choice.

8. Confidence and forecasting

Forward prediction at n cycles uses a random-walk variance model:

σ_prediction(n) = σ_obs × √n

This is the standard model for accumulated jitter in a free-running oscillator and is what the Predict tab displays. It is not a calibrated frequentist interval; it is presented to users as a guide-rail, with documented caveats that real-world predictions degrade beyond ~7 cycles due to compounding tau drift and zeitgeber perturbations.

The default forecast reports the probability mass inside the user's one-cycle σ_obs tolerance. The adaptive predictor uses the same display contract, but compares its learned residual band against the default one-cycle tolerance. That keeps the percentage comparable across modes and prevents the adaptive model from always showing the same confidence decay sequence merely because both numerator and denominator came from its own sigma.

The Analysis tab also surfaces a Phase Position scatter (Position or Residual mode) that lets users compare predicted vs actual onset for each historical entry. In Adaptive mode the per-entry predictions are the particle-filter's weighted-mean predictions made before observing each row (out.fitted on the particle-filter output) — i.e. genuine forward-in-time fits, not retrospective.

9. Co-variates captured per session

For research purposes, each sleep entry can carry:

Core covariates

Zeitgeber bundle (May 2026)

Stored as JSONB on the zeitgebers column of circadia_sleep_entries. All fields optional; missing means "not tracked" (not "false"):

All co-variates are optional, user-reported, and intended as exploratory signals — not ground-truth zeitgeber measurements. Per-user hide controls let users opt out of any zeitgeber they don't track; hidden fields don't appear in either the log form or the Analysis correlation panels.

Per-session derived covariates (computed, not user-reported)

Sleepless periods

Sleepless periods (intentionally skipped sleeps, sometimes lasting 30–48 h in free-running patients) are logged in a separate circadia_wake_periods table to preserve the actual onset/wake timeline. Drift math treats them as gaps; sleep-debt math counts them as ordinary sustained wakefulness.

Per-user settings (cloud-synced)

Settings sync across devices via the user_settings JSONB column on circadia_user_profiles. The Smart estimator and all derived stats honor the user's own thresholds, not a global default.

10. Cohort-side diagnostics

For users analyzing the shared cohort, several additional aggregates are computed in AdminPanel.computeUserStats:

Per-user, recomputed at view time

Cohort-level views

• τ distribution — histogram (0.25 h bins), mean, median, range
• Profile aggregates — self-id, treatments, entrainment status counts
• Median |drift| (unsigned shift magnitude across cohort)
• Median σ_obs (cycle jitter across cohort)
• % with any covariate flag — coverage indicator
• Sleepless gaps — count + mean / max / total hours across cohort

Cohort-vs-individual toggle

The cohort table can be computed two ways:

1. Generic defaults (default view) — every user's Smart estimator re-run with the same thresholds (30 h post-sleepless, 6 h fragmentation). Useful for apples-to-apples comparisons.
2. Per-user settings — each user's stats computed with the thresholds they chose. Useful for "what the user actually sees."

Drill-down

Admin can load any sharing user's anonymized dataset into the normal Log / Chart / Predict / Clock / Calendar / Analysis views (read-only; the admin's own data is untouched). This is the recommended way to inspect individual users. Since 2026-06-09 the preview scores entries with the previewed user's thresholds and timezone (not the viewer's), so the numbers shown match what that user sees in their own app; an estimator-diagnostic panel can also dump the exact entry list and estimate the app computed, as ground truth for offline reproduction.

11. Data structure and sharing

A user who has opted in via "share my data" appears in the admin/research view under an anonymousId (UUID-style). Their identity-linked user_id is never surfaced to admin or research consumers. Per-session co-variates and onset/wake timestamps are exposed in full, but unlinked from any account-level identifier.

Free-text fields not exposed by the shared API:

• Sleep notes
• Wake-period notes
• Profile free-text fields (region, comorbidities_other, treatments_other)

Custom and ad-hoc tag names are a separate opt-in (per-user "share my tag content"). Users sharing data can keep tag content private.

The opt-in is reversible. Revoked sharing deletes the anonymous-share linkage immediately. The underlying sleep data remains under the user's account and is not auto-deleted.

There are three consent items in Circadia, each grantable and withdrawable independently:

• Simple sharing — for the Circadia developer and internal product improvement only. Tag-tuning, model fitting, bug investigation.
• Research-level sharing — same anonymized data as simple, plus pre-consent for future academic research collaborations under a data-use agreement (DUA), plus a structured research profile (age bucket, sex at birth, country, comorbidities, treatments).
• Publication consent — a separate, stricter additive gate that permits a user's anonymized data to be included in publicly-deposited or publication-bound datasets. Publication consent is never implied by simple or research consent; it must be granted explicitly per item. Without it, a user's data is shareable under DUA but not publishable.

Important: Research-level pre-consent does not authorize ad-hoc data transfer. As of May 2026 no academic research collaborations have been initiated. The maintainer will reach out to research-tier sharers individually before any specific collaboration begins.

The DUA / research export (the JSON bundles produced by scripts/export-circadia-shared.mjs) draws from the simple ∪ research union and is appropriate to share with a named researcher under DUA. It is not a publication snapshot — public deposits require the publication tier and a frozen, dated snapshot pipeline that does not yet exist. Do not treat simple-tier OR research-tier data as available for public deposit. See docs/circadia-data-dictionary.md for the full consent-tier table and re-identification caveats.

The reference endpoints are:

• GET /api/circadia/admin/cohort/overview — cohort-level aggregates
• GET /api/circadia/admin/cohort/shared-entries?anonymousId=… — per-user anonymized sleep log, includes covariates + zeitgebers + custom tag IDs + (when shared) tag names

12. Current cohort state (as of 2026-06-17)

All counts pulled live from the production database on the stamped date via computeCohortSummary() (the same function behind the admin overview, so these match the admin panel). (Excludes one operational admin account used for testing imports — auth_users.exclude_from_cohort_stats flag; any data logged there is test data, not user behavior.)

• 102 signups, 95 email-verified. 75 users have logged at least one sleep entry (21,670 distinct sleep onsets across the whole user base).
• 47 users sharing data (41 with research-tier consent, 6 with simple-tier only). 39 of the 47 sharing users have additionally granted publication consent. 43 sharing users have logged at least one sleep entry (4 are zero-entry sharers); 42 of those are τ-ready (≥ 3 within-gate transition pairs, 18–36 h).
• Largest individual logs in the shared cohort (counting distinct sleep onsets, not raw rows — Circadia imports from Fitbit / Sleep As Android can re-import the same file and create duplicate rows; an import-dedupe pass now keeps raw ≈ distinct):
  • 3,754 sleeps across 3,819 days (~10.5 years) — the longest log in the cohort.
  • 2,466 sleeps across 3,062 days (~8.4 years).
  • 2,018 sleeps across 2,238 days (~6.1 years).
  • 1,315 sleeps across 1,062 days (~2.9 years).
  • 1,300 sleeps across 1,601 days (~4.4 years).
  • Several other users in the 100–400-entry range; a long tail of newer users with weeks to a few months of data.
  • These long-history datasets are the strongest single contribution to the τ-estimation work.
• Shared-cohort entry totals: 15,443 raw / 15,439 distinct (only 4 duplicate-import rows remain; fully content-based re-import dedupe remains on the roadmap).
• Observation spans: 1 to 3,819 days per sharing user (median 34 days).
• Sleepless gaps logged (shared cohort): 223 events (mean 29.2 h, median 27.0 h, max 125.6 h) — distinguishable from typical awake stretches (per-user median typically ~16 h).
• Cohort skews adult sighted N24SWD and DSWPD; some ME/CFS overlap; some self-described irregular-sleep-wake / polyphasic patterns.
• Geographic distribution: US-majority but not US-exclusive. The research profile collects country plus an optional free-text region field; these fields are not exported in the shared dataset (see re-identification caveat in the data dictionary).

These numbers move daily — they describe the cohort on the date stamped in the section header. The training corpus described in §6 is a frozen earlier snapshot (2026-05-17), deliberately not updated here. Email if you want a current snapshot for a specific analysis.

13. Known limitations

1. Self-report bias. Onsets and wakes are user-entered; some users log via memory after the fact. There is no actigraphy or PSG ground-truth.
2. No DLMO. Melatonin onset is not measured; τ is inferred from onset timing alone.
3. Cycle counting in long gaps. When gap_h exceeds the upper cycle gate (Smart) or the post-sleepless threshold (Clean), the algorithm cannot infer how many body-cycles elapsed. These pairs are dropped from Smart/Clean drift math rather than imputed. Raw and |drift all| diagnostics include them with modular-wrap math.
4. Sleep-debt model is rough. The 14-day cumulative debt is a linear shortfall vs target — it does not implement the allostatic slow variable of McCauley 2009. Process S uses Borbély 1982 parameters with no individual calibration.
5. Co-variates are correlational only. The dataset does not support causal inference about, e.g., evening screens shifting tau, because exposure is self-reported and unblinded.
6. Particle filter is fit on a small corpus. Long histories shape the model disproportionately. Patterns dissimilar to anything in the training pool may take longer for fits to converge. See §6.
7. Tag-correlation panels use n ≥ 3 per group as their reporting floor. These are exploratory; they should not be treated as statistically calibrated.
8. Polyphasic / multi-sleep rhythms are not yet properly estimated. The gap-space model assumes approximately one main sleep per circadian cycle. Structurally polyphasic logs are detected (§3 step 7) and handled by withholding the headline or collapsing to one main sleep per cycle — an interim measure that can misrepresent genuinely split schedules. A periodogram-based τ estimator (chi-square periodogram or Lomb–Scargle on the sleep/wake series) for sustained multi-sleep rhythms is planned.

14. Key references

Estimator priors and α

• Borbély AA. 1982. A two process model of sleep regulation. Hum Neurobiol 1(3):195–204.
• Czeisler CA et al. 1999. Stability, precision, and near-24-hour period of the human circadian pacemaker. Science 284(5423):2177–81.
• Daan S, Beersma DGM, Borbély AA. 1984. Timing of human sleep: recovery process gated by a circadian pacemaker. Am J Physiol 246(2 Pt 2):R161–83.
• Duffy JF et al. 2011. Sex difference in the near-24-hour intrinsic period of the human circadian timing system. PNAS 108 Suppl 3:15602–8.
• Emens JS et al. 2013. Circadian misalignment in major depressive disorder. Psychiatry Research 207(1–2):37–43.
• Hayakawa T et al. 2005. Clinical analyses of sighted patients with non-24-hour sleep-wake syndrome: a study of 57 consecutively diagnosed cases. Sleep 28(8):945–52.
• Kitamura S et al. 2013. Validity of the Japanese version of the Munich ChronoType Questionnaire. Chronobiology International 30(7):918–25.

Homeostatic / debt

• McCauley P et al. 2009. A new mathematical model for the homeostatic effects of sleep loss on neurobehavioral performance. J Theor Biol 256(2):227–39.
• van Dongen HPA et al. 2003. The cumulative cost of additional wakefulness. Sleep 26(2):117–126.

Zeitgeber correlation backing (used by Analysis tab panels)

• Khalsa SB et al. 2003. A phase response curve to single bright light pulses in human subjects. J Physiol 549(Pt 3):945–52.
• Damiola F et al. 2000. Restricted feeding uncouples circadian oscillators in peripheral tissues from the central pacemaker in the suprachiasmatic nucleus. Genes Dev 14(23):2950–61.
• Stokkan KA et al. 2001. Entrainment of the circadian clock in the liver by feeding. Science 291(5503):490–3.
• Burke TM et al. 2015. Effects of caffeine on the human circadian clock in vivo and in vitro. Sci Transl Med 7(305):305ra146.
• Chang AM et al. 2015. Evening use of light-emitting eReaders negatively affects sleep, circadian timing, and next-morning alertness. PNAS 112(4):1232–7.
• Mason IC et al. 2022. Light exposure during sleep impairs cardiometabolic function. PNAS 119(12):e2113290119.
• Youngstedt SD et al. 2019. Human circadian phase-response curves for exercise. J Physiol 597(8):2253–68.
• Ebrahim IO et al. 2013. Alcohol and sleep I: effects on normal sleep. Alcohol Clin Exp Res 37(4):539–49.

Robust estimation

• Huber PJ. 1964. Robust estimation of a location parameter. Ann Math Statist 35(1):73–101. (The M-estimator used for the headline drift location, §3 step 5.)
• Hampel FR et al. 1986. Robust Statistics: The Approach Based on Influence Functions. Wiley. (MAD-based scale.)

Modeling family

• Kalman RE. 1960. A new approach to linear filtering and prediction problems. J Basic Eng 82(1):35–45. (Lineage for the state-space update.)
• Gordon NJ et al. 1993. Novel approach to nonlinear/non-Gaussian Bayesian state estimation. IEE Proc F 140(2):107–13. (Particle-filter founding paper.)
• Doucet A et al. 2001. Sequential Monte Carlo Methods in Practice. Springer. (General reference for the particle-filter approach used in Adaptive V2.)

15. Contact

Research-level anonymized data may be made available to researchers under a written data-use agreement and the product's current privacy terms. Simple-tier data is developer-only.

Contact: Dayah Dover, dayahdover@gmail.com

Please reach out to discuss before any data flows. Research-tier consent only pre-approves the act of sharing in principle; specific collaborations require a separate conversation.

If you publish using Circadia-derived data, please cite as:

Dover D. Circadia: free-running sleep tracking for N24SWD. Open alpha, 2026. https://circadia.owlandkestrel.com

A formal DOI deposit on Zenodo is planned.