What we found, and how sure we are.
SpaceSift asks one question of three kinds of sky data: when a search says “found it”, how often is it right, and what does it miss? Each investigation below states its data, method and baseline, counts its errors in both directions, names its limits, and shows how to rerun it.
WHAT WE BUILT
An injection-recovery pipeline that has run 72,440 planet searches with known answers; a star classifier matched against 123 real spectra; a live ranking of new asteroid candidates.
WHAT'S NEW
Measuring failure, not just success: completeness maps across stellar variability, with each cleaning method's threshold calibrated on noise-only stars, and a pre-registered test on real Kepler variables (predictions committed before the run).
THE EVIDENCE
On pulsating stars the standard filter found 0 of 106 planets; removing the rhythm first found 105. The star classifier's median error is 1 subclass (old method: 5). The asteroid model ranks better than digest2 but doesn't yet beat it where it matters most.
RERUN IT
Every experiment records its code commit, configuration and random seed. Commands for each are below; the code is on GitHub.
When does a transit search miss a planet that is really there?
Quiet stars: chance of finding a planet
Variable stars: what each cleaning method returns
ERRORS IN BOTH DIRECTIONS
- False alarms (false positives): each method's detection threshold is set so only 1% of searches on noise-only stars beat it (1,800 and 10,800 null searches). But on variable stars where cleaning fails, 60–100% of planet-free searches beat the threshold: the false-alarm rate is only valid on quiet stars.
- Misses (false negatives): half of planets at SNR 10.85 are missed on quiet stars, and 4% of strong signals (SNR > 16) are still missed. On two pulsating Kepler stars in SS-0001, all 100 planted planets were missed.
LIMITATIONS
- Simulated pulsators are two constant sine waves, the ideal case for prewhitening; noise is white. SS-0002B tests real stars, with predictions committed before the run.
- BLS measures a box-shaped depth, so recovered transits are ~15% shallower than injected (explained in SS-0001-depth).
- The in-browser Transit Lab is a JavaScript port checked against the Python pipeline; one demo run is an example, not a result.
Full methodology: data, method, baselines, detailed results and how to rerun
DATA
200 quiet Kepler stars (SS-0001; NASA MAST light curves) and 820 simulated stars spanning pulsation and starspot periods and amplitudes (SS-0002A). 316 real variable Kepler stars for the pre-registered test (SS-0002B, running).
METHOD
Plant planets of known size and orbit, clean the light curve, run a box least-squares (BLS) search, and grade each trial: found, wrong period, the star's own signal, or below threshold. Completeness is fit against signal-to-noise.
BASELINE
The standard pipeline: a 1-day sliding biweight filter, compared against shorter and longer windows, no cleaning, and prewhitening (subtracting the star's rhythms) before filtering.
Rerun these experiments
cd spacesift pip install -e . -c constraints-ci.txt spacesift run configs/SS-0001.yaml # 10,000 injections on 200 quiet Kepler stars spacesift run configs/SS-0001-null.yaml # noise-only searches that set the 1% threshold spacesift analyze experiments/SS-0001 --null experiments/SS-0001-null --far 0.01 spacesift run configs/SS-0002A.yaml && spacesift run configs/SS-0002A-null.yaml spacesift analyze-grid experiments/SS-0002A --null experiments/SS-0002A-null --name far1pct spacesift replay experiments/SS-0001 --star 1868918 --trial 3 # re-run and plot one trial
Each record stores its git commit, package versions, configuration and seed. Full records: Transit Lab → Experiments.
Can a star's type be read reliably from any spectrum someone uploads?
Error per star
Robustness to bad calibration
ERRORS IN BOTH DIRECTIONS
- Too hot: faint SDSS “A0” and “B9” stars come out as B stars (5 and 6.5 subclasses off). They are probably blue horizontal-branch stars, old and metal-poor, which the library lacks.
- Too cool: rare here; the G2 star comes out F8 (too hot by 4) and K7 comes out K5.
LIMITATIONS
- Small test set (11 stars), one per class; “truth” is itself SDSS's template fit, not an independent measurement.
- The library is thin on red dwarfs and has no blue horizontal-branch stars. Temperatures come from the matched type (±a few hundred K); metal content is not measured.
- Pictures of spectra are matched on line positions only, since drawings don't show true line strengths.
Full methodology: data, method, baselines, detailed results and how to rerun
DATA
Templates: 123 real stellar spectra (SDSS spEigenStar-55734, mostly the Indo-US library). Test set: the highest signal-to-noise SDSS DR17 star of each class, O to M (11 stars), chosen by a fixed query before tuning.
METHOD
Fit every template times a smooth curve (absorbing flux calibration and dust), weighting the features astronomers classify by. Errors are counted in subclasses against SDSS's own pipeline type (10 subclasses = one class).
BASELINES
(1) The lab's first classifier: a simplified physics model (blackbody plus ~40 lines). (2) “Temperature from color”: the same templates using only the overall continuum shape.
Rerun this evaluation
node spectra/eval/eval_sdss.mjs # downloads 11 SDSS spectra (~2 MB) on first run node --test spectra/test/ # parsers, template library, picture reader, known stars
Results file: spectra/eval/results.json. Templates are rebuilt from the SDSS file with spectra/tools/build_templates.mjs.
Which unconfirmed candidates are real near-Earth objects?
WHAT THE PROBABILITY MEANS, AND DOESN'T
- P(real NEO) is the chance that a candidate on the Minor Planet Center's confirmation page turns out to be a genuine near-Earth object, rather than junk (artifacts, satellites, stars, lost objects) or an ordinary main-belt asteroid.
- It is not an impact probability. Nothing in this model estimates whether anything will hit Earth. Impact probabilities for known objects come from NASA JPL's Sentry system and are shown separately on the dashboard.
CORRECTION · 11 OCTOBER 2026
The first retrain on 11 October reported a ROC-AUC of 0.97. That number was wrong: some inputs described how each candidate changed on the confirmation page until it left, and leaving the page is the outcome itself, so the model was partly reading the answer. Those inputs are now excluded, and the model only sees what is known when a candidate is first posted. Every number on this page and the dashboard comes from the corrected evaluation.
Ranking quality on newer candidates
Junk excluded while keeping real NEOs
ERRORS IN BOTH DIRECTIONS
LIMITATIONS
- Labels are noisy: “not confirmed” includes some real NEOs that were simply lost; “designated other” is assumed not to be an NEO.
- Few junk examples: most negatives are ordinary main-belt asteroids, and the MPC deletes tracklets of unconfirmed objects, so the hardest junk can't be backfilled.
- One test period (a few hundred candidates over a few weeks), so numbers carry real error bars and may shift as the sky and the surveys change.
- Probabilities are not calibrated (class-balanced training); treat them as a ranking.
Full methodology: data, method, baselines, detailed results and how to rerun
DATA
Twice-daily captures of the MPC confirmation page since July 2026 (3,326 resolved outcomes). Labels from the MPC's Previous-NEOCP list: designated NEO = 1; not confirmed, not real, artificial, star or main-belt = 0.
METHOD
Gradient-boosted trees (LightGBM DART, about 60 inputs: digest2 orbit-class scores, motion, brightness, sky position and observatory, all as known at first posting). Tested on a time split: trained on older candidates, scored once on newer ones. Random-fold cross-validation is computed too but not quoted, because it mixes old and new candidates and flatters the model.
BASELINE
The digest2 score alone, the standard tool observers already use to judge whether a candidate looks like a near-Earth object.
Rerun training and evaluation
# needs digest2 (github.com/Smithsonian/digest2) built into ./d2 pip install numpy scikit-learn lightgbm python src/features.py ./d2 && python src/dataset.py python src/train.py # time-split test, held-out report, model -> models/vetting_model.pkl python src/predict.py ./d2 && python src/make_dashboard.py
Runs automatically every Monday (GitHub Actions: neocp-retrain); results in outputs/training_results.json.