Remove data/ research scripts

[chonknick]

Prunes the data/ directory — 49 model-training / labeling / prototype / benchmark one-off scripts that built the Orange models. Not used by the package or tests; the narrative is preserved in work.log.

• block_chunker.py duplicated pulpie.chunker; the 3 eval scripts that imported it now use from pulpie.chunker import ... (their DATA_DIR is kept — it points at the gitignored runtime dataset/checkpoint location on GPU boxes).
• .gitignore: data/ is now a fully-ignored runtime dir.
• No package code touched; 65 parity tests pass; eval scripts compile.

~12.4K lines removed.

[Copilot]

Pull request overview

This PR removes the repository’s data/ research / training / benchmarking scripts and artifacts (one-off model-building pipeline), while keeping the historical narrative in work.log. It also updates a small set of evaluation scripts to use the canonical chunking utilities from pulpie.chunker, and adjusts .gitignore so data/ becomes a fully runtime-only directory.

Changes:

• Delete the data/ directory contents (training, labeling, prototypes, benchmarks, and metadata files) that are not used by the package/tests.
• Update eval scripts to import chunking helpers from pulpie.chunker instead of data/block_chunker.py.
• Simplify .gitignore to ignore data/ entirely as an untracked runtime directory.

Reviewed changes

Copilot reviewed 52 out of 53 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
eval/eval_latte_large_vs_dripper.py	Switch chunking import to pulpie.chunker (removes reliance on data/ on sys.path).
eval/eval_all_methods_200.py	Switch chunking import to pulpie.chunker (removes reliance on data/ on sys.path).
eval/bench_all_methods.py	Switch chunking import to pulpie.chunker inside run_pulpie().
.gitignore	Ignore data/ as a runtime-only directory (no longer tracks any data/* files).
data/tune_gbm.py	Remove unused one-off GBM tuning script.
data/train_test_split.py	Remove unused split-eval training script.
data/train_potion_classifier.py	Remove unused potion classifier training script.
data/train_gbm.py	Remove unused GBM training/feature-selection script.
data/train_gbm_v3_cc.py	Remove unused fast CC model training script.
data/train_gbm_v2.py	Remove unused GBM v2 training/tuning script.
data/train_final.py	Remove unused final-model training script.
data/train_combined.py	Remove unused combined-dataset training script.
data/test_llm_classifier.py	Remove unused LLM labeling experiment script.
data/test_llm_classifier_v2.py	Remove unused LLM labeling experiment (v2) script.
data/test_ensemble.py	Remove unused GBM+LLM ensemble experiment script.
data/selected_features.json	Remove model-training metadata artifact (no longer tracked).
data/sample_cc.py	Remove unused Common Crawl sampling script.
data/run_label.sh	Remove unused labeling runner shell script.
data/relabel_truncated.py	Remove unused relabeling utility script.
data/prototype_text_classifier.py	Remove unused prototype text-model comparison script.
data/prototype_crf.py	Remove unused CRF prototype script.
data/prototype_crf_v2.py	Remove unused CRF prototype (v2) script.
data/make_wmb_eval_sample.py	Remove unused eval-sample generation script.
data/label_with_dripper.py	Remove unused Dripper labeling script.
data/label_cc.py	Remove unused Bedrock-based CC labeling script.
data/generate_training_data.py	Remove unused WMB training-data generation script.
data/generate_training_data_dom.py	Remove unused DOM-based training-data generation script.
data/generate_cc_training_data.py	Remove unused CC training-data generation script.
data/filter_and_audit_cc.py	Remove unused CC filtering/audit preparation script.
data/estimate_label_noise.py	Remove unused label-noise estimation script.
data/download_warcs.py	Remove unused WARC download helper script.
data/cc_relabel_urls.json	Remove unused relabel URL list artifact.
data/cascade_analysis.py	Remove unused cascade analysis script.
data/block_chunker.py	Remove duplicated chunker implementation (superseded by pulpie.chunker).
data/bench_throughput_real.py	Remove unused throughput benchmark script.
data/bench_tensorrt.py	Remove unused TensorRT benchmark script.
data/bench_onnx_cpu.py	Remove unused ONNX CPU benchmark script.
data/bench_modernbert.py	Remove unused ModernBERT vs EuroBERT benchmark script.
data/bench_gpu.py	Remove unused GPU microbenchmark script.
data/bench_gpu_local.py	Remove unused local-checkpoint GPU benchmark script.
data/bench_gpu_compile.py	Remove unused torch.compile GPU benchmark script.
data/audit_labels_llm.py	Remove unused label-auditing script.
data/analyze_errors.py	Remove unused error analysis script.
data/add_embeddings.py	Remove unused embeddings augmentation script.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.