Issue #5: Analysis Exception: Cannot resolve column name "label" among ()

interviewstreet/spark-stratifier · commit 4692dc6 · investigation verified · 2026-10-11 · investigation cost $0.05

No open PR fixes this and merges clean

4/4
citations verified in code
2 → 0
open PRs shortlisted by cited lines
0 fix
0 merge clean · 0 partly
$0.05
investigation cost

How it breaks

The path is the model's reasoning. Each box is a citation checked against the code.

1stratifier.py:47 ✓
fit() calls stratify_data on raw dataset
2stratifier.py:28 ✓
Hardcoded dataset['label'] lookup for positives
3stratifier.py:29 ✓
Same hardcoded lookup for negatives
4README.md:7 ✓
Docs omit the column-name requirement
where it shows

What's wrong

StratifiedCrossValidator.stratify_data hardcodes the label column name 'label' and ignores the estimator/evaluator's configured labelCol. If the dataset has no column named "label", dataset['label'] raises AnalysisException. The reporter's dataset has columns such as 'features' but no 'label'.
Show full explanationThe report's claim is confirmed: the column must be named 'label'. The README only documents 0/1 label values and never says the column must be called 'label'.
Model confidence: high.

How to reproduce

  1. Create a Spark DataFrame with a 'features' column and a binary target column named something other than 'label', e.g. 'target' with values 0/1.
  2. Build a StratifiedCrossValidator with any estimator (e.g. LogisticRegression(labelCol='target')), a paramGrid, an evaluator and numFolds=3.
  3. Call cv.fit(df).
  4. fit calls _fit, which calls stratify_data, which runs dataset['label'] and raises AnalysisException: Cannot resolve column name "label".
  5. Rename the column to 'label' and the error goes away.

Evidence

Each citation was checked against the code at 4692dc6.

spark_stratifier/stratifier.py3 citations
line 28
verified in code
passes = dataset[dataset['label'] == 1]

Hardcoded 'label' column lookup; this raises AnalysisException when the column is absent.

line 29
verified in code
fails = dataset[dataset['label'] == 0]

Same hardcoded 'label' lookup for the negative class.

line 47
verified in code
stratified_data = self.stratify_data(dataset)

_fit calls stratify_data on the raw input dataset before any use of the estimator's labelCol, so the failure surfaces from fit().

README.md1 citation
line 7
verified in code
Currently, the stratified cross validator works with binary classification problems using labels `0` and `1`.

The docs mention only the label values, not that the column must be named 'label'.

Pull requests

No open PR changes a cited line or mentions the issue.

What this packet does not prove

  • Citations are checked against the code. The explanation that links them, the order of the chain, and any claim about outside systems (APIs, libraries) are the model's reasoning and are not checked.
  • PR verdicts are judged from the diff, not run. No PR was applied or tested.
  • PRs were shortlisted because they change a cited line or mention the issue. A fix in another file, or a PR whose line numbers have drifted, is missed, so this is not proof that no other PR fixes it.