Issue #424: GitHub enrichment: project_type and contributor counts never reach the evaluator
Recommended: merge #425
GitHub enrichment: pass project_type and contributor counts through to the evaluator
How it breaks
The path is the model's reasoning. Each box is a citation checked against the code.
What's wrong
The classification is computed in github.py but is lost twice before it reaches the evaluator. First, the selection LLM receives project_type and contributor_count as input, but the prompt's required JSON output schema omits them.Show full explanation
The code then uses the model's returned objects directly and never re-attaches the computed fields. Second, transform.convert_github_data_to_text never prints project_type, contributor_count or author_commit_count. The rubric rules that depend on project_type therefore cannot fire, and the evaluator has to guess the type from the description. The report's claim about criteria.jinja is confirmed (line 61). The system_message.jinja claim was only partly checked: grep shows project_type on lines 24 and 71, but I did not read those lines.How to reproduce
- Run the pipeline with Ollama qwen3:8b on a GitHub profile whose top repo has more than one human contributor and at least 4 commits by the candidate.
- In github.py, print selected_projects right after json.loads (around line 403). The dicts only have the keys from the template's output schema, with no project_type or contributor_count.
- Call transform.convert_github_data_to_text on the result, or print the evaluator prompt. Each project shows only name, description, URL, stars, forks and language.
- The evaluator has no type information, so the rule at criteria.jinja line 61 (all projects 'self_project') gets applied from a guess.
- To confirm a fix, re-attach the computed fields by repo name after selection and print a Type line in convert_github_data_to_text.
Evidence
Each citation was checked against the code at e0ece80.
Project type is computed per repo (open_source if contributor_count > 1, else self_project). The counts are stored at lines 259-260.
The fields are sent to the selection LLM as input, so they exist before selection.
The LLM-returned project objects are used as-is. Nothing re-attaches project_type or contributor_count from the original data. Only the fallback path at line 424 keeps them.
The output schema (lines 73-98) has author_commit_count and total_commit_count but no project_type or contributor_count, so the model is told to return objects without them.
The evaluator text for each project (lines 895-902) prints name, description, URL, stars, forks and language. It never prints project_type, contributor_count or author_commit_count.
The rubric depends on a project_type value that the evaluator text never contains.
Pull requests
Verdicts are judged from the diff, not run. Merge status is a real git check against e0ece80.
`resolve_selected_projects` rebuilds each selected project from the fetched data (keeping project_type and contributor_count), and transform.py now prints Project Type, Contributors and Author Commits.
`restore_repo_metadata()` re-attaches project_type/contributor_count/commit counts right after the selection JSON is parsed, and `convert_github_data_to_text` now prints a Type line with contributors and author commits, so both losses are closed.
It swaps the LLM's project copies for authoritative `valid_by_name` records (restoring the fields), but transform.py is unchanged so the evaluator text still never prints them.
It pins up to three open_source projects taken from `projects_data` (fields intact), but the remaining projects are still the LLM's objects and transform.py still doesn't print the fields.
It adds Project Type, Contributors and Author Commits to the evaluator text, but github.py is untouched, so LLM-selected projects still lack those fields and would print as N/A.
20 PRs that don't fix itShow
It adds retry and rate-limit handling to the GitHub API fetch and leaves selection and evaluator text unchanged.
It only adds API retry, logging and error-handling fixes in github.py; the selection output and transform.py are unchanged.
It only changes cache-file encoding and error handling.
It changes how project_type is computed and extends the output schema with PR fields, but the schema still omits project_type/contributor_count and transform.py prints neither.
It tidies logging and the `fetch_repo_contributors` return path and does not touch selection output or evaluator text.
It changes console output wording and an undefined variable and does nothing about field loss or evaluator text.
It adds a None guard for malformed JSON and does not re-attach any fields or change evaluator text.
It adds a separate open-source-contributions pipeline and leaves the selection output and the per-project text without project_type or contributor counts.
It fixes the undefined `response` NameError and the contributors return type, not the field loss.
It changes fork stats to come from the parent repo and does not address field loss after selection or the evaluator text.
It adds Repo URL and Live Demo lines to the text and upstream fork details, but still prints no project_type, contributor_count or author commits.
It adds Codeforces/LeetCode scoring and fork handling, and the project text block still omits the classification fields.
Its whitelist only filters by name and still returns the LLM's echoed objects without project_type, and transform.py is untouched.
It adds an external merged-PR section to the text and relaxes the rubric rule, but leaves project_type unprinted and still lost after selection.
It changes fork metadata handling only; selection output and the project text lines are unchanged.
It rewrites prompt text, and nothing visible re-attaches fields in code or prints them in transform.py.
It adds only the live demo URL line to the text and not project_type, contributor_count or author_commit_count.
It adds a resume-side open source contributions section and rubric text, and neither the lost fields nor the project text is changed.
It only makes the fallback paths return the filtered `projects_data`, which already kept the fields, and leaves the success path and the evaluator text unchanged.
It sanitizes `===` delimiters in candidate-controlled fields and adds no classification fields to the text or selection output.
What this packet does not prove
- Citations are checked against the code. The explanation that links them, the order of the chain, and any claim about outside systems (APIs, libraries) are the model's reasoning and are not checked.
- PR verdicts are judged from the diff, not run. No PR was applied or tested.
- PRs were shortlisted because they change a cited line or mention the issue. A fix in another file, or a PR whose line numbers have drifted, is missed, so this is not proof that no other PR fixes it.