Quality gaps & open issues

A 13-agent quality swarm scored the tool 83/100 with 7 of 8 dimensions clean; its one confirmed high-severity defect (exact SPG-number lookups) was fixed the same day, along with two follow-on edge cases. These are the residual, openly-tracked gaps. Faculty and staff are invited to flag more below.

Open items

Reranker weakens SPG (numbered-policy) retrieval

mediumunder evaluation

The Jina reranker improves prose retrieval slightly but hurts the terse, numbered SPG policies (MRR .738 → .558 in the sweep). It is currently mitigated by an exact-number pin and source-balanced retrieval; the reranking strategy (drop it, or rank-fuse it) is being decided from a clean comparison before changing the safety-validated abstention floor.

Cross-source conflict flagging is retrieval-bounded

low-mediummitigated

The assistant flags a school-vs-university conflict only when both sides are actually retrieved. Source-balanced retrieval now guarantees each selected corpus a slot, but reranker precision within a single corpus can still surface the wrong passage for a specific provision (e.g. a particular deadline).

Corpus freshness vs. live university policy

mediumplanned

Cross-source alignment was verified against the February-2024 Provost handbook snapshot; several cited SPGs were revised within the last five months. The fix is a scheduled re-scrape of the live Provost/SPG corpora with an alignment diff — the scraping and alignment machinery already exists.

Open access — authentication deferred

informationalplanned

The site is currently open for testing (no sign-in). A dedicated umich.edu Google sign-in is preserved in the codebase and will be re-enabled; until then, comments below are labeled unverified because the commenter's identity is self-supplied.

Exact-number lookup edge cases

lowfixed

Number-anchored lookups initially missed family numbers (‘201.11’ → 201.11-0/201.11-1) and glued prefixes (‘spg201.92’). Both were fixed and verified this session; listed here for transparency.

Comments

Identity is self-supplied until the umich.edu Google sign-in is added (authentication is deferred), so every comment below is labeled unverified. Until sign-in lands, comments are visible to anyone who can reach this page.

No comments yet — be the first to flag an issue.

Add a comment

Note. This page lists residual, openly-tracked gaps and invites feedback. Comment bodies are stored and displayed verbatim as plain text; no HTML is rendered from user input.