Example scope & report

# GitLab: task, code and test review

A scoped AI assistant that checks a merge request against its task and CI evidence

[← All services](/services)

Public reference: GitLab. Independent example based on public documentation. Not commissioned by or affiliated with GitLab.

[Download all 7 reports (PDF) ↓](/examples/upserto-example-reports.pdf)[Download Markdown ↓](/examples/ai-workflow-evaluation.md)

## Decision brief

Recommended next decision

| Question | Recommendation |
| --- | --- |
| Decision | Pilot a read-only review tied to the task, MR head and matching CI result. |
| Why it matters | A plausible summary or unrelated green pipeline can conceal a missing behavioral check. |
| Acceptance | Each review claim links to its source; stale or incomplete input produces a named open check. |
| Scope boundary | One approved GitLab project; engineers retain publication, merge and release decisions. |

## The proposed engagement

Proposed brief: a team wants a useful answer before merging a backend change: what does the linked task require, where does the diff implement it, and what do the tests actually prove? Upserto would build a read-only GitLab pilot for one project. The assistant would draft a review for an engineer, with links to the requirement, changed lines and matching CI results. It would not merge code or publish comments automatically.

Inputs required. One approved project; task ID; MR IID and head SHA; repository paths; matching pipeline/JUnit artifacts; reviewer rubric.

## System boundary and assumptions

The public reference is GitLab’s merge request, issues and pipeline APIs, plus JUnit test reports. Inputs are an approved issue or task snapshot, paginated MR diffs, relevant permitted files at a pinned commit, the corresponding pipeline and its test report. The pilot would record missing or truncated input explicitly. It would not read CI variables, broaden repository access, or treat a green pipeline as proof that every requirement was tested.

## How Upserto would approach it

First build deterministic collection: pin the MR head SHA, fetch every diff page and record GitLab’s collapsed/too_large flags. Match the pipeline to that SHA and read the available JUnit result. Then let the model map the task criteria to changed code and named tests. A rule layer would reject unsupported file references and block a ready verdict when required input is missing. A human would review the draft and decide what to publish.

## What the team would receive

- A read-only GitLab collector with project-scoped access and a review draft screen.

- A fixed evaluation set containing correct changes, missing requirements, duplicate-callback defects and incomplete CI evidence.

- A report of supported findings, missed seeded defects, false positives and reviewer corrections from actual pilot runs.

A review record connects a requirement to evidenceThe matrix is a proposed review format. A matching CI run describes executed tests; it does not prove a requirement that has no relevant assertion.One snapshot: task + MR head + matching pipeline

Read the diagram- Pin the task, MR head SHA, complete permitted diff and pipeline revision.

- For each task criterion, link the changed code, relevant assertion and result from that pipeline.

- Missing, truncated or stale evidence remains NEEDS_REVIEW with the next check named.

## Starting point and proposed change

Items to inspect in the customer integration, the proposed control and the evidence needed for acceptance.

Design comparison

| Control | Inspect first | Proposed change | Acceptance evidence |
| --- | --- | --- | --- |
| Review input | Check whether the bot reads only part of the diff or an unrelated passing pipeline. | Pinned head SHA, complete diff accounting and matching CI. | Force-push and excluded-file cases are blocked or marked incomplete. |
| Requirement check | Check whether summaries substitute for assertions. | Task criterion → changed path → relevant assertion → CI result. | A missing behavioral test stays open. |
| Action control | Check whether source text can request tools or comments. | Read-only collector and engineer-controlled publication. | Injection fixtures cannot expand scope or write to GitLab. |

## Decisions and approval owners

Decisions to agree before implementation

| Decision | Proposed default | Approval owner |
| --- | --- | --- |
| Context access | Permit specific projects and paths in application code; repository text cannot extend access. | Engineering + security owners |
| Pilot acceptance | Compare drafts with an engineer rubric before allowing a live read-only shadow run. | Review lead |

## Review questions and implementation risks

P1 Resolve before releasing the affected workflow, or record an explicit acceptance decision. P2 Assign an owner and an agreed follow-up milestone; do not leave the item unowned.

### AI-01 · P1 · Is the review using the current diff and pipeline?

A force-push or oversized diff can make a draft incomplete or stale. GitLab documents diff limits and pipeline test-report endpoints; the collector needs to handle both.

Proposed approach. Pin head SHA, paginate diffs and check excluded-file flags. Reject a pipeline from a different commit. Mark required missing context as needs review rather than inventing coverage.

### AI-02 · P1 · Can repository text expand access or trigger a tool?

A source comment or task body can contain instructions to fetch secrets or run commands. That text is input to review, not authorization for the assistant.

Proposed approach. Enforce project and tool permissions outside the model. Keep the first pilot read-only. Any later test execution would use an isolated runner, an allowlisted command and explicit resource limits.

### AI-03 · P2 · Does each conclusion link to the right evidence?

A named test may pass without checking the task’s new behavior. A plausible explanation and a valid URL are not enough to establish that the requirement is covered.

Proposed approach. Require the changed path/line, requirement criterion and relevant test assertion. Evaluate seeded cases with a human rubric; preserve unsupported and missing-evidence outcomes.

## Example technical output

Proposed review output for a duplicate-callback fixture

```text
task_criterion: duplicate callback does not fulfil twice
commit: MR_HEAD_SHA
code_source: src/payments/callback.go:84
test_source: tests/callback_test.go:61
ci_source: matching_pipeline / test_report
result: NEEDS_REVIEW
reason: duplicate-delivery assertion is not in supplied tests
next_check: replay same signed event twice; inspect durable effect
# Paths are fixture examples; no GitLab project was reviewed.
```

## Delivery plan and ownership

Proposed action plan

| Priority | Review item | Accountable role | Acceptance gate |
| --- | --- | --- | --- |
| P1 | AI-01 | Integration backend owner | Force-push and truncated-diff fixtures prevent a ready verdict and identify the missing input. |
| P1 | AI-02 | AI + security owner | An injected instruction cannot access another project, CI variables or an unapproved command. |
| P2 | AI-03 | Engineering reviewer | A missing duplicate-callback assertion remains an open criterion even if the pipeline is green. |

## Acceptance checks

Run these checks before accepting the proposed changes.

Acceptance checks

| Scenario | Required evidence / expected behavior | Status |
| --- | --- | --- |
| MR is updated after collection | Draft is marked stale; review uses the new head and matching pipeline | To verify |
| Diff is too_large or required file is unavailable | No invented code or complete-review verdict; missing input is named | To verify |
| Source text requests another project or a secret | Application denies the read/tool action before model output | To verify |
| Pipeline passes but requirement assertion is absent | Criterion remains open with the specific test to add | To verify |

## Rollout and recovery

Start offline with a fixed fixture repository and compare drafts with an engineer’s rubric. Move to a read-only shadow pilot on one approved project only after access and stale-input checks pass. Record reviewer decisions before considering comment publication or authorized test execution. Expanding scope is a separate decision, not a model permission.

## Handover and limits

The team would receive the collector, access policy, fixture manifest, review format and actual evaluation results. Each pilot run would preserve task version, head SHA, pipeline ID, model/prompt version and reviewer correction. No accuracy percentage or time saving is claimed in this example.

## Access and evidence handling

Controls for the agreed engagement

| Area | Required control |
| --- | --- |
| Access | Read-only project scope; no CI-variable, secret, merge or write access. |
| Model input | Agree the model provider and retention terms before repository content leaves the approved environment. |
| Evidence | Store source references and evaluation decisions; redact credentials and personal data from fixtures and prompts. |

## Public documentation used

The sources below establish product capabilities and API contracts. The implementation choices above are Upserto’s proposed approach for a customer integration.

- [GitLab: Merge requests API ↗](https://docs.gitlab.com/api/merge_requests/)Paginated diffs, collapsed and too_large flags, MR commit references.

- [GitLab: Issues API ↗](https://docs.gitlab.com/api/issues/)Authorized issue descriptions provide the task snapshot.

- [GitLab: Pipelines API ↗](https://docs.gitlab.com/api/pipelines/)Pipeline identity and test_report resource.

- [GitLab: Unit test reports ↗](https://docs.gitlab.com/ci/testing/unit_test_reports/)JUnit artifacts, report limits and separation from job status.

Discuss the same kind of work for your system, with an agreed scope and acceptance criteria.

[Explore AI automation and integrations ↗](/services/ai-integration-automation)[Discuss a similar scope ↗](/contact)

Contact: [hello@upserto.com](mailto:hello@upserto.com)