I’m Stuck Finding Usable Historical Data For A Bayesian PR Risk Model — Looking For Advice On How To Proceed

Hi everyone,

I’m a student working on a research project on risk-aware GitHub PR review. I’m doing the project mostly on my own and I don’t have access to a research lab, large compute budget, or people who can manually annotate thousands of PRs, so I’m trying to find a practical approach that I can actually finish.

The idea is to take a GitHub PR and estimate four types of risk:

1. Bug / correctness

2. Security

3. Compatibility

4. Cross-system / integration

The architecture I’m working with has four separate risk models. They share the same PR characteristics/features, but each risk model has its own historical data, prior, and evidence.

My main problem is the historical data needed for those priors.

At first, I looked for a single PR dataset where I could get reliable PR-level outcomes for all four risks. I couldn’t find one.

I then tried looking for separate datasets for each individual risk model. I thought this would solve the problem, but I keep finding datasets where the labels look relevant at first but don’t actually represent the outcome I need.

For example, SEVRA-plus looked very promising for the Security model:

https://huggingface.co/datasets/RedAI4Code/SEVRA-plus

It contains security-related PR examples with vulnerability/CWE information, but the malicious PRs are deliberately constructed by reversing real CVE security fixes. So although they are useful for evaluating or studying security vulnerabilities, I don’t think I can use their class distribution directly as a real-world prior for ordinary GitHub PRs.

I’ve run into similar issues with other datasets:

– some label the linked issue rather than the PR implementation,

– some label review comments rather than actual PR outcomes,

– some contain artificially constructed vulnerable/failing PRs,

– some only give merge/close status, which doesn’t tell me whether the PR itself was buggy, vulnerable, incompatible, etc.

The distinction between the issue and the PR is especially important for what I am trying to do.

For example, imagine a maintainer opens a security issue, someone creates a PR to fix it, but the PR implementation itself contains a correctness bug and gets rejected. For my problem, I need to know the nature of the PR, not simply inherit the security label from the original issue.

Similarly, a PR could be opened to fix a small bug, get merged, and then later cause a compatibility problem. Again, I care about what happened because of the PR implementation, not just why the PR was originally opened.

Because I couldn’t find a dataset that directly gives me what I need, I tried a practical compromise.

I took 96 real PRs from SWE-Review-Chat, filtered them for sufficient evidence, and used an LLM to annotate the four risk states from the information available in the PR record, such as the description, review discussion, diff context, tests, and lifecycle information.

I’m treating these as weak/model-assisted labels rather than independent ground truth.

The resulting usable outcomes are:

Bug:

30 present / 9 absent

Security:

1 present / 7 absent

Compatibility:

5 present / 12 absent

Cross-system:

4 present / 8 absent

So now I feel like I’ve hit a wall.

I can keep searching for datasets, but so far I haven’t found anything that solves the underlying problem. I also don’t have the resources to manually establish reliable ground truth for thousands of PRs.

I’m therefore looking for advice on “how I should move forward from here”.

Should I continue with the small real dataset I have and explicitly model the uncertainty caused by the sparse risks?

Should I rely on LLM-assisted annotations of real PRs as a practical research compromise, or is there a better low-resource approach that I am missing?

Or is there a completely different way of constructing the historical priors that would make more sense for this problem?

I’m not looking for a perfect dataset at this point. I’m mainly looking for a practical and defensible way to move forward given that I’m a student doing this alone with limited time and resources.

If anyone has worked on GitHub PR datasets, Mining Software Repositories, empirical software engineering, code-review research, or Bayesian risk modelling, I would really appreciate any advice on what you would do in this situation.

Thanks!