Hi everyone,<\/p>\n
I\u2019m a student working on a research project on risk-aware GitHub PR review. I\u2019m doing the project mostly on my own and I don\u2019t have access to a research lab, large compute budget, or people who can manually annotate thousands of PRs, so I\u2019m trying to find a practical approach that I can actually finish.<\/p>\n
The idea is to take a GitHub PR and estimate four types of risk:<\/p>\n
- \n
- \n
Bug \/ correctness<\/p>\n<\/li>\n
- \n
Security<\/p>\n<\/li>\n
- \n
Compatibility<\/p>\n<\/li>\n
- \n
Cross-system \/ integration<\/p>\n<\/li>\n<\/ol>\n
The architecture I\u2019m working with has four separate risk models. They share the same PR characteristics\/features, but each risk model has its own historical data, prior, and evidence.<\/p>\n
My main problem is the historical data needed for those priors.<\/p>\n
At first, I looked for a single PR dataset where I could get reliable PR-level outcomes for all four risks. I couldn\u2019t find one.<\/p>\n
I then tried looking for separate datasets for each individual risk model. I thought this would solve the problem, but I keep finding datasets where the labels look relevant at first but don’t actually represent the outcome I need.<\/p>\n
For example, SEVRA-plus looked very promising for the Security model:<\/p>\n
https:\/\/huggingface.co\/datasets\/RedAI4Code\/SEVRA-plus<\/a><\/p>\n
It contains security-related PR examples with vulnerability\/CWE information, but the malicious PRs are deliberately constructed by reversing real CVE security fixes. So although they are useful for evaluating or studying security vulnerabilities, I don’t think I can use their class distribution directly as a real-world prior for ordinary GitHub PRs.<\/p>\n
I\u2019ve run into similar issues with other datasets:<\/p>\n
– some label the linked issue rather than the PR implementation,<\/p>\n
– some label review comments rather than actual PR outcomes,<\/p>\n
– some contain artificially constructed vulnerable\/failing PRs,<\/p>\n
– some only give merge\/close status, which doesn\u2019t tell me whether the PR itself was buggy, vulnerable, incompatible, etc.<\/p>\n
The distinction between the issue and the PR is especially important for what I am trying to do.<\/p>\n
For example, imagine a maintainer opens a security issue, someone creates a PR to fix it, but the PR implementation itself contains a correctness bug and gets rejected. For my problem, I need to know the nature of the PR, not simply inherit the security label from the original issue.<\/p>\n
Similarly, a PR could be opened to fix a small bug, get merged, and then later cause a compatibility problem. Again, I care about what happened because of the PR implementation, not just why the PR was originally opened.<\/p>\n
Because I couldn’t find a dataset that directly gives me what I need, I tried a practical compromise.<\/p>\n
I took 96 real PRs from SWE-Review-Chat, filtered them for sufficient evidence, and used an LLM to annotate the four risk states from the information available in the PR record, such as the description, review discussion, diff context, tests, and lifecycle information.<\/p>\n
I\u2019m treating these as weak\/model-assisted labels rather than independent ground truth.<\/p>\n
The resulting usable outcomes are:<\/p>\n
Bug:<\/p>\n
30 present \/ 9 absent<\/p>\n
Security:<\/p>\n
1 present \/ 7 absent<\/p>\n
Compatibility:<\/p>\n
5 present \/ 12 absent<\/p>\n
Cross-system:<\/p>\n
4 present \/ 8 absent<\/p>\n
So now I feel like I\u2019ve hit a wall.<\/p>\n
I can keep searching for datasets, but so far I haven’t found anything that solves the underlying problem. I also don’t have the resources to manually establish reliable ground truth for thousands of PRs.<\/p>\n
I\u2019m therefore looking for advice on “how I should move forward from here”.<\/p>\n
Should I continue with the small real dataset I have and explicitly model the uncertainty caused by the sparse risks?<\/p>\n
Should I rely on LLM-assisted annotations of real PRs as a practical research compromise, or is there a better low-resource approach that I am missing?<\/p>\n
Or is there a completely different way of constructing the historical priors that would make more sense for this problem?<\/p>\n
I\u2019m not looking for a perfect dataset at this point. I\u2019m mainly looking for a practical and defensible way to move forward given that I\u2019m a student doing this alone with limited time and resources.<\/p>\n
If anyone has worked on GitHub PR datasets, Mining Software Repositories, empirical software engineering, code-review research, or Bayesian risk modelling, I would really appreciate any advice on what you would do in this situation.<\/p>\n
Thanks!<\/p>\n<\/div>\n
submitted by \/u\/Accomplished-Fun4629 <\/a>
[link]<\/a><\/span> [comments]<\/a><\/span><\/p>
