PSA: open source security considerations in the era of LLMs
I was thinking a bit about the bugs I found in the Piefed codebase yesterday. And these led to an emergency fix by the dev that's now been implemented. codeberg.org/rimu/pyfedi/commi…
And what the real takeaway for me here is that the whole dynamic of how we approach security has now changed in ways most people don't appreciate.
It used to take a lot of effort to find exploits in software projects because you’d have to spend a long time to familiarize yourself with the codebase, then comb through the code looking for mistakes that could be exploited. And to even do that, you'd need a good understanding of the protocols and specifications used by the application.
You basically had to be a domain expert with a deep understanding of how the application works. A random person looking at the source code would have little chance of finding any non trivial problems or figuring out how to actually exploit them.
And in that world, doing a private disclosure made a lot of sense because you did a lot of hard work to find it, and it wasn’t easy for somebody to replicate. This was valuable and dangerous knowledge that had to be communicated in a responsible fashion.
But now, anybody can throw an LLM at the code and it’ll sniff out vulnerabilities and even explain step by step how to exploit these security holes. So, the information itself isn’t really that valuable anymore. If I can throw an LLM at the code and find these problems in a few minutes, anybody else can do the same thing too.
I'm not a Python developer, I don't have any deep knowledge of the Python stack used in Piefed, and on my own, I'd have zero chance of finding these exploits. But once the LLM identifies them, it's very easy for me to verify that they are indeed real exploits, and to realize how they can be used maliciously.
The attacker doesn't even need to have any deep knowledge of programming because the LLM can guide them through the exploit step by step.
Open source projects are particularly vulnerable here since anybody can just grab the source and throw an LLM at it to see if it can find exploits.
I'd argue that raising awareness that this is now the state of things is really important, and I would suggest that running an LLM against the code is minimal due diligence at this point.
Obviously, the LLM vulnerability check is not exhaustive, and if it doesn't find anything that doesn't mean there aren't exploits in the code. But anything it does find should absolutely be checked by the developers.
People should be aware that we're now living in the world where the bar for finding vulnerabilities is far lower than it used to be. And that means security must be taken far more seriously.
security tweaks · 093a466935
pyfedi - Project background: https://join.piefed.social. Flagship instance: https://piefed.socialCodeberg.org
davel
in reply to ☆ Yσɠƚԋσʂ ☆ • • •P ≠ NP scores another win.
unsolved problem in computer science about time complexity
Contributors to Wikimedia projects (Wikimedia Foundation, Inc.)☆ Yσɠƚԋσʂ ☆
in reply to davel • • •mathemachristian[he]
in reply to ☆ Yσɠƚԋσʂ ☆ • • •is this something others agree on as well? Is this how the industry is changing? I don't know what you mean by "takeaway", what are you basing this lesson on?
I however still think it irresponsible. Even if it is very easy, the fact that it wasn't discovered or abused meant no one was looking, but posting it publically made piefed a target. It also meant people didn't have time to assess and react but everything had to happen now and suddenly and a fix had to be rushed.
To be blunt, since this comes on the heels of rimu making a huge ass of himse
... Show more...is this something others agree on as well? Is this how the industry is changing? I don't know what you mean by "takeaway", what are you basing this lesson on?
I however still think it irresponsible. Even if it is very easy, the fact that it wasn't discovered or abused meant no one was looking, but posting it publically made piefed a target. It also meant people didn't have time to assess and react but everything had to happen now and suddenly and a fix had to be rushed.
To be blunt, since this comes on the heels of rimu making a huge ass of himself, it looks like (whether true or not i cant know) a dunk on rimu with little consideration to how others might be affected or who was put at risk. Especially to those who don't know the ins and outs of responsible disclosure practices. And your first response being "If the lead developer was a decent human being, I probably would’ve handled this differently." kind of makes it hard to shake this feeling.
☆ Yσɠƚԋσʂ ☆
in reply to mathemachristian[he] • • •I'm basing this on the demonstration I provided yesterday and the fallout we see from it. Piefed had to go down for maintenance as a result of these vulnerabilities lemmy.ml/post/47393443 and I linked to the dev having to apply the fix in the post.
These were real security issues that anybody could find and exploit with very little effort or programming knowledge. We also don't know that hasn't been discovered or abused until I surfaced it. If there is a vulnerability that's trivial to find and exploit, it should very much be assumed that people are doing so.
Rimu being an ass was a contributing factor that motivated me to look at the codebase. But having reflected on it, I stand by my position that raising awareness of the issue and warning people federating with piefed is far more important in this kind of scenario. If this was a difficult to find exploit that nobody would be reasonably expected to have access to in the wild, then the calculus would be to notify the maintainer and
... Show more...I'm basing this on the demonstration I provided yesterday and the fallout we see from it. Piefed had to go down for maintenance as a result of these vulnerabilities lemmy.ml/post/47393443 and I linked to the dev having to apply the fix in the post.
These were real security issues that anybody could find and exploit with very little effort or programming knowledge. We also don't know that hasn't been discovered or abused until I surfaced it. If there is a vulnerability that's trivial to find and exploit, it should very much be assumed that people are doing so.
Rimu being an ass was a contributing factor that motivated me to look at the codebase. But having reflected on it, I stand by my position that raising awareness of the issue and warning people federating with piefed is far more important in this kind of scenario. If this was a difficult to find exploit that nobody would be reasonably expected to have access to in the wild, then the calculus would be to notify the maintainer and let them fix it quietly on their own time. However, when the likelihood is that this is something that people would already be exploiting because the bar for discovering the flaw is so low, warning the public becomes a bigger concern.
SatansMaggotyCumFart
in reply to mathemachristian[he] • • •f3nyx
in reply to ☆ Yσɠƚԋσʂ ☆ • • •your conclusion is definitely correct. the good thing is that if these LLMs can be used offensively to find exploits, they can also be used defensively to find (and potentially even fix) the same ones.
I'm still of the opinion that AI is a tool that amplifies the abilities of the user, so there will be a mismatch of capability depending on that.
with that argument in mind, open source has the potential to be a huge receiver of benefit, if these tools are used to aid development in that way.
overall, what I think that means (what I hope it means) is that we'll see a much more broad interest in learning secure development as AI research progresses and these tools become more widespread in usage.
☆ Yσɠƚԋσʂ ☆
in reply to f3nyx • • •I think so as well. These tools work both ways, and a project maintainer is in a much better position to use them effectively than a random attacker by virtue of having a deep understanding of what the code is doing. LLM is just a tool that helps you dig through a huge volumes of information, like a large codebase, and surface things that might be of interest. You still need a human to understand what it surfaces and to take meaningful action.
Hopefully this kind of stuff does get people thinking about security a bit more, and how LLMs can be used to help surface issues.
Salamander
in reply to ☆ Yσɠƚԋσʂ ☆ • • •Private disclosure still makes sense to me when you add LLMs into the mix. It is possible that an LLM outputs some plausible-sounding story that over-estimates the actual risk and impact of the exploit. If this story is publicly announced to people who use the software but are not capable of assessing these risks themselves, this can easily have a negative unnecessary consequence - for example, people may bring their server down until an expert or developer provides an assessment or fix.
This is a source of noise, and I don't agree that this is better than private disclosure. Via public disclosure one is applying a lot of pressure to the developer(s) to prioritize whatever is being disclosed, which may not always be the nicest thing to do, especially if the impact is not as significant as th
... Show more...Private disclosure still makes sense to me when you add LLMs into the mix. It is possible that an LLM outputs some plausible-sounding story that over-estimates the actual risk and impact of the exploit. If this story is publicly announced to people who use the software but are not capable of assessing these risks themselves, this can easily have a negative unnecessary consequence - for example, people may bring their server down until an expert or developer provides an assessment or fix.
This is a source of noise, and I don't agree that this is better than private disclosure. Via public disclosure one is applying a lot of pressure to the developer(s) to prioritize whatever is being disclosed, which may not always be the nicest thing to do, especially if the impact is not as significant as the LLM suggests. This may not have been what happened in your case (I don't know the details), but I am thinking about the idea of the average person disclosing publicly LLM-discovered vulnerabilities.
☆ Yσɠƚԋσʂ ☆
in reply to Salamander • • •I'm not talking about a scenario of a hypothetical exploit here. I'm talking about a concrete scenario where somebody finds an exploit and verifies it. In that case, people operating the software need to be aware of the vulnerability in the application they are running. Since the exploit is very easy to find, it should be assumed that malicious people would have found it as well.
You're arguing against a case where you have an unverified exploit that LLM might've hallucinated. This is not the case I'm describing. And this provably did not happen in my case as is clearly evidenced by the fix the dev had to make in their server.
Salamander
in reply to ☆ Yσɠƚԋσʂ ☆ • • •Definitely, disclosing (either private or publicly) a vulnerability that has been verified is significantly better than passing on the LLM output without verifying it.
It isn't my intention to argue one specific case. What I think is that normalizing public disclosure of LLM-inspired vulnerabilities would lead to a wide distribution of cases. We would have some successful cases like yours, and also some cases of the type that I have mentioned. Increase in disclosures will raise the noise floor, and the fact that it is done publicly adds the additional pressure that I mentioned.
I see your point, but I don't agree that the benefit of public awareness offsets the increase in noise. This disagreement isn't rooted in aspects that we can objectively quantify though - we just have a difference of opinion here.
☆ Yσɠƚԋσʂ ☆
in reply to Salamander • • •ghost_laptop
in reply to ☆ Yσɠƚԋσʂ ☆ • • •☆ Yσɠƚԋσʂ ☆
in reply to ghost_laptop • • •Goferking0
in reply to ghost_laptop • • •Stamets
in reply to Goferking0 • • •Eugene V. Debs' Ghost
in reply to Stamets • • •Yeah same, I personally vouched for him, and even he has helped with people making things up about our instance before. I don't get why he defends PieFed like this daily. Even ignoring and omitting important details.
My respect for him has gone down a lot.
Goferking0
in reply to ☆ Yσɠƚԋσʂ ☆ • • •It's also been happening in the Linux kernel with copy fail/dirty frag and the related exploits found once people started looking deeper.
Wild times ahead in good and terrible ways
patatas
in reply to ☆ Yσɠƚԋσʂ ☆ • • •sorry - you used a non-deterministic system to find a bug, and you're saying there's no duty to disclose responsibly because someone else could maybe use a similar system to maybe find that bug?
Edit: elsewhere you say this
While it's possible that someone else has found it, this is not a reason to hand it to malicious people on a platter.
☆ Yσɠƚԋσʂ ☆
in reply to patatas • • •patatas
in reply to ☆ Yσɠƚԋσʂ ☆ • • •☆ Yσɠƚԋσʂ ☆
in reply to patatas • • •patatas
in reply to ☆ Yσɠƚԋσʂ ☆ • • •Sorry, but I do not accept this argument.
Again, you do not know that it was already known to any malicious actor, but by disclosing it publicly, guaranteed that it would be.
This looks very much like a hostile act, doubly so given the overall context. I say this as someone who generally agrees with you on a lot of stuff!
☆ Yσɠƚԋσʂ ☆
in reply to patatas • • •wickedrando
in reply to ☆ Yσɠƚԋσʂ ☆ • • •Nutomic
in reply to ☆ Yσɠƚԋσʂ ☆ • • •☆ Yσɠƚԋσʂ ☆
in reply to Nutomic • • •rimu
in reply to Nutomic • • •Stamets
in reply to Nutomic • • •Brother, Rimu is an enemy of the fediverse. Why? Because he re-created reddit. He basically reinvented shadow banning, controls the narrative of his instance, actively keeps spreading misinformation about other instances and users, will actively suppress certain instances using his platform and lashes out against criticism like a petty tyrant.
I'm tired of this nonsense that "We're all here to leave reddit!" Yes. We are here to leave reddit. So stop giving Spez 2.0 a pass.
Nutomic
in reply to Stamets • • •iByteABit
in reply to Nutomic • • •Worth pointing out here that this is the mentality of the oh so authoritarian and scary communist devs of Lemmy, while the "anti-authoritarian" one is making sure to insert as much as possible of his personal bias hardcoded into the project and then lash out at random people for calling him out on it.
Sorry to continue the infighting a bit but the contradiction here is just glaring. I appreciate your stance and the Fediverse would never be what it is without people like you
Stamets
in reply to Nutomic • • •Invalid comparison. Rimu is not a simple admin, he is a dev of the leading alternate competitor to Lemmy and is personally hardcoding his bias into the project itself.
You want to try that again but with a more accurate comparison?
Nutomic
in reply to Stamets • • •Stamets
in reply to Nutomic • • •Nutomic
in reply to Stamets • • •Stamets
in reply to Nutomic • • •dan
in reply to ☆ Yσɠƚԋσʂ ☆ • • •On the other hand, this means that they should end up more secure. Open-source projects get far, far more vulnerability testing than closed-source projects. Security holes in closed-source systems can exist for years at a time, which is how things like the Pegasus malware work (undisclosed security holes).
☆ Yσɠƚԋσʂ ☆
in reply to dan • • •Stamets
in reply to ☆ Yσɠƚԋσʂ ☆ • • •☆ Yσɠƚԋσʂ ☆
in reply to Stamets • • •