Debian oval links are not working

[Hritik14]

The endpoints at
https://github.com/nexB/vulnerablecode/blob/eec05bb0f796d743e408a1b402df8abfc8344669/vulnerabilities/importers/debian_oval.py#L64-L67

are giving 403.
This is why documentation tests are failing (invalid link): https://github.com/nexB/vulnerablecode/actions/runs/6040203878/job/16390554826

I talked to them over IRC:

20:07 <hritik> Hello, we're working on an open source vulnerabilites database and have been using debain oval sources. Although the endpoints at https://www.debian.org/security/oval/oval-definitions-{release}.xml have started giving 403. Has something changed ?
20:07 <jcristau> yes
20:08 <jcristau> tools using these endpoints were DoSing our infrastructure
20:10 <hritik> jcristau: We try to be as polite as possible. Our project is under development at github.com/nexB/vulnerablecode/ Is there an alternative data source that we can use for oval definitions ?
20:12 <jcristau> hritik: i don't know.  though please 1) set appropriate user-agent, 2) download oval-definitions-{release}.xml.bz2
20:12 <kaiorafael[m]> hritik: you can build this, but I had to do this https://matrix.to/#/!pvJfJpKAVgcLkInkLv:matrix.org/$169357400414234YFyZi:matrix.org?via=matrix.org&via=kde.org&via=ru-matrix.org
20:13 <kaiorafael[m]> Not sure if there is a better way to put these files altogether
20:14 --> HritikVijay[m] (~hritikvma@) has joined #debian-security
20:15 <hritik> kaiorafael[m]: I could not see your message. It says "Tried to load a specific point in this room's timeline, but you do not have permission to view the message in question"
20:16 <-- jandrusk (~jra@d-65-60-194-135.oh.cpe.breezeline.net) has quit (Ping timeout: 480 seconds)
20:17 <kaiorafael[m]> Sorry, pasting here for reference:... (full message at <https://matrix.org/_matrix/media/v3/download/matrix.org/AQEbLRsSQzwMxEEGXJqxRbLZ>)
20:18 <hritik> jcristau: thank you. We will work on setting an appropriate user agent from now on.
20:19 <hritik> jcristau: https://www.debian.org/security/oval/oval-definitions-wheezy.xml.bz2 Also gives 403. it appears that all endpoints under /security/oval/* are 403 :(
20:19 <kaiorafael[m]> My request for the Debian Security folks is that these steps needs to be bit clear so anyone can build their own Oval. Maybe, there is a better way
20:19 <kaiorafael[m]> s/needs/need/
20:22 <hritik> kaiorafael[m]: Cloning (and generate.py) the repositories on regular basis sounds like a cumbersome task. It will also waste the bandwidth for the clones (from both sides). Is this approach a recommendation from Debian Security ?
20:24 <kaiorafael[m]> I don't know. I am a Debian user only and I have no authority to speak on their behalf, but I found this way to generate it. Let's see if they can bring a better solution.
20:25 --> jandrusk (~jra@d-65-60-194-135.oh.cpe.breezeline.net) has joined #debian-security
20:27 <hritik> Is there an issue tracker/mailing list I should raise this at ? Chats on IRC might get lost.
20:27 <jcristau> hritik: thanks.  we'll probably make the bz2 files available again soonish.
20:29 <hritik> jcristau: thank you. we'll check back in soonish ;)

TL;DR

Following are the 2 suggestions from debian security

1. Use .bz2 links
2. Set proper useragent

Related:

• Vulnerablecode useragent #410

[pombredanne]

@Hritik14 Thanks! There is something tickling me though... the oval data seems to be entirely generated from some other data in a Git repo?
If this is the case, may be we would be better served by using the original data?

[pombredanne]

Based on chats with @Hritik14 @pabs3 and others on the #debian-security IRC and here is the gist of things:

• the ultimate origin of debian data is https://salsa.debian.org/security-tracker-team/security-tracker/tree/master/data ... everything else is derived from that data.

• various scripts there generate a sqlite database

• ... and also https://security-tracker.debian.org/tracker/data/json

• https://salsa.debian.org/webmaster-team/webwml/-/blob/master/english/security/oval/generate.py generates the Oval data

• The docs are at https://security-team.debian.org/security_tracker.html

• @pabs3 reported that we have an issue: the hard-coded list of Debian codenames is out of date, it is missing bookworm https://raw.githubusercontent.com/nexB/vulnerablecode/main/vulnerabilities/importers/debian_oval.py

[pombredanne]

@Hritik14 therefore we should consider going further up the food chain rather than using oval data IMHO