[RFC] Advisory search api: access to unprocessed data

[Hritik14]

VulnerableCode currently is one of the best data sources out there for vulnerabilities, although since our upstream data has a lot of inconsistencies/versioning issues/advisory quality issues etc, it makes it looks as though vulnerablecode is not exhaustive.

I'm proposing a VulnerableCode search engine that could search based on advisory's original text instead of purl or vulnerability/advisory. The output would contain all the unstructured advisories and the structured relationship among packages and vulnerabilities. In a sense, this will be Google for VulnerableCode.

For example, the package grocy has no matches in current vulnerablecode deployment, possibly because there was some problem parsing purl from the advisory. Although, we have multiple advisories fetched on grocy (12 in fact).
This is in no manner authoritative data source as VulnerableCode's default search since mere presence of a package name in the advisory does not prove that the package is vulnerable, but it does signal interest in the advisory if there's an interest in the package mentioned.

I've created a mock app that demonstrates this. For the above example, this is how it looks: https://advisory-search.hritik.sh/?q=grocy

[Hritik14]

For search, considering the following:

• Typesense / Opensearch: An additional overhead but with more control, typo and delimiter support in search
• Current implementation with pg_trgm on original advisory text and purls: no additional overhead, with limited search capability.

Typesense / opensearch will need data imports for themselves, this can be done in an improver. Whereas pg_trgm introduces a slight overhead at the time of imports. We could run pg_trgm as an improver at the end of the imports. I'm more inclined towards this approach.

This will further help us build a language model that would train on existing advisory -> vers relationship, to infer advisory -> vers relationship for unprocessed data.

flowchart TD
    A[Upstream advisory sources] --> B[VulnerableCode importers]
    B --> C[(VulnerableCode DB)]

    C --> D1[Structured data<br/>PURLs, packages, vulnerabilities]
    C --> D2[Unstructured data<br/>Original advisory text]

    D2 --> E[Advisory Search Index]

    E --> F{Search backend}

    F -->|Preferred| G[Postgres pg_trgm<br/>on advisory text + PURLs]
    F -->|Alternative| H[Typesense / OpenSearch<br/>separate index + import pipeline]

    G --> I[Search API]
    H --> I

    J[User searches package name<br/>example: grocy] --> I

    I --> K[Find advisories mentioning package]
    K --> L[Return results]

    L --> M[Unstructured advisories<br/>original text]
    L --> N[Structured relationships<br/>packages ↔ vulnerabilities]

    M --> O[Signal interesting advisory]
    N --> O

    O --> P[Not authoritative<br/>mention ≠ confirmed vulnerable]

    P --> Q[Future: train LLM/model based on existing advisory <-> vers relationship]
    Q --> R[Infer advisory → affected versions<br/>for unprocessed advisories]
    R --> C

Loading