Add initial draft spec for version ranges by pombredanne · Pull Request #11 · aboutcode-org/univers

pombredanne · 2021-10-29T16:06:09Z

Here are some of the discussions that led to the creation of this specification:

Signed-off-by: Philippe Ombredanne pombredanne@nexb.com

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Add CI config

Hritik14

There are a few things that caught my eye

Hritik14 · 2021-10-30T08:45:45Z

+- "!=": Version exclusion or inequality comparator. This means a version must
+  not be equal to the provided version and this version must be excluded from
+  the range. For example: "!=1.2.3" means that version "1.2.3" is not part of
+  the range.


this version must be excluded from the range

Should we use the term "excluded from the version-constraint" rather than version range as in an "OR of ANDs" it is not possible to exclude a version explicitly (in the version range specifier) when there are looser constraints available.
Eg: "!=1.2.3, >1.0.0" implies versions should be "greater than 1.0.0 or not equal to 1.2.3" which will consist a set of every possible version.

Eg: "!=1.2.3, >1.0.0" implies versions should be "greater than 1.0.0 or not equal to 1.2.3" which will consist a set of every possible version.

yes, but you may want to write instead "!=1.2.3 & >1.0.0" which means what you intended: ""greater than 1.0.0 and not equal to 1.2.3"

Would you have a an example of something that cannot be expressed with this notation?

Could it be easier to read a "vers" if this is in CNF, e.g., an AND of ORs?
.. possibly using a coma for AND (like is done in pypi) and a pipe for OR?

AND is the default for pypi (there is no OR) with a coma; same for debian, rpm and rust: AND is the first thing, and is a coma

same for node-semver and composer where AND is the default

With that said, the two use cases may be conflicting:

for vulnerabilities, it is common to have a the same vulnerability affect multiple branches, such as httpd 1.x and 2.x. Here a version that's within any of these ranges would be vulnerable. e.g. any woudl mean an OR first.

for dependencies, we are trying to resolve and pick one version and in the common case, all the constrainst would need to be satisfied. Some notations do not even support an OR.

So vulnerabilities, OR is first. Dependencies AND is first.

In order to adopt the most common notations in use, it may be better to use AND first as this would be more natural.
But this would mean that for vulnerabilities multiple "vers" might need to be provided, essentially one for each "branch"; it not seem completely crazy but I am not sure I like it.

For now, I think keeping an OR as a the default is best. But adapting the notation to have a comma mean "AND" may be easier since that's what a coma means elsewhere.

Hritik14 · 2021-10-30T09:04:13Z

+- If the constraint contains a star "*", validate that it is equal to "*".
+  <version-constraint> is "*". Parsing is done and no further processing is
+  needed for this ``vers``.


If there exists something after the "*" should an exception be raised than silently ignoring rest of the part ?

That's a tool implementation issue. A tool may prefer to be strict and report some error, or be lenient and ignore silently the rest. Spec-wise, what is important is what is valid or not valid. That said let me add exactly this phrase.

Hritik14 · 2021-10-30T09:38:03Z

+  - Yield the accumulated list of (comparator and version) that must apply for
+    this constraint
+
+- Finally return the <versioning-scheme> and the list of <version-constraint>


So an input like

vers: semver / <=1.2.3, >2.2 & <2.5

turns into

("semver", ["<=1.2.3", ">2.2&<2.5"]) or better ("semver", ["<=1.2.3", (">2.2","<2.5")])

Right ?
It'll be nice to have short examples of input and output when this is implemented in python (we can have other languages later probably).

I would rather keep the spec free of code snippets. But to your point, a tool could use a list of lists to represent the constraints.... I am working a rev of univers in a branch to match this spec, and this comes out a rather natural way to do model the constraints.

sbs2001 · 2021-10-31T05:34:58Z

+comparison semantics for dependencies. And for security advisories, the lack of
+portable notation for vulnerable package version ranges means that these ranges
+may be either ambiguous or hard to compute and are typically replaced by
+complete enumerations of all versions, such as the NVD CPEs.


I don't think NVD CPEs provide complete enumerations.

You are right, this is a bit more complex indeed.

there are version ranges as part of the configuration https://github.com/nexB/univers/pull/11/files#diff-dda3c48de6da2f63bfa5902e41755b27a2f1d8c186a21dcc2641c42d4f5cfaebR450

and there are daily recomputed concrete ranges available as a feed at https://nvd.nist.gov/vuln/data-feeds#cpeMatch ... which look to me as complete enumerations.

This feed contains both ranges and the enumerations. I wonder if they compute these or add them by hand?

sbs2001 · 2021-10-31T05:41:21Z

+  such as star and caret.
+
+
+Note that there is a closely related problem: the way two versions are compared


Nit: Maybe breakdown the problems into a bullet list ? This sentence is hard to follow

Tell me if this more readable with the latest push.

stevespringett · 2021-10-31T15:50:42Z

+
+The ``<versioning-scheme>`` (such as ``semver``,
+``debian``, etc.) determines:
+


To my understanding, many of the version range schemas are not standards, they are policies or conventions. What happens when those conventions change over time? For example, if in a year, Debian changes how they represent relationships between packages, will that alter the meaning of every vers:debian being used?

Additionally, what happens when one of the schemes adopts another scheme? For example, if Maven were to adopt semver in the future?

@stevespringett hey!
Thank you ++ for the feedback. That's a good point.

Even though there may not always be a spec or clear documentation there is always code in the package management tools that provide something IMHO as good as a spec.

That said, to your point we cannot trust that upstream ecosystems will not change their ways.

A few thoughts:

If they do this in a non-compatible way, this will wreak havoc on their ecosystem. Not a reassuring thought, but at least this makes these events less likely. But they can happen. See Go for instance.

When I helped cleanup the linux kernel licensing using SPDX identifiers there was a concern if the SPDX were to change. The solution was simply to incorporate the few bits we depended on in the kernel process documentation thus making the doc self standing (and this was a good decision as some key license identifiers were deprecated mid-way of this work). To make it short, we can incorporate these external specs as needed to "freeze" them in this spec.

And if they change in the future in a non-compatible, they would essentially create a new versioning scheme, and we could also create a new identifier for this versioning scheme such as vers:debian2, or vers:maven-semver

Unrelated, this makes me think that vers:semver is not a versioning scheme IMHO ... I need to update the text: node-semver is rather it as used in a package.json: e.g. the version are semver but the version ranges are node-specific.

Unrelated, this makes me think that vers:semver is not a versioning scheme IMHO ... I need to update the text: node-semver is rather it as used in a package.json: e.g. the version are semver but the version ranges are node-specific.

I evolved this and now there is rather almost a 1-to-1 between a purl type and a vers scheme

Just as another example of a convention that changed over time is the most widely used database in the world (probably billions of instances, would it be only on our phones, including the pre-smartphone area): sqlite.

In 2015 it changed its versioning scheme to semver. And with a twist to make matter worst: the downloadable archives are not named sqlite-MAJOR.MINOR.PATCHLEVEL, but sqliteMAJORMINORPATCHLEVELWITHPADDINGZEROS. For example 3.37.0 is 3370000. (It also uses its own CVS, Fossil, not git, but that's another concern.)

Clarify that extra characters in a star "*" range can be treated by tools strictly or not. Add URL. Reported-by: Hritik Vijay <hritikxx8@gmail.com> Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne · 2021-10-31T22:27:59Z

+or feeds under "configurations"::
+
+    "versionStartIncluding": "7.3.0",
+    "versionEndExcluding": "7.3.31",


While looking into https://nvd.nist.gov/vuln/data-feeds#cpeMatch there are also:

"versionStartExcluding" : "9.0.0", "versionEndIncluding" : "9.0.46",

and concrete enumeration of versions updated daily.

Reported-by: Shivam Sandbhor <shivam.sandbhor@gmail.com> Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

node-semver is a range notation, while semver is a version string syntax. Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

These recent and closely related specs (they share authors) provide another approach to documeting vulnerable ranges. Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne · 2021-11-09T23:32:20Z

I have been playing more with the code to implement this and what transpires from this is that there seems to be no such thing as a "versioning scheme" that would be different from a purl package type. Even when I thought that semver may have been something that applies to several ecosystem, each seem to have their own small variations that may mean that most code would be shared but still each package type may define as overall scheme.... so I am inclined to reuse the purl type instead of the versioning scheme specified here.

Use pipe and comma separators and drop using ampersand. This avoid a possible ambiguity when using a vers as a query string argument in a URL (such as with a Package URL qualifier). Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Improve documentation for existing versioning schems Explain why we do not use mathematical intervals Fix minor typos and improve formatting Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne · 2021-11-30T14:01:35Z

I am merging this now here to get closure and will start a new PR at https://github.com/package-url/purl-spec for this spec

Originally at aboutcode-org/univers#11 Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Add initial draft spec for version ranges

c427c8e

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne force-pushed the vers-spec branch from c1bee7a to c427c8e Compare October 29, 2021 16:06

pombredanne mentioned this pull request Oct 29, 2021

Version range package-url/vers-spec#9

Closed

pombredanne pushed a commit that referenced this pull request Oct 30, 2021

Merge pull request #11 from nexB/add-ci-config

ae44984

Add CI config

Hritik14 reviewed Oct 30, 2021

View reviewed changes

pombredanne requested a review from sbs2001 October 30, 2021 18:38

sbs2001 reviewed Oct 31, 2021

View reviewed changes

stevespringett reviewed Oct 31, 2021

View reviewed changes

pombredanne added 2 commits October 31, 2021 23:13

Merge remote-tracking branch 'origin/main' into vers-spec

8436f06

Fix typos and refine spec.

9eba23f

Clarify that extra characters in a star "*" range can be treated by tools strictly or not. Add URL. Reported-by: Hritik Vijay <hritikxx8@gmail.com> Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne commented Oct 31, 2021

View reviewed changes

pombredanne added 8 commits October 31, 2021 23:41

Add Arch Linux example

a1b69a8

Reported-by: Shivam Sandbhor <shivam.sandbhor@gmail.com> Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Clarify text on diffrerence bewteen version schemes

15eabf7

Reported-by: Shivam Sandbhor <shivam.sandbhor@gmail.com> Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Fix formatting

50a1ad7

Reported-by: Shivam Sandbhor <shivam.sandbhor@gmail.com> Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Use or later, not and later

446e6d8

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Refine documentation of NVD ranges and approach

2522a26

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Clarify node-semver vs. semver references

5b57c76

node-semver is a range notation, while semver is a version string syntax. Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Fix typo

a6afc04

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Add OSSF OSV and CVE v5 references

f081c6d

These recent and closely related specs (they share authors) provide another approach to documeting vulnerable ranges. Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Switch to use pipe and comma separators

be1dc2a

Use pipe and comma separators and drop using ampersand. This avoid a possible ambiguity when using a vers as a query string argument in a URL (such as with a Package URL qualifier). Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne mentioned this pull request Nov 22, 2021

Adopt "vers" version range spec #12

Merged

Improve doc for existing schemes

b70bbe9

Improve documentation for existing versioning schems Explain why we do not use mathematical intervals Fix minor typos and improve formatting Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne merged commit a52ea99 into main Nov 30, 2021

pombredanne deleted the vers-spec branch November 30, 2021 14:01

pombredanne added a commit to package-url/purl-spec that referenced this pull request Nov 30, 2021

Add mostly universal version range spec draft

69b6770

Originally at aboutcode-org/univers#11 Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

pombredanne mentioned this pull request Nov 30, 2021

Add mostly universal version range spec draft package-url/purl-spec#139

Merged

Hritik14 mentioned this pull request May 18, 2023

NVDImporter not populating affected packages aboutcode-org/vulnerablecode#1194

Open

		such as star and caret.


		Note that there is a closely related problem: the way two versions are compared


		The ``<versioning-scheme>`` (such as ``semver``,
		``debian``, etc.) determines:

Uh oh!

Uh oh!

Conversation

pombredanne commented Oct 29, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Hritik14 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pombredanne commented Nov 9, 2021

Uh oh!

pombredanne commented Nov 30, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

pombredanne commented Oct 29, 2021 •

edited

Loading