From c427c8ea23e286f0e927a08a43f884c254fe6d67 Mon Sep 17 00:00:00 2001 From: Philippe Ombredanne Date: Fri, 29 Oct 2021 18:05:37 +0200 Subject: [PATCH 01/12] Add initial draft spec for version ranges Signed-off-by: Philippe Ombredanne --- VERSION-RANGE-SPEC.rst | 537 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 537 insertions(+) create mode 100644 VERSION-RANGE-SPEC.rst diff --git a/VERSION-RANGE-SPEC.rst b/VERSION-RANGE-SPEC.rst new file mode 100644 index 00000000..25ea40de --- /dev/null +++ b/VERSION-RANGE-SPEC.rst @@ -0,0 +1,537 @@ +====================================================== +vers: a mostly universal version range specifier +====================================================== + +This specification is a new syntax for dependency and vulnerable version ranges. + + +Context +-------- + +Software package version ranges and version constraints are essential: + +- When resolving the dependencies of a package to express which subset of the + versions are supported. For instance a dependency or requirement statement + such as "I require package foo, version 2.0 and later versions" defines a + range of acceptable foo versions. + +- When stating that a known vulnerability or bug affects a range of package + versions. For instance a security advisory such as "vulnerability 123 affects + package bar, version 3.1 and version 4.2 but not version 5" defines a range of + vulnerable "bar" package versions. + +Version ranges can be replaced by a list enumerating all the versions of +interest. But in practice, all the versions may not yet exist when defining an +open version range such as "v2.0 or later". + +Therefore, a version range is a necessary, compact and practical way to +reference multiple versions rather than listing all the versions. + + +Problem +-------- + +Several version range notations exist and have evolved separately to serve the +specific needs of each package ecosystem, vulnerability databases and tools. + +There is no (mostly) universal notation for version ranges and there is no +universal way to compare two versions, even though the concepts that exist in +most version range notations are similar. + +Each package type or ecosystem may define their own ranges notation and version +comparison semantics for dependencies. And for security advisories, the lack of +portable notation for vulnerable package version ranges means that these ranges +may be either ambiguous or hard to compute and are typically replaced by +complete enumerations of all versions, such as the NVD CPEs. + +Because of this, expressing and resolving a version range is often a complex, or +error prone task. + +In particular the need for common notation for version has emerged based on the +usage of Package URLs referencing vulnerable package version ranges such as in +vulnerability databases like VulnerableCode. + +To better understand the problem, here are some of the notations and conventions +in use: + +- Rubygems strongly suggest using semver but does not enforce it + https://guides.rubygems.org/patterns/#semantic-versioning + +- node-semver ranges are used in npms https://github.com/npm/node-semver#ranges + with semantics specific to semver and node and npm. + +- Dart pub versioning scheme is similar to node-semver and the documentation at + https://dart.dev/tools/pub/versioning provides a comprehensive coverage of the + topic of versioning. But version resolution is using its own algorithm. + +- Python uses its own version ranges notation https://www.python.org/dev/peps/pep-0440/ + +- Debian and Ubuntu use their own notation + https://www.debian.org/doc/debian-policy/ch-relationships.html + +- RPM distros use their own range notation + https://rpm-software-management.github.io/rpm/manual/dependencies.html + +- Perl CPAN define its own version range notation similar to this specification: + https://metacpan.org/pod/CPAN::Meta::Spec#Version-Ranges + +- Apache Maven and NuGet use math intervals notation + https://en.wikipedia.org/wiki/Interval_(mathematics) + - Apache Maven http://maven.apache.org/enforcer/enforcer-rules/versionRanges.html + - NuGet https://docs.microsoft.com/en-us/nuget/concepts/package-versioning#version-ranges + +- Gentoo and Alpine Linux use comparison operators similar to this specification: + - Gentoo https://wiki.gentoo.org/wiki/Version_specifier + - Alpine linux https://gitlab.alpinelinux.org/alpine/apk-tools/-/blob/master/src/version.c + +- Go modules https://golang.org/ref/mod#versions use semver versions with + specific version resolution algorithms. + +- Haskell Package Versioning Policy https://pvp.haskell.org/ provides a notation + similar to this specification based on a modified semver with extra notations + such as star and caret. + + +Note that there is a closely related problem: the way two versions are compared +as equal, lesser or greater is complex and package types may have evolved their +own peculiar version comparison procedure. Semver is a prominent specification +in this domain but this is just one of the many versioning schemes. Semver, +Debian, RPM, PyPI, Rubygems, and Composer all have their own subtly different +ways of comparing two versions even though they may seem highly similar at first +glance. + + +Solution +--------- + +A solution to the many version range syntaxes is to design a new notation to +unify them all with: + +- a mostly universal and minimalist notation to express the version range + notations from many different package types and ecosystems. + +- the package type-specific definitions to normalize existing range expressions + to this common notation . + +- the designation of which algorithm or procedure to use when comparing two + versions such that it is possible to resolve if a version is within or + outside of a version range. + +We call this solution "version range specifier" or "vers" and it is described +in this document. + + +Version range specifier +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +A version range specifier (aka. "vers") is a URI string using the ``vers`` +URI-scheme with this syntax:: + + vers:/,,... + +For example to define a set of versions that contains either version ``1.2.3``, +or any versions greater than or equal to ``2.0.0`` but less than ``5.0.0`` using +the `semver` versioning scheme, the version range specifier will be:: + + vers:semver/1.2.3,>=2.0.0&<5.0.0` + +Each ```` in the comma-separated list is either a simple +constraint such as:: + + + +Or a composite constraint grouping multiple ```` joined by +an ampersand such as: + + &... + +The comma is a logical `OR` and the ampersand is a logical `AND`. + +A version range specifier is therefore an "OR of ANDs". This is also called a +"disjunctive normal form" in boolean logic. +See https://en.wikipedia.org/wiki/Disjunctive_normal_form for details. + +``vers`` is the URI-scheme and is an acronym for "VErsion Range Specifier". It +has been selected because it is short, obviously about version and available +for a future formal registration for this URI-scheme at the IANA registry. + + +```` +------------------------ + +The ```` (such as ``semver``, +``debian``, etc.) determines: + +- the specific notation and conventions used for a version string encoded in + this scheme. Versioning schemes often specify a version segments separator and + the meaning of each version segments, such as [major.minor.patch] in semver. + +- how two versions compare as greater or lesser to determine if a version is + within or outside a range. + +- how a versioning scheme-specific range notation can be transformed in the + ``vers`` notation defined here. + +Note: the ```` does not define which symbols to use to +compare versions, and how to compose these as this is the substance of this +specification. + +The ```` is followed by a slash "/". + +Then there are one or more ```` separated by a comma "," +meaning that **any** of these constraints must be satisfied for a version to be +resolved as within this range specifier. + + +```` +---------------------------- + +Each ```` of this comma-separated list can be either a +single constraint or a list of constraints separated by an ampersand "&" as in +``1.2.3,>=2.0.0&<5.0.0``. + +Multiple ```` combined with an ampersand means that **all** +these constraints must be satisfied for a version to be resolved as within this +range. + +Each simple version constraint has this syntax:: + + + +The ```` is one of these comparison operators: + +- "=": Version equality comparator. It is the default and implied if not + present and means that a version must be equal to the provided version. + For example: "=1.2.3". It must be omitted in the canonical representation. + Equality is based on the equality of two lower-cased and normalized version + strings and is not versioning scheme-specific. + +- "!=": Version exclusion or inequality comparator. This means a version must + not be equal to the provided version and this version must be excluded from + the range. For example: "!=1.2.3" means that version "1.2.3" is not part of + the range. + +- "<", "<=": Less than or less-or-equal version comparators points to all + versions less than or equal to the provided version. For example "<=1.2.3" + means less than or equal to "1.2.3". + +- ">", ">=": Greater than or greater-or-equal version comparators points to + all versions greater than or equal to the provided version. For example + ">=1.2.3" means greater than or equal to "1.2.3". + +- The way two version strings are compared using these comparators is defined + by the ````. + +- The structure and meaning of a version string such as "1.2.3" is defined by + the ````. For instance, ``semver`` defines three + dot-separated segments name major, minor and patch. + +- The special star "*" ```` matches any version. This star + constraint must be used **alone** in a version range specifier, exclusive of + any other constraint. For example "vers:debian/\*" resolves to any version. + + +Examples +~~~~~~~~~ + +TODO. + + +Normalized or canonical representation +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +- A version range specifier contains only printable ASCII letters, digits and + punctuation. + +- Spaces are not significant and are removed in the canonical form. For example + "!=1.2.3" and " ! = 1.2. 3" are equivalent. And so are "1.2.3 & < = 2.0.0" and + "1.2.3&<=2.0.0" + +- A version range specifier is case-insensitive and lowercase in canonical form. + +- The ordering of multiple ```` in a range specifier is not + significant. The canonical ordering is by sorting these by lexicographical + order applied with this approach: + + - first to each sub-list of ampersand-separated ````. + - then to the list of comma-separated ````. + +- A version in a ```` can only contain printable ASCII + characters excluding the special characters used as separators and comparators + ``><=!,&*``. If it contains special characters (which should be rare in + practice) the version string in a constraint must be quoted using the URL + quoting rules. + + +Using version range specifiers +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +``vers`` primary usage is to test if a version is within or outside a range. + +An input version is within a range specifier if it matches or satisfies **any** +of the ranges in this spec. To match a range, an input version must match +**all** the version constraints of a range. Otherwise, the input version is +outside of the range specifier. + +Some important usages derived from this primary usage include: + +- **Resolving a version range specifier to a list of concrete versions.** + In this case, the input is the set of known versions of a package (typically + obtained from some package repository or registry). Each version is then + tested individually to check if it is within or outside the ranges. For + example, this can be used to determine which existing package versions are + affected by a known vulnerability if they match the vulnerability version + range specifier. + +- **Selecting one of several versions that are within a range.** + For example, package management tools need to determine what are the dependent + package versions of a package instance based the acceptable version ranges + provided with each dependency. This usually requires deploying heuristics and + algorithms (possibly complex such as sat solvers) that are versioning-scheme- + and tool-specific and outside of the scope for this specification; ``vers`` + scope is limited to provide an input to the resolution process. + + +Parsing version range specifiers +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To parse a version range specifier string: + +- Remove all spaces and tabs. +- Start from left, and split once on colon ":". +- The left hand side is the URI-scheme that must be lowercase. + - Verify that the URI-scheme value is ``vers``. +- The right hand side is the specifier. + +- Split the specifier from left once on colon ":". +- The left hand side is the that must be lowercase. +- The right hand side is the constraints. + +- If the constraint contains a star "*", validate that it is equal to "*". + is "*". Parsing is done and no further processing is + needed for this ``vers``. + +- Split the ranges on comma ",". The result is a list of + strings. +- Remove any empty strings from the list of . + +- For each in this list: + + - Split on ampersand "&". The result is a sub-list of . + - Remove any empty strings from this sub-list. + + - For each in this sub-list: + + - Split the comparator and version starting from left + based on the start of the : + + - If it starts with "=", split it once on "=". + - If it starts with "!=", split it once on "!=". + - If it starts with "<=", split it once on "<=". + - If it starts with ">=", split it once on ">=". + - If it starts with "<", split it once on "<". + - If it starts with ">", split it once on ">". + - Otherwise if there is no split, the comparator is "=" (default) and the + version is the string. + + - After the split, the left hand side is the comparator, and the right hand + is the version. + + - Validate that the version is not empty. + + - If the version contains a percent "%" character, apply URL quoting rules + to unquote this string. + + - Yield the comparator and version for this constraint + + - Yield the accumulated list of (comparator and version) that must apply for + this constraint + +- Finally return the and the list of + + +Notes and caveats +~~~~~~~~~~~~~~~~~~~ + +- Comparing versions from two different versioning schemes is unspecified. Even + though there may be some similarities between the `semver` version of an npm + and the `debian` version of its Debian packaging, these similarities are + specific to each versioning scheme. Tools may report an error in these + cases. + +- Schemes are related to Package URL types (aka. ecosystems) in the sense that + each Package URL type has one versioning scheme, but multiple types can reuse + the same versioning scheme (such as semver). + +- Some package types may use the same versioning scheme such as semver to + compare two versions, but may not use the same notation for version ranges. + + +Some of the known versioning schemes +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +TODO: add details on how to convert to and from ``vers`` for a given versioning +scheme and package type. + +- ``debian``: Debian and Ubuntu https://www.debian.org/doc/debian-policy/ch-relationships.html + The comparators are <<, <=, =, >= and >>. + +- ``rpm``: RPM distros https://rpm-software-management.github.io/rpm/manual/dependencies.html + The version comparison routine of rmpvercmp is also used by archlinux Pacman. + +- ``rubygems``: Rubygems https://guides.rubygems.org/patterns/#semantic-versioning + which is almost but not exactly semver. + +- ``semver``: node-semver as used for npm https://github.com/npm/node-semver#ranges + It is also used by Rust: https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html + and several other package types. `composer` may need its own scheme as this is + not strictly semver. + +- ``python``: Python https://www.python.org/dev/peps/pep-0440/ + +- ``perl``: Perl https://perlmaven.com/how-to-compare-version-numbers-in-perl-and-for-cpan-modules + +- ``go``: Go modules https://golang.org/ref/mod#versions use semver versions + with a specific minimum version resolution algorithm. + +- ``maven``: Apache Maven http://maven.apache.org/enforcer/enforcer-rules/versionRanges.html + +- ``nuget``: NuGet https://docs.microsoft.com/en-us/nuget/concepts/package-versioning#version-ranges + Note that Apache Maven and NuGet are following a similar approach with a + math-derived intervals syntax as in https://en.wikipedia.org/wiki/Interval_(mathematics) + +- ``gentoo``: Gentoo https://wiki.gentoo.org/wiki/Version_specifier + +- ``alpine``: Alpine linux https://gitlab.alpinelinux.org/alpine/apk-tools/-/blob/master/src/version.c + (which might be using Gentoo conventions) + +- ``generic``: a generic version comparison algorithm (which is TBD, likely a + split on punctuation and dealing with digit vs. strings comparisons, like is + done in libversion) + + + +Implementations +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +- Python: https://github.com/nexB/univers +- Yours! + + +Why not reuse existing version range notations? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Most existing version range notations are tied to a specific version string +syntax and are therefore not readily applicable to other contexts. For example, +the use of elements such as tilde and caret ranges in Ruby or Dart notations +implies that a certain structure exists in the version string (semver or semver- +like). The inclusion of these additional comparators is a result of the history +and evolution in a given package ecosystem to address specific needs. + +In practice, the unified and reduced set of comparators and syntax defined for +``vers`` has been designed such that all these notations can be converted to a +``vers`` and back from a ``vers`` to the original notation. + +In contrast, this would not be possible with existing notations. For instance, +the Python notation may not work with npm versions and reciprocally. + +There are likely to be a few rare cases where round tripping from and to +``vers`` may not be possible, and in anycase round tripping to and from ``vers`` +may only produce equivalent results and not strictly the same original strings. + +Another issue with existing version range notations is that, they are primarily +meant to be used for dependency constraints and may not readily be reused for +vulnerable ranges. In particular, a vulnerability may exist for multiple +"branches" of a given package (such as Django 2, 3 and 4) which may not be +possible to express with a notation designed exclusively for dependent versions +resolution. + + +Why not use the NVD CPE Ranges? +############################### + +https://nvd.nist.gov/vuln/vulnerability-detail-pages#divRange + +The NVD CPE Match String Range is a complex specification that goes well beyond +version ranges and is used to match comprehensive configurations across multiple +products and version ranges. The notation is to uses these fields in the CVE API +or feeds under "configurations":: + + "versionStartIncluding": "7.3.0", + "versionEndExcluding": "7.3.31", + +This notation is limited and is compatible can be expressed in ``vers`` but this +does not provide a compact notation like ``vers`` does. + + +Why not use node-semver ranges? +############################### + +https://github.com/npm/node-semver#ranges + +The node semver is very similar to this spec (this is also an OR of ANDs) but it +has a few practical issues: + +- The space means "AND" and significant whitespace in a single string makes + normalization more complicated and may be a source of confusion or errors. The + explicit ampersand used an "AND" operator specified here improves clarity. + +- There is no negation "!=" operator meaning that some version constraints are + difficult to express and require combining < and > comparators. For instance + stating that a vulnerability affects babel 6.2 or later but not babel 7.0 is + possible but difficult. + +- The advanced range syntax has grown to be rather complex using hyphen, stars, + carets and tilde constructs are all tied to the JavaScript and npm ways of + handling versions in their specific ecosystem and furthermore are bound to the + semver semantics. These are not readily reusable elsewhere. And these multiple + comparators and modifiers make the grammar and parsing more complex. + + +Why not use Python pep-0440 ranges? +##################################### + +https://www.python.org/dev/peps/pep-0440/#version-specifiers + +The Python pep-0440 "Version Identification and Dependency Specification" +provides a comprehensive specification for Python package versioning and a +notation for "version specifiers" to express dependencies version constraints. + +This specification is mostly compatible with and is similar to this ``vers`` +spec, but has a richer notation with some aspects specific to the version used +only in the Python ecosystem. + +- In particular pep-0440 uses tilde, triple equal and wildcard star operators + that are specific to how two Python versions are compared. + +- The comma separator between constraints is a logical "AND" rather than an + "OR". The "OR" does not exist in the syntax making some version ranges + harder to express, in particular for vulnerabilities that may affect several + exact versions or version ranges such as when there are multiple release + branches that exist in parallel. For instance a statement such as: Django 1.2 + or later, or Django 2.2 or later or Django 3.2 or later is difficult to + express without an "OR" logic. + + +Why not use Rubygems requirements notation? +############################################################## + +https://guides.rubygems.org/patterns/#declaring-dependencies + +The rubygems specification suggests but does not enforce using semver. It is +similar to this spec's operators with the addition of the "~>" aka. pessimistic +operator or tilde-wakka which is similar to the "tilde" used in node-semver and +implies semver versioning. This makes the notation impractical to reuse +in places that do not use the same semver-like semantics. + + +References +~~~~~~~~~~~~~~~~~~~~ + +Here are some of the discussions that led to the creation of this specification: + +- https://github.com/package-url/purl-spec/issues/66 +- https://github.com/package-url/purl-spec/issues/84 +- https://github.com/package-url/purl-spec/pull/93 +- https://github.com/nexB/vulnerablecode/issues/119 +- https://github.com/nexB/vulnerablecode/issues/140 \ No newline at end of file From 9eba23f7d6c12171cd5d574c2d0d387ba2d34007 Mon Sep 17 00:00:00 2001 From: Philippe Ombredanne Date: Sun, 31 Oct 2021 23:19:59 +0100 Subject: [PATCH 02/12] Fix typos and refine spec. Clarify that extra characters in a star "*" range can be treated by tools strictly or not. Add URL. Reported-by: Hritik Vijay Signed-off-by: Philippe Ombredanne --- VERSION-RANGE-SPEC.rst | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/VERSION-RANGE-SPEC.rst b/VERSION-RANGE-SPEC.rst index 25ea40de..84f4ce84 100644 --- a/VERSION-RANGE-SPEC.rst +++ b/VERSION-RANGE-SPEC.rst @@ -49,7 +49,8 @@ error prone task. In particular the need for common notation for version has emerged based on the usage of Package URLs referencing vulnerable package version ranges such as in -vulnerability databases like VulnerableCode. +vulnerability databases like `VulnerableCode +`_. To better understand the problem, here are some of the notations and conventions in use: @@ -133,7 +134,7 @@ For example to define a set of versions that contains either version ``1.2.3``, or any versions greater than or equal to ``2.0.0`` but less than ``5.0.0`` using the `semver` versioning scheme, the version range specifier will be:: - vers:semver/1.2.3,>=2.0.0&<5.0.0` + vers:semver/1.2.3,>=2.0.0&<5.0.0 Each ```` in the comma-separated list is either a simple constraint such as:: @@ -141,7 +142,7 @@ constraint such as:: Or a composite constraint grouping multiple ```` joined by -an ampersand such as: +an ampersand such as:: &... @@ -208,8 +209,7 @@ The ```` is one of these comparison operators: - "!=": Version exclusion or inequality comparator. This means a version must not be equal to the provided version and this version must be excluded from - the range. For example: "!=1.2.3" means that version "1.2.3" is not part of - the range. + the range. For example: "!=1.2.3" means that version "1.2.3" is excluded. - "<", "<=": Less than or less-or-equal version comparators points to all versions less than or equal to the provided version. For example "<=1.2.3" @@ -303,13 +303,14 @@ To parse a version range specifier string: - Verify that the URI-scheme value is ``vers``. - The right hand side is the specifier. -- Split the specifier from left once on colon ":". +- Split the specifier from left once on a slash "/". - The left hand side is the that must be lowercase. - The right hand side is the constraints. - If the constraint contains a star "*", validate that it is equal to "*". is "*". Parsing is done and no further processing is - needed for this ``vers``. + needed for this ``vers``. A tool may be strict and report an error + if there are extra characters beyond "*" or be lenient. - Split the ranges on comma ",". The result is a list of strings. @@ -454,8 +455,8 @@ https://nvd.nist.gov/vuln/vulnerability-detail-pages#divRange The NVD CPE Match String Range is a complex specification that goes well beyond version ranges and is used to match comprehensive configurations across multiple -products and version ranges. The notation is to uses these fields in the CVE API -or feeds under "configurations":: +products and version ranges. The notation for version ranges uses these two +fields (in the CVE API or feeds) under the "configurations" attribute:: "versionStartIncluding": "7.3.0", "versionEndExcluding": "7.3.31", @@ -474,7 +475,7 @@ has a few practical issues: - The space means "AND" and significant whitespace in a single string makes normalization more complicated and may be a source of confusion or errors. The - explicit ampersand used an "AND" operator specified here improves clarity. + explicit ampersand used as "AND" operator specified here improves clarity. - There is no negation "!=" operator meaning that some version constraints are difficult to express and require combining < and > comparators. For instance From a1b69a8c17a6ccf68cfcc9d4136a246407d55b5e Mon Sep 17 00:00:00 2001 From: Philippe Ombredanne Date: Sun, 31 Oct 2021 23:41:35 +0100 Subject: [PATCH 03/12] Add Arch Linux example Reported-by: Shivam Sandbhor Signed-off-by: Philippe Ombredanne --- VERSION-RANGE-SPEC.rst | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/VERSION-RANGE-SPEC.rst b/VERSION-RANGE-SPEC.rst index 84f4ce84..0598beb2 100644 --- a/VERSION-RANGE-SPEC.rst +++ b/VERSION-RANGE-SPEC.rst @@ -85,6 +85,10 @@ in use: - Gentoo https://wiki.gentoo.org/wiki/Version_specifier - Alpine linux https://gitlab.alpinelinux.org/alpine/apk-tools/-/blob/master/src/version.c +- Arch Linux https://wiki.archlinux.org/title/PKGBUILD#Dependencies use a + simplified notation for its PKGBUILD depends array with comparison operators + similar to this specification. + - Go modules https://golang.org/ref/mod#versions use semver versions with specific version resolution algorithms. From 15eabf729cdf2f7c9fb1fa5fc589655b62761a22 Mon Sep 17 00:00:00 2001 From: Philippe Ombredanne Date: Sun, 31 Oct 2021 23:45:53 +0100 Subject: [PATCH 04/12] Clarify text on diffrerence bewteen version schemes Reported-by: Shivam Sandbhor Signed-off-by: Philippe Ombredanne --- VERSION-RANGE-SPEC.rst | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/VERSION-RANGE-SPEC.rst b/VERSION-RANGE-SPEC.rst index 0598beb2..59913603 100644 --- a/VERSION-RANGE-SPEC.rst +++ b/VERSION-RANGE-SPEC.rst @@ -97,13 +97,17 @@ in use: such as star and caret. -Note that there is a closely related problem: the way two versions are compared -as equal, lesser or greater is complex and package types may have evolved their -own peculiar version comparison procedure. Semver is a prominent specification -in this domain but this is just one of the many versioning schemes. Semver, -Debian, RPM, PyPI, Rubygems, and Composer all have their own subtly different -ways of comparing two versions even though they may seem highly similar at first -glance. +Note that there is a closely related problem as the way two versions are compared +as equal, lesser or greater is often complex. + +Each package ecosystem may have evolved its own peculiar version comparison +procedure. + +For instance, semver is a prominent specification in this domain but this is +just one of the many ways to structure a version string. + +Debian, RPM, PyPI, Rubygems, and Composer have their own subtly different +approach on how to determine which version is greater or lesser. Solution From 50a1ad739d0568a33daaa11c8c6a2ac424933249 Mon Sep 17 00:00:00 2001 From: Philippe Ombredanne Date: Sun, 31 Oct 2021 23:46:28 +0100 Subject: [PATCH 05/12] Fix formatting Reported-by: Shivam Sandbhor Signed-off-by: Philippe Ombredanne --- VERSION-RANGE-SPEC.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/VERSION-RANGE-SPEC.rst b/VERSION-RANGE-SPEC.rst index 59913603..fb59f29a 100644 --- a/VERSION-RANGE-SPEC.rst +++ b/VERSION-RANGE-SPEC.rst @@ -104,7 +104,7 @@ Each package ecosystem may have evolved its own peculiar version comparison procedure. For instance, semver is a prominent specification in this domain but this is -just one of the many ways to structure a version string. +just one of the many ways to structure a version string. Debian, RPM, PyPI, Rubygems, and Composer have their own subtly different approach on how to determine which version is greater or lesser. From 446e6d847fe8bb921f52f227809d625e21759faa Mon Sep 17 00:00:00 2001 From: Philippe Ombredanne Date: Mon, 1 Nov 2021 07:39:51 +0100 Subject: [PATCH 06/12] Use or later, not and later Signed-off-by: Philippe Ombredanne --- VERSION-RANGE-SPEC.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/VERSION-RANGE-SPEC.rst b/VERSION-RANGE-SPEC.rst index fb59f29a..89c5d6ef 100644 --- a/VERSION-RANGE-SPEC.rst +++ b/VERSION-RANGE-SPEC.rst @@ -12,7 +12,7 @@ Software package version ranges and version constraints are essential: - When resolving the dependencies of a package to express which subset of the versions are supported. For instance a dependency or requirement statement - such as "I require package foo, version 2.0 and later versions" defines a + such as "I require package foo, version 2.0 or later versions" defines a range of acceptable foo versions. - When stating that a known vulnerability or bug affects a range of package From 2522a26a33965cab95ffc6739840de1a0c51b81b Mon Sep 17 00:00:00 2001 From: Philippe Ombredanne Date: Mon, 1 Nov 2021 09:27:03 +0100 Subject: [PATCH 07/12] Refine documentation of NVD ranges and approach Signed-off-by: Philippe Ombredanne --- VERSION-RANGE-SPEC.rst | 37 +++++++++++++++++++++++++++---------- 1 file changed, 27 insertions(+), 10 deletions(-) diff --git a/VERSION-RANGE-SPEC.rst b/VERSION-RANGE-SPEC.rst index 89c5d6ef..487bd26f 100644 --- a/VERSION-RANGE-SPEC.rst +++ b/VERSION-RANGE-SPEC.rst @@ -40,9 +40,10 @@ most version range notations are similar. Each package type or ecosystem may define their own ranges notation and version comparison semantics for dependencies. And for security advisories, the lack of -portable notation for vulnerable package version ranges means that these ranges -may be either ambiguous or hard to compute and are typically replaced by -complete enumerations of all versions, such as the NVD CPEs. +a portable and compact notation for vulnerable package version ranges means that +these ranges may be either ambiguous or hard to compute and may be best replaced +by complete enumerations of all impacted versions, such as in the `NVD CPE Match +feed `_. Because of this, expressing and resolving a version range is often a complex, or error prone task. @@ -96,6 +97,10 @@ in use: similar to this specification based on a modified semver with extra notations such as star and caret. +- The NVD https://nvd.nist.gov/vuln/data-feeds#cpeMatch defines CPE ranges as + lists of version start and end either including or excluding the start or end + version. And also provides a concrete enumeration of the available ranges as + a daily feed. Note that there is a closely related problem as the way two versions are compared as equal, lesser or greater is often complex. @@ -459,18 +464,30 @@ resolution. Why not use the NVD CPE Ranges? ############################### -https://nvd.nist.gov/vuln/vulnerability-detail-pages#divRange +- https://nvd.nist.gov/vuln/vulnerability-detail-pages#divRange +- https://nvd.nist.gov/developers/vulnerabilities#divResponse +- https://csrc.nist.gov/schema/nvd/feed/1.1/nvd_cve_feed_json_1.1.schema -The NVD CPE Match String Range is a complex specification that goes well beyond -version ranges and is used to match comprehensive configurations across multiple -products and version ranges. The notation for version ranges uses these two -fields (in the CVE API or feeds) under the "configurations" attribute:: +The version ranges notation defined in the JSON schema of the CVE API payload +uses these four fields: ``versionStartIncluding``, ``versionStartExcluding``, +``versionEndIncluding`` and ``versionEndExcluding``. For example:: "versionStartIncluding": "7.3.0", "versionEndExcluding": "7.3.31", + "versionStartExcluding" : "9.0.0", + "versionEndIncluding" : "9.0.46", -This notation is limited and is compatible can be expressed in ``vers`` but this -does not provide a compact notation like ``vers`` does. +In addition to these ranges, the NVD publishes a list of concrete CPE with +versions resolved for a range with daily updates at +https://nvd.nist.gov/vuln/data-feeds#cpeMatch + +Note that the NVD CVE configuration is a complex specification that goes well +beyond version ranges and is used to match comprehensive configurations across +multiple products and version ranges. ``vers`` focus is exclusively versions. + +The NVD JSON notation is verbose in contrast with ``vers`` that attempts to +provide a compact notation. It provides the same =, <=, < and > comparators +specified in ``vers`` and found in other notations. Why not use node-semver ranges? From 5b57c76289a41d6ceae15e87b37a0abe66d09c77 Mon Sep 17 00:00:00 2001 From: Philippe Ombredanne Date: Mon, 1 Nov 2021 09:58:02 +0100 Subject: [PATCH 08/12] Clarify node-semver vs. semver references node-semver is a range notation, while semver is a version string syntax. Signed-off-by: Philippe Ombredanne --- VERSION-RANGE-SPEC.rst | 95 +++++++++++++++++++++++++----------------- 1 file changed, 56 insertions(+), 39 deletions(-) diff --git a/VERSION-RANGE-SPEC.rst b/VERSION-RANGE-SPEC.rst index 487bd26f..8ec44443 100644 --- a/VERSION-RANGE-SPEC.rst +++ b/VERSION-RANGE-SPEC.rst @@ -56,15 +56,19 @@ vulnerability databases like `VulnerableCode To better understand the problem, here are some of the notations and conventions in use: -- Rubygems strongly suggest using semver but does not enforce it +- ``semver`` https://semver.org/ is a popular specification to structure version + strings, but does not provide a way to express version ranges. + +- Rubygems strongly suggest using ``semver`` for version but does not enforce it. + It use its own notation for version ranges. https://guides.rubygems.org/patterns/#semantic-versioning - node-semver ranges are used in npms https://github.com/npm/node-semver#ranges - with semantics specific to semver and node and npm. + with their own range semantics that are specific to ``semver`` and npm. -- Dart pub versioning scheme is similar to node-semver and the documentation at - https://dart.dev/tools/pub/versioning provides a comprehensive coverage of the - topic of versioning. But version resolution is using its own algorithm. +- Dart pub versioning scheme is similar to ``node-semver`` and the documentation + at https://dart.dev/tools/pub/versioning provides a comprehensive coverage of + the topic of versioning. Version resolution is using its own algorithm. - Python uses its own version ranges notation https://www.python.org/dev/peps/pep-0440/ @@ -74,21 +78,24 @@ in use: - RPM distros use their own range notation https://rpm-software-management.github.io/rpm/manual/dependencies.html -- Perl CPAN define its own version range notation similar to this specification: +- Perl CPAN define its own version range notation similar to this specification https://metacpan.org/pod/CPAN::Meta::Spec#Version-Ranges - Apache Maven and NuGet use math intervals notation https://en.wikipedia.org/wiki/Interval_(mathematics) + - Apache Maven http://maven.apache.org/enforcer/enforcer-rules/versionRanges.html - NuGet https://docs.microsoft.com/en-us/nuget/concepts/package-versioning#version-ranges +- gradle uses Apache Maven notation with extensions + https://docs.gradle.org/current/userguide/single_versions.html + - Gentoo and Alpine Linux use comparison operators similar to this specification: - Gentoo https://wiki.gentoo.org/wiki/Version_specifier - Alpine linux https://gitlab.alpinelinux.org/alpine/apk-tools/-/blob/master/src/version.c -- Arch Linux https://wiki.archlinux.org/title/PKGBUILD#Dependencies use a - simplified notation for its PKGBUILD depends array with comparison operators - similar to this specification. +- Arch Linux https://wiki.archlinux.org/title/PKGBUILD#Dependencies use its + own simplified notation for its PKGBUILD depends array. - Go modules https://golang.org/ref/mod#versions use semver versions with specific version resolution algorithms. @@ -103,16 +110,16 @@ in use: a daily feed. Note that there is a closely related problem as the way two versions are compared -as equal, lesser or greater is often complex. +as equal, lesser or greater is often complex: -Each package ecosystem may have evolved its own peculiar version comparison -procedure. +- Each package ecosystem may have evolved its own peculiar version comparison + procedure. -For instance, semver is a prominent specification in this domain but this is -just one of the many ways to structure a version string. +- For instance, semver is a prominent specification in this domain but this is + just one of the many ways to structure a version string. -Debian, RPM, PyPI, Rubygems, and Composer have their own subtly different -approach on how to determine which version is greater or lesser. +- Debian, RPM, PyPI, Rubygems, and Composer have their own subtly different + approach on how to determine which version is greater or lesser. Solution @@ -121,11 +128,11 @@ Solution A solution to the many version range syntaxes is to design a new notation to unify them all with: -- a mostly universal and minimalist notation to express the version range - notations from many different package types and ecosystems. +- a mostly universal and minimalist, compact notation to express version ranges + from many different package types and ecosystems. - the package type-specific definitions to normalize existing range expressions - to this common notation . + to this common notation. - the designation of which algorithm or procedure to use when comparing two versions such that it is possible to resolve if a version is within or @@ -145,9 +152,9 @@ URI-scheme with this syntax:: For example to define a set of versions that contains either version ``1.2.3``, or any versions greater than or equal to ``2.0.0`` but less than ``5.0.0`` using -the `semver` versioning scheme, the version range specifier will be:: +the ``node-semver`` versioning scheme, the version range specifier will be:: - vers:semver/1.2.3,>=2.0.0&<5.0.0 + vers:node-semver/1.2.3,>=2.0.0&<5.0.0 Each ```` in the comma-separated list is either a simple constraint such as:: @@ -173,7 +180,7 @@ for a future formal registration for this URI-scheme at the IANA registry. ```` ------------------------ -The ```` (such as ``semver``, +The ```` (such as ``node-semver``, ``debian``, etc.) determines: - the specific notation and conventions used for a version string encoded in @@ -368,7 +375,7 @@ Notes and caveats ~~~~~~~~~~~~~~~~~~~ - Comparing versions from two different versioning schemes is unspecified. Even - though there may be some similarities between the `semver` version of an npm + though there may be some similarities between the ``semver`` version of an npm and the `debian` version of its Debian packaging, these similarities are specific to each versioning scheme. Tools may report an error in these cases. @@ -394,12 +401,15 @@ scheme and package type. The version comparison routine of rmpvercmp is also used by archlinux Pacman. - ``rubygems``: Rubygems https://guides.rubygems.org/patterns/#semantic-versioning - which is almost but not exactly semver. + which is almost but not exactly ``node-semver``. -- ``semver``: node-semver as used for npm https://github.com/npm/node-semver#ranges - It is also used by Rust: https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html - and several other package types. `composer` may need its own scheme as this is - not strictly semver. +- ``node-semver``: node-semver is based on semver and is used for npm + https://github.com/npm/node-semver#ranges + A similar scheme is used by Rust https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html + and several other package types that use ``semver``. But most of these related + schemes are not strictly the same as what is implemented in ``node-semver``. + For instance PHP ``composer`` may need its own scheme as this is not strictly + ``node-semver``. - ``python``: Python https://www.python.org/dev/peps/pep-0440/ @@ -417,12 +427,13 @@ scheme and package type. - ``gentoo``: Gentoo https://wiki.gentoo.org/wiki/Version_specifier - ``alpine``: Alpine linux https://gitlab.alpinelinux.org/alpine/apk-tools/-/blob/master/src/version.c - (which might be using Gentoo conventions) + which is using Gentoo-like conventions. - ``generic``: a generic version comparison algorithm (which is TBD, likely a split on punctuation and dealing with digit vs. strings comparisons, like is done in libversion) +TODO: add Rust, composer and archlinux Implementations @@ -493,14 +504,16 @@ specified in ``vers`` and found in other notations. Why not use node-semver ranges? ############################### -https://github.com/npm/node-semver#ranges +- https://github.com/npm/node-semver#ranges -The node semver is very similar to this spec (this is also an OR of ANDs) but it -has a few practical issues: +The node-semver spec is similar to this spec but is an AND of ORs with a few +issues: -- The space means "AND" and significant whitespace in a single string makes - normalization more complicated and may be a source of confusion or errors. The - explicit ampersand used as "AND" operator specified here improves clarity. +- The space means "AND", therefore whitespaces are significant. Having + significant whitespaces in string makes normalization more complicated and may + be a source of confusion if you remove the spaces from the string. Using an + ampersand as an "AND" operator in ``vers`` makes this explicit and avoids any + ambiguity. - There is no negation "!=" operator meaning that some version constraints are difficult to express and require combining < and > comparators. For instance @@ -510,14 +523,18 @@ has a few practical issues: - The advanced range syntax has grown to be rather complex using hyphen, stars, carets and tilde constructs are all tied to the JavaScript and npm ways of handling versions in their specific ecosystem and furthermore are bound to the - semver semantics. These are not readily reusable elsewhere. And these multiple - comparators and modifiers make the grammar and parsing more complex. + semver semantics and the npm implementation. These are not readily reusable + elsewhere. And these multiple comparators and modifiers make the grammar and + parsing more complex. + +Notations that are directly derived from node-semver as used in Rust and PHP +Composer have the same issues. Why not use Python pep-0440 ranges? ##################################### -https://www.python.org/dev/peps/pep-0440/#version-specifiers +- https://www.python.org/dev/peps/pep-0440/#version-specifiers The Python pep-0440 "Version Identification and Dependency Specification" provides a comprehensive specification for Python package versioning and a @@ -542,7 +559,7 @@ only in the Python ecosystem. Why not use Rubygems requirements notation? ############################################################## -https://guides.rubygems.org/patterns/#declaring-dependencies +- https://guides.rubygems.org/patterns/#declaring-dependencies The rubygems specification suggests but does not enforce using semver. It is similar to this spec's operators with the addition of the "~>" aka. pessimistic From a6afc046ad206846c3e275a25e116f0951422118 Mon Sep 17 00:00:00 2001 From: Philippe Ombredanne Date: Mon, 1 Nov 2021 13:04:53 +0100 Subject: [PATCH 09/12] Fix typo Signed-off-by: Philippe Ombredanne --- VERSION-RANGE-SPEC.rst | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/VERSION-RANGE-SPEC.rst b/VERSION-RANGE-SPEC.rst index 8ec44443..6bd2290a 100644 --- a/VERSION-RANGE-SPEC.rst +++ b/VERSION-RANGE-SPEC.rst @@ -461,7 +461,7 @@ In contrast, this would not be possible with existing notations. For instance, the Python notation may not work with npm versions and reciprocally. There are likely to be a few rare cases where round tripping from and to -``vers`` may not be possible, and in anycase round tripping to and from ``vers`` +``vers`` may not be possible, and in any case round tripping to and from ``vers`` may only produce equivalent results and not strictly the same original strings. Another issue with existing version range notations is that, they are primarily @@ -568,6 +568,22 @@ implies semver versioning. This makes the notation impractical to reuse in places that do not use the same semver-like semantics. +Why not use more extra comparators such as a tilde, carret and star? +##################################################################### + +Several existing notations such as node-semver, rubygems or composer specs +provide syntactic shorthand such as: + +- a tilde prefix or ~> prefix as in "~1.3" or "~>1.2.3" +- a caret ^ prefix as in "^ 1.2" +- using a star in a segment of a semver version as in "1.2.*" +- dash-separated ranges as in "1.2 - 1.4" + +These range syntaxes can be reduced to a simpler notation here. Furthermore they +are designed for the structure of a version string (most often semver) as used +in one ecosystem and therefore are not reusable everywhere. + + References ~~~~~~~~~~~~~~~~~~~~ From f081c6dcf9876ca53278a4dea4fd553bd03cc4b1 Mon Sep 17 00:00:00 2001 From: Philippe Ombredanne Date: Mon, 1 Nov 2021 15:55:22 +0100 Subject: [PATCH 10/12] Add OSSF OSV and CVE v5 references These recent and closely related specs (they share authors) provide another approach to documeting vulnerable ranges. Signed-off-by: Philippe Ombredanne --- VERSION-RANGE-SPEC.rst | 26 +++++++++++++++++++++----- 1 file changed, 21 insertions(+), 5 deletions(-) diff --git a/VERSION-RANGE-SPEC.rst b/VERSION-RANGE-SPEC.rst index 6bd2290a..c6a2fbf8 100644 --- a/VERSION-RANGE-SPEC.rst +++ b/VERSION-RANGE-SPEC.rst @@ -109,11 +109,27 @@ in use: version. And also provides a concrete enumeration of the available ranges as a daily feed. -Note that there is a closely related problem as the way two versions are compared -as equal, lesser or greater is often complex: - -- Each package ecosystem may have evolved its own peculiar version comparison - procedure. +- The version 5 of the NVD CVE JSON data format at + https://github.com/CVEProject/cve-schema/blob/master/schema/v5.0/CVE_JSON_5.0.schema#L303 + defines version ranges with a starting version, a versionType, and an upper + limit for the version range as lessThan or lessThanOrEqual. Or an enumeration + of versions. The versionType is defined as ``"The version numbering system + used for specifying the range. This defines the exact semantics of the + comparison (less-than) operation on versions, which is required to understand + the range itself"``. + +- The OSSF OSV schema https://ossf.github.io/osv-schema/ defines vulnerable + ranges with version events using "introduced" and "limit" fields and an + enumeration of all the versions in these ranges, except for semver-based + versions. A range may be ecosystem-specific based on a provided package + "ecosystem" value that ressembles closely the Package URL package "type". + + +The way two versions are compared as equal, lesser or greater is a closely +related topic: + +- Each package ecosystem may have evolved its own peculiar version string + conventions, semantics and comparison procedure. - For instance, semver is a prominent specification in this domain but this is just one of the many ways to structure a version string. From be1dc2adc12d3970b78f7b2a6f309a55efcd04fb Mon Sep 17 00:00:00 2001 From: Philippe Ombredanne Date: Wed, 17 Nov 2021 19:59:34 +0100 Subject: [PATCH 11/12] Switch to use pipe and comma separators Use pipe and comma separators and drop using ampersand. This avoid a possible ambiguity when using a vers as a query string argument in a URL (such as with a Package URL qualifier). Signed-off-by: Philippe Ombredanne --- VERSION-RANGE-SPEC.rst | 289 ++++++++++++++++++++++------------------- 1 file changed, 155 insertions(+), 134 deletions(-) diff --git a/VERSION-RANGE-SPEC.rst b/VERSION-RANGE-SPEC.rst index c6a2fbf8..a98fbefe 100644 --- a/VERSION-RANGE-SPEC.rst +++ b/VERSION-RANGE-SPEC.rst @@ -60,17 +60,19 @@ in use: strings, but does not provide a way to express version ranges. - Rubygems strongly suggest using ``semver`` for version but does not enforce it. - It use its own notation for version ranges. + As a result some use semver and several popular package do not use strict + semver. Runbygems use their own notation for version ranges. https://guides.rubygems.org/patterns/#semantic-versioning -- node-semver ranges are used in npms https://github.com/npm/node-semver#ranges - with their own range semantics that are specific to ``semver`` and npm. +- node-semver ranges are used in npm at https://github.com/npm/node-semver#ranges + with range semantics that are specific to ``semver`` and npm. - Dart pub versioning scheme is similar to ``node-semver`` and the documentation at https://dart.dev/tools/pub/versioning provides a comprehensive coverage of the topic of versioning. Version resolution is using its own algorithm. -- Python uses its own version ranges notation https://www.python.org/dev/peps/pep-0440/ +- Python uses its own version and version ranges notation + https://www.python.org/dev/peps/pep-0440/ - Debian and Ubuntu use their own notation https://www.debian.org/doc/debian-policy/ch-relationships.html @@ -81,7 +83,7 @@ in use: - Perl CPAN define its own version range notation similar to this specification https://metacpan.org/pod/CPAN::Meta::Spec#Version-Ranges -- Apache Maven and NuGet use math intervals notation +- Apache Maven and NuGet use similar math intervals notation https://en.wikipedia.org/wiki/Interval_(mathematics) - Apache Maven http://maven.apache.org/enforcer/enforcer-rules/versionRanges.html @@ -164,28 +166,33 @@ Version range specifier A version range specifier (aka. "vers") is a URI string using the ``vers`` URI-scheme with this syntax:: - vers:/,,... + vers:/|,... For example to define a set of versions that contains either version ``1.2.3``, or any versions greater than or equal to ``2.0.0`` but less than ``5.0.0`` using the ``node-semver`` versioning scheme, the version range specifier will be:: - vers:node-semver/1.2.3,>=2.0.0&<5.0.0 + vers:node-semver/1.2.3|>=2.0.0,<5.0.0 -Each ```` in the comma-separated list is either a simple +Each ```` in the pipe-separated list is either a simple constraint such as:: Or a composite constraint grouping multiple ```` joined by -an ampersand such as:: +a comma such as:: - &... + ,... -The comma is a logical `OR` and the ampersand is a logical `AND`. +The pipe is a logical `OR` and the comma is a logical `AND`. -A version range specifier is therefore an "OR of ANDs". This is also called a -"disjunctive normal form" in boolean logic. +A version range specifier is therefore an "OR of ANDs" where there are two +levels of constraints that a version should satisfy to be part of the range: + +- At the first level, anyone of the constraints should be satisfied +- At the second level, all of the constraints must be satisfied + +This is also called a "disjunctive normal form" in boolean logic. See https://en.wikipedia.org/wiki/Disjunctive_normal_form for details. ``vers`` is the URI-scheme and is an acronym for "VErsion Range Specifier". It @@ -196,39 +203,39 @@ for a future formal registration for this URI-scheme at the IANA registry. ```` ------------------------ -The ```` (such as ``node-semver``, -``debian``, etc.) determines: +The ```` (such as ``npm``, ``deb``, etc.) determines: - the specific notation and conventions used for a version string encoded in this scheme. Versioning schemes often specify a version segments separator and the meaning of each version segments, such as [major.minor.patch] in semver. -- how two versions compare as greater or lesser to determine if a version is - within or outside a range. +- how two versions are compared as greater or lesser to determine if a version + is within or outside a range. - how a versioning scheme-specific range notation can be transformed in the - ``vers`` notation defined here. + ``vers`` simplified notation defined here. -Note: the ```` does not define which symbols to use to -compare versions, and how to compose these as this is the substance of this -specification. +- by convention the versioning scheme should be the same string as the Package + URL package type for a given package ecosystem. It is OK to have other schemes + beyond the purl type and even schemes that are specific to a single package. The ```` is followed by a slash "/". -Then there are one or more ```` separated by a comma "," -meaning that **any** of these constraints must be satisfied for a version to be -resolved as within this range specifier. - ```` ---------------------------- -Each ```` of this comma-separated list can be either a -single constraint or a list of constraints separated by an ampersand "&" as in -``1.2.3,>=2.0.0&<5.0.0``. +After the ```` and "/" there are one or more +```` separated by a pipe "|". The pipe "|" means that +**any** of these constraints must be satisfied for a version to be resolved as +within this version range. -Multiple ```` combined with an ampersand means that **all** -these constraints must be satisfied for a version to be resolved as within this +Each ```` of this pipe-separated list can be either a +single constraint or a list of constraints separated in turn by an comma "," as +in ``1.2.3|>=2.0.0,<5.0.0``. + +Multiple ```` combined with a comma means that **all** these +constraints must be satisfied for a version to be resolved as contained in this range. Each simple version constraint has this syntax:: @@ -241,7 +248,9 @@ The ```` is one of these comparison operators: present and means that a version must be equal to the provided version. For example: "=1.2.3". It must be omitted in the canonical representation. Equality is based on the equality of two lower-cased and normalized version - strings and is not versioning scheme-specific. + strings and is typically not versioning scheme-specific, though some + scheme such as pypi PEP440 may apply some version string normalization + before testing for equality. - "!=": Version exclusion or inequality comparator. This means a version must not be equal to the provided version and this version must be excluded from @@ -263,8 +272,12 @@ The ```` is one of these comparison operators: dot-separated segments name major, minor and patch. - The special star "*" ```` matches any version. This star - constraint must be used **alone** in a version range specifier, exclusive of - any other constraint. For example "vers:debian/\*" resolves to any version. + constraint must be used **alone** in a version range, exclusive of any other + constraint. For example "vers:deb/\*" resolves to any version of a Debian + package. + +- The way each of these comparators work when doing a version comparison is + specific to a versioning scheme. Examples @@ -287,14 +300,14 @@ Normalized or canonical representation - The ordering of multiple ```` in a range specifier is not significant. The canonical ordering is by sorting these by lexicographical - order applied with this approach: + order applied with this two steps approach: - - first to each sub-list of ampersand-separated ````. - - then to the list of comma-separated ````. + - first to each sub-list of comma-separated ````. + - then to the top level list of pipe-separated ````. - A version in a ```` can only contain printable ASCII characters excluding the special characters used as separators and comparators - ``><=!,&*``. If it contains special characters (which should be rare in + ``><=!,&*|``. If it contains special characters (which should be rare in practice) the version string in a constraint must be quoted using the URL quoting rules. @@ -304,28 +317,31 @@ Using version range specifiers ``vers`` primary usage is to test if a version is within or outside a range. -An input version is within a range specifier if it matches or satisfies **any** -of the ranges in this spec. To match a range, an input version must match -**all** the version constraints of a range. Otherwise, the input version is -outside of the range specifier. +An version is within a version range if satisfies or is "contained" in +**any one** of the first level of constraints. To satisfy or be "contained" in +a first level constraint, a version must satisfy or be "contained" in +**all** the second level of constraints. Otherwise, the input version is outside +of the version range. Some important usages derived from this primary usage include: - **Resolving a version range specifier to a list of concrete versions.** In this case, the input is the set of known versions of a package (typically obtained from some package repository or registry). Each version is then - tested individually to check if it is within or outside the ranges. For + tested individually to check if it is within or outside the range. For example, this can be used to determine which existing package versions are affected by a known vulnerability if they match the vulnerability version range specifier. - **Selecting one of several versions that are within a range.** - For example, package management tools need to determine what are the dependent - package versions of a package instance based the acceptable version ranges - provided with each dependency. This usually requires deploying heuristics and - algorithms (possibly complex such as sat solvers) that are versioning-scheme- - and tool-specific and outside of the scope for this specification; ``vers`` - scope is limited to provide an input to the resolution process. + In this case, given several versions that are within a range and several + packages that each expression inter dependencies together with version ranges, + package management tools need to determine and select a set of package versions + that satify all the version ranges of all dependencies. This usually requires + deploying heuristics and algorithms (possibly complex such as sat solvers) + that are ecosystem- and tool-specific and outside of the scope for this + specification; yet ``vers`` could be used in tandem with ``purl`` to provide + an input to a dependencies resolution process. Parsing version range specifiers @@ -341,50 +357,50 @@ To parse a version range specifier string: - Split the specifier from left once on a slash "/". - The left hand side is the that must be lowercase. -- The right hand side is the constraints. +- The right hand side is a list of one or more constraints. -- If the constraint contains a star "*", validate that it is equal to "*". - is "*". Parsing is done and no further processing is - needed for this ``vers``. A tool may be strict and report an error - if there are extra characters beyond "*" or be lenient. +- If the constraints string is equal to "*", the is "*". + Parsing is done and no further processing is needed for this ``vers``. A tool + may be strict and report an error if there are extra characters beyond "*" or + be lenient. -- Split the ranges on comma ",". The result is a list of - strings. -- Remove any empty strings from the list of . +- Split the constraints on pipe "|". The result is a list of top-level + lists. Consecutive pipes should be treated as one. -- For each in this list: +- For each list: - - Split on ampersand "&". The result is a sub-list of . - - Remove any empty strings from this sub-list. + - Split on comma ",". Consecutive commas should be treated as one. The result + is a sub-list of . - For each in this sub-list: - - Split the comparator and version starting from left - based on the start of the : + - Identify the comparator and version based on the + start of the in this sequence: - - If it starts with "=", split it once on "=". - - If it starts with "!=", split it once on "!=". - - If it starts with "<=", split it once on "<=". - - If it starts with ">=", split it once on ">=". - - If it starts with "<", split it once on "<". - - If it starts with ">", split it once on ">". - - Otherwise if there is no split, the comparator is "=" (default) and the - version is the string. + - If it starts with "=", then the comparator is "=" + - If it starts with "!=", then the comparator is "!=". + - If it starts with "<=", then the comparator is "<=". + - If it starts with ">=", then the comparator is ">=". + - If it starts with "<", then the comparator is "<". + - If it starts with ">", then the comparator is ">". + - Else the comparator is "=" (default) and the + version is the full string. - - After the split, the left hand side is the comparator, and the right hand - is the version. + - After the operation and removing the comparator from + string, the remaining string is the version. - Validate that the version is not empty. - If the version contains a percent "%" character, apply URL quoting rules to unquote this string. - - Yield the comparator and version for this constraint + - Append the comparator and version of this constraint to the inner list + of constraints. - - Yield the accumulated list of (comparator and version) that must apply for - this constraint + - Append the accumulated list of (comparator and version) that must apply to + the top level list of constraints. -- Finally return the and the list of +- Finally return the and the nested list of Notes and caveats @@ -393,15 +409,8 @@ Notes and caveats - Comparing versions from two different versioning schemes is unspecified. Even though there may be some similarities between the ``semver`` version of an npm and the `debian` version of its Debian packaging, these similarities are - specific to each versioning scheme. Tools may report an error in these - cases. - -- Schemes are related to Package URL types (aka. ecosystems) in the sense that - each Package URL type has one versioning scheme, but multiple types can reuse - the same versioning scheme (such as semver). - -- Some package types may use the same versioning scheme such as semver to - compare two versions, but may not use the same notation for version ranges. + specific to each versioning scheme. Tools should report an error in these + cases as it does not make sense to perform these comparisons. Some of the known versioning schemes @@ -410,24 +419,25 @@ Some of the known versioning schemes TODO: add details on how to convert to and from ``vers`` for a given versioning scheme and package type. -- ``debian``: Debian and Ubuntu https://www.debian.org/doc/debian-policy/ch-relationships.html +- ``deb``: Debian and Ubuntu https://www.debian.org/doc/debian-policy/ch-relationships.html The comparators are <<, <=, =, >= and >>. - ``rpm``: RPM distros https://rpm-software-management.github.io/rpm/manual/dependencies.html The version comparison routine of rmpvercmp is also used by archlinux Pacman. -- ``rubygems``: Rubygems https://guides.rubygems.org/patterns/#semantic-versioning +- ``gem``: Rubygems https://guides.rubygems.org/patterns/#semantic-versioning which is almost but not exactly ``node-semver``. -- ``node-semver``: node-semver is based on semver and is used for npm - https://github.com/npm/node-semver#ranges - A similar scheme is used by Rust https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html - and several other package types that use ``semver``. But most of these related - schemes are not strictly the same as what is implemented in ``node-semver``. - For instance PHP ``composer`` may need its own scheme as this is not strictly - ``node-semver``. +- ``npm``: npm uses node-semver which is based on semver with its own range + notation https://github.com/npm/node-semver#ranges + A similar but different scheme is used by Rust + https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html + and several other package types may use ``node-semver``-like ranges. But most + of these related schemes are not strictly the same as what is implemented in + ``node-semver``. For instance PHP ``composer`` may need its own scheme as this + is not strictly ``node-semver``. -- ``python``: Python https://www.python.org/dev/peps/pep-0440/ +- ``pypi``: Python https://www.python.org/dev/peps/pep-0440/ - ``perl``: Perl https://perlmaven.com/how-to-compare-version-numbers-in-perl-and-for-cpan-modules @@ -464,33 +474,40 @@ Why not reuse existing version range notations? Most existing version range notations are tied to a specific version string syntax and are therefore not readily applicable to other contexts. For example, -the use of elements such as tilde and caret ranges in Ruby or Dart notations -implies that a certain structure exists in the version string (semver or semver- -like). The inclusion of these additional comparators is a result of the history -and evolution in a given package ecosystem to address specific needs. +the use of elements such as tilde and caret ranges in Rubygems, npm or Dart +notations implies that a certain structure exists in the version string (semver +or semver- like). The inclusion of these additional comparators is a result of +the history and evolution in a given package ecosystem to address specific needs. In practice, the unified and reduced set of comparators and syntax defined for ``vers`` has been designed such that all these notations can be converted to a -``vers`` and back from a ``vers`` to the original notation. +``vers`` and back from a ``vers`` to the original notation. In contrast, this would not be possible with existing notations. For instance, -the Python notation may not work with npm versions and reciprocally. +the Python notation may not work with npm semver versions and reciprocally. There are likely to be a few rare cases where round tripping from and to ``vers`` may not be possible, and in any case round tripping to and from ``vers`` -may only produce equivalent results and not strictly the same original strings. - -Another issue with existing version range notations is that, they are primarily -meant to be used for dependency constraints and may not readily be reused for -vulnerable ranges. In particular, a vulnerability may exist for multiple -"branches" of a given package (such as Django 2, 3 and 4) which may not be -possible to express with a notation designed exclusively for dependent versions +should produce equivalent results and even if not strictly the same original +strings. + +Another issue with existing version range notations is that they are primarily +meant to be used for dependency constraints and may not readily be reusable for +the definitions of vulnerable ranges. In particular, a vulnerability may exist +for multiple "version branches" of a given package such as with Django 2.x and +3.x. Several version range notations have difficulties to communicate these +as typically all the version constraints must be satisfied. In constrast, +a vulnerability can affect multiple disjoint version ranges of a package and any +version satisfying these constraints would be vulnerable: it may not be possible +to express this with a notation designed exclusively for dependent versions resolution. Why not use the NVD CPE Ranges? ############################### +See: + - https://nvd.nist.gov/vuln/vulnerability-detail-pages#divRange - https://nvd.nist.gov/developers/vulnerabilities#divResponse - https://csrc.nist.gov/schema/nvd/feed/1.1/nvd_cve_feed_json_1.1.schema @@ -522,26 +539,27 @@ Why not use node-semver ranges? - https://github.com/npm/node-semver#ranges -The node-semver spec is similar to this spec but is an AND of ORs with a few -issues: +The node-semver spec is similar to this spec but is an AND of ORs constraints +with a few practical issues: - The space means "AND", therefore whitespaces are significant. Having - significant whitespaces in string makes normalization more complicated and may - be a source of confusion if you remove the spaces from the string. Using an - ampersand as an "AND" operator in ``vers`` makes this explicit and avoids any - ambiguity. + significant whitespaces in a string makes normalization more complicated and + may be a source of confusion if you remove the spaces from the string. Using + a comma as an "AND" operator in ``vers`` makes this explicit and avoids the + ambiguity of a space. - There is no negation "!=" operator meaning that some version constraints are difficult to express and require combining < and > comparators. For instance stating that a vulnerability affects babel 6.2 or later but not babel 7.0 is - possible but difficult. + possible but complicated. - The advanced range syntax has grown to be rather complex using hyphen, stars, - carets and tilde constructs are all tied to the JavaScript and npm ways of - handling versions in their specific ecosystem and furthermore are bound to the - semver semantics and the npm implementation. These are not readily reusable - elsewhere. And these multiple comparators and modifiers make the grammar and - parsing more complex. + carets and tilde constructs that are all tied to the JavaScript and npm ways + of handling versions in their ecosystem and are bound furthermore to the + semver semantics and its npm implementation. These are not readily reusable + elsewhere and these extended multiple comparators and modifiers make the + notation grammar more complex to parse for a machine and harder to read for + human. Notations that are directly derived from node-semver as used in Rust and PHP Composer have the same issues. @@ -550,15 +568,17 @@ Composer have the same issues. Why not use Python pep-0440 ranges? ##################################### +See: + - https://www.python.org/dev/peps/pep-0440/#version-specifiers The Python pep-0440 "Version Identification and Dependency Specification" provides a comprehensive specification for Python package versioning and a -notation for "version specifiers" to express dependencies version constraints. +notation for "version specifiers" to express the version constraints of +dependencies. -This specification is mostly compatible with and is similar to this ``vers`` -spec, but has a richer notation with some aspects specific to the version used -only in the Python ecosystem. +This specification is similar to this ``vers`` spec, but has a richer notation +with some aspects specific to the versions used only in the Python ecosystem. - In particular pep-0440 uses tilde, triple equal and wildcard star operators that are specific to how two Python versions are compared. @@ -573,7 +593,7 @@ only in the Python ecosystem. Why not use Rubygems requirements notation? -############################################################## +############################################### - https://guides.rubygems.org/patterns/#declaring-dependencies @@ -584,20 +604,21 @@ implies semver versioning. This makes the notation impractical to reuse in places that do not use the same semver-like semantics. -Why not use more extra comparators such as a tilde, carret and star? -##################################################################### +Why not use fancier comparators such as a tilde, caret and star? +################################################################## -Several existing notations such as node-semver, rubygems or composer specs -provide syntactic shorthand such as: +Several existing notations such as used with npm, gem or python or composer +provide syntactic shorthands such as: -- a tilde prefix or ~> prefix as in "~1.3" or "~>1.2.3" +- a tilde prefix or ~> prefix or =~ as in "~1.3" or "~>1.2.3" - a caret ^ prefix as in "^ 1.2" -- using a star in a segment of a semver version as in "1.2.*" +- using a star in a segment of a version as in "1.2.*" - dash-separated ranges as in "1.2 - 1.4" -These range syntaxes can be reduced to a simpler notation here. Furthermore they -are designed for the structure of a version string (most often semver) as used -in one ecosystem and therefore are not reusable everywhere. +These range syntaxes can typcially be reduced to a set of simpler operators. +Furthermore they are designed for the structure of a version string (most often +semver) as used in one ecosystem and therefore are not reusable in another +ecosystem that would not use the version string conventions. References @@ -609,4 +630,4 @@ Here are some of the discussions that led to the creation of this specification: - https://github.com/package-url/purl-spec/issues/84 - https://github.com/package-url/purl-spec/pull/93 - https://github.com/nexB/vulnerablecode/issues/119 -- https://github.com/nexB/vulnerablecode/issues/140 \ No newline at end of file +- https://github.com/nexB/vulnerablecode/issues/140 From b70bbe9fdab31a8756893580c8691085608221b3 Mon Sep 17 00:00:00 2001 From: Philippe Ombredanne Date: Mon, 22 Nov 2021 12:05:48 +0100 Subject: [PATCH 12/12] Improve doc for existing schemes Improve documentation for existing versioning schems Explain why we do not use mathematical intervals Fix minor typos and improve formatting Signed-off-by: Philippe Ombredanne --- VERSION-RANGE-SPEC.rst | 41 +++++++++++++++++++++++++++-------------- 1 file changed, 27 insertions(+), 14 deletions(-) diff --git a/VERSION-RANGE-SPEC.rst b/VERSION-RANGE-SPEC.rst index a98fbefe..101a6644 100644 --- a/VERSION-RANGE-SPEC.rst +++ b/VERSION-RANGE-SPEC.rst @@ -61,35 +61,38 @@ in use: - Rubygems strongly suggest using ``semver`` for version but does not enforce it. As a result some use semver and several popular package do not use strict - semver. Runbygems use their own notation for version ranges. - https://guides.rubygems.org/patterns/#semantic-versioning + semver. Rubygems use their own notation for version ranges which ressembles + the ``node-semver`` notation with some subtle differences. + See https://guides.rubygems.org/patterns/#semantic-versioning -- node-semver ranges are used in npm at https://github.com/npm/node-semver#ranges +- ``node-semver`` ranges are used in npm at https://github.com/npm/node-semver#ranges with range semantics that are specific to ``semver`` and npm. - Dart pub versioning scheme is similar to ``node-semver`` and the documentation at https://dart.dev/tools/pub/versioning provides a comprehensive coverage of - the topic of versioning. Version resolution is using its own algorithm. + the topic of versioning. Version resolution uses its own algorithm. -- Python uses its own version and version ranges notation +- Python uses its own version and version ranges notation with notable + specificities on how how pre- and post-release suffixes are used https://www.python.org/dev/peps/pep-0440/ -- Debian and Ubuntu use their own notation +- Debian and Ubuntu use their own notation and are remarkabel for their use of + ``epochs`` to disambiguate versions. https://www.debian.org/doc/debian-policy/ch-relationships.html -- RPM distros use their own range notation +- RPM distros use their own range notation and use epochs like Debian. https://rpm-software-management.github.io/rpm/manual/dependencies.html -- Perl CPAN define its own version range notation similar to this specification - https://metacpan.org/pod/CPAN::Meta::Spec#Version-Ranges +- Perl CPAN defines its own version range notation similar to this specification + and uses two-segment versions. https://metacpan.org/pod/CPAN::Meta::Spec#Version-Ranges -- Apache Maven and NuGet use similar math intervals notation +- Apache Maven and NuGet use similar math intervals notation using brackets https://en.wikipedia.org/wiki/Interval_(mathematics) - Apache Maven http://maven.apache.org/enforcer/enforcer-rules/versionRanges.html - NuGet https://docs.microsoft.com/en-us/nuget/concepts/package-versioning#version-ranges -- gradle uses Apache Maven notation with extensions +- gradle uses Apache Maven notation with some extensions https://docs.gradle.org/current/userguide/single_versions.html - Gentoo and Alpine Linux use comparison operators similar to this specification: @@ -133,10 +136,10 @@ related topic: - Each package ecosystem may have evolved its own peculiar version string conventions, semantics and comparison procedure. -- For instance, semver is a prominent specification in this domain but this is +- For instance, ``semver`` is a prominent specification in this domain but this is just one of the many ways to structure a version string. -- Debian, RPM, PyPI, Rubygems, and Composer have their own subtly different +- Debian, RPM, PyPI, Rubygems, and Composer have their own subtly different approach on how to determine which version is greater or lesser. @@ -172,7 +175,7 @@ For example to define a set of versions that contains either version ``1.2.3``, or any versions greater than or equal to ``2.0.0`` but less than ``5.0.0`` using the ``node-semver`` versioning scheme, the version range specifier will be:: - vers:node-semver/1.2.3|>=2.0.0,<5.0.0 + vers:npm/1.2.3|>=2.0.0,<5.0.0 Each ```` in the pipe-separated list is either a simple constraint such as:: @@ -621,6 +624,16 @@ semver) as used in one ecosystem and therefore are not reusable in another ecosystem that would not use the version string conventions. +Why not use mathematical interval notation for ranges? +####################################################### + +Apache Maven and NuGet make use of a mathematical interval with "[" and ")" as a +syntax for version ranges. + +All other notations are using >, <, and = as base symbols for ranges. ``vers`` +reuses this approach because it is more common across package ecosystems. + + References ~~~~~~~~~~~~~~~~~~~~