Occasional errors accessing 'validate' endpoint and fetchcode/package_versions.py versions()/get_response()

[johnmhoran]

Working on the metadata, urls and versions PURL CLI commands (see #247),

• I occasionally get errors when trying to validate a PURL -- often a pkg:deb/debian/2ping (or similar deb/debian) that I'm testing at the time -- with the validate endpoint.
• With versions, I occasionally get a fetchcode/package_versions.py Error while fetching error that seems generated by https://github.com/nexB/fetchcode/blob/master/src/fetchcode/package_versions.py#L512-L534.

I'll add two files that contain examples for metadata and versions in which I ran the same command twice, separated by just a few seconds, and one threw an error while the other ran w/o error. I'm catching and printing response, response.text and response.json() values via my try/except to aid in the troubleshooting.

Before I began to catch the errors, my two most popular were TypeError: 'NoneType' object is not subscriptable and json.decoder.JSONDecodeError: Expecting value: line 2 column 1 (char 1). (Lots of google results for char0 but few for char 1, which some suggest means HTML received when JSON was expected. My file examples will show an incoming HTML conveying a Server 500 error, which might be related.)

[johnmhoran]

Examples from metadata and versions commands are attached.

purlcli-metadata-error-example.txt

purlcli-versions-error-example.txt

[johnmhoran]

@JonoYang Error while fetching 'https://sources.debian.org/api/src/2ping': 503 accompanies one of the two types of versions error -- don't think I've seen it running metadata or urls.

I seem to be unable to prevent this occasional error, which I think is triggered in fetchcode/package_versions.py (https://github.com/nexB/fetchcode/blob/master/src/fetchcode/package_versions.py#L205-L208 and https://github.com/nexB/fetchcode/blob/master/src/fetchcode/package_versions.py#L512-L534 working together). Do you have any suggestions?

(venv) Tue Feb 27, 2024 04:52 PM  /home/jmh/dev/nexb/purldb jmh (247-purlcli-update-validate-and-versions)
$ python -m purldb_toolkit.purlcli versions --purl pkg:gem/bundler-sass --purl pkg:deb/debian/2ping --output -
Error while fetching 'https://sources.debian.org/api/src/2ping': 503
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/jmh/dev/nexb/purldb/purldb-toolkit/src/purldb_toolkit/purlcli.py", line 767, in <module>
    purlcli()
  File "/home/jmh/dev/nexb/purldb/venv/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/home/jmh/dev/nexb/purldb/venv/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/home/jmh/dev/nexb/purldb/venv/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/jmh/dev/nexb/purldb/venv/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/jmh/dev/nexb/purldb/venv/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/home/jmh/dev/nexb/purldb/purldb-toolkit/src/purldb_toolkit/purlcli.py", line 628, in get_versions
    purl_versions = list_versions(purls, output, file, command_name)
  File "/home/jmh/dev/nexb/purldb/purldb-toolkit/src/purldb_toolkit/purlcli.py", line 678, in list_versions
    for package_version_object in list(versions(purl)):
  File "/home/jmh/dev/nexb/purldb/venv/lib/python3.10/site-packages/fetchcode/package_versions.py", line 205, in get_deb_versions_from_purl
    for release in response["versions"]:
TypeError: 'NoneType' object is not subscriptable

(venv) Tue Feb 27, 2024 04:53 PM  /home/jmh/dev/nexb/purldb jmh (247-purlcli-update-validate-and-versions)
$