Add Homebrew Package support by zaheerabbas21 · Pull Request #44 · aboutcode-org/fetchcode

zaheerabbas21 · 2020-10-11T16:29:05Z

Hello, I have added Homebrew support by looking at the previous code and the json API, but could not test locally because I did not know how to generate the homebrew.json json file so that I can pass it as the "expected result" in the test function.

Also I found the API of brew similar to rubygem's API, I have modified the code to match the brew json API format. Also while going through the brew json api docs https://formulae.brew.sh/docs/api/ , I found that the download link can be present in the "URL" or in the "URL"(see the above doc for the docker example) under another dict of "URLs" in the json api (the example of "a2ps" api mentioned in the issue). Please review the code that I have written, if further changes are required, do tell me , I will try to implement it. Also I am new to this repo, so I found no docs related to it (the commit msg doc or the overall working of the project), if there is any doc related to it, i would be happy to read and improve the project further. Thank you

TG1999 · 2020-10-11T19:41:44Z

Homebrew.json will be the expected output for a particular homebrew package, you can look at rubygems.json :)

zaheerabbas21 · 2020-10-11T23:11:35Z

@TG1999, have made a test function and added homebrew.json and homebrew_mock_data.json for node package. The test seems to work fine locally here, do a review when you have time, I am a bit new to open source in general, any feedback or advice would be highly appreciated. I mean I have pushed to the same pull request, should I have created another pull request for this, or is it fine? a bit confused about this. Anyway, do a review when you are free.

TG1999 · 2020-10-12T15:09:42Z

    versions = response.get("versions", [])
    for version in versions:
-        version_purl = PackageURL(type=purl.type, name=name, version=version.get("num"))
+        version_purl = PackageURL(


Please don't change these lines :), they are out of scope of this PR

Actually I have a setting of Format on save on my vscode, it happened unknowingly, will take care in the future. Thanks for the feedback.

TG1999 · 2020-10-12T15:11:33Z

+    base_path = "https://formulae.brew.sh/api/formula"
+    api_url = f"{base_path}/{name}.json"
+    response = get_response(api_url)
+    declared_license = response.get("license") or None


Don't use or None

will change it.

TG1999 · 2020-10-12T15:11:59Z

+    response = get_response(api_url)
+    declared_license = response.get("license") or None
+    homepage_url = response.get("homepage")
+    version = response.get("versions") or None


Don't use or None

will change it.

TG1999 · 2020-10-12T15:12:29Z

+    declared_license = response.get("license") or None
+    homepage_url = response.get("homepage")
+    version = response.get("versions") or None
+    download_url = response.get("urls")["stable"]["url"]


What if response does not have "urls" ?

Should I add try and catch or would you suggest something else?

The documentation of the homebrew json api tells that the download URL will be present inside a stable of URLs, refer here. In the docs, as they have mentioned, if we are using the base path as https://formulae.brew.sh/api/formula.json, then the download URL will be present inside the stable of urls, so I thought of leaving the download URL as such. Please do suggest what should I add it to cover the edge cases, so that it does not fail.

Look how these things are done in the above block of code

Ok, will look into it. Thanks for the feedback, will do the changes by tomorrow.

zaheerabbas21 · 2020-10-12T17:58:26Z

@TG1999, have made the changes following your review, do take a look when you have time. I am really sorry for all those commits, I am having issues pushing and pulling and rebasing it. Will Improve it, please suggest some resource to learn remote git, I know how to handle local git, but don't know why anytime I push some changes, it asks me to pull the data in and then merge conflict occurs. Again sorry for all the commits, will squash it once all the reviewed changes are done. Thanks for your time reviewing it.

TG1999 · 2020-10-14T07:52:21Z

+    declared_license = response.get("license")
+    homepage_url = response.get("homepage")
+    version = response.get("versions")
+    download_path = response.get("urls")["stable"]["url"]


urls = response.get('urls') or {{}} stable_url = urls.get('stable') or {} download_url = stable_url.get('url')

Remove line 345-349 and use this

oh ok, will replace that, sorry for these mistakes, will remember and learn continuing further. Thank you for the feedback

TG1999 · 2020-10-14T07:57:12Z

+    response = get_response(api_url)
+    declared_license = response.get("license")
+    homepage_url = response.get("homepage")
+    version = response.get("versions")


Extraction of version does not seem to be correct

Again sorry will correct this too.

zaheerabbas21 · 2020-10-14T08:36:58Z

@TG1999, have made the changes:

I used double quotes("") instead of single quotes('') in the code that you provided me to replace, so as to maintain the consistency and match the code that was written in the file. I hope that is not a problem
I have not included a case where if the versions are not available ie, I have changed the below line for the versions extraction:
version=response.get("versions")["stable"]. I did not know if I needed to add the case where the versions are not available in the API. If I should, do mention, will change it.
Ran the tests locally, tests are passed successfully.
I am really sorry for not writing code in a production manner, I am new to this, so thanks for helping out, hopefully, will get better at this once I get the hang of it. Again appreciate all the feedback.

TG1999 · 2020-10-14T08:39:50Z

Change the code for version

zaheerabbas21 · 2020-10-14T11:26:44Z

Change the code for version

@TG1999, made the change, give it a look when you are free. :)

zaheerabbas21 · 2020-10-17T09:55:12Z

@TG1999, just an update, have made the requested changes, please give a look when you are free.

steven-esser

@nk4456542 Thanks for this contribution.

See my comment for one slight problem with the code.

Additionally, it would be nice to add a few more test cases to ensure your code if working as it should in all cases.

You can find numerous homebrew forumale examples here: https://formulae.brew.sh/

steven-esser · 2020-10-17T15:50:31Z

+        "type": "homebrew",
+        "namespace": null,
+        "name": "node",
+        "version": null,


We should collect version data; it seems like there is a bug in your code, as I see you attempt to collect this version data in package.py (on line 344-345)

@MaJuRG, Thanks for pointing out the bug, I know this is supposed to be a good first issue, but I am really new to the development side, I appreciate all the help.

But I am stuck here, as you said I changed the version in the tests/data/homebrew.json file. But I do not know how to pass the versions to the Package to yield it, I tried the key version, but did not work. So please tell me how to pass the version variable to the package.

Here is the diff of the test when I ran python -m pytest -vv

Full diff: E [ E {'api_data_url': None, E 'api_url': 'https://formulae.brew.sh/api/formula/node.json', E 'bug_tracking_url': None, E 'code_view_url': None, E 'contains_source_code': None, E 'copyright': None, E 'declared_license': 'MIT', E 'dependencies': [], E 'description': None, E 'download_url': 'https://nodejs.org/dist/v14.13.1/node-v14.13.1.tar.gz', E 'homepage_url': 'https://nodejs.org/', E 'keywords': [], E 'license_expression': None, E 'md5': None, E 'name': 'node', E 'namespace': None, E 'notice_text': None, E 'parties': [], E 'primary_language': None, E 'purl': 'pkg:homebrew/node', E - 'qualifiers': {}, E + 'qualifiers': OrderedDict(), E 'release_date': None, E 'repository_download_url': None, E 'repository_homepage_url': None, E 'root_path': None, E 'sha1': None, E 'sha256': None, E 'sha512': None, E 'size': None, E 'source_packages': [], E 'subpath': None, E 'type': 'homebrew', E 'vcs_url': None, E - 'version': '14.4.0'}, E + 'version': None}, E ]

I also tried to check the above code and see if I could implement it in a way it is implemented above, but did not understand that, I am really sorry for this.
I am really sorry to disturb you, appreciate all the help. Thank you

zaheerabbas21 · 2020-10-22T11:42:58Z

@TG1999, could you please help me, I am stuck here: #44 (comment)

pombredanne · 2021-08-16T10:48:58Z

+    homepage_url = response.get("homepage")
+    versions = response.get("versions") or {}
+    version = versions.get("stable")
+    urls = response.get("urls") or {{}}


{{}} cannot work and does not work.

>>> type({{}}) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: unhashable type: 'dict'

pombredanne · 2022-02-09T14:10:01Z

@nk4456542 I am closing this for now for lack of response. Thank you for trying and and feel free to reopen if you want to finish this

Add Homebrew Package suppoort

3d85342

zaheerabbas21 added 2 commits October 12, 2020 04:27

Add support for homebrew packages

bc24444

Add test to homebrew package and improve the same.

54ad427

TG1999 reviewed Oct 12, 2020

View reviewed changes

Comment thread fetchcode/package.py

TG1999 reviewed Oct 12, 2020

View reviewed changes

zaheerabbas21 added 6 commits October 12, 2020 22:56

Add support for homebrew packages

daa099a

Add Homebrew Package suppoort

d5ae02d

Improve support for Homebrew Packages

ecd26a5

Improve the support for Homebrew Packages

443af63

Improve the support for Homebrew Packages

7cc56bf

Improve support for Homebrew Packages

3cb01dc

TG1999 reviewed Oct 14, 2020

View reviewed changes

Improve the code as per Review

de295e7

zaheerabbas21 added 2 commits October 14, 2020 16:50

Improve the code as per Review

72456f1

Improve the code as per Review

6804289

Change brew to Homebrew

3505cf0

steven-esser suggested changes Oct 17, 2020

View reviewed changes

pombredanne mentioned this pull request Aug 16, 2021

Add support for homebrew packages #47

Closed

pombredanne requested changes Aug 16, 2021

View reviewed changes

pombredanne closed this Feb 9, 2022

Uh oh!

Uh oh!

Conversation

zaheerabbas21 commented Oct 11, 2020

Uh oh!

TG1999 commented Oct 11, 2020

Uh oh!

zaheerabbas21 commented Oct 11, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

TG1999 Oct 12, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TG1999 Oct 12, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TG1999 Oct 12, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zaheerabbas21 commented Oct 12, 2020

Uh oh!

TG1999 Oct 14, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zaheerabbas21 commented Oct 14, 2020

Uh oh!

TG1999 commented Oct 14, 2020

Uh oh!

zaheerabbas21 commented Oct 14, 2020

Uh oh!

zaheerabbas21 commented Oct 17, 2020

Uh oh!

steven-esser left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zaheerabbas21 Oct 17, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zaheerabbas21 commented Oct 22, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pombredanne commented Feb 9, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

TG1999 Oct 12, 2020 •

edited

Loading

TG1999 Oct 12, 2020 •

edited

Loading

TG1999 Oct 12, 2020 •

edited

Loading

TG1999 Oct 14, 2020 •

edited

Loading

zaheerabbas21 Oct 17, 2020 •

edited

Loading