[WIP]integrated with Virtualcodebase of scancode(First Approach) by pratik0316 · Pull Request #151 · aboutcode-org/deltacode

pratik0316 · 2020-03-28T20:51:42Z

In this PR I integrated the virtualcodebases of Scancode to the Deltacode.

Signed-off-by: Pratikrocks <pratikrocks.dey11@gmail.com>

…re_no_lic_change and test_Delta_one_None Signed-off-by: Pratikrocks <pratikrocks.dey11@gmail.com>

Signed-off-by: Pratikrocks <pratikrocks.dey11@gmail.com>

steven-esser · 2020-03-30T15:33:31Z

+        except :
+            pass


We want to handle the exception properly

@MaJuRG okay.
The exceptions were raised when we are getting some invalid sacn paths , and when we counter some some attributes like "fiingerprint" which VirtualCodebase is not supporting

steven-esser · 2020-03-30T15:33:54Z

+        self.new_files_count = 0 #keeps the count of the new file
+        self.old_files_count = 0 #keeps the count of old files
+        self.new_files = [] # a list of [[new file1:Original path],[new file2:Original Path],...]
+        self.old_files = [] # a list of [[old file1:Original path],[old file2:Original Path],...]
+        self.new_files_fingerprint = dict() # map of { {new_file1:fingerprint},{new_file2:fingerprint},...} it will be needed when we need the fingerprints
+        self.old_files_fingerprint = dict() # map of { {old_file1:fingerprint},{old_file2:fingerprint},...}


What is all this stuff and why is it needed?

self.new_files_count and self.old_files_count is to keep the track of the new files ,and old files.
I have added this to make the computation of the statistics easier,otherwise we would have to enumerate the virtual codebase objects to get the cont every time.
self.new_files_fingerprint and self.old_files_fingerprint it keeps a mapping of Resource objects files path from codebase1 and codebase2 with respect to the fingerprints which would be used in similarity comparisons.

@MaJuRG self.new_files and self.old_files it is list of lists comprising of [Resource objects,with their original path] ,Now we need to keep the track of the original path in the align_scans for alignment of the files.

Its fine to enumerate on the Virtualcodebase. Stats should be calculated in the end anyway, and optionally for the user. I hate having carrying around all these unneeded fields.

Also, codebase objects have counts already that we can use. There is no need to track this twice.

@MaJuRG , yes we can get rid of this self.new_files_count and self.old_files_count as they are already present in codebase objects as it is present in the headers of the codebase objects, but for the old files and new files and the fingerprint, I think it is better to have the enumeration done one time and cache those files in an array, else we will again need to enumerate it whenever required.

@MaJuRG used the counts from the codebase , removed the additional variables for the files_count

…and non similarity matching Signed-off-by: Pratikrocks <pratikrocks.dey11@gmail.com>

pratik0316 · 2020-03-31T19:47:47Z

@MaJuRG the test cases like (similarity matching1 and similarity matching2 ,non similarity matching1 ) are passing, the output in json file is appearing as expected.
However the code for this kinda test cases can not be modified

    def test_Delta_create_object_removed(self):
        new = None
        old = models.File({'path': 'path/removed.txt'})

        delta = deltacode.Delta(0, new, old)
        delta.factors.append('removed')

        assert type(delta.new_file) == type(None)
        assert delta.old_file.path == 'path/removed.txt'
        assert 'removed' in delta.factors
        assert delta.score == 0

as they are treating the old and new file as File objects and we are using the ScannedResources object for the old and the file.So should I modify test case like this?

Signed-off-by: Pratikrocks <pratikrocks.dey11@gmail.com>

pratik0316 · 2020-04-12T16:31:12Z

@MaJuRG , I replaced some of the json files which were scanned with ScanCode version 2 with approximately similar files scanned with Scancode Version 3 link , and modified some of the test cases.
This resulted in passing the pytests for test_cli.py.
Kindly review the changes once :)

steven-esser

There is a lot to be done here before it can be considered for merging.

steven-esser · 2020-04-13T22:47:00Z

+        except Exception as exception:
+            click.secho(exception.message ,fg = "red")


What exact exceptions can occur here?

@MaJuRG , the error faced here are following two types:
1.When the paths of the json files for the are not valid.
2.When the VirtualCode base gets an unnecessary fields during the scanning .
However the 2nd error could be resolved if we use scancode versionn3 during the scanning of the json files.

steven-esser · 2020-04-13T22:48:09Z

+        self.new_files_count = 0 #keeps the count of the new file
+        self.old_files_count = 0 #keeps the count of old files
+        self.new_files = [] # a list of [[new file1:Original path],[new file2:Original Path],...]
+        self.old_files = [] # a list of [[old file1:Original path],[old file2:Original Path],...]
+        self.new_files_fingerprint = dict() # map of { {new_file1:fingerprint},{new_file2:fingerprint},...} it will be needed when we need the fingerprints
+        self.old_files_fingerprint = dict() # map of { {old_file1:fingerprint},{old_file2:fingerprint},...}


Its fine to enumerate on the Virtualcodebase. Stats should be calculated in the end anyway, and optionally for the user. I hate having carrying around all these unneeded fields.

steven-esser · 2020-04-13T22:48:55Z

-
-        if self.new.path != '' and self.old.path != '':
+
+        if self.codebase1 != None and self.codebase2 != None:


Can we just have a check for the inverse of this so we dont write everything under an unneeded indentation?

@MaJuRG yeah , trying for it , I think it would be a better approach

steven-esser · 2020-04-13T22:49:26Z

+                try :
+                    self.new_files_fingerprint[obj.path] = obj.fingerprint
+                except AttributeError:
+                    self.new_files_fingerprint[obj.path] = None
+                if obj.is_file:
+                    # increment the new files count
+                    self.new_files_count += 1


Why not use dict.get() method instead of catching exception?

@MaJuRG it would be nice , changing it

@MaJuRG , the dict.get() method can not be used here as obj is not a dictionary it is a Virtualcodebase object.

steven-esser · 2020-04-13T22:49:58Z

+    def enumerate_files_from_codebases(self):
+       """
+       An method which call the utility function get_files for generating the codebase
+       """
+       self.get_files(self.codebase1,is_new = True)
+       self.get_files(self.codebase2,is_new = False)


This function's name is unrelated to what it actually does.

@MaJuRG , I think interchanging the function names get_files and enumerate_files_from_codebases would better match with the situation

changed the function name

steven-esser · 2020-04-13T22:50:42Z

+        self.new_files_count = 0 #keeps the count of the new file
+        self.old_files_count = 0 #keeps the count of old files
+        self.new_files = [] # a list of [[new file1:Original path],[new file2:Original Path],...]
+        self.old_files = [] # a list of [[old file1:Original path],[old file2:Original Path],...]
+        self.new_files_fingerprint = dict() # map of { {new_file1:fingerprint},{new_file2:fingerprint},...} it will be needed when we need the fingerprints
+        self.old_files_fingerprint = dict() # map of { {old_file1:fingerprint},{old_file2:fingerprint},...}


Also, codebase objects have counts already that we can use. There is no need to track this twice.

steven-esser · 2020-04-13T22:51:25Z

+                all_licenses.append(d)
+            return all_licenses
+
+    def new_file_to_dict(self,deltacode):


Why is this not a method of the File class?

@MaJuRG , I will be moving all these methods(license and copyright) to the file class

steven-esser · 2020-04-13T22:51:39Z

+            ])
+
+
+    def old_file_to_dict(self,deltacode):


Why are we repeating this function?

@MaJuRG , I will be truncating this unnecessary functions.

combined redundant functions def old_file_to_dict(self,deltacode) and def new_file_to_dict(self,deltacode) , to a single function.

pratik0316 · 2020-04-14T05:01:54Z

Thanks @MaJuRG for the review , I will be making the suggested changes ASAP :)

Signed-off-by: Pratikrocks <pratikrocks.dey11@gmail.com>

pratik0316 · 2020-04-27T14:19:54Z

+
+
+def fetch_files(location):
+    codebase = VirtualCodebase(location)


This function has been extensively added to test_utils so as to prevent the creation of deltacode object for the test cases when only the Resource files are required

Signed-off-by: Pratikrocks <pratikrocks.dey11@gmail.com>

…te one Signed-off-by: Pratikrocks <pratikrocks.dey11@gmail.com>

Signed-off-by: Pratikrocks <pratikrocks.dey11@gmail.com>

steven-esser · 2021-03-18T19:21:19Z

Closing for now due to stagnation.

pratik0316 · 2021-03-19T02:05:59Z

I was willing to continue this , due to lack of input I discontinued for a moment , anyway if it's anymore required I will work on it :)

steven-esser · 2021-03-19T13:52:06Z

@Pratikrocks Yes this may have fallen through the cracks, sorry about this lack of feedback.

The deltacode repo has been updated since this PR was made with a number of configuration changes. If you can, please rebase your branch, or add your changes to a new branch off of the current develop HEAD. I will make sure to leave prompt feedback if you do!

integrated with Virtualcodebase of scancode

17778de

Signed-off-by: Pratikrocks <pratikrocks.dey11@gmail.com>

pratik0316 force-pushed the VirtualCodebase branch from 969e868 to 17778de Compare March 28, 2020 20:52

fixed for test cases tests/test_deltacode.py::TestDeltacode::test_sco…

d64de2b

…re_no_lic_change and test_Delta_one_None Signed-off-by: Pratikrocks <pratikrocks.dey11@gmail.com>

pratik0316 force-pushed the VirtualCodebase branch from d428273 to d64de2b Compare March 29, 2020 21:32

passed for test_score_no_copyright _no_copyright_changes

60b3017

Signed-off-by: Pratikrocks <pratikrocks.dey11@gmail.com>

pratik0316 force-pushed the VirtualCodebase branch from 76141f6 to 60b3017 Compare March 30, 2020 05:48

changed the passing parameter to_dict()

e40fda3

Signed-off-by: Pratikrocks <pratikrocks.dey11@gmail.com>

pratik0316 force-pushed the VirtualCodebase branch from 0cb65bb to e40fda3 Compare March 30, 2020 15:20

steven-esser suggested changes Mar 30, 2020

View reviewed changes

Fixed the issues for some test cases like similarity matching1 and 2 …

7e78c28

…and non similarity matching Signed-off-by: Pratikrocks <pratikrocks.dey11@gmail.com>

pratik0316 force-pushed the VirtualCodebase branch from 760640b to 7e78c28 Compare March 31, 2020 06:46

test_deltacode_to_dict_modified

1b639c6

Signed-off-by: Pratikrocks <pratikrocks.dey11@gmail.com>

pratik0316 force-pushed the VirtualCodebase branch from be8fc7a to 1b639c6 Compare April 1, 2020 15:19

fixed for test cases: sending None paths in test_deltacode.py

03c8d15

Signed-off-by: Pratikrocks <pratikrocks.dey11@gmail.com>

pratik0316 force-pushed the VirtualCodebase branch from 99a0393 to 03c8d15 Compare April 3, 2020 08:48

pratik0316 requested a review from steven-esser April 3, 2020 12:57

changed the json files which were scanned with version 2 to version 3

c6079d1

Signed-off-by: Pratikrocks <pratikrocks.dey11@gmail.com>

pratik0316 force-pushed the VirtualCodebase branch from ffe2a3e to c6079d1 Compare April 12, 2020 15:33

steven-esser suggested changes Apr 13, 2020

View reviewed changes

pratik0316 changed the title ~~integrated with Virtualcodebase of scancode(First Approach)~~ [WIP]integrated with Virtualcodebase of scancode(First Approach) Apr 23, 2020

changed redundant functions

709ceee

Signed-off-by: Pratikrocks <pratikrocks.dey11@gmail.com>

pratik0316 force-pushed the VirtualCodebase branch from 830154f to 709ceee Compare April 23, 2020 15:15

removed pyvenv.cfg

edc0638

Signed-off-by: Pratikrocks <pratikrocks.dey11@gmail.com>

pratik0316 force-pushed the VirtualCodebase branch from d8fc89c to edc0638 Compare April 23, 2020 15:17

pratik0316 added 3 commits April 24, 2020 01:20

appende the key errors

6a273b9

Signed-off-by: Pratikrocks <pratikrocks.dey11@gmail.com>

fixed some test cases for tests/test_utils.py

730a99a

Signed-off-by: Pratikrocks <pratikrocks.dey11@gmail.com>

added fetch files without creating deltacode object

017c52a

Signed-off-by: Pratikrocks <pratikrocks.dey11@gmail.com>

pratik0316 commented Apr 27, 2020

View reviewed changes

pratik0316 added 5 commits April 28, 2020 12:03

fixed test cases for test_align_tress_zlib_failing

f0affab

Signed-off-by: Pratikrocks <pratikrocks.dey11@gmail.com>

used the files count of the VirtualCodebase instead creating a separa…

1e5e7fb

…te one Signed-off-by: Pratikrocks <pratikrocks.dey11@gmail.com>

fixed for test_update_from_license_info_copyleft_to_different_copyleft

31bd6b0

Signed-off-by: Pratikrocks <pratikrocks.dey11@gmail.com>

fixed the original paths

e26ef87

Signed-off-by: Pratikrocks <pratikrocks.dey11@gmail.com>

few changes in checking conditionals in init.py

652b5f5

Signed-off-by: Pratikrocks <pratikrocks.dey11@gmail.com>

steven-esser closed this Mar 18, 2021

		except Exception as exception:
		click.secho(exception.message ,fg = "red")


		if self.new.path != '' and self.old.path != '':

		if self.codebase1 != None and self.codebase2 != None:



		def fetch_files(location):
		codebase = VirtualCodebase(location)

Uh oh!

Uh oh!

Conversation

pratik0316 commented Mar 28, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pratik0316 commented Mar 31, 2020

Uh oh!

pratik0316 commented Apr 12, 2020

Uh oh!

steven-esser left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pratik0316 commented Apr 14, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

steven-esser commented Mar 18, 2021

Uh oh!

pratik0316 commented Mar 19, 2021

Uh oh!

steven-esser commented Mar 19, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development