From 71d8dad4444f3de7a66a776e8848b0f1d1b1e201 Mon Sep 17 00:00:00 2001 From: Jono Yang Date: Wed, 8 Sep 2021 12:12:48 -0700 Subject: [PATCH] Update READMEs Signed-off-by: Jono Yang --- README.rst | 156 +-------------------------------------- docs/skeleton-usage.rst | 157 ++++++++++++++++++++++++++++++++++++++++ etc/scripts/README.rst | 147 +++++++++++++++++++++++++++++++++++++ 3 files changed, 305 insertions(+), 155 deletions(-) create mode 100644 docs/skeleton-usage.rst create mode 100755 etc/scripts/README.rst diff --git a/README.rst b/README.rst index deaaa341d0..41736895e1 100644 --- a/README.rst +++ b/README.rst @@ -9,161 +9,7 @@ our existing ones as well. Usage ===== -A brand new project -------------------- -.. code-block:: bash - - git init my-new-repo - cd my-new-repo - git pull git@github.com:nexB/skeleton - - # Create the new repo on GitHub, then update your remote - git remote set-url origin git@github.com:nexB/your-new-repo.git - -From here, you can make the appropriate changes to the files for your specific project. - -Update an existing project ---------------------------- -.. code-block:: bash - - cd my-existing-project - git remote add skeleton git@github.com:nexB/skeleton - git fetch skeleton - git merge skeleton/main --allow-unrelated-histories - -This is also the workflow to use when updating the skeleton files in any given repository. - -Customizing ------------ - -You typically want to perform these customizations: - -- remove or update the src/README.rst and tests/README.rst files -- set project info and dependencies in setup.cfg -- check the configure and configure.bat defaults - -Initializing a project ----------------------- - -All projects using the skeleton will be expected to pull all of it dependencies -from thirdparty.aboutcode.org/pypi or the local thirdparty directory, using -requirements.txt and/or requirements-dev.txt to determine what version of a -package to collect. By default, PyPI will not be used to find and collect -packages from. - -In the case where we are starting a new project where we do not have -requirements.txt and requirements-dev.txt and whose dependencies are not yet on -thirdparty.aboutcode.org/pypi, we run the following command after adding and -customizing the skeleton files to your project: - -.. code-block:: bash - - ./configure --init - -This will initialize the virtual environment for the project, pull in the -dependencies from PyPI and add them to the virtual environment. - -Generating requirements.txt and requirements-dev.txt ----------------------------------------------------- - -After the project has been initialized, we can generate the requirements.txt and -requirements-dev.txt files. - -Ensure the virtual environment is enabled. - -.. code-block:: bash - - source venv/bin/activate - -To generate requirements.txt: - -.. code-block:: bash - - python etc/scripts/gen_requirements.py -s venv/lib/python/site-packages/ - -Replace \ with the version number of the Python being used, for example: ``venv/lib/python3.6/site-packages/`` - -To generate requirements-dev.txt after requirements.txt has been generated: - -.. code-block:: bash - ./configure --init --dev - python etc/scripts/gen_requirements_dev.py -s venv/lib/python/site-packages/ - -Note: on Windows, the ``site-packages`` directory is located at ``venv\Lib\site-packages\`` - -.. code-block:: bash - - python .\\etc\\scripts\\gen_requirements.py -s .\\venv\\Lib\\site-packages\\ - .\configure --init --dev - python .\\etc\\scripts\\gen_requirements_dev.py -s .\\venv\\Lib\\site-packages\\ - -Collecting and generating ABOUT files for dependencies ------------------------------------------------------- - -Ensure that the dependencies used by ``etc/scripts/bootstrap.py`` are installed: - -.. code-block:: bash - - pip install -r etc/scripts/requirements.txt - -Once we have requirements.txt and requirements-dev.txt, we can fetch the project -dependencies as wheels and generate ABOUT files for them: - -.. code-block:: bash - - python etc/scripts/bootstrap.py -r requirements.txt -r requirements-dev.txt --with-deps - -There may be issues with the generated ABOUT files, which will have to be -corrected. You can check to see if your corrections are valid by running: - -.. code-block:: bash - - python etc/scripts/check_thirdparty.py -d thirdparty - -Once the wheels are collected and the ABOUT files are generated and correct, -upload them to thirdparty.aboutcode.org/pypi by placing the wheels and ABOUT -files from the thirdparty directory to the pypi directory at -https://github.com/nexB/thirdparty-packages - - -Usage after project initialization ----------------------------------- - -Once the ``requirements.txt`` and ``requirements-dev.txt`` have been generated -and the project dependencies and their ABOUT files have been uploaded to -thirdparty.aboutcode.org/pypi, you can configure the project without using the -``--init`` option. - -If the virtual env for the project becomes polluted, or you would like to remove -it, use the ``--clean`` option: - -.. code-block:: bash - - ./configure --clean - -Then you can run ``./configure`` again to set up the project virtual environment. - -To set up the project for development use: - -.. code-block:: bash - - ./configure --dev - -To update the project dependencies (adding, removing, updating packages, etc.), -update the dependencies in ``setup.cfg``, then run: - -.. code-block:: bash - - ./configure --clean # Remove existing virtual environment - ./configure --init # Create project virtual environment, pull in new dependencies - source venv/bin/activate # Ensure virtual environment is activated - python etc/scripts/gen_requirements.py -s venv/lib/python/site-packages/ # Regenerate requirements.txt - python etc/scripts/gen_requirements_dev.py -s venv/lib/python/site-packages/ # Regenerate requirements-dev.txt - pip install -r etc/scripts/requirements.txt # Install dependencies needed by etc/scripts/bootstrap.py - python etc/scripts/bootstrap.py -r requirements.txt -r requirements-dev.txt --with-deps # Collect dependency wheels and their ABOUT files - -Ensure that the generated ABOUT files are valid, then take the dependency wheels -and ABOUT files and upload them to thirdparty.aboutcode.org/pypi. +Usage instructions can be found in ``docs/skeleton-usage.rst``. Release Notes ============= diff --git a/docs/skeleton-usage.rst b/docs/skeleton-usage.rst new file mode 100644 index 0000000000..7d16259c74 --- /dev/null +++ b/docs/skeleton-usage.rst @@ -0,0 +1,157 @@ +Usage +===== +A brand new project +------------------- +.. code-block:: bash + + git init my-new-repo + cd my-new-repo + git pull git@github.com:nexB/skeleton + + # Create the new repo on GitHub, then update your remote + git remote set-url origin git@github.com:nexB/your-new-repo.git + +From here, you can make the appropriate changes to the files for your specific project. + +Update an existing project +--------------------------- +.. code-block:: bash + + cd my-existing-project + git remote add skeleton git@github.com:nexB/skeleton + git fetch skeleton + git merge skeleton/main --allow-unrelated-histories + +This is also the workflow to use when updating the skeleton files in any given repository. + +Customizing +----------- + +You typically want to perform these customizations: + +- remove or update the src/README.rst and tests/README.rst files +- set project info and dependencies in setup.cfg +- check the configure and configure.bat defaults + +Initializing a project +---------------------- + +All projects using the skeleton will be expected to pull all of it dependencies +from thirdparty.aboutcode.org/pypi or the local thirdparty directory, using +requirements.txt and/or requirements-dev.txt to determine what version of a +package to collect. By default, PyPI will not be used to find and collect +packages from. + +In the case where we are starting a new project where we do not have +requirements.txt and requirements-dev.txt and whose dependencies are not yet on +thirdparty.aboutcode.org/pypi, we run the following command after adding and +customizing the skeleton files to your project: + +.. code-block:: bash + + ./configure --init + +This will initialize the virtual environment for the project, pull in the +dependencies from PyPI and add them to the virtual environment. + +Generating requirements.txt and requirements-dev.txt +---------------------------------------------------- + +After the project has been initialized, we can generate the requirements.txt and +requirements-dev.txt files. + +Ensure the virtual environment is enabled. + +.. code-block:: bash + + source venv/bin/activate + +To generate requirements.txt: + +.. code-block:: bash + + python etc/scripts/gen_requirements.py -s venv/lib/python/site-packages/ + +Replace \ with the version number of the Python being used, for example: ``venv/lib/python3.6/site-packages/`` + +To generate requirements-dev.txt after requirements.txt has been generated: + +.. code-block:: bash + ./configure --init --dev + python etc/scripts/gen_requirements_dev.py -s venv/lib/python/site-packages/ + +Note: on Windows, the ``site-packages`` directory is located at ``venv\Lib\site-packages\`` + +.. code-block:: bash + + python .\\etc\\scripts\\gen_requirements.py -s .\\venv\\Lib\\site-packages\\ + .\configure --init --dev + python .\\etc\\scripts\\gen_requirements_dev.py -s .\\venv\\Lib\\site-packages\\ + +Collecting and generating ABOUT files for dependencies +------------------------------------------------------ + +Ensure that the dependencies used by ``etc/scripts/bootstrap.py`` are installed: + +.. code-block:: bash + + pip install -r etc/scripts/requirements.txt + +Once we have requirements.txt and requirements-dev.txt, we can fetch the project +dependencies as wheels and generate ABOUT files for them: + +.. code-block:: bash + + python etc/scripts/bootstrap.py -r requirements.txt -r requirements-dev.txt --with-deps + +There may be issues with the generated ABOUT files, which will have to be +corrected. You can check to see if your corrections are valid by running: + +.. code-block:: bash + + python etc/scripts/check_thirdparty.py -d thirdparty + +Once the wheels are collected and the ABOUT files are generated and correct, +upload them to thirdparty.aboutcode.org/pypi by placing the wheels and ABOUT +files from the thirdparty directory to the pypi directory at +https://github.com/nexB/thirdparty-packages + + +Usage after project initialization +---------------------------------- + +Once the ``requirements.txt`` and ``requirements-dev.txt`` have been generated +and the project dependencies and their ABOUT files have been uploaded to +thirdparty.aboutcode.org/pypi, you can configure the project without using the +``--init`` option. + +If the virtual env for the project becomes polluted, or you would like to remove +it, use the ``--clean`` option: + +.. code-block:: bash + + ./configure --clean + +Then you can run ``./configure`` again to set up the project virtual environment. + +To set up the project for development use: + +.. code-block:: bash + + ./configure --dev + +To update the project dependencies (adding, removing, updating packages, etc.), +update the dependencies in ``setup.cfg``, then run: + +.. code-block:: bash + + ./configure --clean # Remove existing virtual environment + ./configure --init # Create project virtual environment, pull in new dependencies + source venv/bin/activate # Ensure virtual environment is activated + python etc/scripts/gen_requirements.py -s venv/lib/python/site-packages/ # Regenerate requirements.txt + python etc/scripts/gen_requirements_dev.py -s venv/lib/python/site-packages/ # Regenerate requirements-dev.txt + pip install -r etc/scripts/requirements.txt # Install dependencies needed by etc/scripts/bootstrap.py + python etc/scripts/bootstrap.py -r requirements.txt -r requirements-dev.txt --with-deps # Collect dependency wheels and their ABOUT files + +Ensure that the generated ABOUT files are valid, then take the dependency wheels +and ABOUT files and upload them to thirdparty.aboutcode.org/pypi. diff --git a/etc/scripts/README.rst b/etc/scripts/README.rst new file mode 100755 index 0000000000..4cb6ec7da2 --- /dev/null +++ b/etc/scripts/README.rst @@ -0,0 +1,147 @@ +This directory contains the tools to: + +- manage a directory of thirdparty Python package source, wheels and metadata: + pin, build, update, document and publish to a PyPI-like repo (GitHub release) + +- build and publish scancode releases as wheel, sources and OS-specific bundles. + + +NOTE: These are tested to run ONLY on Linux. + + +Thirdparty packages management scripts +====================================== + +Pre-requisites +-------------- + +* There are two run "modes": + + * To generate or update pip requirement files, you need to start with a clean + virtualenv as instructed below (This is to avoid injecting requirements + specific to the tools here in the main requirements). + + * For other usages, the tools here can run either in their own isolated + virtualenv best or in the the main configured development virtualenv. + These requireements need to be installed:: + + pip install --requirement etc/release/requirements.txt + +TODO: we need to pin the versions of these tools + + + +Generate or update pip requirement files +---------------------------------------- + +Scripts +~~~~~~~ + +**gen_requirements.py**: create/update requirements files from currently + installed requirements. + +**gen_requirements_dev.py** does the same but can subtract the main requirements + to get extra requirements used in only development. + + +Usage +~~~~~ + +The sequence of commands to run are: + + +* Start with these to generate the main pip requirements file:: + + ./configure --clean + ./configure + python etc/release/gen_requirements.py --site-packages-dir + +* You can optionally install or update extra main requirements after the + ./configure step such that these are included in the generated main requirements. + +* Optionally, generate a development pip requirements file by running these:: + + ./configure --clean + ./configure --dev + python etc/release/gen_requirements_dev.py --site-packages-dir + +* You can optionally install or update extra dev requirements after the + ./configure step such that these are included in the generated dev + requirements. + +Notes: we generate development requirements after the main as this step requires +the main requirements.txt to be up-to-date first. See **gen_requirements.py and +gen_requirements_dev.py** --help for details. + +Note: this does NOT hash requirements for now. + +Note: Be aware that if you are using "conditional" requirements (e.g. only for +OS or Python versions) in setup.py/setp.cfg/requirements.txt as these are NOT +yet supported. + + +Populate a thirdparty directory with wheels, sources, .ABOUT and license files +------------------------------------------------------------------------------ + +Scripts +~~~~~~~ + +* **fetch_requirements.py** will fetch package wheels, their ABOUT, LICENSE and + NOTICE files to populate a local a thirdparty directory strictly from our + remote repo and using only pinned packages listed in one or more pip + requirements file(s). Fetch only requirements for specific python versions and + operating systems. Optionally fetch the corresponding source distributions. + +* **publish_files.py** will upload/sync a thirdparty directory of files to our + remote repo. Requires a GitHub personal access token. + +* **build_wheels.py** will build a package binary wheel for multiple OS and + python versions. Optionally wheels that contain native code are built + remotely. Dependent wheels are optionally included. Requires Azure credentials + and tokens if building wheels remotely on multiple operatin systems. + +* **fix_thirdparty.py** will fix a thirdparty directory with a best effort to + add missing wheels, sources archives, create or fetch or fix .ABOUT, .NOTICE + and .LICENSE files. Requires Azure credentials and tokens if requesting the + build of missing wheels remotely on multiple operatin systems. + +* **check_thirdparty.py** will check a thirdparty directory for errors. + +* **bootstrap.py** will bootstrap a thirdparty directory from a requirements + file(s) to add or build missing wheels, sources archives and create .ABOUT, + .NOTICE and .LICENSE files. Requires Azure credentials and tokens if + requesting the build of missing wheels remotely on multiple operatin systems. + + + +Usage +~~~~~ + +See each command line --help option for details. + +* (TODO) **add_package.py** will add or update a Python package including wheels, + sources and ABOUT files and this for multiple Python version and OSes(for use + with upload_packages.py afterwards) You will need an Azure personal access + token for buidling binaries and an optional DejaCode API key to post and fetch + new package versions there. TODO: explain how we use romp + + +Upgrade virtualenv app +---------------------- + +The bundled virtualenv.pyz has to be upgraded by hand and is stored under +etc/thirdparty + +* Fetch https://github.com/pypa/get-virtualenv/raw//public/virtualenv.pyz + for instance https://github.com/pypa/get-virtualenv/raw/20.2.2/public/virtualenv.pyz + and save to thirdparty and update the ABOUT and LICENSE files as needed. + +* This virtualenv app contains also bundled pip, wheel and setuptools that are + essential for the installation to work. + + +Other files +=========== + +The other files and scripts are test, support and utility modules used by the +main scripts documented here.