Infrastructure Guide#
This guide serves as an advanced version of the contributing documentation and contains the information on how we manage MetPy and the infrastructure behind it.
Versioning#
To manage identifying the version of the code, MetPy relies upon setuptools_scm. setuptools_scm
takes the current version of the
source from git tags and any additional commits. For development, the version will have a
string like 0.10.0.post209+gff2e549f.d20190918
, which comes from git describe
. This
version means that the current code is 209 commits past the 0.10.0 tag, on git hash
ff2e549f
, with local changes on top, last made on a date (indicated by d20190918
). For
a release, or non-git repo source dir, the version will just come from the most recent tag
(i.e. v0.10.0
). Our main versioning scheme matches vMajor.Minor.Bugfix
.
Documentation#
MetPy’s documentation is built using sphinx >= 2.1. API documentation is automatically
generated from docstrings, written using the
NumPy docstring standard.
There are also example scripts in the examples
directory, as well as our
User Guide tutorials in the tutorials
. Using the sphinx-gallery
extension, these scripts are executed and turned into a gallery of thumbnails. The
extension also makes these scripts available as Jupyter notebooks.
The documentation is hosted on GitHub Pages. The docs are
built automatically and uploaded upon pushes or merges to GitHub. Commits to main
end up
in our development version docs, while commits to versioned branches will update the
docs for the corresponding version, which are located in the appropriately named subdirectory
on the gh-pages
branch. We only maintain docs at the minor level, not the bugfix one.
The docs rely on the pydata-sphinx-theme
package for styling the docs, which needs to be
installed for any local doc builds. The gh-pages
branch has a GitHub Actions workflow that
handles generating a versions.json
file that controls what versions are displayed in the
selector on the website, as well as update the latest
symlink that points to the latest
version of the docs.
Other Tools#
Continuous integration is performed by
GitHub Actions.
This integration runs the unit tests on Linux for all supported versions of Python, as well
as runs against the minimum package versions, using PyPI packages. This also runs against
a (non-exhaustive) matrix of python versions on macOS and Windows. In addition to these tests,
GitHub actions also builds the documentation and runs the examples across multiple platforms
and Python versions, as well as checks for any broken web links. flake8
(along with a
variety of plugins found in ci/linting.txt
) is also run against the code to check
formatting using another job on GitHub Actions. As part of this linting job, the docs are also
checked using the doc8
tool, and spelling is checked using the codespell
.
Configurations for these are in a variety of files in .github/workflows
.
Test coverage is monitored by codecov.io.
The following services are used to track code quality:
We also maintain custom GitHub actions that automate additional tasks. Besides what’s
mentioned below as part of the release process, we have a script that automatically assigns
the most recent milestone to unmilestoned merged PRs (assign-milestone.yml
).
Additional automation is encouraged, and GitHub Actions, using the javascript and the
actions/github-script
action can greatly streamline the process of automating processes
using the GitHub API. For more information see:
Octokit Docs which is the built-in library for doing GitHub API work in javascript
github-script action repo which is the action that simplifies writing custom scripting
GitHub Actions Docs for all other things relating to GitHub Actions, like available events and workflow syntax
Releasing#
MetPy releases are managed using milestones on GitHub. Each release should have a milestone named with the appropriate version. All issues and Pull Requests that are included, or intended to be included, should be tagged with this milestone. This helps with planning and making sure things are not overlooked.
The release is created from the
GitHub Releases pages, by clicking “Draft a
new release”. The release should use a tag that adheres to our versioning, like v1.0.0. Add
a name for the release (can just be the version), and click the “Auto-generate release notes”
button. This uses the release.yml
template to populate the release notes using PRs that
were merged on the branch since the last release. This also calls out PRs made by new
contributors. Overall contributors are now listed on the GitHub page for a release.
The draft notes should be supplemented at the top with bullets summarizing the highlights of
the release that are of interest to our users.
Once the release notes are completed, click the “Publish release” button. This will actually create the tag on GitHub, triggering the package builds described below as well as new documentation builds.
PyPI#
Once the new release is published on GitHub, this will create the tag, which will trigger
new builds GitHub actions (see release.yml
). This runs no tests, but assumes those were
all working before the release was officially tagged. This action takes care of building
PyPI packages (the wheel and source distribution) and uploads to PyPI.
To build and upload manually (if for some reason it is necessary):
Do a pull locally to grab the new tag. This will ensure that
setuptools_scm
will give you the proper version.(optional) Perform a
git clean -f -x -d
from the root of the repository. This will delete everything not tracked by git, but will also ensure clean source distribution.MANIFEST.in
is set to include/exclude mostly correctly, but could miss some things.Run
python setup.py sdist bdist_wheel
(this requires thatwheel
is installed).Upload using
twine
:twine upload dist/*
, assuming thedist/
directory contains only files for this release. This upload process will include any changes to theREADME
as well as any updated flags fromsetup.py
.
Conda#
MetPy conda packages are automatically produced and uploaded to Anaconda.org thanks to conda-forge. Once the release is built and uploaded to PyPI, conda-forge’s automated bot infrastructure should (within anywhere from 30 minutes to a couple hours) produce a Pull Request on the MetPy feedstock, which handles updating the package. This repository contains the recipe for building MetPy’s conda packages.
If for some reason the bots fail or are delayed, a PR for the version update can be done manually. This should:
Update the version
Update the hash to match that of the new source distribution uploaded to PyPI
Reset the build number to 0 (if necessary)
Update the dependencies (and their versions) as necessary
The Pull Request will test building the packages on all the platforms. Once this succeeds, the Pull Request can be merged, which will trigger the final build and upload of the packages to anaconda.org.