Make release #1059

joshmoore · 2019-03-03T12:08:06Z

close #761

Release make targets using versioneer for automatically bumping the version number. The general strategy is that once all PRs/commits have been merged, a tag is created locally using git. After that someone "preps", "verifies", and then "uploads" (names open for discussion).

The version numbers from versioneer currently follow PEP 440 (though a number of alternatives are available). All non-release tags include a suffix beginning with +:

	version	is_release
from source	0.0.33+20.gac50fe06	False
pip install .	0.0.33+20.gac50fe06	False
pip install -e .	0.0.33+20.gac50fe06	False
pip install dist/*.gz (no tag)	0.0.33+20.gac50fe06	False
pip install dist/*.gz (tagged)	0.0.34	True

This PR does not try to:

automate any of the build steps in travis or elsewhere
automate creation of AUTHORS, ChangeLog files, and/or detect semver-levels ("breaking")
check requirements since that is in flux (can be handled afterwards)

Other minor points worth noting:

setup.py metadata has been moved to setup.cfg and updated with URLs
fixes rename of run_notebooks to run-notebooks (This looks to be untested)
periodic updates of versioneer might be in order
added a make clean target
__is_release_tag__ is still a string

codecov-io · 2019-03-03T12:19:47Z

Codecov Report

Merging #1059 into master will decrease coverage by 2.04%.
The diff coverage is 45.22%.

@@            Coverage Diff            @@
##           master   #1059      +/-   ##
=========================================
- Coverage   90.85%   88.8%   -2.05%     
=========================================
  Files         164     165       +1     
  Lines        6146    6433     +287     
=========================================
+ Hits         5584    5713     +129     
- Misses        562     720     +158

Impacted Files	Coverage Δ
starfish/_version.py	`44.6% <44.6%> (ø)`
starfish/__init__.py	`90% <80%> (-3.75%)`	⬇️
starfish/imagestack/parser/numpy/__init__.py	`81.42% <0%> (-3.65%)`	⬇️
starfish/experiment/builder/defaultproviders.py	`95.12% <0%> (ø)`	⬆️
starfish/imagestack/parser/_tiledata.py	`68.75% <0%> (ø)`	⬆️
starfish/imagestack/parser/tileset/_parser.py	`84.94% <0%> (+0.33%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a59cb6a...9e7e3a7. Read the comment docs.

Makefile

ttung · 2019-03-04T23:52:50Z

Makefile

+	python setup.py sdist;                            \
+	pip install dist/starfish-$$VERSION.tar.gz
+
+release-verify: release-check slow


This doesn't install your to-be-released version, right? So this is checking source against the notebooks? Would it be preferable to use the environment prepped in release-prep and run it there?

Prep does. At the moment, it's very much setup to use the three in order. prep and verify can be combined if you'd prefer.

Would it make sense to make release-verify explicitly depend on release-prep then?

Happy either way. If everyone prefers, maybe it's just release-build and release-publish.

ttung · 2019-03-04T23:53:33Z

Makefile

+	@VERSION=$(shell sh -c "git describe --exact") && \
+	printf '\n# Please execute the following steps\n';\
+	echo git push origin $$VERSION;                   \
+	echo docker push $(DOCKER_IMAGE);                 \


Who owns the docker image rights? I presume not everyone can just push it?

At the moment, I do. Unfortunately, it can't be synced with github teams. I'll pass over to whomever. Can you confirm that you are also @ttung on Docker Hub?

I don't know. :)

Once this lands, I think it would make sense to distribute it to everyone who regularly commits here (yourself, me, @ambrosejcarr, @shanaxel42, @kne42)

I am ambrosejcarr on dockerhub.

Makefile

ttung · 2019-03-04T23:55:53Z

setup.cfg

+[metadata]
+name = starfish
+summary = Pipelines and pipeline components for the analysis of image-based transcriptomics data
+author = Deep Ganguli


I guess we should fix this at some time to include more peeps. :)

Definitely leaving that up to someone else 😄

joshmoore · 2019-03-07T21:22:57Z

~~I'll re-test and then update these instructions tomorrow but~~
In case anyone wants to have a play today: (instructions updated on Mar. 13)

fork this repository and re-clone it to a new location (you can delete all this after testing): git clone git@github.com:XXX/starfish
add my remote and checkout this branch (make-release)
create a docker hub account and then go to the equivalent of https://cloud.docker.com/u/joshmoore/repository/create (name isn't overly important)
test that docker login works
have your pypi password ready for twine'ing

With the above, you should be able to do:

make release-changelog > CHANGELOG.md.next
manually edit CHANGELOG and then commit it. Important are the '##' markers
make release-tag TAG=0.0.34 (no GPG at the moment)
make release-prep
make release-verify
make release-upload These commands need to be run manually in bash.
make clean

cc: @shanaxel42 @ttung @ambrosejcarr

(and yes, once the dust settles, this could become make TAG=0.0.34 release which opens a text editor, and ends with the upload commands).

joshmoore · 2019-03-08T08:52:39Z

Preparation steps:

git clone git@github.com:REPLACE_ME/starfish release-test
git remote add josh git://github.com/joshmoore/starfish
git fetch josh
git checkout josh/make-release
git describe --tags
0.0.33-30-gcdc294d

Also:

docker login
Authenticating with existing credentials...
Login Succeeded

Your repository should be auto-created when you push to Docker Hub, though you can pre-create it if you want.

Then:

make release-changelog > CHANGELOG.md.next
vi CHANGELOG.md.next
mv CHANGELOG.md.next CHANGELOG.md
git commit -a -m "Fake 0.0.34 changelog"

Updated on Mar. 13:
~~git tag -a -m "fake 0.0.34 release" 0.0.34~~

make release-tag TAG=0.0.34

Finally:

$ make release-check
===============================
Releasing version: 0.0.34
===============================

$ make release-prep
===============================
Releasing version: 0.0.34
===============================
if [[ "" = "" ]]; then python -m venv  release-env; else virtualenv -p $(which python)  release-env; fi
release-env/bin/pip install -r REQUIREMENTS-CI.txt
Collecting flake8 (from -r REQUIREMENTS-CI.txt (line 1))
...
  Running setup.py install for starfish ... done
Successfully installed PyWavelets-1.0.2 Send2Trash-1.5.0 click-7.0 cloudpickle-0.8.0 dask-1.1.3 diskcache-3.1.1 ipykernel-5.1.0 ipywidgets-7.4.2 jup
6302
yter-client-5.2.4 mpmath-1.1.0 networkx-2.2 notebook-5.7.5 pillow-5.4.1 prometheus-client-0.6.0 pyyaml-3.13 pyzmq-18.0.1 regional-1.1.2 scikit-image-0.14.2 scikit-learn-0.20.3 semantic-version-2.6.0 showit-1.1.4 slicedimage-1.0.4 starfish-0.0.34 sympy-1.3 terminado-0.8.1 toolz-0.9.0 tornado-6.0.1 trackpy-0.4.1 validators-0.12.4 widgetsnbextension-3.4.2 xarray-0.11.3
You are using pip version 10.0.1, however version 19.0.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

Skipping make verify since it runs slow which is, well ..., slow.

$ make DOCKER_IMAGE=REPLACE_ME/starfish docker release-docker
docker build -t REPLACE_ME/starfish .
Sending build context to Docker daemon  586.2MB
...
Successfully built fd929ecb5290
Successfully tagged REPLACE_ME/starfish:latest
docker run -ti --rm REPLACE_ME/starfish build --fov-count 1 --primary-image-dimensions '{"z": 1}' /tmp/

         _              __ _     _
        | |            / _(_)   | |
     ___| |_ __ _ _ __| |_ _ ___| |__
    / __| __/ _` | '__|  _| / __| '_  `
    \__ \ || (_| | |  | | | \__ \ | | |
    |___/\__\__,_|_|  |_| |_|___/_| |_|


===============================
Releasing version: 0.0.34
===============================
docker tag REPLACE_ME/starfish REPLACE_ME/starfish:0.0.34
docker tag REPLACE_MEx/starfish REPLACE_ME/starfish:0.0.34-1

$ make DOCKER_IMAGE=REPLACE_ME/starfish release-upload
===============================
Releasing version: 0.0.34
===============================

# Please execute the following steps
git push origin 0.0.34
docker push REPLACE_ME/starfish
docker push REPLACE_ME/starfish:0.0.34
docker push REPLACE_ME/starfish:0.0.34-1
twine upload dist/starfish-0.0.34.tar.gz

So running those commands in my case:

$ git push origin 0.0.34
...
 * [new tag]         0.0.34 -> 0.0.34

$ docker push joshmoore/starfish
The push refers to repository [docker.io/joshmoore/starfish]
9641369a265d: Layer already exists
4815c64aeae5: Layer already exists
9c7ec779c39b: Layer already exists
6d2a0d6b9f30: Layer already exists
...
e954ee94518b: Pushing [========================================>          ]  459.6MB/565.8MB
100dfc4274a8: Pushing [=======>                                           ]  275.9MB/1.898GB
...
 docker push joshmoore/starfish:0.0.34
The push refers to repository [docker.io/spacetx/starfish]
9a0ddc8e0ee6: Layer already exists
...
0.0.34: digest: sha256:817bcee2739bae80c07fe8825ff5f46b924d2d2680c6c8f4afa8742afd6231f7 size: 13128

$ docker push joshmoore/starfish:0.0.34-1
The push refers to repository [docker.io/joshmoore/starfish]
...
0f3a12fef684: Layer already exists
0.0.34-1: digest: sha256:358a30c30b2851bde4c96589826519d06d6c885ab2adc9188cb5126d704931ac size: 2839

The suffix -1 is for if the image ever needs to be rebuilt with a new base from the same tag. Below, I upload to test.pypi.org:

$ release-env/bin/twine upload --repository-url https://test.pypi.org/legacy/ dist/starfish-0.0.34.tar.gz
Enter your username: joshmoore
Enter your password:
Uploading distributions to https://test.pypi.org/legacy/
Uploading starfish-0.0.34.tar.gz
100%|███████████████████████████████████████████████████████████████████████████████████████████████| 172k/172k [00:02<00:00, 79.4kB/s]

See results under:

joshmoore · 2019-03-08T08:56:59Z

@ttung @shanaxel42 @ambrosejcarr et al: if you create https://test.pypi.org/account/register/ accounts, I'll add you to the starfish project. Note that even on test.pypi you can never upload a given version more than once (even if you delete it). For testing, you might use dev release numbers or similar form https://www.python.org/dev/peps/pep-0440/

ambrosejcarr · 2019-03-08T13:05:23Z

I'm ambrosejcarr on test pypi

joshmoore · 2019-03-12T08:32:20Z

Fixed conflicts. Ready for re-review (preferably via the testing outlined in #1059 and demonstrated in #1059)

ttung

this looks super cool. can't wait to try it out. please see the comments.

MANIFEST.in

joshmoore · 2019-03-12T20:56:31Z

can't wait to try it out.

So I'm clear: you'd prefer to try it on master?

ttung · 2019-03-12T20:58:37Z

So I'm clear: you'd prefer to try it on master?

The directions are not very clear (and I guess it can be clarified), but I assumed it had to be run from master.

joshmoore · 2019-03-12T21:10:03Z

Happy to expand the testing instructions in #1059 (comment) but I assume that you can work from this branch under the assumption that we're pushing to all the testing versions of remote resources:

pushing to your fork rather than https://github.com/spacetx/starfish
pushing to your docker hub rather than https://hub.docker.com/r/spacetx/starfish
pushing to https://test.pypi.org/project/starfish/ rather than https://pypi.org/project/starfish

Makefile

joshmoore · 2019-03-13T08:37:02Z

Added release-tag target and addressed the other items. Instructions updated (both on the PR and in Makefile). Re-test when ready.

ttung

I had to rebase and update master locally, but I think this is good to go.

ttung · 2019-03-15T19:27:23Z

Makefile

+#
+#  (0) Check out the latest version of the branch for releasing.
+#
+#  (1) `make release-changelog` to print a suggested update to


hilariously, make release-changelog > CHANGELOG.md does horrible things...

Heh, yeah. Likely the only option there would be to create a temp file and overwrite, not printing anything to standard out.

Makefile

joshmoore · 2019-03-16T00:41:16Z

Alright, merging then after a minor wording change. Looking forward to seeing this in action. 😉

joshmoore added 2 commits March 3, 2019 12:37

make release targets and pbr-based versions

884a297

Revert pbr

11cb51b

joshmoore added 6 commits March 3, 2019 14:49

versioneer-based versions

b31d374

Use a str for __is_release_tag__

bb15559

Add Makefile help

8d3d8f3

Remove coverage files on clean

ab977be

Parameterize DOCKER_IMAGE

b997617

Migrate metadata to setup.cfg

a0087c3

joshmoore force-pushed the make-release branch from 50aea10 to a0087c3 Compare March 3, 2019 14:49

joshmoore changed the title ~~Make release (WIP)~~ Make release Mar 3, 2019

joshmoore mentioned this pull request Mar 3, 2019

ease pinned requirements to facilitate installation for users #932

Closed

joshmoore requested a review from ttung March 3, 2019 15:13

ttung requested changes Mar 4, 2019

View reviewed changes

Add twine to ci requirements

cce44e9

ttung mentioned this pull request Mar 7, 2019

make release target #761

Closed

10 tasks

joshmoore added 4 commits March 7, 2019 16:26

Build release in a virtualenv

0dabb75

Add docs and a dirty check

867b463

Split release-docker

41eb63c

Fix release-upload formatting

c6e7845

joshmoore requested a review from ttung March 7, 2019 16:58

Create changelog

cdc294d

Remove quotes from classifiers

d146b3f

Merge branch 'master' into make-release

3a036aa

ambrosejcarr mentioned this pull request Mar 10, 2019

Release schedule napari/napari#106

Closed

Merge branch 'master' into make-release

8b1d3c7

ttung approved these changes Mar 12, 2019

View reviewed changes

MANIFEST.in Outdated Show resolved Hide resolved

Makefile Show resolved Hide resolved

Add newline to MANIFEST.in

71bc91d

ttung reviewed Mar 12, 2019

View reviewed changes

Makefile Outdated Show resolved Hide resolved

ttung reviewed Mar 12, 2019

View reviewed changes

Makefile Outdated Show resolved Hide resolved

ttung reviewed Mar 12, 2019

View reviewed changes

Makefile Outdated Show resolved Hide resolved

ttung reviewed Mar 13, 2019

View reviewed changes

Makefile Show resolved Hide resolved

joshmoore added 4 commits March 13, 2019 08:23

make docs & helper targets

1e142a0

make release-tag and fix changelog

4b7d9b8

Re-order release sections

39e9d38

Add missing EDITOR and use '&&'

7a92261

Add public methods to help

28263fe

ttung reviewed Mar 15, 2019

View reviewed changes

Clarify release-check wording

9e7e3a7

joshmoore merged commit dc138c8 into spacetx:master Mar 16, 2019

joshmoore deleted the make-release branch March 16, 2019 00:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make release #1059

Make release #1059

Make release #1059

Make release #1059

Conversation

This PR does not try to:

Other minor points worth noting:

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment