Click here to launch an
interactive online version of this notebook:
DRILSDOWN Case Study
Notebook your IDV visualization journey
and connect its data sources to Python tools
See the 0:59 YouTube video intro that uses this notebook.¶
Documentation is here, from this github code repo.¶
Sections¶
Install & launch ipython_IDV
Capture images & movies (and optionally, replication bundles)
Python interaction with your IDV session
Publish to a RAMADDA repository
# Install the DRILSDOWN package, and launch
Already a Jupyter user?¶
Install the DRILSDOWN package. The cell below installs the DRILSDOWN package and its dependencies in your existing environment, using this tactic.
[1]:
import sys
!{sys.executable} -m pip install drilsdown --upgrade ### install or upgrade pip package
#NO CONDA YET... !conda install --yes --prefix {sys.prefix} drilsdown ### for conda package
Requirement already up-to-date: drilsdown in /anaconda/envs/drilsdown/lib/python3.6/site-packages (2.4.8)
Requirement not upgraded as not directly required: jython-kernel in /anaconda/envs/drilsdown/lib/python3.6/site-packages (from drilsdown) (2.0)
Requirement not upgraded as not directly required: ipython-IDV>=2.4.8' in /anaconda/envs/drilsdown/lib/python3.6/site-packages (from drilsdown) (2.4.8)
Requirement not upgraded as not directly required: cartopy in /anaconda/envs/drilsdown/lib/python3.6/site-packages (from drilsdown) (0.16.0)
Requirement not upgraded as not directly required: ipykernel in /anaconda/envs/drilsdown/lib/python3.6/site-packages (from drilsdown) (4.8.2)
Requirement not upgraded as not directly required: geoviews in /anaconda/envs/drilsdown/lib/python3.6/site-packages (from drilsdown) (1.5.2)
Requirement not upgraded as not directly required: ipywidgets>=7.1.0rc in /anaconda/envs/drilsdown/lib/python3.6/site-packages (from drilsdown) (7.2.1)
Requirement not upgraded as not directly required: future in /anaconda/envs/drilsdown/lib/python3.6/site-packages (from drilsdown) (0.16.0)
Requirement not upgraded as not directly required: xarray in /anaconda/envs/drilsdown/lib/python3.6/site-packages (from drilsdown) (0.10.8)
Requirement not upgraded as not directly required: ramadda-publish in /anaconda/envs/drilsdown/lib/python3.6/site-packages (from drilsdown) (1.3)
Requirement not upgraded as not directly required: jupyter-client in /anaconda/envs/drilsdown/lib/python3.6/site-packages (from drilsdown) (5.2.3)
Requirement not upgraded as not directly required: holoviews in /anaconda/envs/drilsdown/lib/python3.6/site-packages (from drilsdown) (1.10.7)
Requirement not upgraded as not directly required: MetPy in /anaconda/envs/drilsdown/lib/python3.6/site-packages (from drilsdown) (0.8.0)
Requirement not upgraded as not directly required: idv-teleport in /anaconda/envs/drilsdown/lib/python3.6/site-packages (from drilsdown) (1.6)
Requirement not upgraded as not directly required: six in /anaconda/envs/drilsdown/lib/python3.6/site-packages (from drilsdown) (1.11.0)
Requirement not upgraded as not directly required: pyviz in /anaconda/envs/drilsdown/lib/python3.6/site-packages (from drilsdown) (0.9.9)
Requirement not upgraded as not directly required: IPython>=3.0 in /anaconda/envs/drilsdown/lib/python3.6/site-packages (from jython-kernel->drilsdown) (6.4.0)
Requirement not upgraded as not directly required: pyshp>=1.1.4 in /anaconda/envs/drilsdown/lib/python3.6/site-packages (from cartopy->drilsdown) (1.2.12)
Requirement not upgraded as not directly required: shapely>=1.5.6 in /anaconda/envs/drilsdown/lib/python3.6/site-packages (from cartopy->drilsdown) (1.6.4.post1)
Requirement not upgraded as not directly required: numpy>=1.6 in /anaconda/envs/drilsdown/lib/python3.6/site-packages (from cartopy->drilsdown) (1.14.5)
Requirement not upgraded as not directly required: setuptools>=0.7.2 in /anaconda/envs/drilsdown/lib/python3.6/site-packages (from cartopy->drilsdown) (39.2.0)
Requirement not upgraded as not directly required: traitlets>=4.1.0 in /anaconda/envs/drilsdown/lib/python3.6/site-packages (from ipykernel->drilsdown) (4.3.2)
Requirement not upgraded as not directly required: tornado>=4.0 in /anaconda/envs/drilsdown/lib/python3.6/site-packages (from ipykernel->drilsdown) (4.5.3)
Requirement not upgraded as not directly required: bokeh>=0.12.15 in /anaconda/envs/drilsdown/lib/python3.6/site-packages (from geoviews->drilsdown) (0.13.0)
Requirement not upgraded as not directly required: param>=1.6.1 in /anaconda/envs/drilsdown/lib/python3.6/site-packages (from geoviews->drilsdown) (1.7.0)
Requirement not upgraded as not directly required: widgetsnbextension~=3.2.0 in /anaconda/envs/drilsdown/lib/python3.6/site-packages (from ipywidgets>=7.1.0rc->drilsdown) (3.2.1)
Requirement not upgraded as not directly required: nbformat>=4.2.0 in /anaconda/envs/drilsdown/lib/python3.6/site-packages (from ipywidgets>=7.1.0rc->drilsdown) (4.4.0)
Requirement not upgraded as not directly required: pandas>=0.19.2 in /anaconda/envs/drilsdown/lib/python3.6/site-packages (from xarray->drilsdown) (0.23.1)
Requirement not upgraded as not directly required: jupyter_core in /anaconda/envs/drilsdown/lib/python3.6/site-packages (from jupyter-client->drilsdown) (4.4.0)
Requirement not upgraded as not directly required: pyzmq>=13 in /anaconda/envs/drilsdown/lib/python3.6/site-packages (from jupyter-client->drilsdown) (17.0.0)
Requirement not upgraded as not directly required: python-dateutil>=2.1 in /anaconda/envs/drilsdown/lib/python3.6/site-packages (from jupyter-client->drilsdown) (2.7.3)
Requirement not upgraded as not directly required: pyviz_comms in /anaconda/envs/drilsdown/lib/python3.6/site-packages (from holoviews->drilsdown) (0.5.0)
Requirement not upgraded as not directly required: scipy>=0.14 in /anaconda/envs/drilsdown/lib/python3.6/site-packages (from MetPy->drilsdown) (1.1.0)
Requirement not upgraded as not directly required: pint>=0.8 in /anaconda/envs/drilsdown/lib/python3.6/site-packages (from MetPy->drilsdown) (0.8.1)
Requirement not upgraded as not directly required: matplotlib>=1.4 in /anaconda/envs/drilsdown/lib/python3.6/site-packages (from MetPy->drilsdown) (2.2.2)
Requirement not upgraded as not directly required: simplegeneric>0.8 in /anaconda/envs/drilsdown/lib/python3.6/site-packages (from IPython>=3.0->jython-kernel->drilsdown) (0.8.1)
Requirement not upgraded as not directly required: pygments in /anaconda/envs/drilsdown/lib/python3.6/site-packages (from IPython>=3.0->jython-kernel->drilsdown) (2.2.0)
Requirement not upgraded as not directly required: prompt-toolkit<2.0.0,>=1.0.15 in /anaconda/envs/drilsdown/lib/python3.6/site-packages (from IPython>=3.0->jython-kernel->drilsdown) (1.0.15)
Requirement not upgraded as not directly required: pexpect; sys_platform != "win32" in /anaconda/envs/drilsdown/lib/python3.6/site-packages (from IPython>=3.0->jython-kernel->drilsdown) (4.6.0)
Requirement not upgraded as not directly required: appnope; sys_platform == "darwin" in /anaconda/envs/drilsdown/lib/python3.6/site-packages (from IPython>=3.0->jython-kernel->drilsdown) (0.1.0)
Requirement not upgraded as not directly required: decorator in /anaconda/envs/drilsdown/lib/python3.6/site-packages (from IPython>=3.0->jython-kernel->drilsdown) (4.3.0)
Requirement not upgraded as not directly required: pickleshare in /anaconda/envs/drilsdown/lib/python3.6/site-packages (from IPython>=3.0->jython-kernel->drilsdown) (0.7.4)
Requirement not upgraded as not directly required: jedi>=0.10 in /anaconda/envs/drilsdown/lib/python3.6/site-packages (from IPython>=3.0->jython-kernel->drilsdown) (0.12.0)
Requirement not upgraded as not directly required: backcall in /anaconda/envs/drilsdown/lib/python3.6/site-packages (from IPython>=3.0->jython-kernel->drilsdown) (0.1.0)
Requirement not upgraded as not directly required: ipython-genutils in /anaconda/envs/drilsdown/lib/python3.6/site-packages (from traitlets>=4.1.0->ipykernel->drilsdown) (0.2.0)
Requirement not upgraded as not directly required: PyYAML>=3.10 in /anaconda/envs/drilsdown/lib/python3.6/site-packages (from bokeh>=0.12.15->geoviews->drilsdown) (3.12)
Requirement not upgraded as not directly required: Jinja2>=2.7 in /anaconda/envs/drilsdown/lib/python3.6/site-packages (from bokeh>=0.12.15->geoviews->drilsdown) (2.10)
Requirement not upgraded as not directly required: packaging>=16.8 in /anaconda/envs/drilsdown/lib/python3.6/site-packages (from bokeh>=0.12.15->geoviews->drilsdown) (17.1)
Requirement not upgraded as not directly required: notebook>=4.4.1 in /anaconda/envs/drilsdown/lib/python3.6/site-packages (from widgetsnbextension~=3.2.0->ipywidgets>=7.1.0rc->drilsdown) (5.5.0)
Requirement not upgraded as not directly required: jsonschema!=2.5.0,>=2.4 in /anaconda/envs/drilsdown/lib/python3.6/site-packages (from nbformat>=4.2.0->ipywidgets>=7.1.0rc->drilsdown) (2.6.0)
Requirement not upgraded as not directly required: pytz>=2011k in /anaconda/envs/drilsdown/lib/python3.6/site-packages (from pandas>=0.19.2->xarray->drilsdown) (2018.4)
Requirement not upgraded as not directly required: cycler>=0.10 in /anaconda/envs/drilsdown/lib/python3.6/site-packages/cycler-0.10.0-py3.6.egg (from matplotlib>=1.4->MetPy->drilsdown) (0.10.0)
Requirement not upgraded as not directly required: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /anaconda/envs/drilsdown/lib/python3.6/site-packages (from matplotlib>=1.4->MetPy->drilsdown) (2.2.0)
Requirement not upgraded as not directly required: kiwisolver>=1.0.1 in /anaconda/envs/drilsdown/lib/python3.6/site-packages (from matplotlib>=1.4->MetPy->drilsdown) (1.0.1)
Requirement not upgraded as not directly required: wcwidth in /anaconda/envs/drilsdown/lib/python3.6/site-packages (from prompt-toolkit<2.0.0,>=1.0.15->IPython>=3.0->jython-kernel->drilsdown) (0.1.7)
Requirement not upgraded as not directly required: ptyprocess>=0.5 in /anaconda/envs/drilsdown/lib/python3.6/site-packages (from pexpect; sys_platform != "win32"->IPython>=3.0->jython-kernel->drilsdown) (0.6.0)
Requirement not upgraded as not directly required: parso>=0.2.0 in /anaconda/envs/drilsdown/lib/python3.6/site-packages (from jedi>=0.10->IPython>=3.0->jython-kernel->drilsdown) (0.2.1)
Requirement not upgraded as not directly required: MarkupSafe>=0.23 in /anaconda/envs/drilsdown/lib/python3.6/site-packages (from Jinja2>=2.7->bokeh>=0.12.15->geoviews->drilsdown) (1.0)
Requirement not upgraded as not directly required: nbconvert in /anaconda/envs/drilsdown/lib/python3.6/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.2.0->ipywidgets>=7.1.0rc->drilsdown) (5.3.1)
Requirement not upgraded as not directly required: Send2Trash in /anaconda/envs/drilsdown/lib/python3.6/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.2.0->ipywidgets>=7.1.0rc->drilsdown) (1.5.0)
Requirement not upgraded as not directly required: terminado>=0.8.1 in /anaconda/envs/drilsdown/lib/python3.6/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.2.0->ipywidgets>=7.1.0rc->drilsdown) (0.8.1)
Requirement not upgraded as not directly required: entrypoints>=0.2.2 in /anaconda/envs/drilsdown/lib/python3.6/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.2.0->ipywidgets>=7.1.0rc->drilsdown) (0.2.3)
Requirement not upgraded as not directly required: pandocfilters>=1.4.1 in /anaconda/envs/drilsdown/lib/python3.6/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.2.0->ipywidgets>=7.1.0rc->drilsdown) (1.4.2)
Requirement not upgraded as not directly required: mistune>=0.7.4 in /anaconda/envs/drilsdown/lib/python3.6/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.2.0->ipywidgets>=7.1.0rc->drilsdown) (0.8.3)
Requirement not upgraded as not directly required: testpath in /anaconda/envs/drilsdown/lib/python3.6/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.2.0->ipywidgets>=7.1.0rc->drilsdown) (0.3.1)
Requirement not upgraded as not directly required: bleach in /anaconda/envs/drilsdown/lib/python3.6/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.2.0->ipywidgets>=7.1.0rc->drilsdown) (2.1.3)
Requirement not upgraded as not directly required: html5lib!=1.0b1,!=1.0b2,!=1.0b3,!=1.0b4,!=1.0b5,!=1.0b6,!=1.0b7,!=1.0b8,>=0.99999999pre in /anaconda/envs/drilsdown/lib/python3.6/site-packages (from bleach->nbconvert->notebook>=4.4.1->widgetsnbextension~=3.2.0->ipywidgets>=7.1.0rc->drilsdown) (1.0.1)
Requirement not upgraded as not directly required: webencodings in /anaconda/envs/drilsdown/lib/python3.6/site-packages (from html5lib!=1.0b1,!=1.0b2,!=1.0b3,!=1.0b4,!=1.0b5,!=1.0b6,!=1.0b7,!=1.0b8,>=0.99999999pre->bleach->nbconvert->notebook>=4.4.1->widgetsnbextension~=3.2.0->ipywidgets>=7.1.0rc->drilsdown) (0.5.1)
You are using pip version 10.0.1, however version 18.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
New to Jupyter?¶
Create and use our DRILSDOWN environment in Anaconda.
Uncomment and use the cell below to download our environment.yml file. Then, import the environment (it takes a few minutes as packages are installed). Finally, launch jupyter notebook after selecting the DRILSDOWN environment (channel). More in this science-oriented guide to Jupyter notebooks
[ ]:
# The 3 lines here will download DRILSDOWN's environment.yml file
# Then use Navigator to create environment from it as explained above
#import requests
#r = requests.get('https://raw.githubusercontent.com/Unidata/drilsdown/master/environment.yml', allow_redirects=True)
#open('DRILSDOWN_environment.yml', 'wb').write(r.content)
Launch!¶
After executing this cell, you should see this Control Panel.
Buttons append their outputs to the Control Panel’s results area. Clear Outputs will clear the whole cell.
[2]:
# Load ipython_IDV extension. If successful, It will make a user interface (Control Panel) widget.
%reload_ext ipython_IDV
Code route: You can also get a GUI display of a set of IDV bundles using this example¶
Top
Operate the IDV and capture images, movies, & bundles (state files)¶
1. Installand learn to operate the IDV: Consider my IDV intro .ppt, or Unidata’s documentation and tutorial.¶
2. Launch the IDV on this machine (from your Control Panel, or manually).¶
Linux users may need to set a path for the Launch IDV button to work
3. Display something meaningful in the IDV.¶
Perhaps by loading a bundle from our MapesIDVcollection or some other Catalog, or uncomment the following cell for a standard example.
[2]:
# If you want to load a standard illustration bundle, uncomment this command
#%load_bundle https://geodesystems.com/repository/entry/get/example_grid.xidv?entryid=4728e5eb-fa89-4af8-8162-e2f66a9cd0c9
The cells below capture your IDV session’s images and movies¶
[2]:
%reload_ext ipython_IDV
[4]:
%make_movie -capture legend -caption "This shows that jet streams flow above temperature gradients"
[ ]:
%make_movie -capture legend -caption "My great movie"
For exact replicability, capture a <1 MB sized IDV state bundle and embed it in this notebook¶
[5]:
%save_bundle -embed
Top
## The path to Python
You can also pass data back and forth with the IDV. Currently, this involves reading and writing local netCDF files on disk.
Reading from IDV: example is here with key lines below, commented out¶
[6]:
from ipython_IDV import Idv
data = Idv.export_data(display_id='windspeed')
windspeed=data.variables['WindSpeed']
print(windspeed)
<class 'netCDF4._netCDF4.Variable'>
float32 WindSpeed(time, level, y, x)
units: meters/second
grid_mapping: lambert_conformal_conic
unlimited dimensions:
current shape = (2, 19, 65, 93)
filling off
using our new .to_IDV() method in xarray¶
[ ]:
Top
# Publishing to a RAMADDA repository maybe `ours
Everything in RAMADDA is an entry, with an entry type.¶
1. To publish an IDV bundle entry type, with a quicklook image:¶
In your IDV, install the RAMADDA publisher plugin.
Publisher–>Configure like this
Select the “publish” checkbox in the ipython_IDV Control Panel, and click Make Image button. Then follow the dialog in the IDV session, finding a folder that you have permissions in (like the Contributed folder). An actual example created this way is here.
[ ]:
%reload_ext ipython_IDV
2. To publish an DRILSDOWN Case Study entry type:¶
A DRILSDOWN Case Study folder contains a .ipynb notebook, a .zidv or .xidv IDV bundle, and any other files.¶
In the future, other services could be added to this specific entry type (like composite averages of a set of identically-shaped .zidv files).
Let’s publish this notebook using Python code calls:¶
[ ]:
%publish_notebook "http://weather.rsmas.miami.edu/repository/entry/show?entryid=e322e383-f409-41ee-b50f-e6d3be3625c1"
[ ]:
# Publish in the Contributed folder,
# http://weather.rsmas.miami.edu/repository/entry/show?entryid=e322e383-f409-41ee-b50f-e6d3be3625c1
from ipython_IDV import Ramadda
RAM = Ramadda("http://weather.rsmas.miami.edu/repository/entry/show?entryid=e322e383-f409-41ee-b50f-e6d3be3625c1")
[ ]:
# USER and PASSWORD can be blank for that Contributed folder
import os
os.environ['RAMADDA_USER'] = ""
os.environ['RAMADDA_PASSWORD'] = ""
RAM.publish('Test_Publish', file="./Install_and_demo.ipynb")
Top
## Coding the production of Case Studies
idv_teleport¶
idv_teleport is DRILSDOWN’s Python and command-line scripting tool to batch-generate one, or a set (folder) of data-zipped (.zidv) bundles with user-defined time range and bounding box, like this set.
The -h or -help argument explains its use, or see the repo.
ramadda_publish¶
ramadda_publish publishes files or folders to a RAMADDA. If your folder contains .ipynb notebooks, and is tagged as a DRILSDOWN Case Study, it will be displayed like this library of Case Studies.
Again the -h argument or the repo explains its use.
[ ]:
Top
[1]:
# Time stamp this notebook
!date
Wed Jun 20 15:20:13 MDT 2018
[ ]: