

Install EDEX - BETA Version

EDEX is the Environmental Data Exchange system that represents the backend server for AWIPS. EDEX is only supported for Linux systems: CentOS and RHEL, and ideally, it should be on its own dedicated machine. It requires administrator priviledges to make root-level changes. EDEX can run on a single machine or be spread across multiple machines. To learn more about that please look at Distributed EDEX, Installing Across Multiple Machines

Latest Version

Note: This release is still in beta since the National Weather Service (NWS) is still working on their 23.4.1 release to operations. In the interim, we are waiting on final clearance from the NWS before releasing a production (non-beta) version and source code. - 23.4.1-0.4 - 20.3.2 (Compatible with EOL CentOS/Redhat 7)

View release notes

Version 23.* of CAVE is not compatible with Version 20.* EDEX and vice versa, Version 20.* of CAVE is not compatible with Version 23.* EDEX.

Functionality/Reporting

If you come across issues/bugs/missing functionality, we encourage you to report it using this short form or emailing at support-awips@unidata.ucar.edu.

System requirements

64-bit Rocky/RHEL 8
Bash shell environment
16+ CPU cores (each CPU core can run a decoder in parallel)
24GB RAM
700GB+ Disk Space
A Solid State Drive (SSD) is recommended
- A SSD should be mounted either to /awips2 (to contain the entire EDEX system) or to /awips2/edex/data/hdf5 (to contain the large files in the decoded data store). EDEX can scale to any system by adjusting the incoming LDM data feeds or adjusting the resources (CPU threads) allocated to each data type.

EDEX is only supported for 64-bit Rocky and RHEL 8 Operating Systems.

EDEX is not supported in Debian, Ubuntu, SUSE, Solaris, macOS, or Windows. You may have luck with Fedora Core 12 to 14 and Scientific Linux, but we will not provide support.

Download and Installation Instructions

The first 3 steps should all be run as root

1. Install EDEX

Download and run the installer: awips_install-v23.sh

wget https://downloads.unidata.ucar.edu/awips2/23.4.1/linux/awips_install-v23.sh
chmod 755 awips_install.sh
sudo ./awips_install.sh --edex

awips_install.sh --edex will perform the following steps (it's always a good idea to review downloaded shell scripts):

Checks to see if EDEX is currently running, if so stops the processes with the edex stop command
If EDEX is installed, asks the user if it can be removed and where to backup the data to and does a yum groupremove awips2-server
If the user/group awips:fxalpha does not exist, it gets created
Saves the appropriate yum repo file to /etc/yum.repos.d/awips2.repo
Increases process and file limits for the the awips account in /etc/security/limits.conf
Creates /awips2/data_store if it does not exist already
Runs yum groupinstall awips2-server

If you receive an error relating to yum, then please run

sudo su - -c "[PATH_TO_INSTALL_FILE]/awips_install.sh --edex"

2. EDEX Setup

The external and localhost addresses need to be specified in /etc/hosts

127.0.0.1         localhost    localhost.localdomain
XXX.XXX.XXX.XXX   edex-cloud   edex-cloud.unidata.ucar.edu

3. Configure iptables

This should be a one time configuration change. Configure iptables to allow TCP connections on ports 9581 and 9582 if you want to serve data publicly to CAVE clients and the Python API.

Open Port 9588

If you are running a Registry (Data Delivery) server, you will also want to open port 9588.

To open ports to all connections

vi /etc/sysconfig/iptables

*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 9581 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 9582 -j ACCEPT
#-A INPUT -m state --state NEW -m tcp -p tcp --dport 9588 -j ACCEPT # for registry/dd
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT

To open ports to specific IP addresses

In this example, the IP range 128.117.140.0/24 will match all 128.117.140.* addresses, while 128.117.156.0/24 will match 128.117.156.*.

vi /etc/sysconfig/iptables

*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]
:EXTERNAL - [0:0]
:EDEX - [0:0]
-A INPUT -i lo -j ACCEPT
-A INPUT -p icmp --icmp-type any -j ACCEPT
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A INPUT -s 128.117.140.0/24 -j EDEX
-A INPUT -s 128.117.156.0/24 -j EDEX
-A INPUT -j EXTERNAL
-A EXTERNAL -j REJECT
-A EDEX -m state --state NEW -p tcp --dport 22 -j ACCEPT
-A EDEX -m state --state NEW -p tcp --dport 9581 -j ACCEPT
-A EDEX -m state --state NEW -p tcp --dport 9582 -j ACCEPT
#-A EDEX -m state --state NEW -p tcp --dport 9588 -j ACCEPT # for registry/dd
-A EDEX -j REJECT
COMMIT

Restart iptables

service iptables restart

Troubleshooting

For CentOS 7 error:

Redirecting to /bin/systemctl restart  iptables.service 
Failed to restart iptables.service: Unit iptables.service failed to load: No such file or directory.

The solution is:

yum install iptables-services
systemctl enable iptables
service iptables restart

4. Start EDEX

These steps should be run as user awips with sudo. Switch to the user by running su - awips.

edex start

To manually start, stop, and restart:

systemctl start postgres@awips
systemctl start httpd-pypies
systemctl start qpidd
systemctl start edex_camel@ingest
systemctl start edex_camel@ingestGrib
systemctl start edex_camel@request

The fifth service, edex_ldm, does not run at boot to prevent filling up disk space if EDEX is not running. Start ldm manually:

service edex_ldm start

To restart EDEX

edex restart

Additional Notes

Ensure SELinux is Disabled

vi /etc/sysconfig/selinux

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of these two values:
#     targeted - Targeted processes are protected,
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted

SSD Mount

Though a Solid State Drive is not required, it is strongly encouraged in order to handle the amount of disk IO for real-time IDD feeds.

The simplest configuration would be to mount an 500GB+ SSD to /awips2 to contain both the installed software (approx. 20GB) and the real-time data (approx. 150GB per day).

The default purge rules are configured such that the processed data in /awips2 does not exceed 450GB. The raw data is located in /awips2/data_store, and is scoured every hour and should not exceed 50GB.

If you want to increase EDEX data retention you should mount a large disk to /awips2/edex/data/hdf5 since this will be where the archived processed data exists, and any case studies created.

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        30G  2.5G   26G   9% /
tmpfs            28G     0   28G   0% /dev/shm
/dev/sdc1       788G   81G  667G  11% /awips2
/dev/sdb1       788G   41G  708G  10% /awips2/edex/data/hdf5

Configure LDM Feeds

EDEX installs its own version of the LDM to the directory /awips2/ldm. As with a the default LDM configuration, two files are used to control what IDD feeds are ingested:

Configuration file: /awips2/ldm/etc/ldmd.conf

This file specifies an upstream LDM server to request data from, and what feeds to request:

REQUEST NEXRAD3 "./p(DHR|DPR|DSP|DTA|DAA|DVL|EET|HHC|N0Q|N0S|N0U|OHA|NVW|NTV|NST)." idd.unidata.ucar.edu
REQUEST FNEXRAD|IDS|DDPLUS|UNIWISC ".*" idd.unidata.ucar.edu
REQUEST NGRID ".*" idd.unidata.ucar.edu
REQUEST NOTHER "^TIP... KNES.*" idd.unidata.ucar.edu

Configuration File: /awips2/ldm/etc/pqact.conf

This file specifies the WMO headers and file pattern actions to request:

# Redbook graphics
ANY     ^([PQ][A-Z0-9]{3,5}) (....) (..)(..)(..) !redbook [^/]*/([^/]*)/([^/]*)/([^/]*)/([0-9]{8})
        FILE    -overwrite -close -edex /awips2/data_store/redbook/\8/\4\5Z_\8_\7_\6-\1_\2_(seq).rb.%Y%m%d%H
# NOAAPORT GINI images
NIMAGE  ^(sat[^/]*)/ch[0-9]/([^/]*)/([^/]*)/([^ ]*) ([^/]*)/([^/]*)/([^/]*)/ (T[^ ]*) ([^ ]*) (..)(..)(..)
        FILE    -overwrite -close -edex /awips2/data_store/sat/\(11)\(12)Z_\3_\7_\6-\8_\9_(seq).satz.%Y%m%d%H

See available AWIPS LDM feeds

Configuration File: /awips2/ldm/etc/registry.xml

This file specifies configuration and runtime parameters. If you are pulling in a lot of data, you may want to consider increasing your LDM queue size:

  <queue>
    <path>/awips2/ldm/var/queues/ldm.pq</path>
    <size>24GB</size>
    <slots>default</slots>
  </queue>

Directories to Know

/awips2 - Contains all of the installed AWIPS software.
/awips2/edex/logs - EDEX logs.
/awips2/httpd_pypies/var/log/httpd - httpd-pypies logs.
/awips2/database/data/pg_log - PostgreSQL logs.
/awips2/qpid/log - Qpid logs.
/awips2/edex/data/hdf5 - HDF5 data store.
/awips2/edex/data/utility - Localization store and configuration files.
/awips2/ldm/etc - Location of ldmd.conf and pqact.conf
/awips2/ldm/logs - LDM logs.
/awips2/data_store - Raw data store.
/awips2/data_store/ingest - Manual data ingest endpoint.

What Version is my EDEX?

rpm -qa | grep awips2-edex

Uninstalling EDEX

These are instructions to manually uninstall EDEX. However, the awips_install.sh script will do all of these steps for you if you are installing a newer version of EDEX.

1. Make sure all EDEX processes are stopped

sudo edex stop
sudo edex status

[edex status]
 postgres    :: not running
 pypies      :: not running
 qpid        :: not running
 EDEXingest  :: not running
 EDEXgrib    :: not running
 EDEXrequest :: not running
 ldmadmin    :: not running

2. Backup any important configuration files that you may want to reference

Here are some possible important directories/files to backup:

/awips2/database/data/pg_hba.conf
/awips2/edex/data/utility/*
/awips2/edex/bin/*
/awips2/ldm/*
/awips2/dev/*
/awips2/edex/conf*
/awips2/edex/etc/*
/awips2/edex/logs/*
/usr/bin/edex/*
/etc/init.d/edexServiceList

3. See what AWIPS yum groups are currently installed

In this case the AWIPS EDEX Server group is installed

sudo yum grouplist

Available Environment Groups:
   Minimal Install
   Compute Node
   Infrastructure Server
   File and Print Server
   Cinnamon Desktop
   MATE Desktop
   Basic Web Server
   Virtualization Host
   Server with GUI
   GNOME Desktop
   KDE Plasma Workspaces
   Development and Creative Workstation
Installed Groups:
   AWIPS EDEX Server
   Development Tools
Available Groups:
   AWIPS ADE SERVER
   AWIPS CAVE
   AWIPS Development
   AWIPS EDEX DAT Server
   AWIPS EDEX Database/Request Server
   AWIPS EDEX Decode/Ingest Node (No Database, PyPIES, GFE)
   Cinnamon
   Compatibility Libraries
   Console Internet Tools
   Educational Software
   Electronic Lab
   Fedora Packager
   General Purpose Desktop
   Graphical Administration Tools
   Haskell
   LXQt Desktop
   Legacy UNIX Compatibility
   MATE
   Milkymist
   Scientific Support
   Security Tools
   Smart Card Support
   System Administration Tools
   System Management
   TurboGears application framework
   Xfce

4. Remove any currently installed AWIPS yum groups

sudo yum clean all
sudo yum groupremove "AWIPS EDEX Server"

If you are having trouble removing a group, see the troubleshooting section.

5. Check to make sure all awips rpms have been removed

rpm -qa | grep awips2

If you still have rpms installed, remove them

sudo yum remove awips2-*

6. Remove everything in the /awips2 directory

rm -rf /awips2/*