edit

Purging and Retention

Purge Types

There are two main forms of data puring in AWIPS. The most often thought of is the purging for processed data. This has to do with how long data is stored for after it has been decoded and processed.

The second type of purging has to do with raw data. This has to do with how long data is stored for before it has been decoded.

Processed Data Purging

AWIPS uses a plugin-based purge strategy for processed HDF5 data. This allows the user to change the purge frequency for each plugin individually, and even set purge rules for specific products for a particular plugin. There is also a default purge rules file for those products which do not have specific rules written.

Note: Purging is triggered by a quartz timer event that fires at 30 minutes after each hour.

Purging rules are defined in XML files in the Localization Store. On EDEX, most are located in /awips2/edex/data/utility/common_static/base/purge, and follow the base/site localization pattern (e.g. site purge files are in site/XXX/purge rather than base/purge, where XXX is the site identifier).

Each data set can have a purge rule defined, and the xml file is named after the data set:

ls /awips2/edex/data/utility/common_static/base/purge/

acarsPurgeRules.xml          bufruaPurgeRules.xml         pirepPurgeRules.xml
acarssoundingPurgeRules.xml  ccfpPurgeRules.xml           poessoundingPurgeRules.xml
aggregatePurgeRules.xml      convsigmetPurgeRules.xml     pointsetPurgeRules.xml
airepPurgeRules.xml          cwaPurgeRules.xml            profilerPurgeRules.xml
...

Time-based purge

If a plugin has no XML file, the default rule of 1 day (24 hours) is used, from /awips2/edex/data/utility/common_static/base/purge/defaultPurgeRules.xml:

<purgeRuleSet>
    <defaultRule>
        <period>01-00:00:00</period>
    </defaultRule>
</purgeRuleSet>

Time-based purging is set with the period tag and uses the reference time of the data. The reference time of the data is determined by the decoder.


30-day NEXRAD3 Example

Modify /awips2/edex/data/utility/common_static/base/purge/radarPurgeRules.xml to increase the data retention period from 1 to 31 days:

<purgeRuleSet>
        <defaultRule>
                <period>31-00:00:00</period>
        </defaultRule>
</purgeRuleSet>

Note: you do NOT have to restart EDEX when you change a purge rule!


Frame-Based Purge

Some plugins use frame-base purging, retaining and certain number of product "versions".

/awips2/edex/data/utility/common_static/base/purge/gridPurgeRules.xml

    
<defaultRule>
    <versionsToKeep>2</versionsToKeep>
    <period>07-00:00:00</period>
  </defaultRule>
  <rule>
    <keyValue>LAPS</keyValue>
    <versionsToKeep>30</versionsToKeep>
  </rule>
  <rule regex="true">
    <keyValue>NAM(?:12|20|40)</keyValue>
    <versionsToKeep>2</versionsToKeep>
    <modTimeToWait>00-00:15:00</modTimeToWait>
  </rule>
  ...

In the above example, notice a default rule (2) is specified, as well as specific models with their own rules.
The tag modTimeToWait can be used in conjunction with versionsToKeep and will increase the versionsToKeep by 1 if data matching this rule has been stored within modTimeToWait.


Purge Logs

Data purge events are logged to the file edex-ingest-purge-[yyyymmdd].log, where [yyyymmdd] is the date stamp.

tail -f edex-ingest-purge-20120327.log

--------START LOG PURGE---------
INFO  2012-03-27 00:30:00,027 [DefaultQuartzScheduler_Worker-3] PurgeLogger: EDEX - PURGE LOGS::Skipped file with invalid fileName: afos-trigger.log
INFO  2012-03-27 00:30:00,193 [DefaultQuartzScheduler_Worker-3] PurgeLogger: EDEX - PURGE LOGS::Removed 1 old files
INFO  2012-03-27 00:31:23,155 [DefaultQuartzScheduler_Worker-3] PurgeLogger: EDEX - PURGE LOGS::Archived 14 files
INFO  2012-03-27 00:31:23,155 [DefaultQuartzScheduler_Worker-3] PurgeLogger: EDEX - PURGE LOGS::Skipped processing 1 files
INFO  2012-03-27 00:31:23,155 [DefaultQuartzScheduler_Worker-3] PurgeLogger: EDEX - PURGE LOGS::---------END LOG PURGE-----------

All Purge Rules

To see all purge rule directories (base, site, configured):

find /awips2/edex/data/utility -name purge

/awips2/edex/data/utility/common_static/base/purge

If any overrides have been made, then it's possible that site directories may show up as results from the find command as well.


Raw Data Purging

Raw data are files that have been brought in by the LDM and recognized by an action in the pqact.conf file. These files are written to subdirectories of /awips2/data_store/. This data will wait here until it is purged, from the purging rules defined in /awips2/edex/data/utility/common_static/base/archiver/purger/RAW_DATA.xml.

If the purge time is too short, and the processing latencies on EDEX are too long, it is possible that EDEX will miss some of this data, and the purge times will need to be adjusted by changing the <defaultRetentionHours> or <selectedRetentionHours> tag on the relevent data sets.


Default Retention

The defaultRetentionHours tag is defined at the beginning of the RAW_DATA.xml file. It is the duration that will apply to any piece of data that does not fall under an explicitly defined category.

The default value for our EDEX is 1 hour:

<archive>
  <name>Raw</name>
  <rootDir>/awips2/data_store/</rootDir>
  <defaultRetentionHours>1</defaultRetentionHours>
  <category>
  ...

Selected Retention

Data sets are broken up into categories in the RAW_DATA.xml file. These categories are groupings of similar data. Each category has a selectedRetentionHours tag which specifies how long the matching data will be kept for.

For example, there is a Model category which sets the purge time to 3 hours for all grib, bufrmos, and modelsounding data:

...
<category>
    <name>Model</name>
    <selectedRetentionHours>3</selectedRetentionHours>
    <dataSet>
      <dirPattern>(grib|grib2)/(\d{4})(\d{2})(\d{2})/(\d{2})/(.*)</dirPattern>
      <displayLabel>{1} - {6}</displayLabel>
      <dateGroupIndices>2,3,4,5</dateGroupIndices>
    </dataSet>
    <dataSet>
      <dirPattern>(bufrmos|modelsounding)/(\d{4})(\d{2})(\d{2})/(\d{2})</dirPattern>
      <displayLabel>{1}</displayLabel>
      <dateGroupIndices>2,3,4,5</dateGroupIndices>
    </dataSet>
</category>
...

Logging

Raw data purging can be seen in the purge logs as well (/awips2/edex/logs/edex-ingest-purge-[yyyymmdd].log where [yyyymmdd] is the date stamp).

[centos@tg-atm160027-edex-dev purge]$ grep -i 'archive' /awips2/edex/logs/edex-ingest-purge-20200728.log
INFO  2020-07-28 20:05:23,959 2329 [Purge-Archive] ArchivePurgeManager: EDEX - Start purge of category Raw - Observation, directory "/awips2/data_store/bufrhdw".
INFO  2020-07-28 20:05:23,960 2330 [Purge-Archive] ArchivePurgeManager: EDEX - End purge of category Raw - Observation, directory "/awips2/data_store/bufrhdw", deleted 0 files and directories.
INFO  2020-07-28 20:05:23,961 2331 [Purge-Archive] ArchivePurgeManager: EDEX - Unlocked: "/awips2/data_store/bufrhdw"
INFO  2020-07-28 20:05:23,963 2332 [Purge-Archive] ArchivePurgeManager: EDEX - Locked: "/awips2/data_store/xml"
INFO  2020-07-28 20:05:23,963 2333 [Purge-Archive] ArchivePurgeManager: EDEX - Start purge of category Raw - Products, directory "/awips2/data_store/xml".
INFO  2020-07-28 20:05:23,964 2334 [Purge-Archive] ArchivePurgeManager: EDEX - End purge of category Raw - Products, directory "/awips2/data_store/xml", deleted 5 files and directories.
INFO  2020-07-28 20:05:23,967 2335 [Purge-Archive] ArchivePurgeManager: EDEX - Unlocked: "/awips2/data_store/xml"
INFO  2020-07-28 20:05:23,967 2336 [Purge-Archive] ArchivePurger: EDEX - Raw::Archive Purged 28387 files in 23.8s.
INFO  2020-07-28 20:05:23,979 2337 [Purge-Archive] ArchivePurgeManager: EDEX - Purging directory: "/awips2/edex/data/archive".
INFO  2020-07-28 20:05:23,992 2338 [Purge-Archive] ArchivePurger: EDEX - Processed::Archive Purged 0 files in 25ms.
INFO  2020-07-28 20:05:23,992 2339 [Purge-Archive] ArchivePurger: EDEX - Archive Purge finished.  Time to run: 23.9s
...