Proposals to improve the WFS Temporality Extension

-- VolkerGrabsch with input from Markus Schneider, Eduard Porosnicu, Timo Thomas and Johannes Echterhoff

Proposals for the latest WFS Temporality Extension document:

• Google Docs Working Draft: https://docs.google.com/document/d/1rq-rCoInXRk8t2dV3MU7saYcxr-0onzzGFl4uqfoR1A/edit?usp=sharing

This is based on the OWS-10 engineering report, published 2014-07-16 by OGC:

• PDF: https://portal.opengeospatial.org/files/?artifact_id=58922
• DOCX: https://portal.opengeospatial.org/files/?artifact_id=58922&version=1

Applying WFS-TE to other domains

Due to time constraints in Testbed 10, the WFS-TE draft is specific for the AIXM Temporality Model and handles only use cases of the Aviation domain. However, if we want to push WFS-TE forward as a future OGC standard, it must be applicable to other domains as well.

Question: Is this a worthy goal, or a waste of time?

Other domains with timed data:

• AFX (still Aviation domain, but different temporality model)
• Sensor data (time series)
• ... others?

Comments by Timo:

• We would have to split the document into a generic and an AIXM specific part.
• Generic parts are:
  • DynamicFeatureQuery (for formal reasons)
  • DynamicFilter (without evaluateSchedules)
• Specific parts are:
  • evaluateSchedules
  • includeCorrected
  • includeCanceled
• This is an exciting but expensive task that probably needs funding.

-- JohannesEchterhoff - 26 Feb 2015: I think that the WFS-TE already is a general purpose WFS extension as it is. It's only usable with AIXM data right now, but that's due to a lack of specification regarding dynamic features in ISO standards. What's actually missing is the incorporation of the temporality model itself, as well as likely new UML2XML encoding rules, into the ISO standards - notably ISO 19109! Because it would take ages to achieve this, my suggestion is that we simply move on. If other domains would like to use the same or a similar approach to dynamic features, work on such a baseline can start. Don't get me wrong: I'd love to see this happening - it's just not the WFS-TE that needs to be changed, it's the foundation standards themselves.

Reference implementation

Should we define a reference implementation? What implementations do we have?

Testing

How should a testing environment / compliance check look like?

• Set of scripts / test programs on GitHub?
• Running service instance?

Example B.8 has wrong response

The response of example B.8 is specified as containing two TEMPDELTA timeslices.

However, the query does not contain "includeCorrected", and the second TEMPDELTA (seq: 1, cor: 1) corrects the first one (seq: 1).

So the result should be just one TEMPDELTA, the one with sequenceNumber=1 and correctionNumber=1.

Examples from Annex A.* should use existing Donlon features

Although this is not the topic of Annex A, it would be nice if those queries referred to existing Donlon features. This is easily achieved by changing some UUIDs and time stamps.

Examples from Annex B.* should provide complete responses

Most responses of Annex B.* don't contain actual feature data, but an XML comment stating in which time slices and properties should be there. This should be improved so that the complete expected responses are provided.

Examples and schemas as separate files

Examples and schema files should be provided as separate files. Copying them out of PDF is not practical.

Whitespace issues in example requests

The example requests (Annex A and B) contain whitespace issues, such as a gml:identifier followed by a newline followed by spaces:

          <fes:PropertyIsEqualTo>
            <fes:ValueReference>gml:identifier</fes:ValueReference>
            <fes:Literal>0083defb-b42e-4417-9be2-7aba2db2674d
            </fes:Literal>
          </fes:PropertyIsEqualTo>

These places should be fixed, e.g.

          <fes:PropertyIsEqualTo>
            <fes:ValueReference>gml:identifier</fes:ValueReference>
            <fes:Literal>0083defb-b42e-4417-9be2-7aba2db2674d</fes:Literal>
          </fes:PropertyIsEqualTo>

No PERMDELTAs needed for backup

Use case 8 (page 12) says that for a full history you need all BASELINEs, PERMDELTAs and TEMPDELTAs. However, you need only the BASELINEs and TEMPDELTAs.

Also, the leading comment on Annex A.8 (pages 49-50) should be corrected accordingly.

-- JohannesEchterhoff - 26 Feb 2015: that assumes that the service creates a new BASELINE for each case in which a PERMDELTA has been issued, isn't it? Does the WFS-TE or the Temporality Model require this? Section 3.2 in the Temporality Model should be updated if this was a requirement.

-- VolkerGrabsch - 26 Feb 2015: We should discuss that aspect in the other section "Clarify how to deal with PERMDELTAs". I just wanted point out that the history is already fully determined by a complete set of BASELINEs and TEMPDELTAs. Alternatively, it is also fully determined by a complete set of PERMDELTAs and TEMPDELTAs. You don't need all 3. You can always convert a history of BASELINEs to a history of PERMDELTAs and vice versa.

-- JohannesEchterhoff - 26 Feb 2015: Agreed. We should discuss how best to handle this. It looks like we want to be able to push PERMDELTAs around, even to another WFS, in case of tiny updates. If provenance is relevant, we'd have to see how that would be handled in case that the WFS created a new BASELINE upon receiving a new PERMDELTA.

-- TimoThomas - 09 Mar 2015 I agree with Johannes that it is unclear from the AIXM spec if there is an one-to-one relationship between BASELINEs and PERMDELTAs or not. If there is, BASELINEs and PERMDELTAs do not need to be stored both.

Clarify whether to store SNAPSHOTs in a WFS-TE database

Section 5.2 (page 7) states that no SNAPSHOTs may be stored in the WFS. However, the Donlon data set does contain SNAPSHOT time slices and is used as reference for annex B.

Also, there are known datasets which use SNAPSHOTs for some features (just for reference / basic data), but provide full history (BASELINE+TEMPDELTA) for other features.

There is technically no reason to forbid SNAPSHOTs, as long as we define how to deal with them. So the following possibilities exist:

• (Current status:) We could mandate that only BASELINE and TEMPDELTA are stored in the WFS, and that SNAPSHOTs are only generated by specific queries (SnapshotGeneration), but never inserted e.g. via WFS-T.
• We could also be more liberal by saying that SNAPSHOTs and PERMDELTAs may be stored in addition to BASELINE and TEMPDELTA, but that these time slices should be ignored by all temporal operators.
• Or, we could say that SNAPSHOTs may be stored, and that they may (or must?) be used as pre-calculated information by the SnapshotGeneration to speed up things. (e.g. don't go through BASELINEs and TEMPDELTAs if you can reuse a stored SNAPSHOT, or some part of it)

Either way, we should clarify this in the very beginning and then stick to it throughout the document.

-- JohannesEchterhoff - 26 Feb 2015: First of all, I don't think we should mix SNAPSHOTs with PERMDELTAs. As far as I can see BASELINEs, PERMDELTAs and TEMPDELTAs are the real components that define the state of a dynamic feature. At least that's how I read the Temporality Model. So a WFS should be able to handle them all. Section 5.2 states that "SNAPSHOT time slices cannot be stored in a WFS" and that "a WFS cannot retrieve them from a persistent storage". This does not actually say that you are not allowed to store SNAPSHOTs in your implementation. I'd suggest that we clarify this in the document as follows: a WFS-TE implementation can store SNAPSHOTs internally if it so desires. However, the state of a dynamic feature shall be defined via the other types of time slices. That also means that when inserting and updating a dynamic feature, only these time slice types shall be used (so not SNAPSHOTs). Again, the service can then internally compute SNAPSHOTs if that's how the service is able to realize the WFS-TE. Whether or not a query to generate SNAPSHOTs is answered by looking up pre-calculated SNAPSHOTs or by dynamically creating them is up to the implementation - as long as the implementation guarantees that the resulting SNAPSHOTs are correct (based upon the information at that time found in the BASELINEs, PERMDELTAs and TEMPDELTAs).

-- VolkerGrabsch - 26 Feb 2015: Sorry for the fuzzy question. First of all, I'm not talking about the internal data representation inside the WFS. This is implementation detail and should not be restricted by the standard in any way. We shouldn't care if the implementation internally saves only BASELINEs, saves only PERMDELTAs, caches SNAPSHOTs or generates all possible SNAPSOTs in advance. With "stored in the WFS" I meant: What do you see if you ask the WFS for all time slices of a feature, using a plain, non-temporal query? According to the WFS standard, you see exactly what was inserted/updated by previous WFS-T queries. And those might have inserted SNAPSHOTs. So the real question is: Do we allow for such WFS-T actions? And if we allow for that, should those SNAPSHOTs be ignored by the temporality extension, assuming the user inserted those by accident? Or should they always be used if present, assuming the user inserted those on purpose?

-- TimoThomas - 09 Mar 2015 I support Johannes' point of view. Regarding Volker's latest question, I would say it does not make sense to allow the insertion of SNAPSHOTs, in the same way as it does not make sense to accept any other invalid (or in this case extracted/calculated/compiled) data.

Clarify how to deal with PERMDELTAs

PERMDELTAs were introduced for push systems, while WFS is clearly a pull system. So for normal insertion and querying use cases, we don't need them at all.

So we should clarify whether we allow for insertion of PERMDELTAs via WFS-T. And if we allow for those, we should clearly say whether they should be ignored by any temporal operations. (Since we already have BASELINEs, there is no advantage in using PERMDELTAs. In fact, it would be a performance disadvantage if we were using PERMDELTAs instead of BASELINEs.)

On the other hand, we may consider allowing for special insertion and/or query commands regarding PERMDELTAs. For instance, we could mention the possibility of a PermdeltaGeneration (similar to SnapshotGeneration) so that the output of a WFS query may be use directly for a push system. Also, we may define some kind of PermdeltaInsertion operation for WFS-T (which will internally perform the corresponding BASELINE correction and open a new BASELINE), such that we could pipe from a push system directly into a WFS. Of course, both there operations should be considered "future work".

In case we agree that only BASELINEs and no PERMDELTAs should be stored in the WFS, the referenced Donlon features should be adjusted accordingly (converted from PERMDELTAs to BASELINEs). Reason: In that case, the WFS-TE doesn't specify any means to insert the PERMDELTAs time slices into the WFS. In other words, these Donlon features should represent what's actually inside the WFS, not what we may be able to insert via some PermdeltaInsertion operation in the future.

-- JohannesEchterhoff - 26 Feb 2015: this comes back to the question if there is a requirement for the WFS to generate a BASELINE for each PERMDELTA. If it was, then I'd say PERMDELTAs should be allows for insert/update (no real need imho to define a new operation for this). They should not be removed from the feature state/history, because time slice metadata (like source/origin of the permanent change) can be important. Operations should thus be able to return them, whether that's by default or upon specific request (e.g. via a parameter) can be discussed.

-- VolkerGrabsch - 26 Feb 2015: My understanding of the Temporality Model is that PERMDELTAs were never meant to be stored in a database. They exist just as messages between systems, causing the respective BASELINE updates at the receiver. And I believe they lost their importance over time, because all actual implementors (most prominently NM ADR) actually prefer transfering BASELINEs over PERMDELTAs because they are afraid of consequential errors (accumulating errors) in case a single PERMDELTA was somehow misinterpreted. Even NM's so-called "PERM_DELTA" datasets are just a collection of new BASELINEs. In the context of WFS, I believe that PERMDELTAs are some kind of "undefined behaviour". They are neither expected to be added via WFS-T, nor they are expected to be returned by any WFS query. Nevertheless, it is technically possible and valid to insert these via WFS-T, and then it is reasonable to expect to get them back (at least by plain, non-temporal queries). So we should defined how to handle those.

-- TimoThomas - 09 Mar 2015 To me PERMDELTAs are clearly defined. What is unclear is their exact relationship to BASELINEs (see above). Aren't PERMDELTAs better suited to model the change of features, especially regarding incorporated metadata? E.g. a change caused by construction work, by damage, by degradation, by re-measurement, ... - all this is probably found in the metadata, isn't it?

Add use case: Client wants SNAPSHOTs for a time period

Similar to use cases 1 and 3 (page 12), there should be an additional use case:

The clients wants all data for specific time period (e.g. 2 hours or a full day), for all features within a certain geographic region (e.g. bounding box), as a series of SNAPSHOTs. That way, the client can load this data once and the user can slide forth an back through the time, without the client having to issue a WFS request for each slider position.

The technical description of this functionality (series of SNAPSHOTs for a given time period) is already well-defined by the document (9.3.2 Requirement /req/snapshot-generation/time-period, page 33), but there's no use case mentioned for it.

Clarify omitted correction numbers for non-SNAPSHOT time slices

The WFS-TE document should clearly state that an omitted correctionNumber for non-SNAPSHOT time slices shall be interpreted as correctionNumber=0.

The AIXM Temporality 1.0 is not very clear about correction numbers. It assumes that every non-SNAPSHOT time slice has a sequenceNumber and correctionNumber, but in the examples, the correctionNumber is often obmitted. Obviously there is an implicit convention that a missing correctionNumber means "correctionNumber=0", but this is nowhere explicitly stated. Since it will take a while to get this clarification into AIXM-Temporality, we should provide it in the WFS-TE as a first step.

The related AIXM CCB issue is: https://aixmccb.atlassian.net/browse/AIXM-153?jql=project%20%3D%20AIXM

For the future, we propose to add a business rule ("best practice") which requires an explicit correctionNumber for all non-SNAPSHOT time slices. If there is an agreement, this "best practice" should also be stated in the AIXM Temporality document. In that case, the following XSLT script should be run over the Donlon test data:

<xsl:stylesheet xmlns:aixm="http://www.aixm.aero/schema/5.1" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
  <xsl:output method="xml"/>
  <xsl:template match="aixm:sequenceNumber[../aixm:interpretation!='SNAPSHOT' and not(../aixm:correctionNumber)]">
    <xsl:copy-of select="."/>
    <xsl:text>&#xa;&#9;&#9;&#9;&#9;&#9;</xsl:text>
    <aixm:correctionNumber>0</aixm:correctionNumber>
  </xsl:template>
  <xsl:template match="@*|node()">
    <xsl:copy>
      <xsl:apply-templates select="@*|node()"/>
    </xsl:copy>
  </xsl:template>
</xsl:stylesheet>

Clarify isDefined and featureLifetime

This is an example to make the discussion more clear.

Assume you have a Database 1 that contains a single feature with two BASELINEs:

- Database 1
  - AirportHeliport
    - BASELINE
      - validTime: 2008-2014
      - featureLifetime: 2008-indeterminate
      - point: (0.5, 0.5)
      - name: "Tegel"
    - BASELINE
      - validTime: 2014-2015
      - featureLifetime: 2008-indeterminate
      - point: (0.5, 0.5)
      - name: "Berlin/Tegel"

Now there a second Database 2 that mirrors Database 1, but contains only the latest BASELINE of each feature:

- Database 2 (only recent baselines)
  - AirportHeliport
    - BASELINE
      - validTime: 2014-2015
      - featureLifetime: 2008-indeterminate
      - point: (0.5, 0.5)
      - name: "Berlin/Tegel"

Now, consider the following query:

- Query
  - Filter
    - no timeIndicator (i.e. at any point in time)
    - Bounding Box: ((0, 0), (1, 1))
  - Snapshot
    - timeIndicator: 2009
- Result on Database 1

The Response of Database 1 is pretty straight forward:

<wfs:featureCollection>
  <wfs:hasMember>
    <aixm:AirportHeliport>
      <gml:identifier>...</gml:identifier>
      <aixm:timeSlice>
        ... SNAPSHOT for 2009 ...
      </aixm:timeSlice>
    </aixm:AirportHeliport>
  </wfs:hasMember>
</wfs:featureCollection>

However, the Response of Database 2 is not so clear, as there is no time slice to return. In principle, there are two possible replies

A) The feature is considered to be defined at 2009. The result the the feature without time slices:

<wfs:featureCollection>
  <wfs:hasMember>
    <aixm:AirportHeliport>
      <gml:identifier>...</gml:identifier>
      <!-- no time slices -->
    </aixm:AirportHeliport>
  </wfs:hasMember>
</wfs:featureCollection>

B) The feature is considered to be not defined at 2009. The result does not contain the feature:

<wfs:featureCollection>
  <!-- no feature -->
</wfs:featureCollection>

Clarify featureLifetime property for SNAPSHOTs

Related AIXM CCB Issue: https://aixmccb.atlassian.net/browse/AIXM-154?jql=project%20%3D%20AIXM

The AIXM Temporality 1.0 doesn't clearly say whether a SNAPSHOT contains the featureLifetime property or not, and if so, how it is defined.

The related AIXM CCB issue is: https://aixmccb.atlassian.net/browse/AIXM-154?jql=project%20%3D%20AIXM

Since this is a very visible detail for implementing the WFS-TE, we should find a consensus and write that into our WFS-TE specification. Also, we should adjust the Annex B example responses as needed. Finally, we should adjust the Donlon data set accordingly by raising a pull request.

To make this more concrete, consider a feature with 3 BASELINEs and a lifespan over 3 months:

• interpretation = BASELINE, validTime = [2014-01-01,2014-02-01), featureLifetime = [2014-01-01,indeterminate)
• interpretation = BASELINE, validTime = [2014-02-01,2014-03-01), featureLifetime = [2014-01-01,indeterminate)
• interpretation = BASELINE, validTime = [2014-03-01,2014-04-01), featureLifetime = [2014-01-01,2014-04-01)

(The sequenceNumber and correctionNumber are omitted for brevity. There are no TEMPDELTAs because they are irrelevant for this issue.)

How would the SNAPSHOT for 2014-02-15 look like?

• A: interpretation = BASELINE, validTime = [2014-02-15], no featureLifetime
• B: interpretation = BASELINE, validTime = [2014-02-15], featureLifetime = [2014-01-01,indeterminate)
• C: interpretation = BASELINE, validTime = [2014-02-15], featureLifetime = [2014-01-01,2014-04-01)

For every variant there are good arguments:

• A means: The featureLifetime is out of scope for SNAPSHOTs.
• B means: The featureLifetime is handled like any other property.
• C means: Since we know the total featureLifetime, we should also state so in the SNAPSHOT. (For cases where we don't know this, C is equal to B.)

What the current documents say about this:

• The current WFS-TE seems to favour A, because there is no featureLifetime in the example responses.
• On the other hand, WFS-TE always talks about a feature's lifetime as something that belongs statically to the whole feature, not as something that has different versions/changes over time. So it seems to also have C in mind.
• The current Donlon data set seems to favour B or C, which can be seen in the SNAPSHOT of feature c9bfcce2-76ad-46b7-a1e7-2c246893db56.

Comments from Eddy:

• My preference would go for option B - feature lifetime is treated as any other property.
• However, this does not exclude A since the content of a Snapshot (which properties to include and which not to include) is left at the discretion of the data provider, depending on the intended use of the data.
• For Donlon, I would avoid including featureLifetime in Snapshot examples since that would create the risk for further inconsistencies in the data set, in case of further Timeslices are appended to some features.

-- JohannesEchterhoff - 26 Feb 2015: I wonder if the example is correct, but that's probably due to the documentation of actual requirements regarding the handling of dynamic feature withdrawal / defined end of life in the Temporality Model, section 3.3. The figure there illustrates that once the end of life of a feature is known, all baselines would be corrected to show the updated lifetime. That sounds reasonable to me. When it gets to the creation of SNAPSHOTs, my suggestion is to use the feature lifetime as it is known at the time that the SNAPSHOT is created. So end time indeterminate as long as it is unknown, and the actual time once it is known. That is the same principle as for other dynamic feature properties: with new time slices added after a SNAPSHOT has been created the value of a property reported with that SNAPSHOT can always change. That's a fundamental characteristic of dynamic features.

-- VolkerGrabsch - 26 Feb 2015: This is an interesting aspect. My interpretation of Temporality Model, section 3.3 is slightly different: I believe that only the latest BASELINE gets the final featureLifettme. All previous BASELINEs have only the half-open featureLifetime, with the actual feature start but an indefinite end. However, on a purely technical level, all BASELINE could contain completely different featureLifetime values. Probably we should simply disallow this (e.g. by business rules), but nevertheless a WFS-TE implementation should somehow handle the general case. Either by rigurously refusing to do any temporal operations on features with "strange" featureLifetime. Or by simply taking always the featureLifetime of the latest BASELINE, ignoring all other featureLifetime values. Maybe there are other solutions.

-- TimoThomas - 09 Mar 2015 I don't think the figure in 3.3 supports Johannes point of view. The BASELINE corrections displayed there result from the insertion of subsequent BASELINEs only. Otherwise, the correction numbers should be greater than 1 (first correction by the insertion of the subsequent BASELINE, second correction by the update of featureLifetime.endPosition). Inserting BASELINEs, PERMDELTAs or TEMPDELTAs with the featureLifetime.endPosition set is clearly invalid to me, except it is a decommissioning as defined in 3.3. Regarding the display of featureLifetime in SNAPSHOTs I have no strong opinion.

Reopening a cancelled sequence

Regarding footnote 1 on page 7, what is the reasoning behind that rule?

This is an additional burden on implementation side, as every time slice needs to be checked, not just those with the highest correction number. What is the benefit that arises from this additional special case?

In deegree WFS-TE, reopening is allowed (for now).

-- JohannesEchterhoff - 26 Feb 2015: technically, I think there would be no harm in allowing cancellations to be corrected. So indeed the footnote appears to be too restrictive.

-- TimoThomas - 09 Mar 2015 Yes, the restriction may be dropped, if it is considered to complicate things.

Example:

<aixm:AirportHeliport>
  <aixm:timeSlice>
    <aixm:AirportHeliportTimeSlice>
      <gml:validTime>
        <gml:TimePeriod>
          <gml:beginPosition>2009-01-01T00:00:00Z</gml:beginPosition>
          <gml:endPosition indeterminatePosition="unknown"/>
        </gml:TimePeriod>
      </gml:validTime>
      <aixm:interpretation>BASELINE</aixm:interpretation>
      <aixm:sequenceNumber>1</aixm:sequenceNumber>
      <aixm:designator>OrigDesig</aixm:designator>
    </aixm:AirportHeliportTimeSlice>
  </aixm:timeSlice>
  <aixm:timeSlice>
    <aixm:AirportHeliportTimeSlice>
      <gml:validTime>
        <gml:TimePeriod>
          <gml:beginPosition>2010-01-01T00:00:00Z</gml:beginPosition>
          <gml:endPosition>2010-01-02T00:00:00Z</gml:endPosition>
        </gml:TimePeriod>
      </gml:validTime>
      <aixm:interpretation>TEMPDELTA</aixm:interpretation>
      <aixm:sequenceNumber>200</aixm:sequenceNumber>
      <aixm:designator>NewDesig</aixm:designator>
    </aixm:AirportHeliportTimeSlice>
  </aixm:timeSlice>
  <aixm:timeSlice>
    <aixm:AirportHeliportTimeSlice>
      <gml:validTime>
        <gml:TimePeriod>
          <gml:beginPosition>2010-01-01T00:00:00Z</gml:beginPosition>
          <gml:endPosition>2010-01-02T00:00:00Z</gml:endPosition>
        </gml:TimePeriod>
      </gml:validTime>
      <aixm:interpretation>TEMPDELTA</aixm:interpretation>
      <aixm:sequenceNumber>200</aixm:sequenceNumber>
      <aixm:correctionNumber>1</aixm:correctionNumber>
      <aixm:designator>NewDesigCorrected</aixm:designator>
    </aixm:AirportHeliportTimeSlice>
  </aixm:timeSlice>
  <aixm:timeSlice>
    <aixm:AirportHeliportTimeSlice>
      <gml:validTime nilReason="inapplicable"/>
      <aixm:interpretation>TEMPDELTA</aixm:interpretation>
      <aixm:sequenceNumber>200</aixm:sequenceNumber>
      <aixm:correctionNumber>2</aixm:correctionNumber>
    </aixm:AirportHeliportTimeSlice>
  </aixm:timeSlice>
  <aixm:timeSlice>
    <aixm:AirportHeliportTimeSlice>
      <gml:validTime>
        <gml:TimePeriod>
          <gml:beginPosition>2010-01-01T00:00:00Z</gml:beginPosition>
          <gml:endPosition>2010-01-02T00:00:00Z</gml:endPosition>
        </gml:TimePeriod>
      </gml:validTime>
      <aixm:interpretation>TEMPDELTA</aixm:interpretation>
      <aixm:sequenceNumber>200</aixm:sequenceNumber>
      <aixm:correctionNumber>3</aixm:correctionNumber>
      <aixm:designator>NewDesigResurrected</aixm:designator>
    </aixm:AirportHeliportTimeSlice>
  </aixm:timeSlice>
</aixm:AirportHeliport>

Without the last time slice, the TEMPDELTA is cancelled and a SNAPSHOT would return:

      <aixm:designator>OrigDesig</aixm:designator>

With the last time slice, the TEMPDELTA is not cancelled and a SNAPSHOT would return:

      <aixm:designator>NewDesigResurrected</aixm:designator>

includeCanceled without includeCorrected

Regarding 9.2.3 (page 31), what is the expected behaviour if includeCorrected=false and includeCanceled=true?

• A: show all time slices of all canceled sequences (including the corrected ones)
• B: show only the latest (non-corrected) time slice of each canceled sequence (This is currently implemented in deegree WFS-TE.)
• C: show only the cancelation time slice of each canceled sequence (see "Reopening a cancelled sequence" for why this may be different from the latest)

Telcon result: B ist correct

• includeCorrected means: includeCorrectedTimeSlices
• includeCanceled means: includeCanceledSequences

Example:

• Sequence 1 is normal
• Sequence 2 is canceled
• Sequence 3 is reopened after cancelation
• Sequence 4 is canceled from the very beginning (not very useful, but an interesting edge case)

Seq	Cor	Valid		default	includeCorrected	includeCanceled	includeCorrected+includeCanceled
1	0	ok		no	yes	no	yes
	1	ok		no	yes	no	yes
	2	ok		no	yes	no	yes
	3	ok		yes	yes	yes	yes
2	0	ok		no	no	no	yes
	1	ok		no	no	no	yes
	2	ok		no	no	no	yes
	3	nil		no	no	yes	yes
3	0	ok		no	yes	no	yes
	1	ok		no	yes	no	yes
	2	nil		no	yes	no	yes
	3	ok		yes	yes	yes	yes
4	0	nil		no	no	yes	yes

-- JohannesEchterhoff - 26 Feb 2015: handling of cancellations should be clarified in general.

-- TimoThomas - 09 Mar 2015 C. If re-opening canceled sequences are allowed, I would consider a re-opened sequence to be not canceled, as long as the active time slice of that sequence is not a cancelation. For the (new) case of a canceled + re-opened + canceled sequence, I would refine the definition of 9.2.3 to only return the active time slice (which is a cancelation).

DynamicFeatureFilter without timeIndicator

Regarding 8.2.1 (page 22), what time slices does the featureFilter act on?

• A: the original TEMPDELTA and BASELINE time slices after filtering out corrected and canceled time slices
• B: all possible SNAPSHOT time slices that could be generated (for all times)

At a first glance, the document reads as if A is intended. Also, at a first glance, it seems that A and B will always return the same result. However, consider a feature with the following time slices:

• identifier=23
• interpretation=BASELINE, validTime=[2014-02-01,2014-02-15], somePropery=off
• interpretation=TEMPDELTA, validTime=[2014-02-01,2014-02-15], somePropery=on

Any snapshot generated for the feature will always have somePropery=on. Now, consider the following DynamicFeatureQuery:

• timeIndicator: (omitted)
• featureFilter: identifier=23 and timeSlice/*/somePropery=off
• no transformations
• no projections

Variant A would select feature 23, while variant B would return an empty result.

Another direct consequence is visible by the following queries:

• timeIndicator: (omitted), featureFilter: identifier=23 and timeSlice/*/interpretation=BASELINE
• timeIndicator: (omitted), featureFilter: identifier=23 and timeSlice/*/interpretation=SNAPSHOT

Exactly one of these queries will return the feature while the other will return an empty result.

Implementation specific aspects:

• Simplicity: Variant B is slightly simpler to implement than variant A (about 5-10 fewer lines of code in a well-structured code base).
• Performance: On a database with "usual" indexes, variant A will run faster than variant B. The situation may change when introducing special database indexes aimed at fast SNAPSHOT composition.

State of discussion:

• Johannes favours B
  • -- JohannesEchterhoff - 26 Feb 2015: yes, because the service should filter dynamic features based upon their actual state at any given moment in time. In other words, the service should evaluate the time slices on-the-fly for the client. Omitting the time indicator to me is like providing a time indicator with the feature lifetime being the value. Also, I think that a DynamicFeatureFilter should not be created with the path pointing to the time slice interpretation, because the interpretation is not really a property of the dynamic feature itself. For filtering time slices, better use the TimeSliceProjection.
  • -- VolkerGrabsch - 26 Feb 2015: I fully agree. The "interpretation" example doesn't make any practical sense. I just used it for easier demonstration of the corner cases. I wouldn't explicitly forbid that, I'd just like to let it behave consistently with all rules given (no special rules for filtering by "interpretation" are needed, IMHO).
• Timo favours A, to make the filter language more expressive. Variant B can still be produced by providing a timeInstant with a huge time span, e.g. beginning of universe till end of universe.
• Volker favours B, arguing that leaving out a single parameter (the timeInstant) should not change filter semantics that much (principle of the least surprise).

Query structures

From a bird's eye view, the following query structures are possible: (ignoring projection and transformation clauses, non-filtered clauses and the fact that DynamicFilter internally contains again a normal Filter)

• A: Query containing Filter (OK)
• B: DynamicFeatureQuery containing DynamicFilter (OK)
• C: Query containing DynamicFilter (Invalid)
• D: DynamicFeatureQuery containing Filter (Valid?)
• E: DynamicFeatureQuery containing both Filter and DynamicFilter (Invalid)

Obviously, structures A and B are perfectly valid. Structure C does not violate the XSD schema, because DynamicFilter is in the same substitution group as Filter. However it violates the written GML/WFS standard. Structure E is invalid because it violates the XSD schema.

Open question: Is structure D valid or not? What do we want?

Comments by Timo:

• There is still requirement 9.1.1 (/req/core/include-optional-properties).
• This means that D is the only possibility to express a filter that matches on canceled or corrected time slices (as in A), but still outputs all optional elements (as in B).
• However, it is unclear whether there's a practical use case for that.

-- JohannesEchterhoff - 26 Feb 2015: We should clarify whether D is allowed or not, and in case it was allowed, what that exactly means.

-- TimoThomas - 08 Jun 2015: D also allows the use of SnapshotGeneration and TimesliceProjection in conjunction with a normal WFS Filter. I think we should not forbid that.

Clarify whether extensions elements count as properties

Consider the following example of an imaginary "someext" Extension to AIXM that defines two new elements "a" and "b".

<aixm:AirportHeliport>
  <aixm:timeSlice>
    <aixm:AirportHeliportTimeSlice>
      <gml:validTime>
        <gml:TimePeriod>
          <gml:beginPosition>2009-01-01T00:00:00Z</gml:beginPosition>
          <gml:endPosition indeterminatePosition="unknown"/>
        </gml:TimePeriod>
      </gml:validTime>
      <aixm:interpretation>BASELINE</aixm:interpretation>
      <aixm:designator>OrigDesig</aixm:designator>
      <aixm:extension>
        <someext:AirportHeliportExtension>
          <someext:a>OrigA</someext:a>
          <someext:b>OrigB</someext:b>
        </someext:AirportHeliportExtension>
      </aixm:extension>
    </aixm:AirportHeliportTimeSlice>
  </aixm:timeSlice>
  <aixm:timeSlice>
    <aixm:AirportHeliportTimeSlice>
      <gml:validTime>
        <gml:TimePeriod>
          <gml:beginPosition>2010-01-01T00:00:00Z</gml:beginPosition>
          <gml:endPosition>2010-01-02T00:00:00Z</gml:endPosition>
        </gml:TimePeriod>
      </gml:validTime>
      <aixm:interpretation>TEMPDELTA</aixm:interpretation>
      <aixm:designator>NewDesig</aixm:designator>
      <aixm:extension>
        <someext:AirportHeliportExtension>
          <someext:a>NewA</someext:a>
        </someext:AirportHeliportExtension>
      </aixm:extension>
    </aixm:AirportHeliportTimeSlice>
  </aixm:timeSlice>
</aixm:AirportHeliport>

If the WFS creates a SNAPSHOT for 2010-01-01, what should it return?

A) The extension element is a property much like the designator, so it is replaced at once by the TEMPDELTA:

<aixm:AirportHeliport>
  <aixm:timeSlice>
    <aixm:AirportHeliportTimeSlice>
      <gml:validTime>...</gml:validTime>
      <aixm:interpretation>SNAPSHOT</aixm:interpretation>
      <aixm:designator>NewDesig</aixm:designator>
      <aixm:extension>
        <someext:AirportHeliportExtension>
          <someext:a>NewA</someext:a>
        </someext:AirportHeliportExtension>
      </aixm:extension>
    </aixm:AirportHeliportTimeSlice>
  </aixm:timeSlice>
</aixm:AirportHeliport>

B) The extension element itself is not a property, but the contained elements "a" and "b" are properties, so these are replaced individually by the TEMPDELTA:

<aixm:AirportHeliport>
  <aixm:timeSlice>
    <aixm:AirportHeliportTimeSlice>
      <gml:validTime>...</gml:validTime>
      <aixm:interpretation>SNAPSHOT</aixm:interpretation>
      <aixm:designator>NewDesig</aixm:designator>
      <aixm:extension>
        <someext:AirportHeliportExtension>
          <someext:a>NewA</someext:a>
          <someext:b>OrigB</someext:b>
        </someext:AirportHeliportExtension>
      </aixm:extension>
    </aixm:AirportHeliportTimeSlice>
  </aixm:timeSlice>
</aixm:AirportHeliport>

At the moment, the m-click WFS-TE implements A).

-- TimoThomas - 9 June 2015: Please note that aixm:extension itself is a multi-occurring property. So if we choose A), applying the rule for multi-occurring properties in *DELTA time slices would require to list them all. I do not know what applying B) means here, where there is the special case of multiple overlapping "someext:a" child properties. However, there is another option which I'd prefer: treat extension elements containing the same class of extension (here: someext:AirportHeliportExtension) as a group. That means all elements of a group (here: all aixm:extension/someext:AirportHeliportExtension properties) have to be provided in full in TEMPDELTAs and PERMDELTAs. Motivation: it is likely that the extension mechanism is used to store vendor or customer specific data, where each customer or vendor will introduce new classes of extensions. With this rule, extensions of different classes can play nicely together while the compactness of the data in delta timeslices is still preserved. Example (C):

C) Extensions of same types are treated as individual properties according to the AIXM TM. Example: given two extensions from vendor A and vendor B for AirportHeliport: vendorA:AirportHeliportExtension and vendorB:AirportHeliportExtension. In a BASELINE, always all properties have to be listed. Example:

<aixm:AirportHeliport>
  <aixm:timeSlice>
    <aixm:AirportHeliportTimeSlice>
      <gml:validTime>...</gml:validTime>
      <aixm:interpretation>BASELINE</aixm:interpretation>
      <aixm:designator>NewDesig</aixm:designator>
      <aixm:extension>
        <vendorA:AirportHeliportExtension>
          <vendorA:a>valueX</vendorA:a>
          <vendorA:b>valueY</vendorA:b>
        </vendorA:AirportHeliportExtension>
      </aixm:extension>
      <aixm:extension>
        <vendorB:AirportHeliportExtension>
          <vendorB:propertyC>valueZ</vendorB:propertyC>
        </vendorB:AirportHeliportExtension>
      </aixm:extension>
      <aixm:extension>
        <vendorB:AirportHeliportExtension>
          <vendorB:propertyC>valueZZ</vendorB:propertyC>
        </vendorB:AirportHeliportExtension>
      </aixm:extension>
    </aixm:AirportHeliportTimeSlice>
  </aixm:timeSlice>
</aixm:AirportHeliport>

Now in a PERMDELTA or TEMPDELTA, if the change only affects vendor A's extension, variant C) implies that we only have to list those, and we can leave out vendor B's extensions, as we leave out any other unchanged properties:

<aixm:AirportHeliport>
  <aixm:timeSlice>
    <aixm:AirportHeliportTimeSlice>
      <gml:validTime>...</gml:validTime>
      <aixm:interpretation>TEMPDELTA</aixm:interpretation>
      <aixm:extension>
        <vendorA:AirportHeliportExtension>
          <vendorA:a>newValueX</vendorA:a>
          <vendorA:b>newValueY</vendorA:b>
        </vendorA:AirportHeliportExtension>
      </aixm:extension>
      <!-- vendorB's extensions do not have to be listed as they are unchanged -->
    </aixm:AirportHeliportTimeSlice>
  </aixm:timeSlice>
</aixm:AirportHeliport>

How to query for referenced features

AIXM encodes references between features in properties using an extended XLink standard. This extension introduces 'abstract references', which are not resolvable through the network by nature. Further, references are not bi-directional. E.g. runway features contain references to their airport, but airport features do not contain links to their runways. This is not necessarily involving temporality, i.e. this is an issue for standard WFS/FES as well. But as it is a valid use case, and as it is not trivial, we need to clarify it for WFS-TE. We have the following cases:

a) Constraints on forward references (runway -> airport), e.g. 'Retrieve the runways of an airport given the airport's uuid'

Example: Given a runway associated to an airport:

<aixm:Runway gml:id="uuid.9e51668f-bf8a-4f5b-ba6e-27087972b9b8">
  <gml:identifier codeSpace="urn:uuid:">9e51668f-bf8a-4f5b-ba6e-27087972b9b8</gml:identifier>
  <aixm:timeSlice>
    <aixm:RunwayTimeSlice gml:id="RNWTS2120">
        ...
      <aixm:associatedAirportHeliport xlink:href="urn:uuid:1b54b2d6-a5ff-4e57-94c2-f4047a381c64"/>
    </aixm:RunwayTimeSlice>
  </aixm:timeSlice>
</aixm:Runway>

<aixm:AirportHeliport gml:id="uuid.1b54b2d6-a5ff-4e57-94c2-f4047a381c64">
  <gml:identifier codeSpace="urn:uuid:">1b54b2d6-a5ff-4e57-94c2-f4047a381c64</gml:identifier>
  <aixm:timeSlice>
    <aixm:AirportHeliportTimeSlice gml:id="ahts1EADD">
      ...
    </aixm:AirportHeliportTimeSlice>
  </aixm:timeSlice>
</aixm:AirportHeliport>

There are three options for encoding the constraint in an FES filter:

a.1) using an XPath constraint on the xlink attribute:

<fes:Filter>
  <fes:PropertyIsEqualTo>
  <fes:ValueReference>aixm:associatedAirportHeliport/@xlink:href</fes:ValueReference>
  <fes:Literal>urn:uuid:1b54b2d6-a5ff-4e57-94c2-f4047a381c64</fes:Literal>
  </fes:PropertyIsEqualTo>
</fes:Filter>

Discussion:

• very compact and simple
• only works for abstract references, standard ('concrete') references are not supported and the constraint will not match
• the XPath expression is valid, but not part of the reduced XPath subset defined in FES. However, it shouldn't impose a problem to extend the definition for WFS-TE.

a.2) using the wfs:valueOf() XPath function

<fes:Filter>
  <fes:PropertyIsEqualTo>
  <fes:ValueReference>wfs:valueOf(aixm:timeSlice/*/aixm:associatedAirportHeliport)/gml:identifier</fes:ValueReference>
  <fes:Literal>1b54b2d6-a5ff-4e57-94c2-f4047a381c64</fes:Literal>
  </fes:PropertyIsEqualTo>
</fes:Filter>

Discussion:

• still very compact and simple
• works for both abstract and standard (concrete) references
• Remark: if based on WFS 2.5(?), anticipating CR 03-106, valueOf() is deprecated and auto-dereferencing takes place. There, <fes:ValueReference>aixm:timeSlice/*/aixm:associatedAirportHeliport/aixm:AirportHeliport/gml:identifier</fes:ValueReference> is correct.
• server has to implement wfs:valueOf() or auto-dereferencing, and the definition of wfs:valueOf()/auto-dereferencing has to be extended to support abstract references
• for temporality support, we need to extend wfs:valueOf() by a time parameter: see "Advanced Filtering: valueFor() XPath function" in WFS-TE
• Remark: example will match on all code spaces. To restrict this, the ValueReference could be changed to wfs:valueOf(aixm:timeSlice/*/aixm:associatedAirportHeliport)/gml:identifier[@codeSpace='urn:uuid:'] (if based on WFS 2.0). However, this would also require to extend the reduced XPath subset defined in FES.

a.3) using joins

Standard WFS:

<wfs:Query typeNames="aixm:AirportHeliport aixm:Runway" aliases="airport runway">
  <fes:Filter>
    <fes:And>
      <fes:PropertyIsEqualTo>
        <fes:ValueReference>airport</fes:ValueReference>
        <fes:ValueReference>wfs:valueOf(runway/aixm:timeSlice/*/aixm:associatedAirportHeliport)</fes:ValueReference>
      </fes:PropertyIsEqualTo>
      <fes:PropertyIsEqualTo>
        <fes:ValueReference>airport/gml:identifier</fes:ValueReference>
        <fes:Literal>1b54b2d6-a5ff-4e57-94c2-f4047a381c64</fes:Literal>
      </fes:PropertyIsEqualTo>
    </fes:And>
  </fes:Filter>
</wfs:Query>

WFS-TE:

<wfs-te:DynamicFeatureQuery typeNames="aixm:AirportHeliport aixm:Runway" aliases="airport runway">
  <fes-te:DynamicFeatureFilter>
    <fes-te:timeIndicator>
      ...
    </fes-te:timeIndicator>
    <fes-te:featureFilter>
      <fes:And>
        <fes:PropertyIsEqualTo>
          <fes:ValueReference>airport</fes:ValueReference>
          <fes:ValueReference>wfs:valueOf(runway/aixm:timeSlice/*/aixm:associatedAirportHeliport)</fes:ValueReference>
        </fes:PropertyIsEqualTo>
        <fes:PropertyIsEqualTo>
          <fes:ValueReference>airport/gml:identifier</fes:ValueReference>
          <fes:Literal>1b54b2d6-a5ff-4e57-94c2-f4047a381c64</fes:Literal>
        </fes:PropertyIsEqualTo>
      </fes:And>
    </fes-te:featureFilter>
  </fes-te:DynamicFeatureFilter>
</wfs-te:DynamicFeatureQuery>

Discussion:

• quite verbose
• works for both abstract and standard (concrete) references
• also works for DynamicFeatureQuery with time indicator
• server has to support joins
• Remark: if based on WFS 2.5(?), anticipating CR 03-106, valueOf() is deprecated and auto-dereferencing takes place (simply omit the the outer function call and take its argument)
• server has to implement wfs:valueOf() or auto-dereferencing, and the definition of wfs:valueOf()/auto-dereferencing has to be extended to support abstract references
• Remark: applied to GetFeature, it will return tuples of airport/runway features, which may be more than the client requires (i.e. in this use case he's interested in the runways only). This can be overcome by applying it to GetPropertyValue with valueReference="runway"
• support for joins was not investigated in the development of WFS-TE, but in principle it should work. May need further elaboration.
• Remark: example will match on all code spaces. To restrict this, the ValueReference could be changed to airport/gml:identifier[@codeSpace='urn:uuid:']. However, this would also require to extend the reduced XPath subset defined in FES.

b) Constraints on backward references (airport <- runway), e.g. 'Retrieve the airport of a runway given the runway's uuid'

The following examples are based on the data given in a). As there is no path from the airport to its runways, this is only possible with joins:

Standard WFS:

<wfs:Query typeNames="aixm:AirportHeliport aixm:Runway" aliases="airport runway">
  <fes:Filter>
    <fes:And>
      <fes:PropertyIsEqualTo>
        <fes:ValueReference>airport</fes:ValueReference>
        <fes:ValueReference>wfs:valueOf(runway/aixm:timeSlice/*/aixm:associatedAirportHeliport)</fes:ValueReference>
      </fes:PropertyIsEqualTo>
      <fes:PropertyIsEqualTo>
        <fes:ValueReference>runway/gml:identifier</fes:ValueReference>
        <fes:Literal>9e51668f-bf8a-4f5b-ba6e-27087972b9b8</fes:Literal>
      </fes:PropertyIsEqualTo>
    </fes:And>
  </fes:Filter>
</wfs:Query>   

WFS-TE:

<wfs-te:DynamicFeatureQuery typeNames="aixm:AirportHeliport aixm:Runway" aliases="airport runway">
  <fes-te:DynamicFeatureFilter>
    <fes-te:timeIndicator>
      ...
    </fes-te:timeIndicator>
    <fes-te:featureFilter>
      <fes:And>
        <fes:PropertyIsEqualTo>
          <fes:ValueReference>airport</fes:ValueReference>
          <fes:ValueReference>wfs:valueOf(runway/aixm:timeSlice/*/aixm:associatedAirportHeliport)</fes:ValueReference>
        </fes:PropertyIsEqualTo>
        <fes:PropertyIsEqualTo>
          <fes:ValueReference>runway/gml:identifier</fes:ValueReference>
          <fes:Literal>9e51668f-bf8a-4f5b-ba6e-27087972b9b8</fes:Literal>
        </fes:PropertyIsEqualTo>
      </fes:And>
    </fes-te:featureFilter>
  </fes-te:DynamicFeatureFilter>
</wfs-te:DynamicFeatureQuery>

Discussion:

• quite verbose
• works for both abstract and standard (concrete) references
• also works for DynamicFeatureQuery with time indicator
• server has to support joins
• Remark: if based on WFS 2.5(?), anticipating CR 03-106, valueOf() is deprecated and auto-dereferencing takes place (simply omit the the outer function call and take its argument)
• server has to implement wfs:valueOf() or auto-dereferencing, and the definition of wfs:valueOf()/auto-dereferencing has to be extended to support abstract references
• Remark: applied to GetFeature, it will return tuples of airport/runway features, which may be more than the client requires (i.e. in this use case he's interested in the airport only). This can be overcome by applying it to GetPropertyValue with valueReference="airport"
• support for joins was not investigated in the development of WFS-TE, but in principle it should work. May need further elaboration.
• Remark: example will match on all code spaces. To restrict this, the ValueReference could be changed to runway/gml:identifier[@codeSpace='urn:uuid:']. However, this would also require to extend the reduced XPath subset defined in FES.

Summary:

Joins are the only solution supporting both forward and backward references in a similar manner, however, they are also the most verbose and complex ones.

[Comments from Volker Grabsch, copied from Google Docs]: In the Testbed 11 component "Enrichment Service", we gained some practical experience with resolving references. Backwards references are a tricky beast. The best solution we came up with was to add reverse references where needed. This is possible via the existing schema extension "AIXM_ReverseAssociations" (a.k.a. "MXIA"). We create another, similar schema extension "Event_ReverseAssociations" for Digital NOTAMs.

-- TimoThomas - 24 Jun 2015: I discourage the introduction of backward references because of the implied maintenance. In the above example, this would mean that we need to update the airport feature when there is a runway added to it. One can argue that this is tolerable, as creating a new runway or changing it's association to an airport is a rare operation and affects the airport to. However, we have to consider all other possible references as well. For AirportHeliport, we have also

Taxiway, Apron, Road, TouchDownLiftOff, NonMovementArea, SurveyControlPoint, WorkArea, AirportHeliportCollocation, AirportHotSpot, RulesProcedures, RadarSystem, AirportGroundService, InformationService, GroundTrafficControlService, AirTrafficControlService, NavigationSystemCheckpoint, Navaid, DesignatedPoint, SignificantPoint, AeronauticalGroundLight, Unit, ObstacleAreaOrigin, Procedure, SafeAltitudeArea, FlightConditionElementChoice, FlightRoutingElementChoice

referencing it. For the feature OrgansiationAuthority, there are

AirportHeliport, AuthorityForAirspace, AirspaceActivation, RadarSystem, SpecialDate, PropertiesWithSchedule, NavaidEquipment, SpecialNavigationStation, SpecialNavigationSystem, Unit, OrganisationAuthority, ObstacleAreaOrigin, VerticalStructure, FlightConditionElementChoice, Route, AerialRefuelling

referencing it. These are just a few examples, there are more. Please note that there is an reference from the Object PropertiesWithSchedule to OrganisationAuthority. Objects do not have a gml:identifier, which means we cannot formulate an abstract reference from OrganisationAuthority to a PropertiesWithSchedule object. Switching to concrete references based on the gml:id is not option in my opinion, as the consequences are vast.

-- VolkerGrabsch - 30 Jun 2015: I agree that we should discourage adding all possible backward references. This would be a maintenance nightmare without much benefit. In particular, this would completely destory the utility of the "resolveDepth" parameter. However, I still think that for the most important use cases (get all features related to some base feature, for portrayal on the map), the "resolveDepth" is a very elegant solution as you don't have specify any joins explicitly. This removes burden from the client to have intimate knowledge about the AIXM structures and thus plays well with generic clients.

On the other hand, for queries where you do want to follow specific links, I believe that (a.1) is an ugly hack, although not as bad as it seems due to the convention that AIXM should always use the urn:uuid: codespace. (a.3) seem to be too complicated, and (a.2) with or without auto-dereferencing should be the way to go.

To clarify, I propose backward references only if applied sporatically and exactly where you need them. It's a kind of "manually managed index" for reverse references.</verbatim>