Redesign of the DatePartitionedQueryPlanner (formerly know as the FederatedQueryPlanner) #2717
Conversation
Updated TODOs
what fields are unindexed in each range
This reverts commit 549130b.
warehouse/query-core/src/main/java/datawave/query/planner/DatePartitionedQueryPlanner.java
Outdated
Show resolved
Hide resolved
warehouse/query-core/src/main/java/datawave/query/planner/DatePartitionedQueryPlanner.java
Show resolved
Hide resolved
warehouse/query-core/src/test/java/datawave/query/IndexHoleQueryTest.java
Show resolved
Hide resolved
|
We should probably adjust the marker concept to be one day earlier than the last old index entry timestamp. Since we usually deploy during the day, any deployment where we ceased indexing something would only be a partial day. I replicated this with a test case and any data from the last indexed date that did not get indexed (i.e. post hypothetical deployment) would be missed. |
|
In other tests, I'm getting empty ranges. We're still checking the fi for the field with the Hole marker, so the AndIterator appears to be defeating. Haven't fully stepped through, though, but we need more end-to-end tests up in here (up in here). |
...se/query-core/src/main/java/datawave/query/jexl/visitors/PushdownUnindexedFieldsVisitor.java
Outdated
Show resolved
Hide resolved
...se/query-core/src/main/java/datawave/query/jexl/visitors/PushdownUnindexedFieldsVisitor.java
Show resolved
Hide resolved
...se/query-core/src/main/java/datawave/query/jexl/visitors/PushdownUnindexedFieldsVisitor.java
Outdated
Show resolved
Hide resolved
...se/query-core/src/main/java/datawave/query/jexl/visitors/PushdownUnindexedFieldsVisitor.java
Outdated
Show resolved
Hide resolved
...se/query-core/src/main/java/datawave/query/jexl/visitors/PushdownUnindexedFieldsVisitor.java
Outdated
Show resolved
Hide resolved
|
|
||
| @Override | ||
| public Object visit(ASTEQNode node, Object data) { | ||
| if (isIndexed(node) && missingIndexRange(node)) { |
There was a problem hiding this comment.
I'm not sure that this logic will handle queries with arithmetic or methods very well. The field or the value may be null.
See the FunctionalSetTest for examples
There was a problem hiding this comment.
but those methods will not be used in the index lookup so do they matter?
There was a problem hiding this comment.
Sorry, I think, I see what you mean.... I will reevaluate what I am missing.
...se/query-core/src/main/java/datawave/query/jexl/visitors/PushdownUnindexedFieldsVisitor.java
Outdated
Show resolved
Hide resolved
I changed this to use evaluation only instead of index value hole markers in the query to avoid the field index lookups for unindexed fields. |
warehouse/query-core/src/main/java/datawave/query/planner/DatePartitionedQueryPlanner.java
Outdated
Show resolved
Hide resolved
warehouse/query-core/src/main/java/datawave/query/planner/DatePartitionedQueryPlanner.java
Outdated
Show resolved
Hide resolved
warehouse/query-core/src/main/java/datawave/query/planner/DatePartitionedQueryPlanner.java
Outdated
Show resolved
Hide resolved
warehouse/query-core/src/main/java/datawave/query/planner/FederatedQueryIterable.java
Outdated
Show resolved
Hide resolved
warehouse/query-core/src/main/java/datawave/query/planner/DatePartitionedQueryPlanner.java
Outdated
Show resolved
Hide resolved
...se/query-core/src/main/java/datawave/query/jexl/visitors/PushdownUnindexedFieldsVisitor.java
Outdated
Show resolved
Hide resolved
...se/query-core/src/main/java/datawave/query/jexl/visitors/PushdownUnindexedFieldsVisitor.java
Show resolved
Hide resolved
...se/query-core/src/main/java/datawave/query/jexl/visitors/PushdownUnindexedFieldsVisitor.java
Show resolved
Hide resolved
| /** | ||
| * Visitor meant to 'push down' predicates for expressions that are not executable against the global OR field index. | ||
| */ | ||
| public class PushdownUnindexedFieldsVisitor extends RebuildingVisitor { |
There was a problem hiding this comment.
Will you be adding any tests for this visitor?
warehouse/query-core/src/main/java/datawave/query/planner/DatePartitionedQueryPlanner.java
Outdated
Show resolved
Hide resolved
No description provided.