Context and Rendered Template Fields in StartTriggerArgs

[HadarSha]

Hello,

We have started experimenting with the new experimental features of Triggering Deferral from Task Start and Exiting deferred task from Trigger - they seem to be working well for what they do, but the former has two very challenging limitations at the moment:

1. It seems like there is no way to access the Airflow Context when using a trigger directly, unless, of course, you start from a worker by implementing the execute method and set start_trigger_args to False). Once you set start_trigger_args to True, the execute method (understandably) does not actually get called, neither does pre_execute and in fact it seems that there is nowhere "in the middle" to obtain any access to the Airflow Context.
   This can be a limitation in case you need to access anything from the Context during runtime, but mostly present itself in the usage of rendered templated fields, which is interrelated to the context and gets rendered somewhere between the __init__ method of the operator running the task, and its pre_execute method (where the templated fields have already been rendered). This brings us to the second issue...
2. It seems like when using StartTriggerArgs, the templated fields never get rendered, and you basically cannot use templating at all. This is a severe limitation even in relatively light and simple operators that don't do much work (e.g. operator to run a SQL query only makes a call to the database and doesn't warrant a worker, hence a classic case for using the triggerer directly, but the SQL query cannot be templated...)

Last but not least - it seems that you do get access to a somewhat half-baked TaskInstance object, that doesn't quite look and behave like the one you could obtain from the Context, probably for the exact same reasons you aren't getting the Context to begin with.

All in all, it's an incredibly useful feature, but I'd like to enquire about the possibility of adding a way to reason about templating or rendering without having to go through the execute method. That would unlock a lot of new possibilities that are currently not present with this feature.

The documentation does not deal with this issue or mentions it in any way, AFAICS. This is just my own experience with having tried to implement it in various operators, with partial success due to the above.

Cheers!

[adrianrego]

Ran into the same limitations recently and wasn't clear from the documentation if it was possible or not. Definitely an area for improvement and further discussion.

[potiuk]

Indeed - if any of you would like to contribute to the documentation explaining that - that would be awesome - this is super-easy (click "suggest a change on this page" at the page you want to improve and PR will be opened automatically for you @adrianrego @HadarSha - this is how open-source contribution works, Airflow is created by 3000+ contributors, often people like you who missed some docs and added it after they figured out how to do stuff.

[potiuk]

Later it can be improved, maybe - but documenting it is the least you can help wuth (unless you would like to add those features and lead implementation of those).