AIP-84 Authentications and Permissions

[pierrejeambrun]

Meta issue for Authentications and Permissions for the API.

I Multiple PRs to add back permissions to all endpoints (the ones from the legacy API that were removed during the migration)

Public API

• assets @rawwar AIP-84 - Add Auth for Assets #47136
• asset aliases @rawwar AIP 84 - Add auth for asset alias #47241
  • associated assets @rawwar AIP-84 filter authorized asset_alias in asset endpoints #47561
• backfills @vatsrahul1001 AIP-84 Add Auth for backfill #47482
• config @jason810496 AIP-84 | Add Auth for Config #47208
• connections @rawwar AIP 84 - Add auth to connections #47194
• dag_parsing @rawwar AIP 84 - Add Auth to xcom, ti, dag_warning, dag_stats, dag_source, dag_run, dag_parsing #47532
• dag_report @rawwar
• dag_run @rawwar AIP 84 - Add Auth to xcom, ti, dag_warning, dag_stats, dag_source, dag_run, dag_parsing #47532
• dag_sources @rawwar AIP 84 - Add Auth to xcom, ti, dag_warning, dag_stats, dag_source, dag_run, dag_parsing #47532
• dag_stats @rawwar AIP 84 - Add Auth to xcom, ti, dag_warning, dag_stats, dag_source, dag_run, dag_parsing #47532
• dag_tags @vatsrahul1001
• dag_versions @vatsrahul1001
• dag_warning @rawwar AIP 84 - Add Auth to xcom, ti, dag_warning, dag_stats, dag_source, dag_run, dag_parsing #47532
• dags @jason810496 AIP-84 | Add Auth for Dags #47062, AIP-84 | Add Auth for Dags #47433
• event_logs @jason810496 AIP-84 | Add Auth for Event Logs #47246
• extra_links @jason810496 AIP-84 | Add Auth for Extra Link #47251
• import_error @jason810496 AIP-84 | Add Auth for Import Error #47270
• job @jason810496 AIP-84 | Add Auth for Job #47271
• log @jason810496 AIP-84 | Add Auth for Log #47302
• login (is already not an authenticated endpoint and need to remain like this)
• monitor (is already not an authenticated endpoint and need to remain like this)
• plugins @pierrejeambrun AIP-84 Add Auth to plugins #47504
• pools @rawwar AIP 84 - Add Auth to Pools #47192
• providers @pierrejeambrun AIP-84 Providers permissions #47505
• task_instances @rawwar AIP 84 - Add Auth to xcom, ti, dag_warning, dag_stats, dag_source, dag_run, dag_parsing #47532
• tasks @vatsrahul1001
• variables @@jason810496 AIP-84 | Add Auth for Variable #47211
• version (is already not an authenticated endpoint and need to remain like this)
• xcom @rawwar AIP 84 - Add Auth to xcom, ti, dag_warning, dag_stats, dag_source, dag_run, dag_parsing #47532
• @jason810496 AIP-84 Add ui auth endpoint to expose FAB links from NavBar #47457

Private / UI API (<=> views.py) @Lee-W

• assets @Lee-W feat(AIP-84): Add Auth for ui assets #47485
• backfills (depends on: backfills) @Lee-W feat(AIP-84): add auth to /ui/backfills #47657
• config @Lee-W feat(AIP-84): Add auth for ui config #47487
• dags @Lee-W feat(AIP-84): add auth to /ui/recent_dag_runs #47623
• dependencies @Lee-W feat(AIP-84): add auth to /ui/dependencies #47625
• structure @Lee-W feat(AIP-84): add auth to /ui/structure/structure_data #47631
• dashboard @Lee-W feat(AIP-84): add auth to /ui/historical_metrics_data #47650
• grid @Lee-W feat(AIP-84): add auth to /ui/grid #47651

Execution API

• assets @Lee-W feat(AIP-84): add auth to /execution/assets endpoints #47658
• connections @Lee-W feat(AIP-84): add auth to /execution/connections #47661
• health @Lee-W (probably don't need it?)
• task_instances @Lee-W feat(AIP-84): add auth to /execution/task-instances #47664
• variables
• xcoms

misc

• AIP-84 | Fix test_python_client after Adding Permission to API #47388

II Additional Tasks

• AIP-84 Support Auth Backends for API authentication #47055 @vincbeck

Default to SimpleAuthManager for development time of airflow 3 as long as FABAuthManager does not support JWT auth, this is the only way we can progress on this while #44849 is not completed

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[jason810496]

Hi @pierrejeambrun, could you please assign this ticket to me? Thanks!

[pierrejeambrun]

Hello @jason810496,

There has been some update on this one, it is not well scoped and depends on the work that Vincent is doing on the FABAuthManager and JWT backend. I do not recommend to take this one at this point. Removing the good first issue tag

[jason810496]

Gotcha, I will take other tickets. Thanks !

[pierrejeambrun]

Related PR #42634

[pierrejeambrun]

cc: @vincbeck

[pierrejeambrun]

This is now unblocked since #44884 has been implemented. We can start implementing the permissions for different endpoints.

[pierrejeambrun]

@jason810496 if you still have some interest for this let me know so I can assign you :)

[jason810496]

@jason810496 if you still have some interest for this let me know so I can assign you :)

Sure! I can work on this issue, I will look into the context tomorrow then ask if I not sure about the details.

[pierrejeambrun]

Great. Don't hesitate to do 1 PR per 'resource'. (for instance for DagRun endpoints, for taskinstance endpoints, connection enpoints etc...).

I think @rawwar will provide some help too so you can split the work easily. :)

[jason810496]

I will start with the Dag endpoint first and provide an update here before working on the other endpoints. I’ll also list all the necessary endpoints later.

Before proceeding further, I’d like to double-check something. After reviewing #44884, it seems that we might need a general middleware or a Depends function (in FastAPI) as an authentication layer to read the Bearer token from authorization request headers. This authentication layer should leverage the AuthManager initialized in the FastAPI app (accessible via request.app.state.auth_manager).
Somehow like action_logging in airflow.api_fastapi.logging.decorators

Update:
Continue finish airflow.api_fastapi.core_api.security and take airflow.api_connexion.security as reference.

[jason810496]

I’ll also list all the necessary endpoints later.

Endpoints:

• configuration
• connection
• dag
• asset
• pool
• variable
• view
• custom_view

[pierrejeambrun]

Thanks @jason810496 meta issue updated

[rawwar]

@jason810496 , I'll pick asset, pool and variable. I'll work on them this week.

[jason810496]

@jason810496 , I'll pick asset, pool and variable. I'll work on them this week.

Then I'll take configuration and connection also, if I am fast enough I can take the rest also ! ( Not sure I have enough time currently )

• dag AIP-84 | Add Auth for Dags #47062
• configuration AIP-84 | Add Auth for Config #47208
• connection variable AIP-84 | Add Auth for Variable #47211

[Lee-W]

I just updated the current status in the descpriont

[Lee-W]

@jason810496 @rawwar @vatsrahul1001 , are there any other known dependencies in public auths? It would be nice if we could list them so that we know what to do first 🙂 I'll take a look at the non-assigned ones later. as for the format, you can take a look at the UI part.

[rawwar]

@jason810496 @rawwar @vatsrahul1001 , are there any other known dependencies in public auths? It would be nice if we could list them so that we know what to do first 🙂 I'll take a look at the non-assigned ones later. as for the format, you can take a look at the UI part.

except asset_alias,dag_report my rest of the PR's depend on dags

[jason810496]

@jason810496 @rawwar @vatsrahul1001 , are there any other known dependencies in public auths? It would be nice if we could list them so that we know what to do first 🙂 I'll take a look at the non-assigned ones later. as for the format, you can take a look at the UI part.

Still working on fixing the Kubernetes tests. If not resolved today, I’ll continue over the weekend. I have idea of how to fix it but need to test it locally. ( Adapt #47460 idea to k8s system test )

[rawwar]

@pierrejeambrun, dag report is a new endpoint in AF3. What permissions should a user need to access it?

Should this be a separate permission or should it be a sub-entity of dags?

[vatsrahul1001]

@pierrejeambrun, I see that the dag_tags and tasks endpoints are unassigned. I can pick them up.

[Lee-W]

@vatsrahul1001 just assigned and updated the list. Thanks!

[jason810496]

The DAG-related blocker #47433 has just been resolved!
Entities that depend on DAGs can now rebase to the latest main branch and leverage ReadableDagsFilterDep and EditableDagsFilterDep from the airflow.api_fastapi.core_api.security module to obtain the get_permitted_dag_ids filter.

[rawwar]

The DAG-related blocker #47433 has just been resolved!

Great job @jason810496 , @Lee-W

[Lee-W]

@pierrejeambrun do we need auth for execution/health?

[rawwar]

@pierrejeambrun do we need auth for execution/health?

I don't think we should have auth for health endpoints, as monitoring tools can use them. Maybe we should have been calling it /monitor to be consistent with others?

[Lee-W]

Yep, I don't think we should have auth either. as for whether to call it /monitor, I think /health is quite commonly used and probably shouldn't be changed?