Display Multiple Configuration Files in the Airflow-Docker dropdown

[huntcsg]

Acceptance Critera

1. The airflow-docker dropdown displaying the config.json file would include other configuration files found

Questions:

1. How will the plugin know what files to include?
2. How should the files be partitioned
3. How should it handle json decode errors

[huntcsg]

@DanCardin @gordonfierce thoughts?

[DanCardin]

Like the idea, but the existing get_config usage is probably more specific to the way we currently have conceived config to be done to really be ergonomic for more than just this one config file.

If you want to facilitate more generic config file usage, I'd suggest it simultaneously make that function more generic, or else rename it and compose that particular config getting from something more generic.

Couple questions:

• Do you have a specific config file usecase in mind that you're trying to support displaying? or is this theoretical.
• right now it's hardcoded to json highlighting, if this is theoretical for arbitrary user-specified config, you'll probably want to introspect the type of file instead of just asserting that json is the one true config file type.

[huntcsg]

Re get_config I think it just needs to be relative to the dags folder by default. That would be enough to make it more general purpose.

The use case is for a company "shared" airflow that might have multiple departments that want to use it.

For example:

.
└── dags
    ├── config.json
    ├── team-1
    │   └── config.json
    ├── team-2
    │   └── config.json
    └── team-3
        └── config.json

There might be a global config.json file (e.g. with globally relevant things), but then each department would have it's own config.

That way, team-1 won't risk breaking team-2 when they update their config.

I don't care about json vs. other file formats/hightlighting (I think that would be orthogonal to this change).

So options are:

In configuration:

# airflow.cfg
[airflowdocker]
static_config_files=
    config.json
    team-1/config.json
    team-2/config.json
    team-3/config.json

In an static airflow_variable

# src/airflow_docker/views/config.py
static_config_files = Variable.get(airflow_docker.config.STATIC_CONFIG_FILES_VAR)

In an configurable airflow_variable

# src/airflow_docker/views/config.py
static_config_files = Variable.get(
    airflow_docker.get_default(
        'static_config_files_var', 
        airflow_docker.config.STATIC_CONFIG_FILES_VAR
    )
)

os.walk

# src/airflow_docker/views/config.py
def gather_static_config(directory=None):  # defaults to dags folder
    static_config_files = []
    for stuff in os.walk(directory):
        ....
    return static_config_files

Once we decide on this, then we can decide on the view page design (just a list of highlighted files, or maybe drop downs with options).

[DanCardin]

Your example was helpful. If we're still talking about config.json as we've generally conceived it up til now, it almost seems like we should be automatically adjusting the lookup location based on the location of the caller's dag file. e.g. team-3/dag.py would pull from team-3/config.py and traverse upward until it finds one or hits the base dag folder. At which point I'd definitely want os.walk.

Otherwise the static config option would be my fallback.

However if you were talking about more general and arbitrary config/data files. then i think we'd need to make changes to get_config

• get_config's signature is get_config(path=None, file='config.json'). If it was for more general/arbitrary config, it should probably be get_config(file='config.json', path=None) so you only have to specify the specific file in the general case.
• get_config inserts the "airflow-docker" key into the config automatically.
• syntax highlighting should be generalized and not assume json