Add sample DAG, test ExasolToS3Operator, and update documentation (#22632) #47478
Conversation
boring-cyborg
bot
added
area:providers
provider:amazon
jieyao-MilestoneHub
force-pushed
the
feature-exasol-sample-dag
branch
from
afc543f to
a6e6146
Compare
|
Hey @vincbeck, I’d love to get your thoughts on this. I’ve tested the connection setup, and this PR mainly focuses on documentation. Adding a test file could help validate the setup better. While working on example_exasol_to_s3.py, I noticed exasol_to_s3 could be refactored for clearer separation of concerns, making testing easier. Thinking of including this in the PR—does that sound good to you? I’ll adjust the docs as needed once we decide. Looking forward to your thoughts! 🚀 |
That sounds perfect to me :) |
|
hanks for your feedback, @vincbeck! 😊 Glad to hear you’re on board with the changes. I’ll add the test file and refactor exasol_to_s3 for better separation of concerns and easier testing. I’ll also update the docs accordingly—let me know if you have any suggestions! 🚀 |
This commit adds a detailed Exasol connection setup guide for Apache Airflow, outlining the configuration steps, prerequisites, and troubleshooting tips. The document is structured in reStructuredText (.rst) format for better readability and integration with documentation tools. Requires review to ensure clarity and completeness.
- Optimized `ExasolToS3Operator` to separate Exasol and S3 operations. - Improved unit tests for `ExasolExportOperator`, `S3UploadOperator`, and `ExasolToS3Operator`. - Added `example_exasol_to_s3` DAG to validate Exasol connection. - Created `exasol_connection.rst` with setup instructions, avoiding credit card requirement.
jieyao-MilestoneHub
force-pushed
the
feature-exasol-sample-dag
branch
from
a6e6146 to
b75b646
Compare
|
Hey @vincbeck, thanks for your input! Just wanted to confirm—if this kind of introduction (how to test the Exasol connection without needing a credit card) is what you were expecting. If so, do you have any thoughts on where it would be best placed? Let me know what you think! 🚀 |
| tags=["exasol", "test"], | ||
| ) as dag: | ||
|
|
||
| test_exasol_connection = ExasolExportOperator( |
There was a problem hiding this comment.
test_exasol_connection does not describe well what this operator does
| This DAG uses the ExasolToS3Operator to execute a simple SQL query on Exasol | ||
| and simulate an S3 upload. The main purpose is to verify that the Exasol connection | ||
| settings (exasol_default) are working correctly. S3 connection is simulated via dummy | ||
| parameters (aws_default, dummy bucket and key). |
There was a problem hiding this comment.
An example DAG is a DAG that should work out of the box (if you have valid credentials of course). I would rephrase this comment and just mention what this DAG is doing. This DAG should work for someone who has exasol and AWS permissions.
| @@ -0,0 +1,152 @@ | |||
| .. Licensed to the Apache Software Foundation (ASF) under one | |||
There was a problem hiding this comment.
There should no doc in example_dags folder. This kind of documentation should go in providers/exasol/docs
| @@ -0,0 +1,174 @@ | |||
| .. _exasol_airflow_setup: | |||
There was a problem hiding this comment.
Please do not duplicate documentation. Plus, this documentation is related to Exasol, it has nothing to do with Amazon. As I mentioned in my previous comment, this should go in Exasol documentation
| @@ -31,29 +32,165 @@ | |||
| from airflow.utils.context import Context | |||
|
|
|||
|
|
|||
| class ExasolExportOperator(BaseOperator): | |||
There was a problem hiding this comment.
We already have ExasolToS3Operator, are you sure we need this operator as well?
| raise | ||
|
|
||
|
|
||
| class S3UploadOperator(BaseOperator): |
There was a problem hiding this comment.
S3CreateObjectOperator already exists
| def execute(self, context: Context): | ||
| exasol_hook = ExasolHook(exasol_conn_id=self.exasol_conn_id) | ||
| s3_hook = S3Hook(aws_conn_id=self.aws_conn_id) | ||
| def execute(self, context: Context) -> str: |
There was a problem hiding this comment.
Why modifying the implementation? Did you notice anything wrong with the previous implementation?
Summary
Adds documentation for ExasolToS3Operator to enhance clarity and usability.
Changes Introduced
Related Issues
related: #22632