Patch/abhi tablename : Adding import query type property and modify tableName property for redshift and postgres plugin #607

AbhishekKumar9984 · 2025-06-13T11:38:27Z

Adding import query type property and modify tableName property for redshift and postgres plugin

…t in redshift and Postgres plugin.

vikasrathee-cs · 2025-06-18T07:15:22Z

amazon-redshift-plugin/src/main/java/io/cdap/plugin/amazon/redshift/RedshiftSchemaReader.java

+  }
+
+  /**
+   * Helper method for type mapping when fetching schema by table name.


this is not proper javadoc format, use that one

changed and used javadoc format.

vikasrathee-cs · 2025-06-18T07:18:18Z

amazon-redshift-plugin/src/main/java/io/cdap/plugin/amazon/redshift/RedshiftSchemaReader.java

+  /**
+   * Override: Fetches schema fields for a specific table using database metadata.
+   */
+  @Override


It is there in parent class, you can use that one, instead you can call getSchema method of this class inside

vikasrathee-cs · 2025-06-18T07:19:54Z

amazon-redshift-plugin/src/main/java/io/cdap/plugin/amazon/redshift/RedshiftSource.java

+  public void configurePipeline(PipelineConfigurer pipelineConfigurer) {
+    FailureCollector collector = pipelineConfigurer.getStageConfigurer().getFailureCollector();
+    if (!sourceConfig.containsMacro("tableName") && !sourceConfig.containsMacro("importQuery")) {
+      if ((sourceConfig.getTableName() == null || sourceConfig.getTableName().isEmpty())


Use Strings method for null empty check

vikasrathee-cs · 2025-06-18T07:20:15Z

amazon-redshift-plugin/src/main/java/io/cdap/plugin/amazon/redshift/RedshiftSource.java

+      if ((sourceConfig.getTableName() == null || sourceConfig.getTableName().isEmpty())
+              && (sourceConfig.getImportQuery() == null || sourceConfig.getImportQuery().isEmpty())) {
+        collector.addFailure(
+                "Either 'tableName' or 'importQuery' must be specified for the PostgreSQL source.",


Remove DB name from here

vikasrathee-cs · 2025-06-18T07:23:17Z

amazon-redshift-plugin/src/main/java/io/cdap/plugin/amazon/redshift/RedshiftSource.java

@@ -88,6 +107,11 @@ protected LineageRecorder getLineageRecorder(BatchSourceContext context) {
    return new LineageRecorder(context, assetBuilder.build());
  }

+  public DatabaseMetaData getDatabaseMetadata(Connection connection) throws SQLException {
+    return (DatabaseMetaData) connection.getMetaData().getColumns(null,
+            null, redshiftSourceConfig.getTableName(), null);


schema is not handled here, if customer is passing "schema.tablename", it will not work

for now this method is not used so no need to handle we can remove this method .

vikasrathee-cs · 2025-06-18T07:24:45Z

amazon-redshift-plugin/widgets/Redshift-batchsource.json

+    {
+      "name": "ImportQuery",
+      "condition": {
+        "expression": "importQueryType == 'importQuery'"


use condition importQueryType != 'tableName', it will be backward compatible then

vikasrathee-cs · 2025-06-18T07:25:36Z

cloudsql-mysql-plugin/widgets/CloudSQLMySQL-batchsource.json

@@ -251,6 +275,30 @@
          "name": "port"
        }
      ]
+    },


no need to add filters as it is hidden

keeping it so that if further we have to make changes for this we can just changed the hidden property.
but if not required , will removed..

vikasrathee-cs · 2025-06-18T07:27:25Z

database-commons/src/main/java/io/cdap/plugin/db/CommonSchemaReader.java

+  /**
+   * Resolves the actual table name from the database in a case-insensitive way.
+   */
+  private String resolveTableName(DatabaseMetaData dbMetaData, String schema, String userTableName)


remove this method, as it can create issues in case sensitive DBs

removed and back to previous method .

vikasrathee-cs · 2025-06-18T07:31:10Z

database-commons/src/main/java/io/cdap/plugin/db/source/AbstractDBSource.java

@@ -163,7 +164,14 @@ public Schema getSchema() throws SQLException {
    try (Connection connection = getConnection()) {
      executeInitQueries(connection, sourceConfig.getInitQueries());
      String query = sourceConfig.getImportQuery();
-      return loadSchemaFromDB(connection, query);
+      if (query != null && !query.isEmpty()) {


Add Strings class method for null check

vikasrathee-cs · 2025-06-18T07:33:45Z

database-commons/src/main/java/io/cdap/plugin/db/source/AbstractDBSource.java

@@ -174,9 +182,14 @@ private Schema loadSchemaFromDB(Connection connection, String query) throws SQLE
      query = removeConditionsClause(query);
    }
    ResultSet resultSet = statement.executeQuery(query);
+    DatabaseMetaData databaseMetaData = getDatabaseMetadata(connection);


you are not using it, remove it

vikasrathee-cs · 2025-06-18T07:35:25Z

database-commons/src/main/java/io/cdap/plugin/db/source/AbstractDBSource.java

+      if (!Strings.isNullOrEmpty(importQuery)) {
+        return loadSchemaFromDB(connection, importQuery);
+      } else {
+        String query = String.format("SELECT * FROM %s LIMIT 1", tableName);


we have fetch schema from database metadata in case of tableName, this should not behave like this

vikasrathee-cs · 2025-06-18T07:37:49Z

postgresql-plugin/src/main/java/io/cdap/plugin/postgres/PostgresSchemaReader.java

@@ -73,7 +72,7 @@ public Schema getSchema(ResultSetMetaData metadata, int index) throws SQLExcepti
        return Schema.of(Schema.Type.STRING);
      }
    }
-    
+


vikasrathee-cs · 2025-06-18T07:38:29Z

postgresql-plugin/src/main/java/io/cdap/plugin/postgres/PostgresSource.java

+      if ((sourceConfig.getTableName() == null || sourceConfig.getTableName().isEmpty())
+              && (sourceConfig.getImportQuery() == null || sourceConfig.getImportQuery().isEmpty())) {
+        collector.addFailure(
+                "Either 'tableName' or 'importQuery' must be specified for the PostgreSQL source.",


remove db name

psainics and others added 5 commits June 11, 2025 15:01

add tablename code [wip]

d158bd6

FIXUP

205ffe5

fix missing calls

f148b99

adding import query type property to db plugin.which will only reflec…

b956661

…t in redshift and Postgres plugin.

resloved null pointer error during running the pipline.

53874e9

vikasrathee-cs reviewed Jun 18, 2025

View reviewed changes

resolved Pr comments.

f4d0f7c

Patch/abhi tablename : Adding import query type property and modify tableName property for redshift and postgres plugin #607

Are you sure you want to change the base?

Patch/abhi tablename : Adding import query type property and modify tableName property for redshift and postgres plugin #607

Uh oh!

Conversation

AbhishekKumar9984 commented Jun 13, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vikasrathee-cs Jun 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

vikasrathee-cs Jun 18, 2025 •

edited

Loading