Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

source 读取数据返回乱序，未按照StarRocksSourceOptions设置的scan.columns顺序返回，TableSchema设置了也没用 #201

Open
hangupwd opened this issue Mar 14, 2023 · 0 comments · May be fixed by #317
Open

source 读取数据返回乱序，未按照StarRocksSourceOptions设置的scan.columns顺序返回，TableSchema设置了也没用 #201

hangupwd opened this issue Mar 14, 2023 · 0 comments · May be fixed by #317

Comments

@hangupwd
Copy link

我在使用connector source读取starrocks数据时，返回的数据乱序

版本信息
starrocks-v2.3 flink-1.14.2 connector-1.2.5_flink-1.14_2.11

传入的StarRocksSourceOptions属性

scan.columns=c_sbm_a457bc598b214148a9e74d212a8b3847,uuid_318bec51528a459483bdbb19b593126b,c_zzyj_f6fa7c9a546140189e0c866cfc6363b9,c_psjssj_f880d4d094014fc9b4319416dab1a38c,c_qdzt_c936ceee39234cd292d763452aa82dd5,c_dwbm_d59ba3be586d40259524320c2aecee87,c_psjd_df485556c82e4298bc6713270e46e933,c_psrwlx_f5217e02a20d4a00a0381b60f9481ec0,c_psrwmc_e64b8bde2af4473da4c85c1601f988c7,c_pskssj_fddc0c49eb8246f49a588e1dd2452e0a,c_pszt_d89e639fb64f488fa1393c1210a57268,c_bhsz_b702504e391d4e93b8373ee35e0ee867,c_gh_f24c13864ce04d7fa2567e85f00cd9a1,c_ljscbz_b6162004a7714eb2a1162dadc4bade8e,c_wjghrwid_fbe9b92fd2db4730b1f95f5e2e3bdb37,c_psfs_c8c49283e11c48ef8616ef1ae8c866d1,c_zdmj_e046f49aa5da404981cc0b3206fd9257,c_dlxlzd_b0f17329f82548e5b06039588132f17e,c_cjrmc_c1ca861d89f949c3a8f0e73618bdf8e2,c_scbs_cd3533fd72b7435a9b13fd1da0848a67,c_sfbc_aa9ce0c4933348bfa0e2cd622b59c7db,c_xldjh_be8a75e5ac794df7b9f894ab8e2bad78,c_jkxlzd_edd46a8a6ab0402ca7d0430806730176,c_xmid_af419689630f44cd8571452321bdbe7b,c_famc_c609e2f7d6d0417d8ad1308fc32a6ea8,c_bdrl_ef7b6694534449f2be3f1ff53ca4e335,c_xgsj_d40fd2fa45ce4049b36ce96d1d98cb9c,c_xgrmc_eeb2d63900a94a8d91cfdd06da724827,c_dydj_ea55ae82d2e14c46b1ba6a77502fa36a

TableSchema为

root
 |-- c_sbm_a457bc598b214148a9e74d212a8b3847: STRING
 |-- uuid_318bec51528a459483bdbb19b593126b: STRING
 |-- c_zzyj_f6fa7c9a546140189e0c866cfc6363b9: STRING
 |-- c_psjssj_f880d4d094014fc9b4319416dab1a38c: STRING
 |-- c_qdzt_c936ceee39234cd292d763452aa82dd5: STRING
 |-- c_dwbm_d59ba3be586d40259524320c2aecee87: STRING
 |-- c_psjd_df485556c82e4298bc6713270e46e933: STRING
 |-- c_psrwlx_f5217e02a20d4a00a0381b60f9481ec0: STRING
 |-- c_psrwmc_e64b8bde2af4473da4c85c1601f988c7: STRING
 |-- c_pskssj_fddc0c49eb8246f49a588e1dd2452e0a: STRING
 |-- c_pszt_d89e639fb64f488fa1393c1210a57268: STRING
 |-- c_bhsz_b702504e391d4e93b8373ee35e0ee867: STRING
 |-- c_gh_f24c13864ce04d7fa2567e85f00cd9a1: STRING
 |-- c_ljscbz_b6162004a7714eb2a1162dadc4bade8e: STRING
 |-- c_wjghrwid_fbe9b92fd2db4730b1f95f5e2e3bdb37: STRING
 |-- c_psfs_c8c49283e11c48ef8616ef1ae8c866d1: STRING
 |-- c_zdmj_e046f49aa5da404981cc0b3206fd9257: STRING
 |-- c_dlxlzd_b0f17329f82548e5b06039588132f17e: STRING
 |-- c_cjrmc_c1ca861d89f949c3a8f0e73618bdf8e2: STRING
 |-- c_scbs_cd3533fd72b7435a9b13fd1da0848a67: STRING
 |-- c_sfbc_aa9ce0c4933348bfa0e2cd622b59c7db: STRING
 |-- c_xldjh_be8a75e5ac794df7b9f894ab8e2bad78: STRING
 |-- c_jkxlzd_edd46a8a6ab0402ca7d0430806730176: STRING
 |-- c_xmid_af419689630f44cd8571452321bdbe7b: STRING
 |-- c_famc_c609e2f7d6d0417d8ad1308fc32a6ea8: STRING
 |-- c_bdrl_ef7b6694534449f2be3f1ff53ca4e335: STRING
 |-- c_xgsj_d40fd2fa45ce4049b36ce96d1d98cb9c: STRING
 |-- c_xgrmc_eeb2d63900a94a8d91cfdd06da724827: STRING
 |-- c_dydj_ea55ae82d2e14c46b1ba6a77502fa36a: STRING

建表语句

CREATE TABLE `collective_20144` (
  `c_sbm_a457bc598b214148a9e74d212a8b3847` varchar(65533) NULL COMMENT "",
  `uuid_318bec51528a459483bdbb19b593126b` varchar(65533) NULL COMMENT "",
  `c_zzyj_f6fa7c9a546140189e0c866cfc6363b9` varchar(65533) NULL COMMENT "",
  `c_psjssj_f880d4d094014fc9b4319416dab1a38c` varchar(65533) NULL COMMENT "",
  `c_qdzt_c936ceee39234cd292d763452aa82dd5` varchar(65533) NULL COMMENT "",
  `c_dwbm_d59ba3be586d40259524320c2aecee87` varchar(65533) NULL COMMENT "",
  `c_sjly_dcb220e987b54c01a43956629a67fe23` varchar(65533) NULL COMMENT "",
  `c_psjd_df485556c82e4298bc6713270e46e933` varchar(65533) NULL COMMENT "",
  `c_psrwlx_f5217e02a20d4a00a0381b60f9481ec0` varchar(65533) NULL COMMENT "",
  `c_psrwmc_e64b8bde2af4473da4c85c1601f988c7` varchar(65533) NULL COMMENT "",
  `c_cjsj_e6527c6fc2f4498389d25192bea190d8` varchar(65533) NULL COMMENT "",
  `c_pskssj_fddc0c49eb8246f49a588e1dd2452e0a` varchar(65533) NULL COMMENT "",
  `c_cjrid_f421b54a6fe543c4a3e53061c78717a5` varchar(65533) NULL COMMENT "",
  `c_lgsbb_d73cc28bc78f4eb591b24c2087dc1dd1` varchar(65533) NULL COMMENT "",
  `c_pszt_d89e639fb64f488fa1393c1210a57268` varchar(65533) NULL COMMENT "",
  `c_xgsj_d201e5e614c2468fa6d5c3cf7a0ddd30` varchar(65533) NULL COMMENT "",
  `c_bhsz_b702504e391d4e93b8373ee35e0ee867` varchar(65533) NULL COMMENT "",
  `c_gh_f24c13864ce04d7fa2567e85f00cd9a1` varchar(65533) NULL COMMENT "",
  `c_jbm_e1394ec9dc044c2f86ac912f8ef7c979` varchar(65533) NULL COMMENT "",
  `c_ljscbz_b6162004a7714eb2a1162dadc4bade8e` varchar(65533) NULL COMMENT "",
  `c_zj_e9dfa249e00642eb9f37d7bdba3806ef` varchar(65533) NULL COMMENT "",
  `c_wjghrwid_fbe9b92fd2db4730b1f95f5e2e3bdb37` varchar(65533) NULL COMMENT "",
  `c_xgrid_d304c4f9323e436a8fd69a1aa6fcaad6` varchar(65533) NULL COMMENT "",
  `c_psfs_c8c49283e11c48ef8616ef1ae8c866d1` varchar(65533) NULL COMMENT "",
  `c_cjsj_f326995b57c14afab0d82a5ab90e612e` varchar(65533) NULL COMMENT "",
  `c_zdmj_e046f49aa5da404981cc0b3206fd9257` varchar(65533) NULL COMMENT "",
  `c_jbm_db4b4b2caf0e4d95b2481e4b7fd74106` varchar(65533) NULL COMMENT "",
  `c_dlxlzd_b0f17329f82548e5b06039588132f17e` varchar(65533) NULL COMMENT "",
  `c_cjrid_e0e7877a88654e9ebe7a2fb232f8eb29` varchar(65533) NULL COMMENT "",
  `c_cjrmc_c1ca861d89f949c3a8f0e73618bdf8e2` varchar(65533) NULL COMMENT "",
  `c_sjly_f68669e0c685486cb9f9b921d3b6314d` varchar(65533) NULL COMMENT "",
  `c_scbs_cd3533fd72b7435a9b13fd1da0848a67` varchar(65533) NULL COMMENT "",
  `c_zj_e0a9f0da68584411a5242b9c4765a1d7` varchar(65533) NULL COMMENT "",
  `c_sfbc_aa9ce0c4933348bfa0e2cd622b59c7db` varchar(65533) NULL COMMENT "",
  `c_xldjh_be8a75e5ac794df7b9f894ab8e2bad78` varchar(65533) NULL COMMENT "",
  `c_lgsbb_f82463c04b4246c1b3eda36e8c2d278d` varchar(65533) NULL COMMENT "",
  `c_jkxlzd_edd46a8a6ab0402ca7d0430806730176` varchar(65533) NULL COMMENT "",
  `c_rwid_be95655dcae4456fbfce48eaf33ddcf6` varchar(65533) NULL COMMENT "",
  `c_xmid_af419689630f44cd8571452321bdbe7b` varchar(65533) NULL COMMENT "",
  `c_sbm_a005d17cfbc54b5e84e4b7df55496ba8` varchar(65533) NULL COMMENT "",
  `c_famc_c609e2f7d6d0417d8ad1308fc32a6ea8` varchar(65533) NULL COMMENT "",
  `c_bdrl_ef7b6694534449f2be3f1ff53ca4e335` varchar(65533) NULL COMMENT "",
  `c_xgsj_d40fd2fa45ce4049b36ce96d1d98cb9c` varchar(65533) NULL COMMENT "",
  `c_xgrid_fc36bb876de54b85a264ea01d1bc8a50` varchar(65533) NULL COMMENT "",
  `c_xgrmc_eeb2d63900a94a8d91cfdd06da724827` varchar(65533) NULL COMMENT "",
  `c_dydj_ea55ae82d2e14c46b1ba6a77502fa36a` varchar(65533) NULL COMMENT ""
) ENGINE=OLAP 
DUPLICATE KEY(`c_sbm_a457bc598b214148a9e74d212a8b3847`)
COMMENT "OLAP"
DISTRIBUTED BY HASH(`c_sbm_a457bc598b214148a9e74d212a8b3847`, `uuid_318bec51528a459483bdbb19b593126b`) BUCKETS 10 
PROPERTIES (
"replication_num" = "1",
"in_memory" = "false",
"storage_format" = "DEFAULT"
);

最后返回的数据未按照传入的scan.columns 和table scheam顺序返回，这个问题我需要怎么调整
@Jin-H Jin-H linked a pull request Dec 13, 2023 that will close this issue
Open
11 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[Enhancement]: support reading some columns Jin-H/starrocks-connector-for-apache-flink
1 participant
@hangupwd