Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Do not review] Decouple import batch size from resources in blob file #4788

Open
wants to merge 122 commits into
base: main
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
122 commits
Select commit Hold shift + click to select a range
47c370f
Raw resources in the Lake and Resource Id integer map
SergeyGaluzo Sep 27, 2024
5aca029
perf tweeks
SergeyGaluzo Oct 2, 2024
de5a156
Merge history separation with separated raw resources (#4685)
SergeyGaluzo Oct 17, 2024
e37ffd4
Merge resource id map
SergeyGaluzo Oct 17, 2024
5d8d03a
Fix after merge
SergeyGaluzo Oct 18, 2024
cc94008
Fix after merge
SergeyGaluzo Oct 18, 2024
d6ce380
adjustments
SergeyGaluzo Oct 18, 2024
2950bcf
tests
SergeyGaluzo Oct 18, 2024
8e74731
Merge from main
SergeyGaluzo Oct 18, 2024
c3ccf69
Merge fixes
SergeyGaluzo Oct 18, 2024
b0ed1eb
Ignore NULL reference resource tyoe
SergeyGaluzo Oct 18, 2024
51a942d
test fix
SergeyGaluzo Oct 18, 2024
38e869b
default number of columns = 13
SergeyGaluzo Oct 19, 2024
bb535af
commented 84 diff
SergeyGaluzo Oct 19, 2024
7060321
set end date on enqueue
SergeyGaluzo Oct 22, 2024
427cdfa
rename secondary to adls
SergeyGaluzo Oct 22, 2024
02a545f
defrag formatting
SergeyGaluzo Oct 27, 2024
fbbaf20
set timeout to 0 for getting frag
SergeyGaluzo Oct 28, 2024
3f5dcd9
isnull on input param
SergeyGaluzo Oct 28, 2024
e6e3fe9
Merge branch 'main' into users/sergal/rawmaplakeandhistoryandresource…
SergeyGaluzo Oct 29, 2024
5569ea1
after merge from main
SergeyGaluzo Oct 30, 2024
2648ecd
Merge branch 'users/sergal/rawmaplakeandhistoryandresourceidmap' into…
SergeyGaluzo Oct 31, 2024
037061d
Reference separation
SergeyGaluzo Oct 31, 2024
99c6620
file id
SergeyGaluzo Nov 1, 2024
0b226bf
small things
SergeyGaluzo Nov 1, 2024
7f286f9
Do not write raw resource
SergeyGaluzo Nov 7, 2024
873583e
Added missing file id in history insert
SergeyGaluzo Nov 18, 2024
ce28891
hard delete tests
SergeyGaluzo Nov 19, 2024
848195c
change capture
SergeyGaluzo Nov 19, 2024
61b1c06
FK
SergeyGaluzo Nov 19, 2024
65b21dc
Comments
SergeyGaluzo Nov 19, 2024
2b28ed4
resource list lake and 768
SergeyGaluzo Nov 20, 2024
6c46e43
no lake on list
SergeyGaluzo Nov 20, 2024
2d8a618
no extra lines
SergeyGaluzo Nov 20, 2024
38e6c45
small cleanup
SergeyGaluzo Nov 21, 2024
90132fb
84 in sync
SergeyGaluzo Nov 21, 2024
77d4950
diff
SergeyGaluzo Nov 22, 2024
7f17c96
GetResources back
SergeyGaluzo Nov 25, 2024
8b660a0
Fix diff
SergeyGaluzo Dec 4, 2024
9992090
Use ResourceListLake correctly
SergeyGaluzo Dec 4, 2024
daa5c53
Added @ResourceListLake to update resource search params
SergeyGaluzo Dec 5, 2024
cfa461c
tweeking
SergeyGaluzo Dec 11, 2024
5373db8
fix diff
SergeyGaluzo Dec 11, 2024
ad7576a
resourceids
SergeyGaluzo Dec 11, 2024
8867758
filter out null ref resourec type
SergeyGaluzo Dec 12, 2024
e56f599
handle search params updates in parallel with data move
SergeyGaluzo Dec 12, 2024
c59a7ce
Added id join in include queries
SergeyGaluzo Dec 12, 2024
83c812a
84 diff fix
SergeyGaluzo Dec 13, 2024
3019c0f
no CR on IsResourceRef
SergeyGaluzo Dec 14, 2024
46892b1
Removed not needed alias
SergeyGaluzo Dec 14, 2024
4d65b4d
no alias
SergeyGaluzo Dec 15, 2024
73d02be
fixed RespurceIns trigger
SergeyGaluzo Dec 16, 2024
4bbb393
Add ResourceTbl history update into temporary ResourceUpd trigger.
SergeyGaluzo Dec 17, 2024
a79c4c3
Final hard delete logic
SergeyGaluzo Dec 17, 2024
730260b
final delete invisible history
SergeyGaluzo Dec 17, 2024
9197191
DataLakeStore flag
SergeyGaluzo Dec 17, 2024
4183235
names
SergeyGaluzo Dec 18, 2024
0347536
names
SergeyGaluzo Dec 18, 2024
ba9415b
Current resource view
SergeyGaluzo Dec 18, 2024
88f831b
nicer
SergeyGaluzo Dec 18, 2024
039a148
spelling
SergeyGaluzo Dec 18, 2024
37b83b0
84 -> 85
SergeyGaluzo Dec 18, 2024
b975e1a
85
SergeyGaluzo Dec 18, 2024
a7f7ea3
merge from CurrentResource branch
SergeyGaluzo Dec 18, 2024
2f34299
85
SergeyGaluzo Dec 18, 2024
0389733
Changed order of columns for clustered key on resource tables to supp…
SergeyGaluzo Dec 20, 2024
826d08b
DELETE inside loop
SergeyGaluzo Dec 20, 2024
bd453b2
old views always and proc to move
SergeyGaluzo Dec 20, 2024
cb5c967
Merge from main after 84
SergeyGaluzo Dec 20, 2024
e6c5268
optimized stored procedure
SergeyGaluzo Dec 21, 2024
f5764a4
export
SergeyGaluzo Dec 22, 2024
cde0918
Simple sort value logic
SergeyGaluzo Dec 23, 2024
5840834
plus
SergeyGaluzo Dec 23, 2024
c9450ff
Merge from filed count branch
SergeyGaluzo Dec 23, 2024
22174f7
11->12 plus comments
SergeyGaluzo Dec 23, 2024
ff69585
null check
SergeyGaluzo Dec 24, 2024
a8b7de2
adls message
SergeyGaluzo Dec 24, 2024
53d5930
message
SergeyGaluzo Dec 24, 2024
ccdc3e5
log
SergeyGaluzo Dec 24, 2024
8b35734
2 columns back
SergeyGaluzo Dec 25, 2024
b17416b
exporter with file/blob API
SergeyGaluzo Dec 26, 2024
567b839
group read
SergeyGaluzo Dec 26, 2024
d8fc0d7
Accomodate CurrentResource view
SergeyGaluzo Dec 27, 2024
0f00915
Switched search to set file reads
SergeyGaluzo Dec 27, 2024
1a8d6d4
moved large logic outside of reader
SergeyGaluzo Dec 27, 2024
ba9bfc0
refactored to use set reads
SergeyGaluzo Dec 27, 2024
0b633b3
alarms
SergeyGaluzo Dec 27, 2024
b751b14
Corrected != null on SqlBytes
SergeyGaluzo Dec 27, 2024
24bcfaf
removed redundant select
SergeyGaluzo Dec 27, 2024
7d5368e
check for adls client is null after returning empty
SergeyGaluzo Dec 28, 2024
3db2a4d
name typo
SergeyGaluzo Dec 28, 2024
e5f833f
Removed file system
SergeyGaluzo Dec 28, 2024
747e1d9
Hard delete check for invisible resource
SergeyGaluzo Dec 30, 2024
dad9d90
hard delete fix
SergeyGaluzo Jan 1, 2025
890d5ca
move alter inside try/catch
SergeyGaluzo Jan 2, 2025
d00ee7d
Added ResourceTbl to update trigger
SergeyGaluzo Jan 3, 2025
7212ebc
Added diag
SergeyGaluzo Jan 3, 2025
55b4bba
Added checks of Tbls
SergeyGaluzo Jan 3, 2025
9a01079
sending zero raw resource for reindex
SergeyGaluzo Jan 3, 2025
6983e72
Configurable sleep
SergeyGaluzo Jan 3, 2025
5dfcfac
constraint name correction
SergeyGaluzo Jan 5, 2025
88cbc06
rebatched
SergeyGaluzo Jan 6, 2025
daa1ad9
removed commit with failure call
SergeyGaluzo Jan 6, 2025
3913db5
Removed redundant tostring
SergeyGaluzo Jan 9, 2025
a16381d
ensure not null
SergeyGaluzo Jan 9, 2025
806f877
more ensure
SergeyGaluzo Jan 9, 2025
b2e2d7e
more ensure
SergeyGaluzo Jan 9, 2025
49bbdbd
import tests with raw in lake
SergeyGaluzo Jan 10, 2025
5f44cc7
minus data lake
SergeyGaluzo Jan 12, 2025
99a8c7a
minus data lake
SergeyGaluzo Jan 12, 2025
609ff9e
Removed not used stored proc
SergeyGaluzo Jan 13, 2025
983a5da
reverted formatting change
SergeyGaluzo Jan 13, 2025
feb8562
comments
SergeyGaluzo Jan 13, 2025
a309c92
Removed item2
SergeyGaluzo Jan 13, 2025
f850188
Cient -> Client
SergeyGaluzo Jan 13, 2025
24430d0
removed local var
SergeyGaluzo Jan 13, 2025
9e1d8a6
revert disable indexes
SergeyGaluzo Jan 14, 2025
8665848
WIP: adls test
Jan 17, 2025
a440879
made uploads to blob parallel
Jan 18, 2025
3f03441
download parallel during export
Jan 19, 2025
bb1d219
export changes
Jan 19, 2025
d128d68
adls testing
Jan 19, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
minus data lake
  • Loading branch information
SergeyGaluzo committed Jan 12, 2025
commit 5f44cc76537a9c7bc54532d8f4f72416ddcb142b
1 change: 0 additions & 1 deletion tools/Exporter/Exporter.csproj
Original file line number Diff line number Diff line change
@@ -13,7 +13,6 @@
</ItemGroup>

<ItemGroup>
<PackageReference Include="Azure.Storage.Files.DataLake" />
<PackageReference Include="System.Configuration.ConfigurationManager" />
<PackageReference Include="Microsoft.Health.SqlServer" />
</ItemGroup>
Loading