Skip to content

Commit 7407581

Browse files
author
Haik
committed
npgsqlcopy add
1 parent 3fdc708 commit 7407581

15 files changed

+684
-57
lines changed

‎Readme.md

Lines changed: 105 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -1,21 +1,46 @@
1-
# Pandatech.EFCore.PostgresExtensions
2-
Pandatech.EFCore.PostgresExtensions is a NuGet package that enhances Entity Framework Core with support for PostgreSQL-specific syntax for update operations.
1+
- [1. Pandatech.EFCore.PostgresExtensions](#1-pandatechefcorepostgresextensions)
2+
- [1.1. Features](#11-features)
3+
- [1.2. Installation](#12-installation)
4+
- [1.3. Usage](#13-usage)
5+
- [1.3.1. Row-Level Locking](#131-row-level-locking)
6+
- [1.3.2. Npgsql COPY Integration](#132-npgsql-copy-integration)
7+
- [1.3.2.1. Benchmarks](#1321-benchmarks)
8+
- [1.3.2.1.1. General Benchmark Results](#13211-general-benchmark-results)
9+
- [1.3.2.1.2. Detailed Benchmark Results](#13212-detailed-benchmark-results)
10+
- [1.3.2.1.3. Efficiency Comparison](#13213-efficiency-comparison)
11+
- [1.3.2.1.4. Additional Notes](#13214-additional-notes)
12+
- [1.4. License](#14-license)
313

4-
## Introduction
5-
You can install the Pandatech.EFCore.PostgresExtensions NuGet package via the NuGet Package Manager UI or the Package Manager Console using the following command:
14+
# 1. Pandatech.EFCore.PostgresExtensions
15+
16+
Pandatech.EFCore.PostgresExtensions is an advanced NuGet package designed to enhance PostgreSQL functionalities within
17+
Entity Framework Core, leveraging specific features not covered by the official Npgsql.EntityFrameworkCore.PostgreSQL
18+
package. This package introduces optimized row-level locking mechanisms and an efficient, typed version of the
19+
PostgreSQL COPY operation, adhering to EF Core syntax for seamless integration into your projects.
20+
21+
## 1.1. Features
22+
23+
1. **Row-Level Locking**: Implements the PostgreSQL `FOR UPDATE` feature, providing three lock
24+
behaviors - `Wait`, `Skip`, and
25+
`NoWait`, to facilitate advanced transaction control and concurrency management.
26+
2. **Npgsql COPY Integration**: Offers a high-performance, typed interface for the PostgreSQL COPY command, allowing for
27+
bulk data operations within the EF Core framework. This feature significantly enhances data insertion speeds and
28+
efficiency.
29+
30+
## 1.2. Installation
31+
32+
To install Pandatech.EFCore.PostgresExtensions, use the following NuGet command:
33+
34+
```bash
635
Install-Package Pandatech.EFCore.PostgresExtensions
36+
```
37+
38+
## 1.3. Usage
739

8-
## Features
9-
Adds support for PostgreSQL-specific update syntax.
10-
Simplifies handling of update operations when working with PostgreSQL databases.
40+
### 1.3.1. Row-Level Locking
1141

12-
## Installation
13-
1. Install Pandatech.EFCore.PostgresExtensions Package
14-
```Install-Package Pandatech.EFCore.PostgresExtensions```
15-
16-
2. Enable Query Locks
42+
Configure your DbContext to use Npgsql and enable query locks:
1743

18-
Inside the AddDbContext or AddDbContextPool method, after calling UseNpgsql(), call the UseQueryLocks() method on the DbContextOptionsBuilder to enable query locks.
1944
```csharp
2045
services.AddDbContext<MyDbContext>(options =>
2146
{
@@ -24,36 +49,82 @@ services.AddDbContext<MyDbContext>(options =>
2449
});
2550
```
2651

27-
## Usage
28-
Use the provided ForUpdate extension method on IQueryable within your application to apply PostgreSQL-specific update syntax.
52+
Within a transaction scope, apply the desired lock behavior using the `ForUpdate` extension method:
53+
2954
```csharp
30-
using Pandatech.EFCore.PostgresExtensions;
31-
using Microsoft.EntityFrameworkCore;
55+
using var transaction = _dbContext.Database.BeginTransaction();
56+
try
57+
{
58+
var entityToUpdate = _dbContext.Entities
59+
.Where(e => e.Id == id)
60+
.ForUpdate(LockBehavior.NoWait) // Or use LockBehavior.Default (Wait)/ LockBehavior.SkipLocked
61+
.FirstOrDefault();
3262

33-
// Inside your service or repository method
34-
using (var transaction = _dbContext.Database.BeginTransaction())
63+
// Perform updates on entityToUpdate
64+
await _dbContext.SaveChangesAsync();
65+
transaction.Commit();
66+
}
67+
catch (Exception ex)
3568
{
36-
try
37-
{
38-
// Use the ForUpdate extension method on IQueryable inside the transaction scope
39-
var entityToUpdate = _dbContext.Entities
40-
.Where(e => e.Id == id)
41-
.ForUpdate()
42-
.FirstOrDefault();
69+
transaction.Rollback();
70+
// Handle exception
71+
}
72+
```
4373

44-
// Perform updates on entityToUpdate
74+
### 1.3.2. Npgsql COPY Integration
4575

46-
await _dbContext.SaveChangesAsync();
76+
For bulk data operations, use the `BulkInsert` or `BulkInsertAsync` extension methods:
4777

48-
transaction.Commit();
49-
}
50-
catch (Exception ex)
78+
```csharp
79+
public async Task BulkInsertExampleAsync()
80+
{
81+
var users = new List<UserEntity>();
82+
for (int i = 0; i < 10000; i++)
5183
{
52-
transaction.Rollback();
53-
// Handle exception
84+
users.Add(new UserEntity { /* Initialization */ });
5485
}
86+
87+
await dbContext.Users.BulkInsertAsync(users); // Or use BulkInsert for synchronous operation
88+
// It also saves changes to the database
5589
}
5690
```
57-
## License
91+
92+
#### 1.3.2.1. Benchmarks
93+
94+
The integration of the Npgsql COPY command showcases significant performance improvements compared to traditional EF
95+
Core and Dapper methods:
96+
97+
##### 1.3.2.1.1. General Benchmark Results
98+
99+
| Caption | Big O Notation | 1M Rows | Batch Size |
100+
|------------|----------------|-------------|------------|
101+
| BulkInsert | O(log n) | 350.000 r/s | No batch |
102+
| Dapper | O(n) | 20.000 r/s | 1500 |
103+
| EFCore | O(n) | 10.600 r/s | 1500 |
104+
105+
##### 1.3.2.1.2. Detailed Benchmark Results
106+
107+
| Operation | BulkInsert | Dapper | EF Core |
108+
|-------------|------------|--------|---------|
109+
| Insert 10K | 76ms | 535ms | 884ms |
110+
| Insert 100K | 405ms | 5.47s | 8.58s |
111+
| Insert 1M | 2.87s | 55.85s | 94.57s |
112+
113+
##### 1.3.2.1.3. Efficiency Comparison
114+
115+
| RowsCount | BulkInsert Efficiency | Dapper Efficiency |
116+
|-----------|----------------------------|---------------------------|
117+
| 10K | 11.63x faster than EF Core | 1.65x faster than EF Core |
118+
| 100K | 21.17x faster than EF Core | 1.57x faster than EF Core |
119+
| 1M | 32.95x faster than EF Core | 1.69x faster than EF Core |
120+
121+
##### 1.3.2.1.4. Additional Notes
122+
123+
- The `BulkInsert` feature currently does not support entity properties intended for `JSON` storage.
124+
125+
- The performance metrics provided above are based on benchmarks conducted under controlled conditions. Real-world
126+
performance may vary based on specific use cases and configurations.
127+
128+
## 1.4. License
58129

59130
Pandatech.EFCore.PostgresExtensions is licensed under the MIT License.

‎img.png

3.53 KB
Loading

‎src/EFCore.PostgresExtensions/EFCore.PostgresExtensions.csproj

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -8,22 +8,22 @@
88
<PackageReadmeFile>Readme.md</PackageReadmeFile>
99
<Authors>Pandatech</Authors>
1010
<Copyright>MIT</Copyright>
11-
<Version>1.0.0</Version>
11+
<Version>2.0.0</Version>
1212
<PackageId>Pandatech.EFCore.PostgresExtensions</PackageId>
1313
<Title>Pandatech.EFCore.PostgresExtensions</Title>
14-
<PackageTags>Pandatech, library, EntityFrameworkCore, PostgreSQL, For Update, Lock, LockingSyntax</PackageTags>
15-
<Description>The Pandatech.EFCore.PostgresExtensions library enriches Entity Framework Core applications with advanced PostgreSQL functionalities, starting with the ForUpdate locking syntax. Designed for seamless integration, this NuGet package aims to enhance the efficiency and capabilities of EF Core models when working with PostgreSQL, with the potential for further PostgreSQL-specific extensions.</Description>
14+
<PackageTags>Pandatech, library, EntityFrameworkCore, PostgreSQL, For Update, Lock, LockingSyntax, Bulk insert, BinaryCopy</PackageTags>
15+
<Description>The Pandatech.EFCore.PostgresExtensions library enriches Entity Framework Core applications with advanced PostgreSQL functionalities, starting with the ForUpdate locking syntax and BulkInsert function. Designed for seamless integration, this NuGet package aims to enhance the efficiency and capabilities of EF Core models when working with PostgreSQL, with the potential for further PostgreSQL-specific extensions.</Description>
1616
<RepositoryUrl>https://github.com/PandaTechAM/be-lib-efcore-postgres-extensions</RepositoryUrl>
17-
<PackageReleaseNotes>InitialCommit</PackageReleaseNotes>
17+
<PackageReleaseNotes>Npgsql copy feature</PackageReleaseNotes>
1818
</PropertyGroup>
1919

2020
<ItemGroup>
21-
<None Include="..\..\pandatech.png" Pack="true" PackagePath="\" />
22-
<None Include="..\..\Readme.md" Pack="true" PackagePath="\" />
21+
<None Include="..\..\pandatech.png" Pack="true" PackagePath="\"/>
22+
<None Include="..\..\Readme.md" Pack="true" PackagePath="\"/>
2323
</ItemGroup>
2424

2525
<ItemGroup>
26-
<PackageReference Include="Microsoft.EntityFrameworkCore.Relational" Version="8.0.3" />
26+
<PackageReference Include="Npgsql.EntityFrameworkCore.PostgreSQL" Version="8.0.2"/>
2727
</ItemGroup>
2828

2929
</Project>

‎src/EFCore.PostgresExtensions/Extensions/BulkInsertExtension/BulkInsertExtensionSync.cs

Lines changed: 159 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,159 @@
1+
using System.Collections;
2+
using System.Diagnostics;
3+
using System.Reflection;
4+
using Microsoft.EntityFrameworkCore;
5+
using Microsoft.EntityFrameworkCore.Metadata;
6+
using Microsoft.Extensions.Logging;
7+
using Npgsql;
8+
using Npgsql.EntityFrameworkCore.PostgreSQL.Storage.Internal.Mapping;
9+
10+
namespace EFCore.PostgresExtensions.Extensions.BulkInsertExtension;
11+
12+
public static class BulkInsertExtension
13+
{
14+
public static ILogger? Logger { get; set; }
15+
16+
public static async Task BulkInsertAsync<T>(this DbSet<T> dbSet, List<T> entities,
17+
bool pkGeneratedByDb = true) where T : class
18+
{
19+
var context = PrepareBulkInsertOperation(dbSet, entities, pkGeneratedByDb, out var sp, out var properties,
20+
out var columnCount, out var sql, out var propertyInfos, out var propertyTypes);
21+
22+
var connection = new NpgsqlConnection(context.Database.GetConnectionString());
23+
await connection.OpenAsync();
24+
25+
await using var writer = await connection.BeginBinaryImportAsync(sql);
26+
27+
for (var entity = 0; entity < entities.Count; entity++)
28+
{
29+
var item = entities[entity];
30+
var values = propertyInfos.Select(property => property!.GetValue(item)).ToList();
31+
32+
ConvertEnumValue<T>(columnCount, propertyTypes, properties, values);
33+
34+
await writer.StartRowAsync();
35+
36+
for (var i = 0; i < columnCount; i++)
37+
{
38+
await writer.WriteAsync(values[i]);
39+
}
40+
}
41+
42+
await writer.CompleteAsync();
43+
await connection.CloseAsync();
44+
sp.Stop();
45+
46+
Logger?.LogInformation("Binary copy completed successfully. Total time: {Milliseconds} ms",
47+
sp.ElapsedMilliseconds);
48+
}
49+
50+
public static void BulkInsert<T>(this DbSet<T> dbSet, List<T> entities,
51+
bool pkGeneratedByDb = true) where T : class
52+
{
53+
var context = PrepareBulkInsertOperation(dbSet, entities, pkGeneratedByDb, out var sp, out var properties,
54+
out var columnCount, out var sql, out var propertyInfos, out var propertyTypes);
55+
56+
var connection = new NpgsqlConnection(context.Database.GetConnectionString());
57+
connection.Open();
58+
59+
using var writer = connection.BeginBinaryImport(sql);
60+
61+
for (var entity = 0; entity < entities.Count; entity++)
62+
{
63+
var item = entities[entity];
64+
var values = propertyInfos.Select(property => property!.GetValue(item)).ToList();
65+
66+
ConvertEnumValue<T>(columnCount, propertyTypes, properties, values);
67+
68+
writer.StartRow();
69+
70+
for (var i = 0; i < columnCount; i++)
71+
{
72+
writer.Write(values[i]);
73+
}
74+
}
75+
76+
writer.Complete();
77+
connection.Close();
78+
sp.Stop();
79+
80+
Logger?.LogInformation("Binary copy completed successfully. Total time: {Milliseconds} ms",
81+
sp.ElapsedMilliseconds);
82+
}
83+
84+
private static void ConvertEnumValue<T>(int columnCount, IReadOnlyList<Type> propertyTypes,
85+
IReadOnlyList<IProperty> properties, IList<object?> values) where T : class
86+
{
87+
for (var i = 0; i < columnCount; i++)
88+
{
89+
if (propertyTypes[i].IsEnum)
90+
{
91+
values[i] = Convert.ChangeType(values[i], Enum.GetUnderlyingType(propertyTypes[i]));
92+
continue;
93+
}
94+
95+
// Check for generic types, specifically lists, and ensure the generic type is an enum
96+
if (!propertyTypes[i].IsGenericType || propertyTypes[i].GetGenericTypeDefinition() != typeof(List<>) ||
97+
!propertyTypes[i].GetGenericArguments()[0].IsEnum) continue;
98+
99+
var enumMapping = properties[i].FindTypeMapping();
100+
101+
// Only proceed if the mapping is for an array type, as expected for lists
102+
if (enumMapping is not NpgsqlArrayTypeMapping) continue;
103+
104+
var list = (IList)values[i]!;
105+
var underlyingType = Enum.GetUnderlyingType(propertyTypes[i].GetGenericArguments()[0]);
106+
107+
var convertedList = (from object item in list select Convert.ChangeType(item, underlyingType)).ToList();
108+
values[i] = convertedList;
109+
}
110+
}
111+
112+
113+
private static DbContext PrepareBulkInsertOperation<T>(DbSet<T> dbSet, List<T> entities, bool pkGeneratedByDb,
114+
out Stopwatch sp, out List<IProperty> properties, out int columnCount, out string sql,
115+
out List<PropertyInfo?> propertyInfos, out List<Type> propertyTypes) where T : class
116+
{
117+
sp = Stopwatch.StartNew();
118+
var context = dbSet.GetDbContext();
119+
120+
121+
if (entities == null || entities.Count == 0)
122+
throw new ArgumentException("The model list cannot be null or empty.");
123+
124+
if (context == null) throw new ArgumentNullException(nameof(context), "The DbContext instance cannot be null.");
125+
126+
127+
var entityType = context.Model.FindEntityType(typeof(T))! ??
128+
throw new InvalidOperationException("Entity type not found.");
129+
130+
var tableName = entityType.GetTableName() ??
131+
throw new InvalidOperationException("Table name is null or empty.");
132+
133+
properties = entityType.GetProperties().ToList();
134+
135+
if (pkGeneratedByDb)
136+
properties = properties.Where(x => !x.IsKey()).ToList();
137+
138+
var columnNames = properties.Select(x => $"\"{x.GetColumnName()}\"").ToList();
139+
140+
if (columnNames.Count == 0)
141+
throw new InvalidOperationException("Column names are null or empty.");
142+
143+
144+
columnCount = columnNames.Count;
145+
var rowCount = entities.Count;
146+
147+
Logger?.LogDebug(
148+
"Column names found successfully. \n Total column count: {ColumnCount} \n Total row count: {RowCount}",
149+
columnCount, rowCount);
150+
151+
sql = $"COPY \"{tableName}\" ({string.Join(", ", columnNames)}) FROM STDIN (FORMAT BINARY)";
152+
153+
Logger?.LogInformation("SQL query created successfully. Sql query: {Sql}", sql);
154+
155+
propertyInfos = properties.Select(x => x.PropertyInfo).ToList();
156+
propertyTypes = propertyInfos.Select(x => x!.PropertyType).ToList();
157+
return context;
158+
}
159+
}

‎src/EFCore.PostgresExtensions/Extensions/DbSetExtensions.cs

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
using Microsoft.EntityFrameworkCore;
2+
using Microsoft.EntityFrameworkCore.Infrastructure;
3+
4+
namespace EFCore.PostgresExtensions.Extensions;
5+
6+
public static class DbSetExtensions
7+
{
8+
public static DbContext GetDbContext<T>(this DbSet<T> dbSet) where T : class
9+
{
10+
var infrastructure = dbSet as IInfrastructure<IServiceProvider>;
11+
var serviceProvider = infrastructure.Instance;
12+
var currentDbContext = serviceProvider.GetService(typeof(ICurrentDbContext)) as ICurrentDbContext;
13+
return currentDbContext.Context;
14+
}
15+
}

0 commit comments

Comments
 (0)