Skip to content

Storage improvements #73

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 38 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
001f28e
Add a warning in readme to sync ipfs cluster before registring the pr…
faza Jul 8, 2025
27fbd27
Calculate storage cost only when storage price mb is not zero
faza Jul 8, 2025
c76ab2d
Handle DeleteRepository and PackfileDeleted events
faza Jul 8, 2025
f3fd632
Fix attribute parsing and handle empty repository scenario in delete …
faza Jul 10, 2025
934ab22
Get owner id from release attribute in case of DeleteRepositoryPackfi…
faza Jul 10, 2025
c184610
Integrate storage module in git lfs endpoint
faza Jul 15, 2025
2b48819
Add event hander for LFS object update and pin to pinata if enabled
faza Jul 15, 2025
eedf2ab
Do not store lfs objects in nested directories
faza Jul 15, 2025
9fd1cde
Handle lfs objects in cache manager
faza Jul 15, 2025
723945e
Update readme and config files with cache configuration for lfs objects
faza Jul 15, 2025
bfb31ef
Infer the public address from the lfs batch request instead of manual…
faza Jul 16, 2025
ecb1692
Return http status ok on root endpoint and also include a health chec…
faza Jul 16, 2025
f60a94a
Add support for lfs objects in challenge handler
faza Jul 16, 2025
75a45dd
Refactor merkle root calculation
faza Jul 16, 2025
c95229b
Fixes in lfs object update handler
faza Jul 16, 2025
361fade
Implement batch tx manager
faza Jul 17, 2025
be8e0c2
Don't create websocket connections by default for events which are ha…
faza Jul 17, 2025
96a28a3
fix lfs objects env var
faza Jul 17, 2025
43ec946
Handle deletion of lfs objects when repository is deleted
faza Jul 17, 2025
1c170f6
Check the message has been executed successfully in the each handler …
faza Jul 17, 2025
9cbd391
Add the remaining state update checks
faza Jul 17, 2025
80d9fc9
Refactor and clean up code
faza Jul 17, 2025
a9ef5c6
Add missing methods and fix errors
faza Jul 17, 2025
ca455ad
Initialize batch tx manager outside gitopia proxy client so that we c…
faza Jul 17, 2025
ae6a957
Use batch tx processor in migration script also
faza Jul 17, 2025
a25011d
fix packfile update check
faza Jul 18, 2025
c50f5ad
fix release asset and lfs object update check
faza Jul 18, 2025
e8436b6
feat(build): Exclude profiling tools in production build
faza Jul 18, 2025
d47a2ea
Handle lfs objects in migration script
faza Jul 20, 2025
d123198
do balance check on startup
faza Jul 21, 2025
045f627
fix config for production environment
faza Jul 21, 2025
8dd6853
Improve event proccesing by subscribing to multiple events in a singl…
faza Jul 21, 2025
29da847
Handle the default limit of 5 subscriptions per client on the server
faza Jul 21, 2025
1b58e5b
check auth only for upload operation in lfs handler
faza Jul 21, 2025
18c3cc4
use the correct lfs objects path in the migration script
faza Jul 21, 2025
058227a
Implement lfs lock endpoints to prevent authentication error warnings…
faza Jul 21, 2025
9d8c7b0
Add all provider related transactions to storage provider cli commands
faza Jul 23, 2025
b7be3c0
Pass previous state of the repo or release asset in the update request
faza Jul 24, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
87 changes: 87 additions & 0 deletions app/batch_tx_manager.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
package app

import (
"context"
"sync"
"time"

sdk "github.com/cosmos/cosmos-sdk/types"
"github.com/gitopia/gitopia-go"
"github.com/sirupsen/logrus"
)

type BatchTxManager struct {
client gitopia.Client
txQueue []sdk.Msg
queueMutex sync.Mutex
ticker *time.Ticker
stopCh chan struct{}
batchTimeout time.Duration
}

func NewBatchTxManager(client gitopia.Client, batchTimeout time.Duration) *BatchTxManager {
return &BatchTxManager{
client: client,
txQueue: make([]sdk.Msg, 0),
ticker: time.NewTicker(batchTimeout),
stopCh: make(chan struct{}),
batchTimeout: batchTimeout,
}
}

func (b *BatchTxManager) Start() {
go b.processBatches()
}

func (b *BatchTxManager) Stop() {
b.ticker.Stop()
close(b.stopCh)
// Process any remaining messages
b.processBatch()
}

func (b *BatchTxManager) AddToBatch(ctx context.Context, msgs ...sdk.Msg) error {
b.queueMutex.Lock()
defer b.queueMutex.Unlock()

b.txQueue = append(b.txQueue, msgs...)
return nil
}

func (b *BatchTxManager) processBatches() {
for {
select {
case <-b.ticker.C:
b.processBatch()
case <-b.stopCh:
return
}
}
}

func (b *BatchTxManager) processBatch() {
b.queueMutex.Lock()
if len(b.txQueue) == 0 {
b.queueMutex.Unlock()
return
}

// Take a copy of the current batch and reset the queue
batch := make([]sdk.Msg, len(b.txQueue))
copy(batch, b.txQueue)
b.txQueue = b.txQueue[:0]
b.queueMutex.Unlock()

// Process the batch
if len(batch) > 0 {
// Use background context since the original context might be canceled
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()

// Send the batch transaction
if err := b.client.BroadcastTxAndWait(ctx, batch...); err != nil {
// Log the error using logrus
logrus.WithError(err).Error("error broadcasting batch transaction")
}
}
}
Loading