feat: PDP #227

magik6k · 2024-09-30T19:09:41Z

Reading:

Roadmap:

hannahhoward

I've mostly put in comments to check for understanding before I start writing code. But generally LGTM -- looks like it's compliant with the API we agreed on :)

pdp/auth.go

cuhttp/server.go

pdp/handlers.go

hannahhoward · 2024-10-20T23:16:41Z

pdp/handlers_upload.go

+
+	// Parse request body
+	var req struct {
+		PieceCID string `json:"pieceCid"`


It looks like you're not requiring piececid v2 here, not in the put handler. Long term, I think we should require this. From the standpoint of the SP, I imagine them to want to filter at the piece level, and I imagine one of the main filter critieria to be size. They should know exactly what size piece they're about to accept before they accept it.

Unless I'm wrong, a PieceCID v1 doesn't specify that, and absent a size parameter, they could be accepting anything from 32 bytes to 256 MB.

Yes, PieceCIDv2 is still todo, can quickly update if that's a blocker

pdp/handlers_upload.go

lib/pieceprovider/piecepark_reader.go

hannahhoward · 2024-10-21T02:52:49Z

am I correct that the retrieval handler already has been modified to support PDP pieces through the parked piece reader?

hannahhoward · 2024-10-21T03:09:27Z

am I correct in believing that the next step on the API is that add root will send a location header to a creation status API for the root, then that will be used to get the root ID which is then possible to pass to the root detail and delete root APIs? Not crazy urgent just want to know.

magik6k

Github thinks I have an unsubmitted review here and can't discard it..

magik6k · 2024-10-21T08:05:00Z

am I correct that the retrieval handler already has been modified to support PDP pieces through the parked piece reader?

Yes, correct, retrievals work already. No ACL mechanism yet, but it's not going to be hard to implement once we have a spec (you can see that all retrieval requests quite quickly become aware what piece they are reading from + ACLs should be quite cachable)

magik6k · 2024-10-21T08:10:28Z

am I correct in believing that the next step on the API is that add root will send a location header to a creation status API for the root, then that will be used to get the root ID which is then possible to pass to the root detail and delete root APIs? Not crazy urgent just want to know.

Other than delete all the things should be there. Operations on PDP proofsets (other than create) you should probably watch through eth events. I can also make the message-sending endpoints output the message hash somehow, but you still want to watch for all the events to not miss any contract changes made by the SP

hannahhoward · 2024-10-27T22:55:46Z

Upon further review, it's possible we have a design issue here that we need a solution for.

Please let me know if I am understanding the sequence of events correctly when a user uploads a pdp piece:

The piece is put in the “Stash”
A parked piece record is created
Asychronously, the piece moves from the stash to permaneant storage (Task: https://github.com/filecoin-project/curio/blob/feat/pdp/tasks/piece/task_park_piece.go)
After that, again asynchronously, the notify task is called, which hits the notify URL. (Task: https://github.com/filecoin-project/curio/blob/feat/pdp/tasks/pdp/notify_task.go)

Why this matters: a basic gaurantee of all hot storage, which storacha tries to offer, is read-on-write. The basic expectation is from the users point of view is that when an upload finishes, it should be retrievable.

One thing that is not clear I'd like to clarify: are steps 3 & 4 likely to run in a couple seconds or much longer? If it's only a few seconds, we can work around this. Fortunately our upload process includes a confirmation step AFTER the upload, and what we can do is essentially block the confirmation until step 4 completes. But that only works if it's a short period of time before the piece is fully retrievable.

If not, we really need to revisit the architecture and find a way to shorten the process.

hannahhoward · 2024-10-28T05:33:23Z

Follow-up: a pull based approach for checking for piece cid -- #304

* feat(pdp): add handler to check for piece cid status * feat(pdp): add handler to check for upload exists

Allows accepting JWTs that are signed with Ed25519 as well as ECDSA

x509 does not parse to pointer type on ed25519

* Update PDP to new proof scheduling WIP * add listener addr to pdp tables * generate bindings for new proving_schedule iface * Complete draft of new challenge window scheduling * Try to get tests building * No fancy ALTER tables * Review response * Fix * Fix sql syntax --------- Co-authored-by: zenground0 <[email protected]>

* Update contract wrappers * Update to pdp code freeze -- next pp now called after first add * Better initializing of init_ready flag * Name too long * Bad db select arg from copy paste * update pdp contract addr * Fix invalid insert statement * Remove unnecessary call that fails * Fix old sql error * SQL schema tweak * Fix write to bogus column * YASQLSE * DB scan param tweaking * Debuging bad call to listener * Lets try this * Fix attempt 1 * Safer init proving period challenge epoch * Less aggressive buffer * Print out init epoch * more logging * fix sql update bug * pass challenge epoch to pdptool * drand integration * flexible challenge window extension * Update contract abi * Update to refactored contract fee calc endpoint * Multiply by basefee * debug * Correct logging * Provide value during gas estimation * Tweak * Better understanding of proof fee * correct printing * more logging * bigger buffer * logging for debug of next pp * fix bug * Cleanup * Fix incorrect challenge range when adding new roots --------- Co-authored-by: zenground0 <[email protected]>

* Core api and prove task logic changes * Update pdptool to process deletes * SQL ordering fix * NoSQL just track removals on chain * Log removals * correct pdptool output * correct delete api interface * remove * remove * Update tasks/pdp/task_prove.go Co-authored-by: Łukasz Magiera <[email protected]> * review response * Fix up --------- Co-authored-by: zenground0 <[email protected]> Co-authored-by: Łukasz Magiera <[email protected]>

cmd/pdptool/main.go

magik6k · 2025-01-29T18:51:23Z

tasks/message/watch_eth.go

+
+	// Assign pending transactions with null owner to ourselves
+	{
+		n, err := mw.db.Exec(ctx, `UPDATE message_waits_eth SET waiter_machine_id = $1 WHERE waiter_machine_id IS NULL AND tx_status = 'pending'`, machineID)


Could use an index on this table like https://github.com/filecoin-project/curio/pull/392/files

magik6k · 2025-01-29T18:54:09Z

tasks/pdp/task_prove.go

+
+		proofStr += "] ] ]"
+
+		log.Infof("PDP Prove Task: proofSetID: %d, taskID: %d, proofs: %s", proofSetID, taskID, proofStr)


Should drop the verbosity here

magik6k · 2025-01-29T18:59:32Z

tasks/pdp/task_prove.go

+	return address, nil
+}
+
+func (p *ProveTask) cleanupDeletedRoots(ctx context.Context, proofSetID int64, pdpVerifier *contract.PDPVerifier) error {


There is also the pdp_piecerefs table, and I'm not entirely sure what's the best way to approach this because:

On upload the piece is only referred in pdp_piecerefs (+the piecepark tables)

Only after upload is complete and data is verified the PDP service can add the piece to a proofset

Technically one piece can be in multiple proofsets / roots (as a subroot)

So probably:

Add a flag to pdp_piecerefs which is optionally settable through the delete api which tells curio to drop the piece reference when all pdp_proofset_roots are gone

Possibly add an endpoint which lists all pdp_proofset_roots for a given piece (so the service can know why a given piece may be "locked")

magik6k · 2025-01-29T19:04:40Z

Did a deeper review pass with focus on whether this will impact other curio functionality - looks like this should be safe to merge, generally most invasive changes are in piece_park, but default behaviors don't change there, and in retrieval paths, but there PDP is plugged in as a fallback, so it shouldn't impact normal operation.

magik6k force-pushed the feat/pdp branch 2 times, most recently from 07a5b22 to 5c3f546 Compare October 4, 2024 18:35

magik6k force-pushed the feat/pdp branch from 3ec47d6 to 54af620 Compare October 13, 2024 07:34

hannahhoward approved these changes Oct 20, 2024

View reviewed changes

magik6k commented Oct 21, 2024

View reviewed changes

magik6k force-pushed the feat/pdp branch 4 times, most recently from fa8d937 to a3a1c3a Compare October 24, 2024 13:28

magik6k added 15 commits October 30, 2024 11:15

pdp: Inital scaffold

fc7399d

pdp: Drop unneded handlers

40aad8c

pdp: wip uploads

e3ad509

pdp: Proper pieceref tracking

ececf7c

storage: Implement local stash storage

ed8c966

storage: Make Local use a single URL

afa54ec

pdp: Complete upload

794d167

piecepark: Special long-term storage mode

6f36c6b

dealdata: Handle custore://

041675a

pdp: Notify task

4daac0c

retrieval: Foundations for PiecePark readers

cf7ae93

retrieval: Parked piece /piece retrievals

2082fe9

pdptool for interacting with curio pdp endpoints

cea21f7

pdp webui: Basic page, adding services through UI

76634fd

pdp webui: Remove service button

14487f2

magik6k and others added 11 commits October 30, 2024 11:15

pdp: Update contarcts

b910e95

make gen, consistent max piece size

ecc52c9

fix local path test

2e4f9b9

pdp: Implement fees

2adff15

pdp chall lookback in prove

16ccaad

update to new proving scheme

4ac1e85

small fixes

2cea05f

pdp: Support arbitrary hash funcs in uploads

7513819

pdp: api readme

ff3d92f

fix lint

3190e7c

feat(pdp): add endpoint to poll for completed upload (#304)

4cb6643

* feat(pdp): add handler to check for piece cid status * feat(pdp): add handler to check for upload exists

magik6k force-pushed the feat/pdp branch from 8fe488c to 4cb6643 Compare October 30, 2024 04:15

magik6k and others added 4 commits November 1, 2024 12:13

enable chain sched with just eth sender

143411a

feat(pdp): add ed25519 support for JWT (#323)

4b70e1d

Allows accepting JWTs that are signed with Ed25519 as well as ECDSA

fix(pdp): fix ed25519 auth (#324)

921be6f

x509 does not parse to pointer type on ed25519

Base automatically changed from feat/market to main January 7, 2025 16:57

ZenGround0 and others added 4 commits January 28, 2025 14:07

Merge remote-tracking branch 'origin/main' into feat/pdp2

db1cf8e

make gen

092e15f

magik6k commented Jan 29, 2025

View reviewed changes

cmd/pdptool/main.go Outdated Show resolved Hide resolved

magik6k commented Jan 29, 2025

View reviewed changes

magik6k changed the title ~~[WIP] feat: PDP~~ feat: PDP Jan 29, 2025

drop some unused things

dce9f17

magik6k added 2 commits January 29, 2025 22:32

crypto rand in pdptool

a29c049

reduce log verbosity, ethwaits index

c2316aa

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: PDP #227

feat: PDP #227

magik6k commented Sep 30, 2024 •

edited by ZenGround0

Loading

hannahhoward left a comment

hannahhoward Oct 20, 2024

magik6k Oct 21, 2024

hannahhoward commented Oct 21, 2024

hannahhoward commented Oct 21, 2024

magik6k left a comment •

edited

Loading

magik6k commented Oct 21, 2024

magik6k commented Oct 21, 2024

hannahhoward commented Oct 27, 2024 •

edited

Loading

hannahhoward commented Oct 28, 2024

magik6k Jan 29, 2025

magik6k Jan 29, 2025

magik6k Jan 29, 2025

magik6k commented Jan 29, 2025


		proofStr += "] ] ]"

		log.Infof("PDP Prove Task: proofSetID: %d, taskID: %d, proofs: %s", proofSetID, taskID, proofStr)

feat: PDP #227

Are you sure you want to change the base?

feat: PDP #227

Conversation

magik6k commented Sep 30, 2024 • edited by ZenGround0 Loading

hannahhoward left a comment

Choose a reason for hiding this comment

hannahhoward Oct 20, 2024

Choose a reason for hiding this comment

magik6k Oct 21, 2024

Choose a reason for hiding this comment

hannahhoward commented Oct 21, 2024

hannahhoward commented Oct 21, 2024

magik6k left a comment • edited Loading

Choose a reason for hiding this comment

magik6k commented Oct 21, 2024

magik6k commented Oct 21, 2024

hannahhoward commented Oct 27, 2024 • edited Loading

hannahhoward commented Oct 28, 2024

magik6k Jan 29, 2025

Choose a reason for hiding this comment

magik6k Jan 29, 2025

Choose a reason for hiding this comment

magik6k Jan 29, 2025

Choose a reason for hiding this comment

magik6k commented Jan 29, 2025

magik6k commented Sep 30, 2024 •

edited by ZenGround0

Loading

magik6k left a comment •

edited

Loading

hannahhoward commented Oct 27, 2024 •

edited

Loading