Skip to content

Conversation

@pzhao5
Copy link
Contributor

@pzhao5 pzhao5 commented Nov 2, 2018

Saving the following data into S3 so that lambda could process it.

  1. Data format as gensis_hash/data/file_index. File contains a BatchSize of data.
  2. Genesis Hash as gensis_hash/hash/file_index.
  3. Each data and genesis hash is 1:1 mapping. And there is not treasure within.
  4. Config data(single file) so that lambda knows how to process it.
  5. Treasure map data (single file).

When lambda attach to the data, it would first download config data and treasure map.
And then start to upload each data/genesis hash in pairs.
When arrive to a particular index that need to skip (for inserting the treasure), Lambda will then insert it with treasure rather than the data.

@pzhao5 pzhao5 requested a review from rfornea November 2, 2018 00:43
/*GetObjectKeyForData will return object key for particular data. startIndex is the smallest index in data. */
func GetObjectKeyForData(genesisHash string, startIndex int, totalCount int, isReserveIteration bool, batchSize int) string {
index := 0
if isReserveIteration {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit isReverseIteration

rfornea
rfornea previously approved these changes Nov 2, 2018
@rfornea
Copy link
Contributor

rfornea commented Nov 2, 2018

Went ahead and approved this but we should see if we can figure out what the deal is with Travis

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants