Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
240 commits
Select commit Hold shift + click to select a range
45d4abf
Add local cache for json blocks as well as possibility to query Bitco…
dominusmi Jun 2, 2023
b9c8803
Refactor address changes (from Bitcoin.22 migration). See https://git…
dominusmi Jun 2, 2023
7bdff7d
Update license
dominusmi Jun 2, 2023
a1402cc
Add gzip natively
dominusmi Jun 5, 2023
b10bd99
Add bitocin request timeout argument
dominusmi Jun 5, 2023
159e9cb
Fix missing paramiko error
dominusmi Jun 5, 2023
c1ca00e
Add print statement after CSV and before sorting
dominusmi Jun 6, 2023
dc92eee
Add script to create pk -> hash relationships
dominusmi Jun 7, 2023
f6a6f72
Add extra logs and parameters for compute entities
dominusmi Jun 8, 2023
7210741
Add header writing, fix OP_CHECKSIG error, and add try catch to avoid…
dominusmi Jun 9, 2023
3eb555e
Add batched reading of file
dominusmi Jun 9, 2023
7b42381
Fix batch write, add better logs
dominusmi Jun 10, 2023
ab31cb0
Clean up synchronize script
dominusmi Jun 27, 2023
780ada0
Move from using REST api to using Bolt. Only migrated necessary funct…
dominusmi Jun 27, 2023
5f0eca7
Use tqdm instead of printing
dominusmi Jun 28, 2023
984c683
Remove prints
dominusmi Jun 28, 2023
9902328
Update requirements.txt
dominusmi Jun 28, 2023
911f3fc
Add bc-create-entities.py script
dominusmi Jun 28, 2023
72bc41f
Add protocol
dominusmi Jun 28, 2023
ce76f3e
Refactor code into separate script & src
dominusmi Jun 28, 2023
4e7550c
Add incremental PK generated addresses
dominusmi Jun 28, 2023
25367cd
Fix error when genesis block not existent
dominusmi Jun 29, 2023
2df88dc
Fix all id based errors
dominusmi Jun 29, 2023
5ca7cc6
Add address module which mostly deals with pk -> addresses
dominusmi Jun 29, 2023
0e600e9
Move script code to src module
dominusmi Jun 29, 2023
cc478cd
Transform synchronize into a yield
dominusmi Jun 29, 2023
ab27aca
Fix entities query
dominusmi Jun 29, 2023
983eccd
Add daemon
dominusmi Jun 29, 2023
a0bfa7a
Add neo4j requirement
dominusmi Jun 29, 2023
6dc4cfa
Add log message
dominusmi Jun 30, 2023
2b4f787
use set instead of list to avoid duplicates
dominusmi Jun 30, 2023
6a268a6
Fix max_height upper bound in entities thread function
dominusmi Jun 30, 2023
173f7a2
Argument int conversion
dominusmi Jun 30, 2023
d555311
Make entities saving into tqdm if wished
dominusmi Jun 30, 2023
ddffc0d
Fix argument
dominusmi Jun 30, 2023
3612316
Add description
dominusmi Jun 30, 2023
074c374
Add dumping of state in case of unexpected error
dominusmi Jun 30, 2023
bfd5b5d
Add delete-entities script
dominusmi Jul 1, 2023
aaa7971
Fix argument type
dominusmi Jul 1, 2023
f718b03
Try to improve performance
dominusmi Jul 1, 2023
9fa6a01
Fix progress bar
dominusmi Jul 1, 2023
5d05e32
Fix progress bar
dominusmi Jul 1, 2023
1f49da4
Change delete from relationships to entities
dominusmi Jul 2, 2023
1df5fa6
Fix naming
dominusmi Jul 2, 2023
3c93127
Fix KeyError bug - not all addresses were moved during entity update …
dominusmi Jul 3, 2023
6f9a385
Add import
dominusmi Jul 4, 2023
0e85e01
Add resume argument to provide dump file
dominusmi Jul 4, 2023
4b7c4e8
Add shebang for python
dominusmi Jul 4, 2023
6a10d3e
Remove txid return
dominusmi Jul 5, 2023
7600596
Completely refactor query style
dominusmi Jul 5, 2023
dd267c4
Remove count of transactions which takes too long
dominusmi Jul 5, 2023
bddd611
Test with transactions
dominusmi Jul 5, 2023
07ada1f
Increase time between dumps
dominusmi Jul 5, 2023
eb54ee8
Do not use blocks at all
dominusmi Jul 5, 2023
87ac1e0
Add retrial if the thread crashed
dominusmi Jul 5, 2023
613742a
Remove named node and leave native transactions to 1000
dominusmi Jul 5, 2023
d6ac9b8
Leave native fetch size
dominusmi Jul 5, 2023
4af7729
Fix not using index
dominusmi Jul 5, 2023
acec0eb
Update compute entities
dominusmi Jul 9, 2023
5b58aa3
Improve query performance and add type hints
dominusmi Jul 9, 2023
6dd061e
Add parameters
dominusmi Jul 9, 2023
f279d67
Remov eentities
dominusmi Jul 9, 2023
bd92e79
Typo
dominusmi Jul 9, 2023
5a7a572
Fix progress bar
dominusmi Jul 9, 2023
c13cd53
Add some logs
dominusmi Jul 9, 2023
ec725cb
Add some logs
dominusmi Jul 9, 2023
3f1f698
many fixes
dominusmi Jul 9, 2023
89dcd7e
Increase initial cache size
dominusmi Jul 9, 2023
5c386d7
Ugh
dominusmi Jul 9, 2023
96ae7d7
Add raised error when txid_n greater than possible
dominusmi Jul 9, 2023
ae67f1e
Try to add log
dominusmi Jul 9, 2023
7207128
Try to add log
dominusmi Jul 9, 2023
b5a89d4
Remove bad try / except
dominusmi Jul 9, 2023
47f0723
Potential fix
dominusmi Jul 9, 2023
5f775b0
(:
dominusmi Jul 9, 2023
1b20760
(:
dominusmi Jul 9, 2023
94e45d6
(:
dominusmi Jul 9, 2023
b1f6268
Remove log
dominusmi Jul 9, 2023
f52a420
Read larger chunks from file, but save small batches
dominusmi Jul 9, 2023
e7d9eca
Remove try/except, add logs
dominusmi Jul 10, 2023
2c835ad
Fix try/except
dominusmi Jul 10, 2023
cb3ee12
Add dumping of state
dominusmi Jul 10, 2023
ef49fb8
Update README.md
dominusmi Jul 10, 2023
052d55c
Use binary search in index
dominusmi Jul 10, 2023
355bd0f
Add resume capability, and improve read performance by caching file h…
dominusmi Jul 10, 2023
4f05843
Hotfix number of iterations between dumps
dominusmi Jul 10, 2023
a0df8cd
Add datetime in compute entities dump filename
dominusmi Jul 10, 2023
fcb3482
Write header csv, entity csv, and change path to input_path
dominusmi Jul 12, 2023
4a4decd
Update schema image and readme queries
dominusmi Jul 12, 2023
40b2662
Add GENERATES relationship
dominusmi Jul 12, 2023
efafb8b
Fix possible address None in group if address is not valid
dominusmi Jul 15, 2023
cb75591
Update README.md contraints
dominusmi Jul 15, 2023
dad4d3e
Fix entities grouping, max blocks mechanism, and add tests
dominusmi Jul 17, 2023
c229fb5
Update README
dominusmi Jul 17, 2023
4c2484a
Add test for three cases
dominusmi Jul 17, 2023
bd9141f
Add entity name concatenation
dominusmi Jul 17, 2023
bf13f9a
Fix entity computation mistake
dominusmi Jul 17, 2023
b88aff3
Make sort input not default in compute entities
dominusmi Jul 18, 2023
d079dc2
Update README.md
dominusmi Jul 18, 2023
d36c29f
Add picture
dominusmi Jul 18, 2023
4e9b7ed
Add white background
dominusmi Jul 18, 2023
9f089d1
Remove sort_input entirely
dominusmi Jul 21, 2023
c21ed99
Add log for end of cache
dominusmi Jul 21, 2023
da5558d
Add initial indexing
dominusmi Jul 21, 2023
ec07ca2
Add warning message if using pypy
dominusmi Jul 24, 2023
adf4376
Indexing updates
dominusmi Jul 24, 2023
ac049e9
Performance improvements
dominusmi Jul 24, 2023
ee14895
Make library PyPy compatible
dominusmi Jul 24, 2023
f42d67d
Fix init
dominusmi Jul 24, 2023
3cb1464
Remove count lines and replace with approximate line count
dominusmi Jul 24, 2023
e773be4
Add queuing system [WIP]
dominusmi Jul 24, 2023
901bdfa
Something is wrong
dominusmi Jul 25, 2023
31ec512
Fixed + added max distance
dominusmi Jul 25, 2023
3590cff
Performance improvements
dominusmi Jul 25, 2023
636614f
Fix dynamic_distance measurement
dominusmi Jul 25, 2023
4bc3df4
Performance is key
dominusmi Jul 25, 2023
dae2a8d
remove @profile
dominusmi Jul 25, 2023
dae56df
Perf
dominusmi Jul 25, 2023
e8463b4
Missing distance
dominusmi Jul 25, 2023
2f6c1a3
Missing distance
dominusmi Jul 25, 2023
a9a9ac5
missing count_tx increment
dominusmi Jul 25, 2023
c68b1e9
Skip 0 output fills
dominusmi Jul 25, 2023
ac2b363
Change distance computation from 3 to 4 characters
dominusmi Jul 25, 2023
0b715eb
Refactor queue mechanism
dominusmi Jul 26, 2023
2698fed
Full queue based v1 ready
dominusmi Jul 26, 2023
b1585d6
Remove @profile
dominusmi Jul 26, 2023
5537122
Remove @profile
dominusmi Jul 26, 2023
c5a03f3
Add aveg group length to queue size calculation
dominusmi Jul 26, 2023
2a168da
Small bug fix
dominusmi Jul 26, 2023
9ae6d20
God help me
dominusmi Jul 26, 2023
19ae655
Groupds broken and reformed during processing
dominusmi Jul 27, 2023
be87333
Improve queue display info
dominusmi Jul 27, 2023
0c0f96d
Display imporvements
dominusmi Jul 27, 2023
6ba9939
Fix minor progress display errors
dominusmi Jul 28, 2023
0fb4be7
Add merge entities process
dominusmi Jul 28, 2023
9a1042f
Fix warning int -> long
dominusmi Aug 1, 2023
5040608
Add explicit flush
dominusmi Aug 1, 2023
c56f620
Replace hash-based uniqueness in pk-to-addresses. TODO add uniqueness
dominusmi Aug 1, 2023
653bc53
Change default parameters
dominusmi Aug 2, 2023
74fc4ce
Update README
dominusmi Aug 2, 2023
8dac48f
Make csv writer terminate with \n instead of \r\n
dominusmi Aug 3, 2023
3028a66
Minor fix
dominusmi Aug 3, 2023
b8e3993
Add OS disclaimer
dominusmi Aug 4, 2023
113ae6c
Add section regarding work done in this fork
dominusmi Aug 4, 2023
3ea790e
Fix link
dominusmi Aug 4, 2023
a2be444
Fix entity index restarting
dominusmi Aug 4, 2023
d8e8f8a
Name
dominusmi Aug 4, 2023
696c5fd
Add merge entities log
dominusmi Aug 4, 2023
015cf8e
Fix case when no rel_entity_address.csv exists
dominusmi Aug 4, 2023
0b4d6d5
Remove Cargo.lock and add it to .gitignore
dominusmi Aug 5, 2023
ecd9203
Add script find_direct_entities.py
dominusmi Aug 10, 2023
f94f00b
Fix bugs and improve performance
dominusmi Aug 10, 2023
898f597
Improve help, and always write final log
dominusmi Aug 11, 2023
2729ae5
Type conversion
dominusmi Aug 11, 2023
56b84b9
Fix race condition and add transaction logging
dominusmi Aug 11, 2023
577d018
Add log message
dominusmi Aug 11, 2023
f0123f5
Improve log message
dominusmi Aug 11, 2023
fe181a3
Improve log message
dominusmi Aug 11, 2023
02965dd
Add progress saved through pickle
dominusmi Aug 16, 2023
aa115f4
Fix interval
dominusmi Aug 16, 2023
74d857a
Refactoring for understandability
dominusmi Aug 16, 2023
22688f0
Add bi-directional tree expansion
dominusmi Aug 16, 2023
e09292f
Fix entity connection, make path "direct", add match query in result,…
dominusmi Aug 17, 2023
b24650b
Update README
dominusmi Aug 21, 2023
3d5111e
Fix re-import of Bitcoingraph
dominusmi Aug 21, 2023
d872d5f
Rename
dominusmi Aug 21, 2023
740a7d4
Add script to add USD value as well as CSV
dominusmi Aug 21, 2023
8a7ee6a
Add shebangs
dominusmi Aug 21, 2023
ab5c4f3
Fix bcgraph-pk-to-addresses
dominusmi Aug 21, 2023
c991b39
Fix very last output group of file not being used for compute entities
dominusmi Aug 21, 2023
301e224
Fix bc-daemon.py
dominusmi Aug 21, 2023
554b24b
Make bc-daemon due atomic updates
dominusmi Aug 23, 2023
e8596f3
Add test for entities queries
dominusmi Aug 23, 2023
0f29a92
Add generates to transaction addresses for entities
dominusmi Aug 23, 2023
b33f3df
Add generates to entities, and add simple script entities_process.sh
dominusmi Aug 23, 2023
cd8e37f
Remove unused scripts
dominusmi Aug 23, 2023
1fd728b
Improve memory use of merge entities scripts
dominusmi Aug 23, 2023
dba17c2
Fix merge-entities warnigns and small bug
dominusmi Aug 23, 2023
eb22711
Apply /tmp fix to sort function
dominusmi Aug 28, 2023
cfc7224
Test documentation.md
dominusmi Aug 28, 2023
abe4598
Update readme
dominusmi Aug 28, 2023
f27dca3
Update readme
dominusmi Aug 28, 2023
c079352
Raise error if sorting fails
dominusmi Aug 28, 2023
e76af76
Complete refactoring: blocks downloaded entirely from bitcoind instea…
dominusmi Sep 7, 2023
c3eb2ca
Workflow
dominusmi Sep 7, 2023
cc698c5
Remove health check
dominusmi Sep 7, 2023
9f0f4d7
Fix neo4j ip
dominusmi Sep 7, 2023
99ad2ae
Fix graph config
dominusmi Sep 7, 2023
61c40c9
Try with hostname?
dominusmi Sep 7, 2023
8a31921
Try with compose?
dominusmi Sep 7, 2023
132bd13
Try with compose?
dominusmi Sep 7, 2023
f968618
Try with compose?
dominusmi Sep 7, 2023
4610eda
Try with compose?
dominusmi Sep 7, 2023
0e0aea5
Try with compose?
dominusmi Sep 7, 2023
56ba68a
invalid password
dominusmi Sep 7, 2023
66dcc83
invalid password
dominusmi Sep 7, 2023
4d46c9b
invalid password
dominusmi Sep 7, 2023
ae3da9c
invalid password
dominusmi Sep 7, 2023
c60b763
Move delete nodes earlier on
dominusmi Sep 7, 2023
56d4cde
Improve name
dominusmi Sep 7, 2023
ae282bf
Add logging
dominusmi Sep 7, 2023
73827a0
Add entities logging
dominusmi Sep 8, 2023
d67e8b1
Add more logging
dominusmi Sep 8, 2023
046141c
bc-daemon save entities use int instead of str
dominusmi Sep 8, 2023
a2ff08a
Fix log import position
dominusmi Sep 9, 2023
2022803
Re-add tqdm to export
dominusmi Sep 9, 2023
9b7d3de
Fix writer
dominusmi Sep 9, 2023
3ebda4b
Use string entity_id, and optimize query
dominusmi Sep 9, 2023
37f77d3
Add minA as entity_id again
dominusmi Sep 9, 2023
da9bb83
Update tests
dominusmi Sep 9, 2023
6663c7e
Save entities separated into two calls interlaced with python
dominusmi Sep 9, 2023
29f0f1d
Fix address type
dominusmi Sep 9, 2023
42aeba5
Code cleanup
dominusmi Sep 9, 2023
c95e81f
Add datetime in logging
dominusmi Sep 9, 2023
828f95a
Merge pull request #2 from dominusmi/save_entity_two_queries
dominusmi Sep 9, 2023
cb50ff6
Fix logger
dominusmi Sep 9, 2023
1305e7e
Add "lag" required argument to bc-daemon
dominusmi Sep 12, 2023
ee1eefd
Remove plain-header
dominusmi Sep 14, 2023
ff4b984
Remove separate_header
dominusmi Sep 14, 2023
5bfc03b
Remove no deduplication of transactions
dominusmi Sep 14, 2023
b5e42c4
Add caching to RPC call too
dominusmi Sep 14, 2023
3f0838c
Add paramiko config example
dominusmi Sep 14, 2023
8b73204
Missing sys exist
dominusmi Sep 14, 2023
f96469f
Extra {0} in command
dominusmi Sep 14, 2023
0d36604
Fix tmp path for sort function
dominusmi Sep 14, 2023
edc591a
Fix tmp directory issue and add --sort-only to bcgraph-export
dominusmi Sep 20, 2023
a433d64
Add logging message
dominusmi Sep 20, 2023
dea74ac
Remove tmp directory before rewriting into it
dominusmi Sep 20, 2023
1190863
Re-create tmp folder after delete
dominusmi Sep 20, 2023
36459aa
Make blockchain connection lazy instead of eager
dominusmi Sep 20, 2023
4a4beb3
Create copy of rel_entity_address.csv for processing
dominusmi Sep 22, 2023
66d0c35
Add execution permission for all executables
dominusmi Sep 24, 2023
3ded391
Save sorted file in output, with extensions .sorted
dominusmi Sep 24, 2023
43e399f
During entity merge, largest is kept and all addresses are connected …
dominusmi Oct 10, 2023
8c01656
Fix script not ending when max_height is reached
dominusmi Oct 11, 2023
9ad9b0b
Improve logs
dominusmi Oct 11, 2023
459bac1
add bitcoind compose
dominusmi Oct 17, 2023
c1d0c84
Add resume capability - missing merge with existing CSVs
dominusmi Oct 17, 2023
8575032
Add fully functional resume capability
dominusmi Oct 18, 2023
b52e58e
Add backslash to bash command
dominusmi Oct 18, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 38 additions & 0 deletions .github/workflows/workflow.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
name: Run tests

on:
push:
branches:
- master
- 'r&d'
pull_request:
branches:
- 'master'
workflow_dispatch:

jobs:
test:
runs-on: ubuntu-latest

steps:
- name: Checkout repository
uses: actions/checkout@v2

- name: Start neo4j
run: docker compose up -d

- name: Set up Python
uses: actions/setup-python@v2
with:
python-version: 3.9

- name: Install dependencies
run: |
pip install .
pip install pytest

- name: Test
working-directory: tests
env:
NEO4J_HOST: localhost
run: python -m pytest
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -51,3 +51,6 @@ docs/_build/
# Test log
/testlog.log
/venv_cpython/
/scripts/merge-entities/target/

Cargo.lock
332 changes: 209 additions & 123 deletions README.md

Large diffs are not rendered by default.

Loading