feat: added block handler to handle indexing of reserve balance #4

frazarshad · 2024-04-25T17:00:50Z

This PR adds a block handler to the subql node.
This block handler is responsible for querying the total ist balance in reserve and indexing that value

rabi-siddique · 2024-04-26T04:16:04Z

src/mappings/utils.ts

+  const response = await crossFetch(url);
+
+  if (response.status >= 400) {
+    const errorText = await response.text();


What's the purpose of checking the response status? What if it's in the 300 range?

Why not use try/catch? That way you don't need to check for response status on line 104? If there is an error, the catch block can throw an error accordingly.

crossFetch wont automatically throw an error in cases where it receives a proper response. such as when we hit this URL https://main-a.api.agoric.net/cosssmos/auth/v1beta1/module_accountss
We receive a 501 but it doesnt throw an error. in those cases we manually check if we have an error code and throw an error ourselves

What if it's in the 300 range?

3XX are for redirect codes which are not errrors

crossFetch wont automatically throw an error in cases where it receives a proper response. such as when we hit this URL https://main-a.api.agoric.net/cosssmos/auth/v1beta1/module_accountss

Interesting. I reckon this is something wrong with the endpoint we're hitting, right?

3XX are for redirect codes which are not errrors

Yeah. However, it does not indicate a successful response. Perhaps we should handle it too.

Interesting. I reckon this is something wrong with the endpoint we're hitting, right?

This is the case with most API responses actually. a lot of libraries dont throw an error implicitly and expect you to do it yourself

Yeah. However, it does not indicate a successful response. Perhaps we should handle it too.

300 are successful responses in the sense that we can actually obtain useful responses from them. compared to 400s and 500s which just represent a failure

300 are successful responses in the sense that we can actually obtain useful responses from them. compared to 400s and 500s which just represent a failure

In that regard, 400 and 500 are also useful responses converting us what's wrong :D

@rabi-siddique we look for errors in 400+ range. Here we are just checking for errors in response.

rabi-siddique · 2024-04-26T04:46:44Z

docker-compose.yml

@@ -37,6 +37,7 @@ services:
      - --db-schema=app
      - --workers=4
      - --batch-size=30
+      - --unsafe
      # - --log-level=debug


Why unsafe? Isn't it unsafe? :D

subql uses a restricted version of node to create a controlled env for running the code. sort of like endojs for hardened js

Therefore a lot of functionality (such as making api calls) is restricted. to enable it you need to run it in unsafe mode. you can read more about it here

rabi-siddique · 2024-04-26T04:55:35Z

package.json

@@ -46,6 +46,7 @@
  "dependencies": {
    "@subql/types-cosmos": "^3.2.3",
    "@types/node": "^17.0.21",
+    "cross-fetch": "^4.0.0",
    "pino": "^7.8.0",


Why not use axios?

After comparing axios and cross-fetch, I think axios has more features compared to cross-fetch. It's easier to use, has automatic JSON parsing, and more popular :D

cross-fetch does have the advantage of being lightweight at 9.8kB minified, compared to axios's 30.5kB.

axios unfortunately does not work in the restricted environment properly. i've tried a node-fetch as well which did not work

only cross-fetch seems to work properly

rabi-siddique · 2024-04-26T04:59:52Z

schema.graphql

+type ReserveBalance @entity {
+  id: ID!
+  blockHeight: BigInt!
+  address: String!


@toliaqat I am not sure about it. But what if the blockHeight is much larger than what a BigInt type can handle? Should we also use a String type for blockHeight?

BigInt can store super long numbers such as: 10 ^(10^8) which i dont think we can physically reach

BigInt is good enough.

toliaqat · 2024-04-27T07:26:32Z

src/mappings/mappingHandlers.ts

@@ -40,6 +42,22 @@ BigInt.prototype.toJSON = function () {
  return this.toString();
 };

+const API_ENDPOINT = 'https://main-a.api.agoric.net:443'


There is a parameter in project.js called network which has an endpoint URL list. Project.js is complied into project.yaml. Is there a was way to read network.endpoint list instead of creating another variable here?

toliaqat · 2024-04-27T07:27:56Z

src/mappings/utils.ts

+  const response = await crossFetch(url);
+
+  if (response.status >= 400) {
+    const errorText = await response.text();


@rabi-siddique we look for errors in 400+ range. Here we are just checking for errors in response.

toliaqat

The more I think, i feel the API call for each block will slow down our indexing. I think instead of indexing this reserve balance, let's call the API in the inter-dashboard frontend when rendering the Reserve page. What do you think?

frazarshad · 2024-04-29T05:00:57Z

The more I think, i feel the API call for each block will slow down our indexing. I think instead of indexing this reserve balance, let's call the API in the inter-dashboard frontend when rendering the Reserve page. What do you think?

@toliaqat while i think thats a fair point, do you think it would be inconsistent that some of our api calls are made to the indexer while some are made to the chain

toliaqat · 2024-04-30T02:26:16Z

@toliaqat while i think thats a fair point, do you think it would be inconsistent that some of our api calls are made to the indexer while some are made to the chain

True, and this can be resolved when we have closed this issue #7
For now, I believe, we cannot afford to make our indexing slow down.
All blocks should all the data that we can index if done properly without calling the chain's RPC directly from this code base.

feat: added block handler

4dd9ddd

frazarshad requested a review from toliaqat April 25, 2024 17:02

chore: proper error handling for cross-fetch package

240fe23

frazarshad requested a review from rabi-siddique April 25, 2024 17:57

rabi-siddique reviewed Apr 26, 2024

View reviewed changes

toliaqat reviewed Apr 27, 2024

View reviewed changes

toliaqat requested changes Apr 27, 2024

View reviewed changes

toliaqat closed this Apr 30, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: added block handler to handle indexing of reserve balance #4

feat: added block handler to handle indexing of reserve balance #4

frazarshad commented Apr 25, 2024 •

edited

Loading

rabi-siddique Apr 26, 2024

rabi-siddique Apr 26, 2024

frazarshad Apr 26, 2024

frazarshad Apr 26, 2024

rabi-siddique Apr 26, 2024

rabi-siddique Apr 26, 2024

frazarshad Apr 26, 2024 •

edited

Loading

frazarshad Apr 26, 2024

rabi-siddique Apr 26, 2024

toliaqat Apr 27, 2024

rabi-siddique Apr 26, 2024

frazarshad Apr 26, 2024

rabi-siddique Apr 26, 2024

frazarshad Apr 26, 2024

rabi-siddique Apr 26, 2024

frazarshad Apr 26, 2024

toliaqat Apr 27, 2024

toliaqat Apr 27, 2024

toliaqat Apr 27, 2024

toliaqat left a comment

frazarshad commented Apr 29, 2024

toliaqat commented Apr 30, 2024

feat: added block handler to handle indexing of reserve balance #4

feat: added block handler to handle indexing of reserve balance #4

Conversation

frazarshad commented Apr 25, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

frazarshad Apr 26, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

toliaqat left a comment

Choose a reason for hiding this comment

frazarshad commented Apr 29, 2024

toliaqat commented Apr 30, 2024

frazarshad commented Apr 25, 2024 •

edited

Loading

frazarshad Apr 26, 2024 •

edited

Loading