Cascading calculations across a directed graph

[sancarn]

Hello,

Another pro-open source dev here 😄

I work for a UK water company, in waste water and one of the things I've been dreaming of for the last few years is a network risk system based on cascading calculations throughout a sewer network (directed graph).

The dream is to have some DSL which allows the definition of rules which looks like the following:

with Dwelling {
  population = 2.8
  flow = population * 160 / 1000 / 24 / 60 / 60  // m^3/s
}

with Manhole {
  population = self.upstream.map(e=>e.population).sum
  flow = self.upstream.map(e=>e.flow).sum
}

with Pipe {
  population = upstream.map(e=>e.population).sum
  flow = upstream.map(e=>e.flow).sum
  area = getArea(shape, width, height) 
  velocity = flow / area
  meetsSelfCleansing = 1.7 ≤ velocity
  pubCount = self.upstream.map(e=>e.pubCount).sum
  if(!meetsSelfCleansing) blockageLikelihood = pubCount/blockagePerPub
  meetsErosionLimit = velocity<3
  …
}

Attributes defined in the specified scope will be outputted as different columns in an output table. This DSL would be converted into some data flow:

And the calculations would ripple through the input network data, doing as many of the calculations in parallel as possible.

---

I was wondering, and I apologise for how open ended this is:

• Does SWMM / PySWMM allow for this kind of "custom property definition"? In the past I've mainly seen PySWMM examples which modify flow characteristics, rather than addition of and simulation of additional characteristics/attributes flowing throughout the network.
• If SWMM doesn't have this capability, do you know any software which might?
• If I were to create this myself, are there any learnings which you could advise e.g. technologies useful specifically for dealing with large model networks.

[cbuahin]

@sancarn, your question is not an open ended question at all. I can see how the capabilities you seek would be useful for several applications. Unfortunately, the SWMM API does not allow you to do this easily at the moment. However, you can combine the SWMM API (or parse the SWMM input file yourself) with a python package like Networkx to accomplish the goal you seek.

I would recommend you address specific questions about pyswmm to the developers of pyswmm on their repository. I hope this is helpful.

[sancarn]

However, you can combine the SWMM API (or parse the SWMM input file yourself) with a python package like Networkx to accomplish the goal you seek.

My initial thoughts were to rebuild the network in IndraDB. Having a graph database at my back should provide some nice performance, and especially used as a rust library, should have native speed too. The main concern with any technology really is running out of memory, which I can imagine might happen if one uses an in-memory graphdb.

I would recommend you address specific questions about pyswmm to the developers of pyswmm on their repository. I hope this is helpful.

Fair enough, I wasn't sure where the overlap was.

[cbuahin]

I am not familiar with IndraDB, but from the little I have read, it seems it will work. How big of a network are you dealing with? How many nodes and links? It must be huge if you are running out of memory.

[sancarn]

Just estimating here but it will probably be upwards of 10 million nodes and 20 million edges at full size? There may have been other reasons why I ran out of memory in previous experiments though.

[cbuahin]

Fascinating! That is indeed a pretty large model. I would like to chat about it offline if you have time. Let me know how I can reach out.

[sancarn]

I should mention that that spans across our entire waste water network for the company. When we actually perform simulations (usually inside ICM) we cut the models down specifically to hydraulically interacting areas. We have about 200 separate hydraulic models. Our biggest hydraulic model being 50k nodes and 50k links.

I'll message you on linkedin :)