Starting Anubis proof-of-work without enabling scripts for entire site

[jimbobmcgee]

There is a recently pervasive anti-scraping tool being deployed to certain websites which attempts to make the browser perform a JS-based proof-of-work before entering a website.

However noble that goal, it is deployed directly to the target website, and its artefacts are not served from a dedicated domain, which has the unfortunate side-effect of requiring enabling scripts for the domain of the protected website before one can reasonably evaluate the website content.

If one runs UBO with scripts disabled or with inline/1p/3p-scripts blocked, it renders the target website non-functional. If one enables/unblocks the scripts for the domain, any other script within that domain are unblocked.

In most cases, the specific scripts that would be run are all under a very specific $HOSTNAME/.within.website/x/cmd/anubis/ directory.

Obviously, dynamic rules such as...

• example.com https://example.com/.within.website/x/cmd/anubis/static/ script noop
• example.org https://example.org/.within.website/x/cmd/anubis/static/ script noop
• example.net https://example.net/.within.website/x/cmd/anubis/static/ script noop

...work well enough, but are tedious to maintain.

Is there any way to permit/no-op based on URL pattern, without having to duplicate the pattern for each target website?

I thought maybe something akin to one of...

• @@||*$path=/.within.website/x/cmd/anubis$_,script,xhr
• @@/.within.website/x/cmd/anubis^1p,script,xhr
• * ^.*[/][.]within[.]website[/]x[/]cmd[/]anubis[/].*$ script noop

...but I can't seem to get myself all the way there (static filters are observably not my strong suit!).

[freezer2022]

jimbobmcgee: Obviously, dynamic rules such as......work well enough, but are tedious to maintain.

Some related discussions/informations here:

• Implement support for hostname wildcards (*. and +.) in Dynamic URL filtering uBlock-issues#3682
• [Feature request] Add RegEx for dynamic filterrules uBlock-issues#2242
• Allow to set dynamic rules to the subdomain or domain in uBO panel uBlock-issues#730
• Unify dynamic filtering rule syntax uBlock-issues#685
• Add basic hostname wildcards to dynamic filtering rules uBlock-issues#439
• [Feature] Option to make dynamic rules use the page base domain for the rule source instead of the full hostname gorhill/uBlock#3297
• dynamic URL filtering + subdomain matching gorhill/uBlock#1281

jimbobmcgee: Is there any way to permit/no-op based on URL pattern, without having to duplicate the pattern for each target website? I thought maybe something akin to one of...

I misread a bit your request previously, therefore some other guy has already fixed your exception filters below my comment.

---

In general, blocking that script prevents the website from loading, therefore either it must be allowed, or perhaps it can be bypassed / hacked by some uBO scriptlets or redirection, by the way, the script appears on almost 28000 websites:
https://publicwww.com/websites/%22.within.website%2Fx%22/2

[jimbobmcgee]

I wondered about a possible scriptlet. That said, I don't think a scriptlet can override an existing dynamic rule (I run * * 1p-script block * * 3p-frame block * * 3p-script block * * inline-script block by default, and manually set domains to noop when I want to run scripts).

I started out by writing a GreaseMonkey userscript to recognise the interstitial page for Anubis, scrape and download the script file directly using XHR, and eval it (with suitable warnings/user prompts ahead of time).

It worked well enough for a while -- the GM XHR and eval are treated as behind-the-scenes by UBO, and is mostly enabled by default, even with the above dynamic rule.

But each iteration of Anubis seems to add another challenge. First it was its attempt to load UI languages as separate scripts, which I was able to overcome with a minor string-replace. Now it seems to also want to download a separate SHA2 hashing module, and it seems to do so that in a way that is not treated as behind-the-scenes any more.

So I am back to trying to override the dynamic rule with a path-based specifier.

[krystian3w]

The first two exception filters in the uBo/EasyList syntax have mistakes:

• path is not a valid network filter option https://github.com/gorhill/ublock/wiki/Static-filter-syntax#static-network-filtering (so the engine thinks it's a strange URI from the URL set)
• in the second one you started writing network filter options but ate $ (so they are considered part of the URI in the URL set).

Test these in the my filters tab:

@@/.within.website/x/cmd/anubis$_,script,inline-script,xhr
@@/.within.website/x/cmd/anubis^$1p,script,inline-script,xhr

To fix the third one, you may want to read: https://github.com/gorhill/ublock/wiki/Dynamic-URL-filtering (when Anubis resources are visible as blocked).

---

Looking at it Eclipse-Community/r3dfox#30, the inserted string is supposedly of type document or frame, not xhr or script.

[stephenhawk8054]

As said here: uBlockOrigin/uBlock-issues#2242 (comment)

Dynamic rules are designed to be as fast as possible, and with minimal overhead as possible. That's why it can be used as point-and-click and can be turned on/off to block/unblock instantly without causing any performance overhead.

Any further features should put the performance as the 1st, most important consideration before anything else. If the new features cause absolutely no further JS overhead, it can be considered.

Otherwise, things like wildcard, regex... would complicate and stress the performance more, which is opposite to the goal of dynamic rules.

If you want more complicated and complex adjustments, use static filters for that.

---

Up until now I still don't understand what you want to achieve, there's no example URLs and what are the steps to reproduce for us to give any inputs.

[jimbobmcgee]

@stephenhawk8054

Up until now I still don't understand what you want to achieve, there's no example URLs and what are the steps to reproduce for us to give any inputs.

OK. Consider a UBO instance running with...

* * 1p-script block
* * 3p-frame block
* * 3p-script block
* * inline-script block

Now, visit https://openwrt.org/ (or any of the 30K or so websites returned by https://publicwww.com/websites/%22.within.website%2Fx%2Fcmd%2Fanubis%22/).

You should see the Anubis interstitial page "checking if you are a bot".

It wants to do this by kicking off a JS function (to perform a bunch of SHA256 hashing), but can't because of the UBO settings (as expected).

Now, consider you want to allow the Anubis JS to run. You do this by greying (setting to noop) the explicit website domain (or 1p-script) in the UI and reloading. Anubis is passed, the website loads (again, as expected).

But, now, during that reload, any first-party script that happens to be in that domain is now also allowed to run.

There is no reasonable window (as in, time period) between Anubis completing and the rest of the domain's scripts running, in which one could reliably inspect the "real" content of the website and make an informed decision. Once 1p-script is greyed, the door is already open.

If you want more complicated and complex adjustments, use static filters for that.

As I understand it (thank to others in the discussion), static filters do not override dynamic rules, and there is no equivalent static filter that provides the above four rules that can itself be easily overridden by UI.

@krystian3w's above amendments to my work-in-progress filter patterns do accurately identify Anubis scripts, so I can use a /.within.website/x/cmd/anubis^$1p,all static filter to block Anubis, but I can't reason a way to tell a static filter to allow it when scripts are dynamically blocked.

The equivalent @@/.within.website/x/cmd/anubis^$1p,all does not match until after 1p-script is dynamically greyed, but that filter is no longer necessary by that point.

For a static filter to work, there would need to be some way to tell UBO to explicitly apply a noop as an override to the existing dynamic block. Conceptually, something like @@/.within.website/x/cmd/anubis^$1p,all,dynamic or @@/.within.website/x/cmd/anubis^$1p,all,noop.

[stephenhawk8054]

For the above site, you can open the logger and select which you want to noop based on URL

That's the most we can support. There are thousands of ways a site can implement their own captchas, we won't add any further supports just to tailor for each case, especially if the implementation means sacrificing the performance.

[jimbobmcgee]

@stephenhawk8054

you can open the logger and select...

Again, this doesn't apply until after 1p-script is greyed, by which point it is already too late.

There are thousands of ways a site can implement their own captchas, we won't add any further supports just to tailor for each case

I haven't asked you to tailor for specific captchas; I have demonstrated a gap in the available toolset which prevents me from solving this for myself. This discussion simply confirms that I haven't missed anything in my assessment of that gap.

The question still stands as to whether or not the current/growing popularity of tools such as Anubis is enough to reassess whether adding any support for overriding dynamic rules is useful enough in the general case.

At first glance, the options for doing so appear to be:

1. dynamic rules supporting wildcard hosts (e.g. * */path/to/resource script noop), so that path-based rules can be applied in My rules
2. static filters supporting a flag which can override dynamic (e.g. @@/path/to/resource^$noop), so that existing support for path-based logic can be used to override dynamic rules
3. An Anubis-specific scriptlet (discounted due to maintenance effort / understandable lack of desire for tailored cases)

...there may well be others I haven't considered.

especially if the implementation means sacrificing the performance.

Performance is obviously a consideration, but any added feature is always going to add some overhead, so this shouldn't be a factor until after determining if there is a need.

Now, it may be that the existence of * * 1p-script block means that a subsequent override (dynamic or static) couldn't ever be applied. It may be that intersection of (sites that deploy Anubis)∩(people who run * * 1p-script block) is infinitesimal so as not to bother (and I am just in the unlucky bracket). I don't think either are the case, but both are valid blockers.

If the answer is there is a need, but we can't do it (fast|reliably|well) enough, then so be it.

[stephenhawk8054]

Again, this doesn't apply until after 1p-script is greyed, by which point it is already too late.

Did you test it? I didn't need to noop 1p-script first. That script runs normally for me. This is what appears in the My rules pane when I just noop that script

* * 1p-script block
* * 3p-frame block
* * 3p-script block
* * inline-script block
openwrt.org https://openwrt.org/.within.website/ script noop

It's based on the priority of dynamic rules written in the wiki.

---

Performance is obviously a consideration, but any added feature is always going to add some overhead, so this shouldn't be a factor until after determining if there is a need.

It will always be the first factor for dynamic rules, regardless of your need.

---

The question still stands as to whether or not the current/growing popularity of tools such as Anubis is enough to reassess whether adding any support for overriding dynamic rules is useful enough in the general case.

No, it's not. As said in the comment linked above, any hostname wildcards will require changing the whole dynamic rules engine and it won't be performant any more.

Blocking 1p-script will always cause massive issues and breakages. There won't be further supports for users who are willing to take these issues.

---

Seems like my words don't hold any weights here. I will unsubscribe from this discussion. Any other volunteers who are willing to help can chime in. Good luck.

[jimbobmcgee]

@stephenhawk8054

Seems like my words don't hold any weights here

That's not the case at all, and I appreciate the discussion.

---

I didn't need to noop 1p-script first. That script runs normally for me

You are missing my point here, though (or I am not explaining it well enough). Consider the actual user workflow here when you encounter a webpage protected by this thing. Right now, you have to:

1. Either...
   1. Open UBO pop-up dialog, open Dashboard
   2. Navigate to My rules
   3. Build Anubis-specific rule based on URL and path...
      1. Go back to web page, copy URL
      2. Paste URL into Temporary rules
      3. Fix up paste so it only contains the domain/origin
      4. Paste URL again
      5. Fix up paste so it contains the base URL
      6. Add the magic incantation of /.within.website/x/cmd/anubis/ * noop
      7. Save temporary rule
   4. Go back to web page, reload
2. Or...
   1. Open UBO pop-up dialog
      1. Set 1p-scripts (or current domain) to noop/grey
      2. Open the logger
      3. Refresh the page (realistically stop here, because you've already let everything in already)
   2. Scroll the logger to find the /.within.website/x/cmd/anubis-prefixed entry; click it
   3. Navigate to URL rule
      1. Change the Type to * (or rinse/repeat for every resource type)
      2. Noop/grey the exact entry for https://example.com/.within.website/x/cmd/anubis/
      3. Refresh the page

Now do this every time you encounter this -- which, if you happen to be browsing search results in a particularly Linux-y subject -- is maybe 60% of all the results?

What fun.

And you end up with...

* * 1p-script block
* * 3p-frame block
* * 3p-script block
* * inline-script block
example-website-a.tld https://example-website-a.tld/.within.website/x/cmd/anubis/ * noop
example-website-b.tld https://example-website-b.tld/.within.website/x/cmd/anubis/ * noop
example-website-c.tld https://example-website-c.tld/.within.website/x/cmd/anubis/ * noop
example-website-d.tld https://example-website-d.tld/.within.website/x/cmd/anubis/ * noop
example-website-e.tld https://example-website-e.tld/.within.website/x/cmd/anubis/ * noop
example-website-f.tld https://example-website-f.tld/.within.website/x/cmd/anubis/ * noop
example-website-g.tld https://example-website-g.tld/.within.website/x/cmd/anubis/ * noop
example-website-h.tld https://example-website-h.tld/.within.website/x/cmd/anubis/ * noop
example-website-i.tld https://example-website-i.tld/.within.website/x/cmd/anubis/ * noop
...

...which becomes pretty unwieldy, pretty quickly.

It's bad enough for the reCAPTCHA challenges, hCaptcha challenges, Cloudflare challenges, etc. but at least those are coming from relatively-fixed domains and can be managed in a single pass.

These blasted things are coming from inside the house. You cannot noop them without nooping the whole site.

Whereas one of...

*   */.within.website/x/cmd/anubis/    *    noop
@@/.within.website/x/cmd/anubis/^$1p,all,noop

...would be far more manageable. Set once. Never a problem again.

If it can't be done, then it can't be done. But I'm still going to register an interest in it.

---

any hostname wildcards will require changing the whole dynamic rules engine

Fair enough re. dynamic rules. What about a $noop-like in static rules? Same problem...?

---

Looking at it from another angle, what about some sort of templating for generating dynamic rules based on patterns, instead of having to trawl through the logger each time one of these gates is raised?

The UI could present the pre-configured templates in a dropdown from the same pop-up dialog where the usual temporary overrides are displayed; the user action realises/vivifies the template substituting template values based on the current page, and adds to the list of dynamic rules.

The templates would be edited in advance from the dashboard, and would be at the behest of the end-user; I wouldn't expect UBO to ship any pre-written.

The performance issue is satiated because the patterns are resolved at edit-time, not per-request. The engine does not need to be changed, because the vivified patterns are just normal dynamic rules. The user workflow is reduced to a couple of clicks (popup, template dropdown, refresh).

[stephenhawk8054]

The engine does not need to be changed, because the vivified patterns are just normal dynamic rules.

Did you even read the comment in the link I gave you?

There is no tokenization taking place in dynamic filtering rules, it's all lookup tables using hostnames as keys, hence why it's fast compared to static filtering -- and using lookup tables means no possibility of regex or wildcard. I do not intend to change this.

---

I said

That's the most we can support

Blocking 1p-script will always cause massive issues and breakages. There won't be further supports for users who are willing to take these issues.

There is no point in arguing this further. If you are willing to take that much breakages, you are willing to take time to fix your own issues.

---

The UI could present the pre-configured templates in a dropdown from the same pop-up dialog where the usual temporary overrides are displayed; the user action realises/vivifies the template substituting template values based on the current page, and adds to the list of dynamic rules.

The templates would be edited in advance from the dashboard, and would be at the behest of the end-user; I wouldn't expect UBO to ship any pre-written.

If you really want this, feel free to make PRs and see whether gorhill accepts it. Don't make others do the work for your personal demands.

We will have to spend time and maintain these kinds of codes for the usage of only you here. Meanwhile, we have tons of other issues everyday to resolve in our little free time.

---

I said I will unsubscribe and won't discuss in this further. DON'T tag me please.