Restore the useLocalCrawling & maxDepth settings for indexed documents

[vincentkelleher]

Description

This re-introduces the useLocalCrawling & maxDepth configuration parameters for document indexing as they were ignored since the JSON to YAML configuration migration.

Checklist

• I've read the contributing guide
• The relevant docs, if any, have been updated or created
• The relevant tests, if any, have been updated or created

Tests

DocsService tests are skipped and commented at the moment 👉 https://github.com/continuedev/continue/blob/bbb81ff032608e03a2208be908c1394da228ad6a/core/indexing/docs/DocsService.skip.ts

[cubic-dev-ai]

Your cubic subscription is currently inactive. Please reactivate your subscription to receive AI reviews and use cubic.

[netlify]

✅ Deploy Preview for continuedev ready!

Name	Link
🔨 Latest commit	16b80d9
🔍 Latest deploy log	https://app.netlify.com/projects/continuedev/deploys/68516b828b9d6c00083ce358
😎 Deploy Preview	https://deploy-preview-5958--continuedev.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

[vincentkelleher]

I have read the CLA Document and I hereby sign the CLA

[recurseml]

😱 Found 1 issue. Time to roll up your sleeves! 😱

[vincentkelleher]

All tests are green 🎉

So recurseml, is that enough sleeve rolling for you ? 😏 🤣

[vincentkelleher]

Bump

Could someone do a quick review of this PR ? 😇
It's very simple and is highly needed in my team 😊

[vincentkelleher]

This PR is now one month old, anyone to check it and merge it ? 😢

[RomneyDa]

@vincentkelleher do you need the maxDepth param specifically?
We want to merge this with useLocalCrawling but maybe deprecate maxDepth in favor of an allowList/blockList pattern

Apologies for the delays!

[vincentkelleher]

@RomneyDa I was just aware of maxDepth because it was there historically.

I imagine allowList/blockList would be a list of regex ?

Thanks for the feedback 😊

[RomneyDa]

Got it! So would it solve your issue if I merged and then removed maxdepth and kept uselocalCrawling?

(Or if you'd like to)

Yes, I think glob patterns for allow/block

[vincentkelleher]

@RomneyDa I have the feeling that maxDepth requires less thinking and is safer as you won't explicitly know how many pages will be indexed by each glob, don't you think ?

[RomneyDa]

we're thinking also about adding a maxPages to give a more direct limit, but I think people generally either want to index all docs that match a pattern (with a hard limit perhaps). maxDepth doesn't create any hard limit, it could yield tens of thousands of pages in somewhat edge casey scenarios

[RomneyDa]

would maxPages and useLocalCrawling be sufficient?
The other issue with maxDepth is it's not super clear how it works, i.e. as a dev I can't keep a 3-link-deep map of the docs pages I want in my head

[vincentkelleher]

It's true that there are clearly two types of limits:

• hard limits with maxPages
• soft limits with maxDepth, allowList and blockList

Usually having a max depth of 1 seems like a reasonable case as you want everything directly linked to the subject of the page, doing more than 1 or 2 would directly bring you to indexing the whole website in most cases IMHO. I also think that having an allow or block list would be about the same, if not worse, than a max depth over 1 as you will have to clearly know the sitemap.

Having a maximum number of pages would be a good guard-rail to avoid using too many hardware resources, that seems like a good feature 👍

I would go for useLocalCrawling, maxDepth and maxPages 😇

[RomneyDa]

@vincentkelleher appreciate the feedback! Do you currently have cases for which you set maxDepth > 1?

[vincentkelleher]

@RomneyDa I don't have any in mind right now 🤔

[RomneyDa]

Adds the maxDepth and useLocalCrawling params back in for YAML docs config

[RomneyDa]

Reinstates missing YAML docs config params

[RomneyDa]

After running by team opened a new PR to remove maxDepth for YAML, opened a ticket to add maxPages/allow/block list or similar to replace. Will leave useLocalCrawling. Thanks for the contribution!

[sestinj]

🎉 This PR is included in version 1.1.0 🎉

The release is available on:

• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot 📦🚀