You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Agent tools: let navigate return before the page settles (waitUntil)
#361
Hi! I've been driving Lilo through lilo-mcp (0.21.0) for scraping-style tasks and compared it with a Playwright-style browser on the same searches. Lilo held up much better against bot walls, but almost all the time difference came from one place: navigate always waits for the page to settle.
What I measured
waitForLoad (Lilo/Agent/AgentTools.swift) waits for the load, then for no request in flight, a still DOM (200 ms) and a quiet network (500 ms), up to 5 s past the load. On pages that keep sending ads/analytics requests, that wait is spent after the data I need is already there:
Site / flow
navigate
javascript_toollocation.href = … + wait_for url
Coupang search (result list in the first HTML)
6.8 s
1.5 s
Naver Shopping search, nav only (median of 15)
6.48 s
1.38 s
Naver Shopping, nav + scroll-collect all items (median)
8.98 s
4.05 s
Both paths extracted the same items. On Coupang the product list was complete at load. On Naver the rest of the list only loads on scroll, so settling doesn't add data there either.
The workaround works, but it loses what makes navigate nice: the final URL/title, redirect following, and the "URL did not change" warning. It also has to be explained to every agent.
Proposal
An optional waitUntil argument on navigate, default unchanged:
"settled" (default): today's behavior.
"load": return once the load finished (!engine.isLoading after it started), skipping the DOM/network quiet checks.
"commit" (maybe): return once the new URL is committed, for agents that will wait_for a selector anyway.
The reply would stay the same (final URL, title, warnings), so an agent that picks "load" and then wait_fors what it needs gets both speed and the safety net. It'd be a small change in the tool schema, the navigate case and waitForLoad, plus a test in LiloTests/Agent/.
Would you take a PR like this? Or would you rather tune the default wait itself, e.g. a shorter cap past load? I'm happy to implement either.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
Hi! I've been driving Lilo through
lilo-mcp(0.21.0) for scraping-style tasks and compared it with a Playwright-style browser on the same searches. Lilo held up much better against bot walls, but almost all the time difference came from one place:navigatealways waits for the page to settle.What I measured
waitForLoad(Lilo/Agent/AgentTools.swift) waits for the load, then for no request in flight, a still DOM (200 ms) and a quiet network (500 ms), up to 5 s past the load. On pages that keep sending ads/analytics requests, that wait is spent after the data I need is already there:navigatejavascript_toollocation.href = …+wait_forurlBoth paths extracted the same items. On Coupang the product list was complete at
load. On Naver the rest of the list only loads on scroll, so settling doesn't add data there either.The workaround works, but it loses what makes
navigatenice: the final URL/title, redirect following, and the "URL did not change" warning. It also has to be explained to every agent.Proposal
An optional
waitUntilargument onnavigate, default unchanged:"settled"(default): today's behavior."load": return once the load finished (!engine.isLoadingafter it started), skipping the DOM/network quiet checks."commit"(maybe): return once the new URL is committed, for agents that willwait_fora selector anyway.The reply would stay the same (final URL, title, warnings), so an agent that picks
"load"and thenwait_fors what it needs gets both speed and the safety net. It'd be a small change in the tool schema, thenavigatecase andwaitForLoad, plus a test inLiloTests/Agent/.Would you take a PR like this? Or would you rather tune the default wait itself, e.g. a shorter cap past
load? I'm happy to implement either.All reactions