msglist: Fetch newer messages despite previous muted batch #1989

sm-sayedi · 2025-11-15T07:57:18Z

Fixes: #1256

Screen recordings

First batch - Empty

Before

Empty.first.batch.-.Before.mov

After

Empty.first.batch.-.After.mov

First batch - Less than a screenful

Before

First.batch.with.few.messages.-.Before.mov

After

First.batch.with.few.messages.-.After.mov

The nested `try` block doesn't seem to be making any difference, so good to remove.

This change is necessary when there is a need to fetch more messages in both directions, older and newer, and when fetching in one direction avoids fetching in another direction at the same time, because of the `if (busyFetchingMore) return` line in both `fetchOlder` and `fetchNewer`. This scenario happens when a conversation is opened in its first unread, such that the number of messages in the initial batch is so low (because they're muted in one way or another) that it's already past the certain point where the scroll metrics listener in the widget code triggers both `fetchOlder` and `fetchNewer`. In 2025-11, that code first calls `fetchOlder` then `fetchNewer`, and for the reason mentioned above, `fetchNewer` will not fetch any new messages. But that's fine, because as soon as older messages from `fetchOlder` arrives, there will be a change in the scroll metrics, so `fetchNewer` will be called again, fetching new messages. But imagine if the number of messages in the initial batch occupies less than a screenful, and then `fetchOlder` returns no messages or a few messages that combined with the initial batch messages are still less than a screenful; in that case, there will be no change in the scroll metrics to call `fetchNewer`. With the three fetch request types being separated, especially the two request types for older and newer messages, each direction can fetch more messages independently without interfering with one another. Another benefit of this change is that if there's a backoff in one direction, it will not affect the other one. Fixes: zulip#1256

chrisbobbe

Thanks! Here's a review of the first three commits:

542996e msglist [nfc]: Mention fetchNewer in the MessageListView dartdoc
ce56456 msglist [nfc]: Add a new point to _MessageSequence.messages dartdoc
3c10bed msglist [nfc]: Remove one nested try block in _fetchMore

and a partial review of the fourth:

03d61ca msglist: Fetch newer messages despite previous muted batch

For that fourth commit, can you say briefly in the commit message why it's not a complete fix for the issue, to help orient the reader to what comes next?

chrisbobbe · 2025-11-21T00:08:05Z

lib/model/message_list.dart


      processResult(result);
+    } catch (e) {
+      hasFetchError = true;


msglist [nfc]: Remove one nested try block in `_fetchMore` The nested `try` block doesn't seem to be making any difference, so good to remove.

This isn't NFC: now, hasFetchError is about what happens when data is fetched and processed (with processResult), not just about what happens when data is fetched. That doesn't seem desirable, just from looking at its name.

chrisbobbe · 2025-11-21T01:09:37Z

lib/model/message_list.dart

+  /// The ID of the oldest known message so far in this narrow.
+  ///
+  /// This will be `null` if no messages of this narrow are fetched yet.
+  /// Having a non-null value for this doesn't always mean [haveOldest] is `true`.
+  ///
+  /// The related message may not appear in [messages] because it
+  /// is muted in one way or another.
+  int? get oldMessageId => _oldMessageId;


The ID of the oldest known message so far in this narrow.

There isn't necessarily a message with this ID in the narrow, though, right? In a quick skim, I don't see event-handling code to update _oldMessageId when the corresponding message is deleted or moved out of the narrow. We should avoid saying the message is in the narrow when it might not be.

gnprice

Thanks!

Similar to #1951 (review), here's some high-level comments before I go on vacation.

gnprice · 2025-11-26T00:39:14Z

lib/widgets/message_list.dart

        // TODO perhaps offer mark-as-read even when not done fetching?
-        MarkAsReadWidget(narrow: widget.narrow),
+        if (model.messages.isNotEmpty)
+          MarkAsReadWidget(narrow: widget.narrow),


What's the relationship of this change to the main changes happening in this commit? What's the user-visible change in behavior it causes?

So with the new changes in the commit, if the initial newest batch is all muted (model.messages.isEmpty), and then when older messages are being fetched, the "Mark as read" widget will be shown along with a progress indicator and no messages, so I thought it may be a good idea to hide "Mark as read" until there are visible messages populated.

Before (this one-line change) After ((this one-line change)

Hmm, I see.

What does that situation look like before this branch / before even the first batch is fetched? I believe we show a different progress indicator (centered), and don't show the mark-read button.

Can we have that behavior continue when we've had some fetch requests finish, but haven't yet actually found any messages? I think I'd like to think of that state as equivalent to when the initial fetch is still going.

IOW I'd like to think of it as "still working on fetching messages", like in the doc comment on MessageListView.fetched (in main):

/// Whether [messages] and [items] represent the results of a fetch. /// /// This allows the UI to distinguish "still working on fetching messages" /// from "there are in fact no messages here". bool get fetched => switch (_status) { FetchingStatus.unstarted || FetchingStatus.fetchInitial => false, _ => true, };

Yeah, I think that will be an improvement. It will also avoid the abrupt change from a centered progress indicator to a bottom-aligned indicator.

gnprice · 2025-11-26T00:44:02Z

lib/model/message_list.dart

+  FetchingInitialStatus _fetchInitialStatus = .unstarted;
+  FetchingMoreStatus _fetchOlderStatus = .unstarted;
+  FetchingMoreStatus _fetchNewerStatus = .unstarted;


Another benefit of this change is that if there's a backoff in one direction, it will not affect the other one.

Hmm — that sounds like a problem, not a benefit. 🙂 The fetches are all going to the same server, so if one type of fetch has trouble that suggests we should hold off on our next requests, then the same need applies to requests at the other end of the list.

(In fact ideally we'd be sharing backoff information across all requests on a given account. See also #946. But these are potentially some of the more frequent requests, so it's good that we at least share it among these.)

More broadly:

But imagine if the number of messages in the initial batch occupies less than a screenful, and then `fetchOlder` returns no messages or a few messages that combined with the initial batch messages are still less than a screenful; in that case, there will be no change in the scroll metrics to call `fetchNewer`.

This change feels like the wrong layer for solving this problem.

When the fetchOlder returns, that will notify the view-model's listeners, right? What if we have the _MessageListState react to that by looking to see if more fetching is needed, and calling fetchOlder/fetchNewer if so?

IOW, we could have its _modelChanged method call the same logic that's at the end of _handleScrollMetrics. Probably that means pulling that part out as its own smaller helper method.

gnprice · 2025-11-26T00:54:05Z

lib/model/message_list.dart

+    } while (visibleMessageCount < kMessageListFetchBatchSize / 2
+             && this.generation == generation);


If we have the widget state react in the way I suggested in my last comment just now, does that also let us skip these loops? We'd have the state retain responsibility for taking the initiative to call fetchOlder and fetchNewer when potentially needed.

sm-sayedi force-pushed the 1256-fetch-older-fails branch 4 times, most recently from 22e9f22 to f920082 Compare November 19, 2025 18:01

sm-sayedi added 3 commits November 20, 2025 15:17

msglist [nfc]: Mention fetchNewer in the MessageListView dartdoc

542996e

msglist [nfc]: Add a new point to _MessageSequence.messages dartdoc

ce56456

msglist [nfc]: Remove one nested try block in _fetchMore

3c10bed

The nested `try` block doesn't seem to be making any difference, so good to remove.

sm-sayedi force-pushed the 1256-fetch-older-fails branch from f920082 to a53cbfa Compare November 20, 2025 19:20

sm-sayedi marked this pull request as ready for review November 20, 2025 19:23

msglist: Fetch newer messages despite previous muted batch

03d61ca

sm-sayedi force-pushed the 1256-fetch-older-fails branch from a53cbfa to ef1fcc3 Compare November 20, 2025 19:50

sm-sayedi added the maintainer review PR ready for review by Zulip maintainers label Nov 20, 2025

sm-sayedi requested a review from chrisbobbe November 20, 2025 19:51

sm-sayedi assigned chrisbobbe Nov 20, 2025

sm-sayedi force-pushed the 1256-fetch-older-fails branch from ef1fcc3 to 127f690 Compare November 20, 2025 20:02

chrisbobbe reviewed Nov 21, 2025

View reviewed changes

gnprice reviewed Nov 26, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

msglist: Fetch newer messages despite previous muted batch #1989

msglist: Fetch newer messages despite previous muted batch #1989

Uh oh!

sm-sayedi commented Nov 15, 2025

Uh oh!

chrisbobbe left a comment •

edited

Loading

Uh oh!

chrisbobbe Nov 21, 2025

Uh oh!

chrisbobbe Nov 21, 2025

Uh oh!

gnprice left a comment

Uh oh!

gnprice Nov 26, 2025

Uh oh!

sm-sayedi Nov 26, 2025

Uh oh!

gnprice Nov 26, 2025

Uh oh!

sm-sayedi Nov 26, 2025

Uh oh!

gnprice Nov 26, 2025

Uh oh!

gnprice Nov 26, 2025

Uh oh!

gnprice Nov 26, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		} while (visibleMessageCount < kMessageListFetchBatchSize / 2
		&& this.generation == generation);

msglist: Fetch newer messages despite previous muted batch #1989

Are you sure you want to change the base?

msglist: Fetch newer messages despite previous muted batch #1989

Uh oh!

Conversation

sm-sayedi commented Nov 15, 2025

First batch - Empty

Before

After

First batch - Less than a screenful

Before

After

Uh oh!

chrisbobbe left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gnprice left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

chrisbobbe left a comment •

edited

Loading