A way of disallowing duplicate link bookmarks?

[Alanon202]

Nate, I’ve just opened a new issue with the visual suggestions like you suggested. I didn’t add any suggestions that pertain to features you already have planned.

Now, I’ve been testing out the new bookmark feature, I have two questions and thought it was better to put them here, instead of opening a second issue or resurrecting the issue you just closed.

First, is a minor tweak idea. I was wondering if you’d consider making the URL in the popup editable? For example, because of the existing preview of the URL I was able to notice tracking or referral parameters, and I prefer to remove them. It would be great if it was possible to just delete them before importing to Readeck.

Second is a more difficult thing to pull off, and I’m not sure if Readeck exposes a way to do this easily. Would it be possible for the bookmark popup to refuse to make a duplicate of an existing article and instead allow the user to edit the existing entry?

The use case for something like this would be, I just saved an article but forgot to add labels, or missed the timer and it saved before I made the adjustment. Or as I’m scrolling in the browser/newsfeed I change my mind and want to delete an article I just added, in all those instances it’s more convenient to reshare an article and edit/update/delete than it is to switch apps, find the article in a list then do what you want to do, then trace back to where you were.

Or alternatively, if this is too much or you feel the juice isn’t really worth the squeeze, would it be possible to somehow indicate in the bookmark popup if an article has already been saved? Thus far this has been a major nag with Readeck, since the browser extension doesn’t indicate this, and I’ve on multiple accidentally added the same article.

[NateEaton]

Stefan,

I was surprised the first time I noticed Readeck allows adding the same URL multiple times. I started to dig into a way to prevent it but realized that at least for now it can be useful in terms of dev/test - especially as I was working on the Add/View/Archive buttons for the Add Bookmark bottom sheet.

The author of Readeck feels strongly about keeping the content extracted from the source for each bookmark immutable which is why he doesn't support a way to re-load the content of a given bookmark. One implication of this is that the content for a given URL added today might look different than the content for a bookmark of the same URL added a year ago. So I get the value some might see in being able to add the same URL multiple times.

The API centers on the unique ID for each bookmark rather that the URL so I'm not sure duplicate detection directly in an API call is possible but I'll look into the feasibility of doing an internal check before adding a bookmark. I could see adding a setting for whether to allow duplicates with an add-time override capability. Or the ability to detect a duplicate and prompt for whether to add it again or update elements (title, labels, favorite, article status) of the existing one. The complexity for that would be the possibility that there might already be multiple instances of the same URL. At a minimum, I think a warning that the URL already exists is feasible as long as there's a way to efficiently detect there's already a bookmark with the same URL.

As for the ability to change the URL itself, that was an oversight on my part as that's possible with ReadeckApp. I already planned a fix for the intent-based add function today: a focus change if you click into the title or labels fields (and after today the URL field) stops the auto close timer but I want to allow any click including just tapping on the bottom sheet outside a field to stop the timer. I'll also fix the URL field so that it's editable.

Regards,
Nate

[NateEaton]

Stefan,

I was surprised the first time I noticed Readeck allows adding the same URL multiple times. I started to dig into a way to prevent it but realized that at least for now it can be useful in terms of dev/test - especially as I was working on the Add/View/Archive buttons for the Add Bookmark bottom sheet.

The author of Readeck feels strongly about keeping the content extracted from the source for each bookmark immutable which is why he doesn't support a way to re-load the content of a given bookmark. One implication of this is that the content for a given URL added today might look different than the content for a bookmark of the same URL added a year ago. So I get the value some might see in being able to add the same URL multiple times.

The API centers on the unique ID for each bookmark rather that the URL so I'm not sure duplicate detection directly in an API call is possible but I'll look into the feasibility of doing an internal check before adding a bookmark. I could see adding a setting for whether to allow duplicates with an add-time override capability. Or the ability to detect a duplicate and prompt for whether to add it again or update elements (title, labels, favorite, article status) of the existing one. The complexity for that would be the possibility that there might already be multiple instances of the same URL.

As for the ability to change the URL itself, I think that was an oversight on my part as that's possible with ReadeckApp. I already planned a fix for the intent-based add function today - a focus change if you click into the title or labels fields (and after today the URL field) stops the auto close timer but I want to allow any click including just tapping on the bottom sheet outside a field to stop the timer. I'll also fix the URL field so that it's editable.

Regards,
Nate

BTW, not sure if you noticed but using the FAB + to add a bookmark within the app now detects if there is a URL on the clipboard and if so prepopulates the form. It also tries to intelligently parse out a title if there is text followed by a URL. That could lead to inadvertently using clipboard contents if you had copied a bunch of text that happened to end in a URL but it handles when other apps share title followed by URL (for example, how Instapaper does).

[NateEaton]

On a related note, a feature I was working on but hadn't necessarily planned to share is a way to find articles that have errors, either a broken link or that content extraction failed on.

Given how far back my bookmarks from Pocket go, I have lots of those since when I imported them into Readeck there are ones that either no longer exist or that Readeck couldn't extract content from. They basically fall into these categories:
• Bookmarks where the link no longer works but it's because the site reorganized and it just exists at a different URL. A number of photography articles from places like 500px fall into this category.
• Ones where the site no longer exists but the content is available on archive.org. I've seen discussion threads on the Readeck repo with people asking for a way to automate extracting from archive.org to replace extracted content.
• Ones where the link works but for whatever reason Readeck had an issue when importing it, either getting a web page error or not being able to extract the content. For the ones where the bookmark is fine and there is no error but no extracted content to view, that was what led me to implement the feature where opening a bookmark to read automatically puts it in webview mode to show the original web content if there's no content available from the server.
This morning, I discovered that at least one instance like this does actually have content at the server, but something went wrong in MyDeck when it tried to sync the content for that bookmark. I have a backlog item to go into the bookmark detail dialog that's displayed when you click the info icon in reading view and add a button to have it try re-pulling content from the server. The button would only show up if the local metadata indicated no content available and a toast message would be displayed indicating whether the server responded with content or not. And, of course, if there actually was content to pull down, then in returning to the reading view the menu would now allow switching from original view to article view.

Across all of these, I have plenty of bookmarks that, in looking back now, I don't necessarily need any longer. Rather than going through 2400 bookmarks one at a time figuring out what to do with them, I want to start with low-hanging fruit, clear out ones that are dead links where their content isn't even available any longer, etc.

My first step in solving this will be to have a way to find the ones that have issues. I have at least 700 bookmarks that the API indicates have no content. Some of those may be videos or photos but I'm sure the rest fall into the categories above. My original plan was to just add some Gmail like search qualifiers that I could type into the existing search field to filter the main list. However, at the rate I've been able to get major features implemented, I'm tempted to just go ahead and do the Readeck-style filter panel sooner rather than later as it has filter controls for things like no content and bookmarks with errors.

Is this something you would find useful in the near term?

As for what to do with links that fall under the first category above, it obviously takes manual research to find the new link. Automating work on fixing these would be complex and a one-off so I will probably just manually work through fixing them up, but the one piece I would like to come up with a way to automate is for the "replacement" bookmark to have a way to set the date added for it to match the original date added metadata. Also having it have the same labels as the original would be good, too.

In terms of broken links where the content is still available on archive.org, I am playing around with the idea of putting a button on the bookmark detail page for ones where Readeck recorded an error during import due to being a broken link. Clicking the button would kick off something like the following:
• Do a background test of an archive.org URL built from the original URL and the date as of when the original bookmark was added.
• If that background check fails, pop up a dialogue offering to delete the current bookmark or to add a descriptive label like "unrecoverable" and mark it as archived.
• If the background check passes, then automatically

• add the new bookmark, setting it with the labels from the original bookmark plus a descriptive label like "archive.org"
• set the date added to the add date of the original bookmark (if possible) and then
• delete the original bookmark
  Given this would be a significant departure from the main function of this application, the other solution I'm considering is to do an export of all the bookmarks from Readeck, then create automation using the exported CSV to identify bookmarks with issues, and go through the steps above, then reload the server from the updated CSV. The one thing I've got to figure out is whether the Readeck export will give me information on which ones it didn't find content for and/or had errors with. If I can't get what I need from the Readeck export, I may put an enriched export option somewhere in the app, perhaps in the settings.

Is any of this functionality you would find useful? Thoughts on the proposed approaches?

[Alanon202]

Stefan,

I was surprised the first time I noticed Readeck allows adding the same URL multiple times. I started to dig into a way to prevent it but realized that at least for now it can be useful in terms of dev/test - especially as I was working on the Add/View/Archive buttons for the Add Bookmark bottom sheet.

The author of Readeck feels strongly about keeping the content extracted from the source for each bookmark immutable which is why he doesn't support a way to re-load the content of a given bookmark. One implication of this is that the content for a given URL added today might look different than the content for a bookmark of the same URL added a year ago. So I get the value some might see in being able to add the same URL multiple times.

Yes, Readeck has some quirks, but it’s probably the most performant of all these apps, and works so well that it’s worth it, IMHO. somehow trust the maintainer’s judgement. It took one look at Omnivore’s docker setup and one test of Linkwarden to know that Readeck is quite a different beast.

As for the ability to change the URL itself, I think that was an oversight on my part as that's possible with ReadeckApp. I already planned a fix for the intent-based add function today - a focus change if you click into the title or labels fields (and after today the URL field) stops the auto close timer but I want to allow any click including just tapping on the bottom sheet outside a field to stop the timer. I'll also fix the URL field so that it's editable.

Nice touch for the focus change! The first time I used the bookmark saver I’d missed the timer. I was going to mention something about it but I figured I’d wait a bit and see how I adapt to it.

BTW, not sure if you noticed but using the FAB + to add a bookmark within the app now detects if there is a URL on the clipboard and if so prepopulates the form. It also tries to intelligently parse out a title if there is text followed by a URL. That could lead to inadvertently using clipboard contents if you had copied a bunch of text that happened to end in a URL but it handles when other apps share title followed by URL (for example, how Instapaper does).

I did not notice that, but I’ll have to test it out a bit!. I can’t remember off the top of my head but there is an app that shares title plus url with the articles, that would be terrific if it can handle it out of the box.

---

On a related note, a feature I was working on but hadn't necessarily planned to share is a way to find articles that have errors, either a broken link or that content extraction failed on.

Given how far back my bookmarks from Pocket go, I have lots of those since when I imported them into Readeck there are ones that either no longer exist or that Readeck couldn't extract content from. They basically fall into these categories: • Bookmarks where the link no longer works but it's because the site reorganized and it just exists at a different URL. A number of photography articles from places like 500px fall into this category. • Ones where the site no longer exists but the content is available on archive.org. I've seen discussion threads on the Readeck repo with people asking for a way to automate extracting from archive.org to replace extracted content. • Ones where the link works but for whatever reason Readeck had an issue when importing it, either getting a web page error or not being able to extract the content. For the ones where the bookmark is fine and there is no error but no extracted content to view, that was what led me to implement the feature where opening a bookmark to read automatically puts it in webview mode to show the original web content if there's no content available from the server. This morning, I discovered that at least one instance like this does actually have content at the server, but something went wrong in MyDeck when it tried to sync the content for that bookmark. I have a backlog item to go into the bookmark detail dialog that's displayed when you click the info icon in reading view and add a button to have it try re-pulling content from the server. The button would only show up if the local metadata indicated no content available and a toast message would be displayed indicating whether the server responded with content or not. And, of course, if there actually was content to pull down, then in returning to the reading view the menu would now allow switching from original view to article view.

Across all of these, I have plenty of bookmarks that, in looking back now, I don't necessarily need any longer. Rather than going through 2400 bookmarks one at a time figuring out what to do with them, I want to start with low-hanging fruit, clear out ones that are dead links where their content isn't even available any longer, etc.

My first step in solving this will be to have a way to find the ones that have issues. I have at least 700 bookmarks that the API indicates have no content. Some of those may be videos or photos but I'm sure the rest fall into the categories above. My original plan was to just add some Gmail like search qualifiers that I could type into the existing search field to filter the main list. However, at the rate I've been able to get major features implemented, I'm tempted to just go ahead and do the Readeck-style filter panel sooner rather than later as it has filter controls for things like no content and bookmarks with errors.

Is this something you would find useful in the near term?

I think some aspects of your solutions will certainly be useful to me. I’m actually in the process of migrating my stuff and finding a new home for my bookmarks. First I was a Pocket user, then a paying customer, but then I figured out exactly what you did, Pocket was charging me for a couple of fonts and the privilege of still not archiving my links. So I promptly moved over to Wallabag and opted to support the project by hosting it with them rather than self host. I never did go back to Pocket.

Fast forward a couple of years, and I have 4k bookmarks and thousands of annotations and highlights in Wallabag. However, there seems to be less and less interest in making it better, upgrading its capabilities etc. The worst bit is that I’ve made all these annotations and highlights in Wallabag only to learn that in 5 years of using it they never added the possibility to export/reimport your annotations and with the activity on the project, they probably never will.

So I have all this prior work that I basically cannot migrate, since they won’t dump the DB for me, a random user. My last resort is to hope readeck can somehow add the option to import that content into it. I have toyed around with logseq and Obsidian, and I’m still trying to arrive at a setup that might allow me to retain links to the content and articles.

I even tried Readwise, and wanted to like all its features, even importeda ll of the highlights into it, but that thing is such a resource hog...

So yeah, I noticed several issues with the imports from Wallabag, flagged them to the team and to their credit they were extremely quick to deal with it. So I’d suggest you do pop in and try and see if can fix some of the parsing/import issues that fall under their purview. AFAIK the Readeck extension is at times much more powerful for retrieving pages as it parses the text on the opened page it doesn’t act like a bot. If push comes to shove, one way to deal with your issues is to brute force it, use puppeteer and script it to open URLs from a link into a new tab, then click the extension and load it up on Readeck that way. If a site fails, you could perhaps make a custom recipe for bypassing the error, which might work.

Now as it happens my situation is a bit different. The main plus of Wallabag was always that it by default saves the article text. During import, Readeck will apparently get the text from Wallabag, import it, then get all the images and download them as well and integrate it all. I did get errors on a small percentage of articles, there were text parsing issues with some articles, etc. minor stuff really.

So after manually fixing those issues, officially, I’m error free. I would still like to know whether this is true or are there edge cases not even Readeck sees. My primitive way of trying to solve this is by read time. Usually articles with errors or partially scraped ones are short to read. By sorting by length I was able to ferret out some of those exceptions.

As for what to do with links that fall under the first category above, it obviously takes manual research to find the new link. Automating work on fixing these would be complex and a one-off so I will probably just manually work through fixing them up, but the one piece I would like to come up with a way to automate is for the "replacement" bookmark to have a way to set the date added for it to match the original date added metadata. Also having it have the same labels as the original would be good, too.

1000+ on both the re-fetch idea and the original metadata! I tried to inquire about both with the Readeck team but sadly this isn’t something they’re particularly interested in. For me chronology is a great thing to have and all the fixes to my imports have disturbed that a bit. I would love to be able to restore their date.

Even worse, I know Readeck has a more refined parser than Wallabag on some websites. In many imported articles there are formatting blunders and errors like subscribe text that Wallabag picked up, and I know Readeck wouldn’t. It would be a great option to just have a refetch button on every article, that would try to refetch it and merge existing metadata.

In terms of broken links where the content is still available on archive.org, I am playing around with the idea of putting a button on the bookmark detail page for ones where Readeck recorded an error during import due to being a broken link. Clicking the button would kick off something like the following: • Do a background test of an archive.org URL built from the original URL and the date as of when the original bookmark was added. • If that background check fails, pop up a dialogue offering to delete the current bookmark or to add a descriptive label like "unrecoverable" and mark it as archived. • If the background check passes, then automatically

* add the new bookmark, setting it with the labels from the original bookmark plus a descriptive label like "archive.org"

* set the date added to the add date of the original bookmark (if possible) and then

* delete the original bookmark

The one thing I’d be worried about is the bot filter of archive. I know Wallabag never was able to scrape its links for the content. But since there isn’t really any other service that comes close, I suppose it’s worth a shot, but only once you check if indeed there are no issues with Readeck in bot mode and archive.org The extension I’m pretty sure would work.

  Given this would be a significant departure from the main function of this application, the other solution I'm considering is to do an export of all the bookmarks from Readeck, then create automation using the exported CSV to identify bookmarks with issues, and go through the steps above, then reload the server from the updated CSV. The one thing I've got to figure out is whether the Readeck export will give me information on which ones it didn't find content for and/or had errors with. If I can't get what I need from the Readeck export, I may put an enriched export option somewhere in the app, perhaps in the settings.

Is any of this functionality you would find useful? Thoughts on the proposed approaches?

I think your overall approach is good, trying to eliminate as many variables as possible and then start eliminating the root cause. You are, of course, correct in that most of these features are an integration on top of Readeck, and surpass the available options. I’m not sure whether you’d want that in the future, but that’s a different story.

A precondition for all these manipulations is, I think, a refetch settings and "preserve metadata" to embed post-hoc. Because if you restore a bunch of articles and the dates get scrambled and chronology lost, what’s the point.

[Alanon202]

@NateEaton One thing I just realised about the bookmark feature - it would be nice to get a way to fill existing tags? I mean a system where typing "his" would automatically suggest "history" and I could select that and have the list populate accordingly. I’m always worried that I’ll type the wrong letter and create a new unique tag for no reason.

On a different note, about the design and overall features like text, theming, animations, this app I think does a great job of it. It has the right amount of reading settings, properly implements Material You, has slick but quick animations, and it’s also a Kotlin app.. I know you can’t just lift the code or anything, but it could be a useful template of sorts.

[NateEaton]

Great idea. I'll add it as an enhancement issue. As for Material You, I do have a rework planned based on your earlier note but I'll also checkout ReadYou. I notice they acknowledge a bunch of other open source projects and they have a GPL license so there might be some opportunity to build on what they have done there in MyDeck.