Adds a dynamic group mirroring the file system (#10930)

[ErwanGou]

Closes #10930

Changed or Added files

• DirectoryGroup.java
• DirectoryGroupTest.java
• GroupNodeViewModel.java
• GroupDialog.fxml
• GroupDialogView.java
• GroupDialogViewModel.java
• GroupDialogViewModelTest.java
• JabRef_en.properties
• DirectoryUpdateListener.java
• DirectoryUpdateMonitor.java
• DummyDirectoryUpdateMonitor.java
• DefaultDirectoryUpdateMonitor.java
• JabRefGUI.java
• GroupTreeViewModel.java
• GroupTreeViewModelTest.java
• GroupTreeView.java
• CHANGELOG.md

The class DirectoryGroup.java contains the new type of group. There are some tests in DirectoryGroupTest.java but most of the feature cannot be tested because it needs the WatchService.

The changes in GroupNodeViewModel.java allow directory groups to be edited or removed. The root node of the mirrored structure can be dragged.

The changes in GroupDialog.fxml, GroupDialogView.java and GroupDialogViewModel.java add the button to mirror the user's local structure within JabRef. The GroupDialogViewModelTest.java was updated to fit the new constructor and to test the new feature.

The new version of JabRef_en.properties contains the English text displayed to the user when trying to mirror its local structure.

The files DirectoryUpdateListener.java, DirectoryUpdateMonitor.java, DummyDirectoryUpdateMonitor.java and DefaultDirectoryUpdateMonitor.java implement a watcher for directories. The file JabRefGUI.java is now able to initialize this watcher.

The file GroupTreeViewModel.java can now create a directory structure. It needed new attributes in the constructor so GroupTreeViewModelTest.java has been updated.

The changes in GroupTreeView.java correct a bug : the user could use "Sort subgroups Z-A" on groups that are not sortable.

In the file CHANGELOG.md there is the description of what was changed in the code.

Next steps

There is an issue with the watcher on Windows : the user can't rename, move or delete a local directory which contains a folder that is not empty because it is detected as opened in an application.

I chose not to allow the user to add entries in a directory group because it is supposed to work only automatically, but due to this choice they are not sortable. About this point I think the existing implementation of sortAlphabeticallyRecursive() and sortReverseAlphabeticallyRecursive() in GroupTreeViewModel.java is a bit weird :

• It is recursive but never checks if the subgroup is sortable so we can actually sort Directory Groups if we sort the AllEntries node.
• Furthermore I didn't get why it is mandatory that group.canAddEntriesIn() should be true because it seems the sorted method never uses that.

Steps to test

Start JabRef and open a library, empty or not :

You can add a new group by clicking on the 'Add group' button. A dialog should show up. Select 'Directory structure' under the 'Collect by' title :

Here you can select the root folder of the structure you want to mirror by clicking on the folder icon button :

Once the root is set it should automatically fill the 'Root path'. It also fills the 'Name' of the group if and only if it was empty and set the hierarchical context to 'Union', which means a group contains the entries of its subgroups -it is not mandatory to use that but it is recommended because it is how local folders work. You can also write or paste the root path instead of selecting the folder but remember that the 'OK' button won't be active until a valid path and a valid name are provided. You can add a description, an icon, a color or even change the hierarchical context if you want :

Then click on the 'OK' button, your local structure should appear :

The entries created with the local PDFs should appear a little while later :

Now feel free to test the feature. You can either add, delete, rename or move PDFs or folders in your local directory structure, you should see the consequences on your library groups. You can also edit those groups with a right-click. Remember that if you remove a group or change its type it won't adapt to your local changes anymore. If you create an entry that has a file within the mirrored structure it should automatically be added to the respective group. The feature also works if you have multiple opened libraries.

Mandatory checks

• I own the copyright of the code submitted and I license it under the MIT license
• I manually tested my changes in running JabRef (always required)
• I added JUnit tests for changes (if applicable)
• I added screenshots in the PR description (if change is visible to the user)
• I described the change in CHANGELOG.md in a way that is understandable for the average user (if change is visible to the user)
• I checked the user documentation: Is the information available and up to date? If not, I created an issue at https://github.com/JabRef/user-documentation/issues or, even better, I submitted a pull request updating file(s) in https://github.com/JabRef/user-documentation/tree/main/en.

[github-actions]

Hey @ErwanGou!

Thank you for contributing to JabRef! Your help is truly appreciated ❤️.

We have automatic checks in place, based on which you will soon get automated feedback if any of them are failing. We also use TragBot with custom rules that scans your changes and provides some preliminary comments, before a maintainer takes a look. TragBot is still learning, and may not always be accurate. In the "Files changed" tab, you can go through its comments and just click on "Resolve conversation" if you are sure that it is incorrect, or comment on the conversation if you are doubtful.

Please re-check our contribution guide in case of any other doubts related to our contribution workflow.

[koppor]

Proposal to refinement, because "structure" alone is a very vague term.

Suggested change

	- We added groups that mirror the local structure. [#10930](https://github.com/JabRef/jabref/issues/10930)
	- We added groups that mirror the local file directory structure. [#10930](https://github.com/JabRef/jabref/issues/10930)

[ErwanGou]

Thank you for your review, I applied the change.

[ErwanGou]

I recently added the last features and modified the PR message. Apart from what I explained in the Next steps section --which can't be fixed-- it should work properly. I won't commit new code anymore to this branch, you can review it freely now.

[InAnYan]

@ErwanGou can you fix merge conflicts in CHANGELOG.md? If you believe that no more commits is needed, just keep track on conflicts and our review

[InAnYan]

CachyOS (Arch Linux) x86_64.

I can't add a PDF to an empty DirectoryGroup.

Unfortunately, getting error:

java.lang.UnsupportedOperationException: Don't know how to serialize grouporg.jabref.model.groups.DirectoryGroup
	at org.jabref.jablib/org.jabref.logic.exporter.GroupSerializer.serializeGroup(GroupSerializer.java:152)
	at org.jabref.jablib/org.jabref.logic.exporter.GroupSerializer.serializeTree(GroupSerializer.java:119)
	at org.jabref.jablib/org.jabref.logic.exporter.GroupSerializer.serializeTree(GroupSerializer.java:123)
	at org.jabref.jablib/org.jabref.logic.exporter.MetaDataSerializer.serializeGroups(MetaDataSerializer.java:152)
	at org.jabref.jablib/org.jabref.logic.exporter.MetaDataSerializer.lambda$getSerializedStringMap$9(MetaDataSerializer.java:74)
	at java.base/java.util.Optional.ifPresent(Optional.java:178)
	at org.jabref.jablib/org.jabref.logic.exporter.MetaDataSerializer.getSerializedStringMap(MetaDataSerializer.java:73)
	at org.jabref.jablib/org.jabref.logic.exporter.BibDatabaseWriter.writeMetaData(BibDatabaseWriter.java:281)
	at org.jabref.jablib/org.jabref.logic.exporter.BibDatabaseWriter.writePartOfDatabase(BibDatabaseWriter.java:240)
	at org.jabref.jablib/org.jabref.logic.exporter.BibDatabaseWriter.writeDatabase(BibDatabaseWriter.java:184)
	at org.jabref/org.jabref.gui.exporter.SaveDatabaseAction.saveDatabase(SaveDatabaseAction.java:276)
	at org.jabref/org.jabref.gui.exporter.SaveDatabaseAction.save(SaveDatabaseAction.java:240)
	at org.jabref/org.jabref.gui.exporter.SaveDatabaseAction.save(SaveDatabaseAction.java:215)
	at org.jabref/org.jabref.gui.exporter.SaveDatabaseAction.save(SaveDatabaseAction.java:83)
	at org.jabref/org.jabref.gui.exporter.SaveAction.execute(SaveAction.java:58)
	at org.jabref/org.jabref.gui.actions.JabRefAction.lambda$new$1(JabRefAction.java:25)
	at [email protected]/org.controlsfx.control.action.Action.handle(Action.java:423)
	at [email protected]/org.controlsfx.control.action.Action.handle(Action.java:64)
	at [email protected]/com.sun.javafx.event.CompositeEventHandler.dispatchBubblingEvent(CompositeEventHandler.java:86)
	at [email protected]/com.sun.javafx.event.EventHandlerManager.dispatchBubblingEvent(EventHandlerManager.java:232)
	at [email protected]/com.sun.javafx.event.EventHandlerManager.dispatchBubblingEvent(EventHandlerManager.java:189)
	at [email protected]/com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(BasicEventDispatcher.java:58)
	at [email protected]/com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(EventDispatchChainImpl.java:114)
	at [email protected]/com.sun.javafx.event.EventUtil.fireEventImpl(EventUtil.java:74)
	at [email protected]/com.sun.javafx.event.EventUtil.fireEvent(EventUtil.java:49)
	at [email protected]/javafx.event.Event.fireEvent(Event.java:199)
	at [email protected]/javafx.scene.control.MenuItem.fire(MenuItem.java:459)
	at [email protected]/com.sun.javafx.scene.control.ControlAcceleratorSupport.lambda$doAcceleratorInstall$0(ControlAcceleratorSupport.java:215)
	at [email protected]/com.sun.javafx.scene.KeyboardShortcutsHandler.processAccelerators(KeyboardShortcutsHandler.java:383)
	at [email protected]/com.sun.javafx.scene.KeyboardShortcutsHandler.dispatchBubblingEvent(KeyboardShortcutsHandler.java:163)
	at [email protected]/com.sun.javafx.event.CompositeEventDispatcher.dispatchBubblingEvent(CompositeEventDispatcher.java:59)
	at [email protected]/com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(BasicEventDispatcher.java:58)
	at [email protected]/com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(EventDispatchChainImpl.java:114)
	at [email protected]/com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(BasicEventDispatcher.java:56)
	at [email protected]/com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(EventDispatchChainImpl.java:114)
	at [email protected]/com.sun.javafx.event.EventUtil.fireEventImpl(EventUtil.java:74)
	at [email protected]/com.sun.javafx.event.EventUtil.fireEvent(EventUtil.java:54)
	at [email protected]/javafx.scene.Scene.processKeyEvent(Scene.java:2257)
	at [email protected]/javafx.scene.Scene$ScenePeerListener.keyEvent(Scene.java:2791)
	at [email protected]/com.sun.javafx.tk.quantum.GlassViewEventHandler$KeyEventNotification.get(GlassViewEventHandler.java:176)
	at [email protected]/com.sun.javafx.tk.quantum.GlassViewEventHandler$KeyEventNotification.get(GlassViewEventHandler.java:120)
	at [email protected]/com.sun.javafx.tk.quantum.QuantumToolkit.runWithoutRenderLock(QuantumToolkit.java:424)
	at [email protected]/com.sun.javafx.tk.quantum.GlassViewEventHandler.handleKeyEvent(GlassViewEventHandler.java:204)
	at [email protected]/com.sun.glass.ui.View.handleKeyEvent(View.java:563)
	at [email protected]/com.sun.glass.ui.View.notifyKey(View.java:1004)
	at [email protected]/com.sun.glass.ui.gtk.GtkApplication._runLoop(Native Method)
	at [email protected]/com.sun.glass.ui.gtk.GtkApplication.lambda$runLoop$0(GtkApplication.java:240)
	at java.base/java.lang.Thread.run(Thread.java:1447)

[ErwanGou]

@InAnYan
I couldn't reproduce this error --maybe because I use Windows, I tried all the ways to add a PDF to an empty group and it never failed-- but thanks to the error message I could find where it came from and it should be fixed : the class GroupSerializer.java build a string representation of a group but it was not implemented for DirectoryGroup.
However I just discovered that this kind of string is used to create a group from scraps in GroupsParser.java so a new error will probably show up. I will need to modify tons of constructors to make it work properly because I need a DirectoryUpdateMonitor and an entire BibDatabaseContext --not only the Metadata-- to initialize the new group type. I will do it shortly but I won't commit that until someone asks me to. I really think that there is no other choice but I am a bit reluctant to modify so many constructors.

[ErwanGou]

@InAnYan
I think I fixed the issue with GroupsParser.java. As I said I had to modify many constructors so in total it impacted 87 files. It should work properly now though I can't test it because of an error during :jabref:run. This error was fixed recently in the main repository but I can't pull due to conflicts with two of the 87 files I modified : I need to commit first and then resolve the conflicts. I won't commit those changes until I get your approbation.

[InAnYan]

I will not be available during next 2-3 hours, so, if you have time fixing the conflicts now, you can do this before my aprobation.

Thanks for looking into the issue!

[ErwanGou]

In fact I managed to pull the most recent version of Jabref without committing my changes so I can now run :jabref:run. I tested the app and at first glance it works correctly so the files I modified shouldn't have brought new errors.

[ErwanGou]

@InAnYan
I recently fixed a slight bug with the user interface. Since the PDF import is a background task, when a DirectoryGroup is created it first doesn't contain any entry, so after its creation it is automatically selected and no entry is shown. Now, after the background task has finished, the new group is selected again to display the entries. I also added a message for the user, I don't know if it is relevant.
I still haven't commited the 87 files that fix your bug, I'm afraid that it will make your task more complicated, as there would be a lot of changes. Please, tell me when it's ok for you.

[InAnYan]

Thanks for working on this PR, but, sadly, there is a new error

Please check your library file for wrong syntax.

Group tree could not be parsed. If you save the BibTeX library, all groups will be lost. Caused by: Unknown group: DirectoryGroup:/mnt/data/test/with_pdfs;2;/mnt/data/test/with_pdfs;1;;;;

[InAnYan]

Error log:

Could not parse entry
java.io.IOException: Error in line 20: Expected { or ( but received B
	at org.jabref.jablib/org.jabref.logic.importer.fileformat.BibtexParser.consume(BibtexParser.java:1197)
	at org.jabref.jablib/org.jabref.logic.importer.fileformat.BibtexParser.parseEntry(BibtexParser.java:712)
	at org.jabref.jablib/org.jabref.logic.importer.fileformat.BibtexParser.parseAndAddEntry(BibtexParser.java:339)
	at org.jabref.jablib/org.jabref.logic.importer.fileformat.BibtexParser.parseFileContent(BibtexParser.java:265)
	at org.jabref.jablib/org.jabref.logic.importer.fileformat.BibtexParser.parse(BibtexParser.java:183)
	at org.jabref.jablib/org.jabref.logic.importer.fileformat.BibtexParser.parseEntries(BibtexParser.java:148)
	at org.jabref.jablib/org.jabref.logic.importer.Parser.parseEntries(Parser.java:18)
	at org.jabref.jablib/org.jabref.logic.importer.fileformat.pdf.PdfVerbatimBibtexImporter.importDatabase(PdfVerbatimBibtexImporter.java:34)
	at org.jabref.jablib/org.jabref.logic.importer.fileformat.pdf.PdfMergeMetadataImporter.extractCandidatesFromPdf(PdfMergeMetadataImporter.java:103)
	at org.jabref.jablib/org.jabref.logic.importer.fileformat.pdf.PdfMergeMetadataImporter.importDatabase(PdfMergeMetadataImporter.java:82)
	at org.jabref.jablib/org.jabref.logic.importer.fileformat.pdf.PdfImporter.importDatabase(PdfImporter.java:52)
	at org.jabref.jablib/org.jabref.logic.importer.fileformat.pdf.PdfMergeMetadataImporter.importDatabase(PdfMergeMetadataImporter.java:175)
	at org.jabref.jablib/org.jabref.logic.externalfiles.ExternalFilesContentImporter.importPDFContent(ExternalFilesContentImporter.java:24)
	at org.jabref/org.jabref.gui.externalfiles.ImportHandler$1.call(ImportHandler.java:147)
	at org.jabref/org.jabref.gui.externalfiles.ImportHandler$1.call(ImportHandler.java:109)
	at org.jabref/org.jabref.gui.util.UiTaskExecutor$1.call(UiTaskExecutor.java:188)
	at [email protected]/javafx.concurrent.Task$TaskCallable.call(Task.java:1407)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:328)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:545)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:328)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1095)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:619)
	at java.base/java.lang.Thread.run(Thread.java:1447)