Project goals

[EricTheMagician]

The goal of this project is to provide a quality fuse filesystem experience with google drive, that supports both upload and download, Team Drives, good support for home users witht assymetric internet connections, and most importantly, for it to bear the seal of approval from the Wife.

What currently works is a file upload / download ( writing / reading from GDrive )

• Upload supports resuming, and can be "streamed" while in the upload queue. Currently does not support writing in the middle of a file, whiich is normally fine if you are planning on using it for backups and storage and not as a real filesystem.

• Downloads can be cached to disk. Currently, the disk checking is only done once when the program is first started. This is the current feature being worked on.

• Updating the filesystem if an external change has occured on GDrive.

• If you would like to see a feature, please post in this issue for now.

Currently, the list of features that I plan on implementing (probably in order):

• Delete items when the download cache is full, not only on start up.
• Headless login. This is technically already supported, but not cleanly. You can open the link on a different computer and then wget or curl the link with the redirected page with localhost.
• Write documentation for building DriveFS.
• Link mongocxx statically with DriveFS as it's a pain to compile and most OSs don't provide mongocxx
• Write a read me

Long term goal (no particular order, but higher priority for rclone):

• Support rclone encryption natively. (This means that you should be able to decrypt your filesystem without a seperate rclone process on top of DriveFS to decrypt your filesystem ). This is a very long term goal, and comes with it's own set of challenges, particularly since I'm not fluent in GO and it's difficult for me to go through the rclone code.
• Encfs support
• Define a folder id as root
• Support resizing of the downloaded cache files.
• Support pinning files to drive.
• Adding DriveFS to the Arch User Repository

[somerandom48]

This is awesome! I am currently still running your GDriveF4JS as my main mount for GDrive. In my opinion it is still the best available option of mounting GDrive on Linux.

I love the idea of native rclone encryption, but could I request you also look into native encfs support as that is the encryption method that I am using, and I know other do as well.

Thanks for all your work.

[EricTheMagician]

Good to know that people still use my node-gdrive project. I found it had way too many issues for me was was unstable, so this should be much more stable, and run much more smoothly.

For encfs, it should be much simpler. The current incarnation of drivvefs is based on my acdfs + encfs, but I never released because it was also unstable, but i had a bug in the encfs portion. the name encrding/decoding worked, but the reading/writing was slightly different and made it incompatible.

I'll probably just commit the encfs code as is at somepoint and if someone wants to look into it, a pull request would be more than appreciated. Actually, the code is lost. It would have been a good reference for both encfs and rclone. So I might do encfs first since that's already in c/c++, but man it was not fun to go through understanding the ssl library.

[hjone72]

Really looking forward to this! I also found that your GDriveF4JS has been by far the best. Plus easiest for me to make changes to, after making a few small adjustments it is stable as a rock! My only problem with your old solution was the ls speeds.

I assume that is because of the FUSE libraries you had to use?
Will this version be faster in that regard?

[EricTheMagician]

Thanks @hjone72! Can you elaborate on the speed of ls? I haven't used it in a while. I don't recall it being slow, so if you could give a more concrete example, I can compare on my current setup. I'm using rclone to decrypt and it's slows it down, but natively, it's quite fast. I'm also running it in debug mode under gdb. Finally, since people are still using the node version, what's the write throughput that you guys are getting, from copying a file to node gdrive. On my test laptop, it's super fast since I have an SSD but on my VPS, since it's mechanical, and I'm running it on unraid, it's super slow, like 10mbytes/s.

…

On Fri, Jun 1, 2018, 10:10 PM hjone72 ***@***.***> wrote: Really looking forward to this! I also found that your GDriveF4JS has been by far the best. Plus easiest for me to make changes to, after making a few small adjustments it is stable as a rock! My only problem with your old solution was the ls speeds. I assume that is because of the FUSE libraries you had to use? Will this version be faster in that regard? — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#1 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AATvbBG5kePrKWSbtX6_cLqQiLIp7MRdks5t4fQSgaJpZM4UWC8m> .

[hjone72]

hmmm not sure how best to explain it. I have a fairly large GDrive with lots of files and folders (200,000+). When software that scans over those files it can take quite a significant amount of time. Or when running ls -R it can hault for various periods. This could also be to do with the nature of node (being single threaded). I guess the slowness mostly occurs when trying to perform many actions at once...

Download speed is the most important for me, and the download speed with the current node version has been fantastic! I don't use the upload feature so can't comment.

[EricTheMagician]

@hjone72
Ok I see. I don't have such a large library on GDrive. ~10k files with gdb attached for debugging, and code compiled in debug mode (not reldeb), it takes about 3 secs using find to find all the files, and ~0.5 seconds if I pipe the output to wc (word count).

In the past, I had ~100k files on ACD, and using a similar filesystem, I managed to get it close to ~5 seconds to go through all those files, with code compiled in release, and I think I had got it down to about 1 second when I stopped using smart pointers.

So I'll go right ahead and assume that when you say a significant amount of time, it's way more than 10 seconds. Maybe 10-15 minutes, if not more?

[hjone72]

I have never actually let it finish, because it has taken so long. The very fast times you are talking about sound amazing!
Happy to run tests for you on my larger drive when you get to that stage.

As I said before, the most important things for me is having a cache and high download speed.

Really looking forward to trying this out!

[Gawthorne]

I found that when using the high-level libfuse API within node, listing files is magnitudes faster than when using the low-level API. I put it to the fact that the high-level API only calls node a few times whereas the low-level API is constantly calling hooks within node.. However the high-level API being synchronous is terrible for reads/writes due to all the IO blocking... So I found neither really ideal. :(

Just my two cents.

[EricTheMagician]

It's not using node. It's pure C++, so there shouldn't be any performance problems with the lowlevel api here.

I wrote some build instructions on the wiki:
https://github.com/thejinx0r/DriveFS/wiki/Compiling

I'm still testing it out.
Since I'm mostly testing uploads right now,
I'm not sure how it will behave once the cache is full while downloading.

It should clear it, but I haven't tested it yet.
Just note that it will show a message that the cache is full and it will start deleting, though it will only update the size once it has cleared up ~10% of the cache. So if you download or upload while the cache is clearing, it will look like the cache is increasing, when it's really not.

[Gawthorne]

I was referring to the slow-downs hjone72 was facing when using your node-gdrive project. But awesome! I attempted to get this to build a little while ago but couldn't find the special sauce to get it to compile. Thanks for the instructions.

…

On Wed., 4 Jul. 2018, 1:58 am Eric Yen, ***@***.***> wrote: It's not using node. It's pure C++, so there shouldn't be any performance problems with the lowlevel api here. I wrote some build instructions on the wiki: https://github.com/thejinx0r/DriveFS/wiki/Compiling I'm still testing it out. Since I'm mostly testing uploads right now, I'm not sure how it will behave once the cache is full while downloading. It should clear it, but I haven't tested it yet. Just note that it will show a message that the cache is full and it will start deleting, though it will only update the size once it has cleared up ~10% of the cache. So if you download or upload while the cache is clearing, it will look like the cache is increasing, when it's really not. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACCzZ8sFOnKHIy7Lypu9ftoL1YEdQVrHks5uC5SvgaJpZM4UWC8m> .

[hjone72]

Wondering if there is any chance you could post a sample config file?

[EricTheMagician]

@hjone72
https://github.com/thejinx0r/DriveFS/wiki/Sample-Config

[BerriJ]

Maybe you could add "Adding DriveFS to the AUR" as a long term goal?