upgrade to 0.0.3

Author: wesley1001

README.md

@@ -1,85 +1,97 @@
 BlobFs
 =====
 ![blobfs demo](doc/blobfs-demo.gif)
-BlobFs is a distributed [FUSE](http://fuse.sourceforge.net) based file system backed by [Microsoft azure blob storage service](https://azure.microsoft.com/en-us/services/storage/blobs/). It allows you to mount the containers/blobs in the storage account as a Linux local folder. It support the cluster mode.
+BlobFs is a distributed [FUSE](http://fuse.sourceforge.net) based file system backed by [Microsoft azure blob storage service](https://azure.microsoft.com/en-us/services/storage/blobs/). It allows you to mount the containers/blobs in the storage account as a the local folder/driver. , no matter it is a Linux system or a Windows system. It support the cluster mode. you can mount the blob container (or part of it) across multiple linux and windows nodes.
+
+## Important Notes:
+* Here is the linux/mac version of the blobfs, please find the windows version of the blobfs from [blobfs-win](https://github.com/wesley1975/blobfs-win).
+* For the core libraries of the blobfs, you can find it from the bloblib(https://github.com/wesley1975/bloblib). which is responsible for handling all the underlying azure blob storage operations
+* If you are interested in contributing, please contact me via [email protected]
 
 ## Project Goals
-The main goal of the project is to make azure storage service easy to use for Linux box.
+Object storage is one of the most fundamental topic you'll encounter when you decide to start your cloud journey.The main goal of the project is to make azure storage service easy to use for Linux and windows box.
+
+## Key Updates:
+base on the lots of feedbacks, in version 0.0.3, I made these major updates:
+* Ported the blobfs to windows platform. It became a universal solution. 
+* Improve the performance of list/rename/delete operation by enabling the multi-threading way. now it can list/rename/delete the thousands of items within few seconds.
+* By requests of many users, I changed the queue from service bus to the Azure Queue storage,This will greatly simplify the configuration.
+* Various bugs fixed, It is now more stable. you can use in production environment, but it's at your own risk.
 
 ## Features:
 * Implemented these fuse functions: getattr, readdir, open, release, read, flush, create, mkdir, rename, rmdir, unlink, truncate, write, symlink, readlink.
 * Allow mount multiple containers (or part of them) as the local folder.
 * Cluster enabled: It supports mount the same containers/blobs across multiple nodes. Theses files can be shared via these nodes. The caches of these nodes are synchronized via service bus.
 * Use blob leases as the distributed locking mechanism across multiple nodes. The blob will be locked exclusively when it is written. 
-* File’s attribute is cached for better performance, the cache are synchronized via service bus.
+* File’s attribute is cached for better performance, the cache are synchronized via azure queuq storage.
 * The contents are pre-cached by chunks when there is read operation. This will eliminate the times of http request and increase the performance greatly. 
 * Multi-part uploads are used for the write operation. Data is buffered firstly and then be uploaded if the buffer size exceed the threshold. This also can eliminate the times of http request and increase the performance greatly. 
 * You can edit the file content on the fly, especially recommend for the small file, It does not need download, edit and then upload.
 * Append mode is supported, you can append the new line to the existing blob directly. this is more friendly for logging operation. And it can change the block blob to append blob automatically.
 * Use server-side copy for move, rename operations, more efficient for big files and folders.
-* Support the link function
+* Support the symbol link function
 
 ## Architecture and introduction
 
 This is the logical architecture of blobfs:
 
 ![blobfs Logical Architecture](doc/blobfs-arch.jpg)
 * Blobfs uses the blob leases to safe the write operation in the distributed environment, there is a dedicated thread to renew the lease automatically.
-* For each of the node, there is local cache in it’s memory, the cache will store the file attributes. Once the file is changed by the node, the node will send a message to the topic of the service bus. And then other nodes will receive the notification via the dedicated subscription of the topic.
+* For each of the node, there is local cache in it’s memory, the cache will store the file attributes. Once the file is changed by the node, the node will send a message to the Azure Queue storage. And then other nodes will receive the message and process it.
 
 ## installation
-I strongly recommend to test and verify it in you environment before you use it.
+Installation now is very easy. But I strongly recommend to test and verify it in you environment before you use it. it's at your own risk.
 ### 1.Install fuse
     yum install fuse fuse-devel
-### 2.Install blobfs without cluster mode enabled
+### 2.Install blobfs
 #### 2.1 get the azure account connection string, refer this [link](https://docs.microsoft.com/en-us/azure/storage/storage-create-storage-account)
 #### 2.2 Edit configuration file: 
 	Open blobfs.conf
 	change the setting of :
     Storage_Connection_String = your-storage-account -connection-string
     blob_prefix = /  (e.g. /container1/folder1/)
     mount_point = /mnt/blobfs (make sure the path exists in you node)
-    cluster_enabled = false
-You can also modify other settings if needed
-### 3.Install blobfs with cluster mode enabled
-Additionally, you should do these actions
-#### 3.1 Create a service bus topic. Refer this [link](https://docs.microsoft.com/en-us/azure/service-bus-messaging/service-bus-create-topics-subscriptions)
-#### 3.2 create a subscription for each of your node, refer this [link](https://docs.microsoft.com/en-us/azure/service-bus-messaging/service-bus-create-topics-subscriptions)
-#### 3.3 Edit configuration file:
-	Open blobfs.conf, 
-	change the setting of :
-    Storage_Connection_String = your-storage-account -connection-string
-    blob_prefix = /  (e.g. /container1/folder1/) 
-    mount_point = /mnt/blobfs (make sure the path exists in you node)
     cluster_enabled = true
-    service_bus_connection_string = your-servicebus-connection-string
-    service_bus_topic = your-blobfs-topic
-    service_bus_subscription = subscription of the dedicated node
-	You can modify other settings if needed
+the cluster mode is enabled by default, You can also modify other settings if needed
+
 ### final.Start the blobfs service
-    nohup java -jar uber-blobfs-0.0.1-SNAPSHOT.jar
+    nohup java -jar blobfs-0.0.3jar
+It is highly recommended that you should use [supervisord](http://supervisord.org/) to manage the blobfs services.
 
 ## Tips
 * the block blob is read only by default. marked with read only flag. e.g. r--r--r--
 * the append blob is marked with the read and write flag. e.g rw-rw-rw-
 
+## How to create a append blob
+* CLI way
+	touch append.log	// this will create a empty block blob.
+	echo 'new line here' >> append.log  
+		//this will change the underlying block blob to append blob automatically. 
+		// you also can issue this command against a existing file, this also works, but the time depends on the size of the file.
+		
+* Programming way
+	FileWriter fw = new FileWriter("/mnt/blobfs/container1/append.log", true)  //java 1.7+
+	applendFile = open("/mnt/blobfs/container1/append.log",' a+') //python
+	...
+
 ## Performance Test
-* simply do a dd testing (single thread mode). 
+* simply do a dd testing (single thread mode).  so the performance depends on the machine and network.
 ![blobfs performance test](doc/blobfs-perf.gif)
 
 ## Dependency
 * FUSE (Filesystem in Userspace) is an OS mechanism for unix-like OS that lets non-privileged users create their own file systems without editing kernel code.
 * [Java Native Runtime (JNR)](https://github.com/jnr/jnr-ffi) is high-performance Java API for binding native libraries and native memory.
 * [jnr-fuse](https://github.com/SerCeMan/jnr-fuse) is FUSE implementation in Java using Java Native Runtime (JNR).
 
-## Limitation
+## Limitation and known issues:
 * Due to the overhead of fuse system, the performance will be expected slower than native file system. 
 * For the cp command, the blobfs will use read out - then write in to new blob mode. this will spent more time for large files/folders.
-* For the page blob, currently, blobfs doest not support the copy operation. it may casue file interruption.  
+* For the page blob, currently, should be, but it is not well tested yet. it may casue file interruption.  
 
 ## Supported platforms
 -Linux : x86, x64
 -MacOS (via osxfuse): x86, x64  (should be, but not tested yet)
+-windows: [blobfs-win](https://github.com/wesley1975/blobfs-win)
 
 ## Command Line Usage
     blobfs  -h
@@ -90,10 +102,11 @@ Additionally, you should do these actions
     -m,--mount-point <arg>   Desired local mount point for BlobFs.
     -o <arg>                 FUSE mount options
 
+	for more fuse mount options, you can find [here](http://manpages.ubuntu.com/manpages/xenial/man8/mount.fuse.8.html).
 
 ## License
 	Copyright (C) 2017 Wesley Wu [email protected]
-	This code is licensed under the The MIT License (MIT). 
+	This code is licensed under The General Public License version 3
 
 ## FeedBack
 	Your feedbacks are highly appreciated! :)

README.md.old

@@ -0,0 +1,99 @@
+BlobFs
+=====
+![blobfs demo](doc/blobfs-demo.gif)
+BlobFs is a distributed [FUSE](http://fuse.sourceforge.net) based file system backed by [Microsoft azure blob storage service](https://azure.microsoft.com/en-us/services/storage/blobs/). It allows you to mount the containers/blobs in the storage account as a the local folder/driver. It support the cluster mode.
+
+## Project Goals
+The main goal of the project is to make azure storage service easy to use for Linux box.
+
+## Features:
+* Implemented these fuse functions: getattr, readdir, open, release, read, flush, create, mkdir, rename, rmdir, unlink, truncate, write, symlink, readlink.
+* Allow mount multiple containers (or part of them) as the local folder.
+* Cluster enabled: It supports mount the same containers/blobs across multiple nodes. Theses files can be shared via these nodes. The caches of these nodes are synchronized via service bus.
+* Use blob leases as the distributed locking mechanism across multiple nodes. The blob will be locked exclusively when it is written. 
+* File’s attribute is cached for better performance, the cache are synchronized via service bus.
+* The contents are pre-cached by chunks when there is read operation. This will eliminate the times of http request and increase the performance greatly. 
+* Multi-part uploads are used for the write operation. Data is buffered firstly and then be uploaded if the buffer size exceed the threshold. This also can eliminate the times of http request and increase the performance greatly. 
+* You can edit the file content on the fly, especially recommend for the small file, It does not need download, edit and then upload.
+* Append mode is supported, you can append the new line to the existing blob directly. this is more friendly for logging operation. And it can change the block blob to append blob automatically.
+* Use server-side copy for move, rename operations, more efficient for big files and folders.
+* Support the link function
+
+## Architecture and introduction
+
+This is the logical architecture of blobfs:
+
+![blobfs Logical Architecture](doc/blobfs-arch.jpg)
+* Blobfs uses the blob leases to safe the write operation in the distributed environment, there is a dedicated thread to renew the lease automatically.
+* For each of the node, there is local cache in it’s memory, the cache will store the file attributes. Once the file is changed by the node, the node will send a message to the topic of the service bus. And then other nodes will receive the notification via the dedicated subscription of the topic.
+
+## installation
+I strongly recommend to test and verify it in you environment before you use it.
+### 1.Install fuse
+    yum install fuse fuse-devel
+### 2.Install blobfs without cluster mode enabled
+#### 2.1 get the azure account connection string, refer this [link](https://docs.microsoft.com/en-us/azure/storage/storage-create-storage-account)
+#### 2.2 Edit configuration file: 
+	Open blobfs.conf
+	change the setting of :
+    Storage_Connection_String = your-storage-account -connection-string
+    blob_prefix = /  (e.g. /container1/folder1/)
+    mount_point = /mnt/blobfs (make sure the path exists in you node)
+    cluster_enabled = false
+You can also modify other settings if needed
+### 3.Install blobfs with cluster mode enabled
+Additionally, you should do these actions
+#### 3.1 Create a service bus topic. Refer this [link](https://docs.microsoft.com/en-us/azure/service-bus-messaging/service-bus-create-topics-subscriptions)
+#### 3.2 create a subscription for each of your node, refer this [link](https://docs.microsoft.com/en-us/azure/service-bus-messaging/service-bus-create-topics-subscriptions)
+#### 3.3 Edit configuration file:
+	Open blobfs.conf, 
+	change the setting of :
+    Storage_Connection_String = your-storage-account -connection-string
+    blob_prefix = /  (e.g. /container1/folder1/) 
+    mount_point = /mnt/blobfs (make sure the path exists in you node)
+    cluster_enabled = true
+    service_bus_connection_string = your-servicebus-connection-string
+    service_bus_topic = your-blobfs-topic
+    service_bus_subscription = subscription of the dedicated node
+	You can modify other settings if needed
+### final.Start the blobfs service
+    nohup java -jar uber-blobfs-0.0.1-SNAPSHOT.jar
+
+## Tips
+* the block blob is read only by default. marked with read only flag. e.g. r--r--r--
+* the append blob is marked with the read and write flag. e.g rw-rw-rw-
+
+## Performance Test
+* simply do a dd testing (single thread mode). 
+![blobfs performance test](doc/blobfs-perf.gif)
+
+## Dependency
+* FUSE (Filesystem in Userspace) is an OS mechanism for unix-like OS that lets non-privileged users create their own file systems without editing kernel code.
+* [Java Native Runtime (JNR)](https://github.com/jnr/jnr-ffi) is high-performance Java API for binding native libraries and native memory.
+* [jnr-fuse](https://github.com/SerCeMan/jnr-fuse) is FUSE implementation in Java using Java Native Runtime (JNR).
+
+## Limitation
+* Due to the overhead of fuse system, the performance will be expected slower than native file system. 
+* For the cp command, the blobfs will use read out - then write in to new blob mode. this will spent more time for large files/folders.
+* For the page blob, currently, blobfs doest not support the copy operation. it may casue file interruption.  
+
+## Supported platforms
+-Linux : x86, x64
+-MacOS (via osxfuse): x86, x64  (should be, but not tested yet)
+
+## Command Line Usage
+    blobfs  -h
+    -b,--blob-prefix <arg>   The prefix of the blobs that will be used as the
+                          mounted BlobFS root (e.g., /container1/blob1/;
+                          defaults to /)
+    -h,--help                Print this help
+    -m,--mount-point <arg>   Desired local mount point for BlobFs.
+    -o <arg>                 FUSE mount options
+
+
+## License
+	Copyright (C) 2017 Wesley Wu [email protected]
+	This code is licensed under the The MIT License (MIT). 
+	
+## FeedBack
+	Your feedbacks are highly appreciated! :)

bin/blobfs-0.0.3.jar

@@ -0,0 +1,49 @@
+# ================================================================================
+# ++++++++++++++++++ blobfs configurations  +++++++++++++++++++++++++++++++++++++++
+# ================================================================================
+# Azure storage account connection string
+Storage_Connection_String = DefaultEndpointsProtocol=https;AccountName=dev2107;AccountKey=wJNRBizfsyQ2JUBBBJjBFAsxjBmvVpkK+61ZqKV/n3HaI/jSEdS3oGoyBGdmvBhVVeUy5S2hlGSAuipeyXXSww==;EndpointSuffix=core.chinacloudapi.cn
+
+# the name of the queue, which will be used for synchronizing the file cache across the nodes.
+queue_name = blobfs
+
+# the prefix of the blobs that will be used as the mounted blobfs root,
+# e.g., /container1/blob1/; defaults to /
+blob_prefix = /
+
+# Desired local mount point for BlobFs, linux and osx
+mount_point = /mnt/blobfs
+
+# Desired local mount point for BlobFs, windows
+win_mount_point = Y:\
+
+# the user id for the BlogFs, default value is the caller user id
+# default value is :-1.
+uid = -1
+
+# the group id for the BlogFs, default value is the group id of the caller
+# default value is :-1.
+gid = -1
+
+
+# In the Debug Mode, the debug messages will be displayed in the console, currently this option is ignored.
+debug_enabled =  true
+
+# supports five logging levels: TRACE < DEBUG < INFO < WARNING < ERROR.
+log_level =  ERROR
+
+# cache 
+cache_enabled = true
+
+# in seconds
+cache_TTL = 180
+
+# if one source of the blobs will be mounted by more than one host, you should enable the cluster mode
+# in the cluster mode, blobfs will manage the cache distributedly
+cluster_enabled = true
+
+# change the block blob to append blob automatically,
+# this will be triggered when you open a read only file with the append mode.
+# caution for large blob, this will consume more time
+# e.g., echo "new line" >> readonlyfile
+auto_change_block_blob_to_append_blob = true