FTP/SFTP support for loading data (JSON, CSV, XML)

[neo-technology-build-agent]

Issue by aethant
Wednesday Feb 22, 2017 at 01:49 GMT
Originally opened as neo4j-contrib/neo4j-apoc-procedures#292

---

Not sure whether this is a bug report, or a feature request or a request for clarification - is it possible to do an apoc.load call, using FTP/SFTP as a url? If so, how would you pass in the username and password?

I'm not really fluent in Java to delve into the code myself, but I'd imagine the underlying mechanics would involve opening the stream and reading it, just like any other file. Perhaps username and password could be passed in with the options object, just like the separator?

If it is currently possible, can someone point me in the right direction of how? If not, can I please officially request support for this? It'd save like 3 steps on a project I'm currently working on.

[neo-technology-build-agent]

Comment by abhishekunotech
Wednesday Feb 22, 2017 at 10:54 GMT

---

@aethant : If you do want to delve into the Java of this, Current implementation of apoc.load eventually traces back to creation of a HTTPClient Instance in Java using the URL you pass to apoc.load()...,

You can use an FTPClient instance of Java, in its place, if you have the resources to implement that and re-compile APOC locally.

[neo-technology-build-agent]

Comment by sarmbruster
Wednesday Feb 22, 2017 at 11:03 GMT

---

I think it should be easy to change https://github.com/neo4j-contrib/neo4j-apoc-procedures/blob/3.1/src/main/java/apoc/export/util/FileUtils.java#L18 to also check for ftp. By default the JVM does have URLStreamHandlers for http, https and ftp aboard. For any other protocols it is possible to supply additional URLStreamHandlers to the JVM, see https://docs.oracle.com/javase/8/docs/api/java/net/URL.html#URL-java.lang.String-java.lang.String-int-java.lang.String-

I think it's actually a good idea to change apoc code to be agnostic of url schemes and leave it up to the JVM if it can be loaded or not.

[neo-technology-build-agent]

Comment by abhishekunotech
Wednesday Feb 22, 2017 at 11:12 GMT

---

@sarmbruster, it will still require changes at https://github.com/neo4j-contrib/neo4j-apoc-procedures/blob/3.1/src/main/java/apoc/util/Util.java#L223, to create Clients specific to protocol.

It might require a bit more than a simple patch, to rework the requests to be url-scheme agnostic.

But, I see where you would want to go with it.

[neo-technology-build-agent]

Comment by jexp
Thursday Jun 29, 2017 at 07:45 GMT

---

FTP at least should be easy to check via ftp:// URLs
Not sure about URL-Handlers for sftp