how to provide directories to python programs? #25

dckc · 2016-12-26T06:42:18Z

The example for cloudabi for FreeBSD shows passing a directory:

tmpdir: !file
  path: tmp-unittest

I tried to follow this example but I got:

% cloudabi-run /usr/local/x86_64-unknown-cloudabi/bin/python3 < zv.yaml
Fatal Python error: can't create sys.argdata
IsADirectoryError: [Errno 31] Is a directory: 3

Current thread 0x00007fffffffcd20 (most recent call first):
Abort

This is the yaml spec I tried:

% cat zv.yaml
%TAG ! tag:nuxi.nl,2015:cloudabi/
---
stderr: !fd stderr               # Let Python write backtraces to stderr.
path: !file                      # Search path for Python modules.
    path: /usr/local/x86_64-unknown-cloudabi/lib/python3.6
cwd: !file                       # Local (in-development) python modules.
    path: .
args:                            # Arguments exposed as sys.argdata.
    stuff: junk
    cwd: !file
        path: .
command: >-                      # Inline Python script, like 'python -c'.
    from sys import argdata, stderr;
    import main;
    print(argdata['cwd'], file=stderr)

The text was updated successfully, but these errors were encountered:

@dckc

Previous versions of Python allowed this, but newer ones seem to throw IsADirectoryError. This should fix issue #25. Reported by: @dckc

dckc · 2016-12-26T14:28:52Z

Thanks for the quick fix.

It's not clear to me how to try it out.

I'm running a FreeBSD vm, but pkg install x86_64-unknown-cloudabi-python-3.6.0_31 says "The most recent version of packages are already installed".

I suppose I should be able to check out this repository and build the one package for the one architecture. Here's hoping I can figure it out...

dckc · 2016-12-26T14:44:09Z

I'm using the FreeBSD-11.0-STABLE box. This environment doesn't seems to supply python in the expected way:

vagrant@:~/cloudabi-ports % python3 build_packages.py python
... build many other packages successfully...

INFO:src.package:BUILD ninja
INFO:src.distfile:CHECKSUM /home/vagrant/cloudabi-ports/_obj/distfiles/ninja/v1.7.1.tar.gz
WARNING:src.distfile:[Errno 2] No such file or directory: '/home/vagrant/cloudabi-ports/_obj/distfiles/ninja/v1.7.1.tar.gz'
INFO:src.distfile:FETCH https://github.com/ninja-build/ninja/archive/v1.7.1.tar.gz
INFO:src.distfile:CHECKSUM /home/vagrant/cloudabi-ports/_obj/distfiles/ninja/v1.7.1.tar.gz
env: python: No such file or directory
Traceback (most recent call last):
  File "build_packages.py", line 66, in <module>
    build_package(target_packages[(name, arch)])
  File "build_packages.py", line 59, in build_package
    catalog.insert(package, version, catalog.package(package, version))
  File "/home/vagrant/cloudabi-ports/src/catalog.py", line 292, in package
    package.build()
  File "/home/vagrant/cloudabi-ports/src/package.py", line 111, in build
    }, self._lib_depends)
  File "/home/vagrant/cloudabi-ports/src/package.py", line 191, in initialize_buildroot
    dep.build()
  File "/home/vagrant/cloudabi-ports/src/package.py", line 64, in build
    self._resource_directory))
  File "<string>", line 9, in build
  File "/home/vagrant/cloudabi-ports/src/builder.py", line 172, in run
    self._builder.run(self._path, command)
  File "/home/vagrant/cloudabi-ports/src/builder.py", line 350, in run
    ] + command)
  File "/usr/local/lib/python3.5/subprocess.py", line 581, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['env', 'CC=/usr/bin/cc', 'CXX=/usr/bin/c++', 'CFLAGS=-O2 -I/usr/obj/cloudabi-ports/include', 'CXXFLAGS=-O2 -I/usr/obj/cloudabi-ports/include', 'LDFLAGS=-L/usr/obj/cloudabi-ports/lib', 'PATH=/usr/obj/cloudabi-ports/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/home/vagrant/bin', './configure.py', '--bootstrap']' returned non-zero exit status 127

Some package info:

vagrant@:~/cloudabi-ports % pkg query %n-%v|grep python
python27-2.7.12
python3-3_3
python35-3.5.2
x86_64-unknown-cloudabi-python-3.6.0_31

I don't expect it's relevant, but I commented out the stuff I'm not interested in building:

vagrant@:~/cloudabi-ports % git diff
diff --git a/build_packages.py b/build_packages.py
index b576dda..6b93633 100755
--- a/build_packages.py
+++ b/build_packages.py
@@ -42,14 +42,14 @@ for filename in util.walk_files(DIR_REPOSITORY):
 target_packages = repo.get_target_packages()
 
 catalogs = {
-    ArchLinuxCatalog(None, DIR_PACKAGES_ARCHLINUX),
-    CygwinCatalog(None, DIR_PACKAGES_CYGWIN),
-    DebianCatalog(None, DIR_PACKAGES_DEBIAN),
+#    ArchLinuxCatalog(None, DIR_PACKAGES_ARCHLINUX),
+#    CygwinCatalog(None, DIR_PACKAGES_CYGWIN),
+#    DebianCatalog(None, DIR_PACKAGES_DEBIAN),
     FreeBSDCatalog(None, DIR_PACKAGES_FREEBSD),
-    HomebrewCatalog(None, DIR_PACKAGES_HOMEBREW, 'http://example.com/'),
-    NetBSDCatalog(None, DIR_PACKAGES_NETBSD),
-    OpenBSDCatalog(None, DIR_PACKAGES_OPENBSD),
-    RedHatCatalog(None, DIR_PACKAGES_REDHAT),
+#    HomebrewCatalog(None, DIR_PACKAGES_HOMEBREW, 'http://example.com/'),
+#    NetBSDCatalog(None, DIR_PACKAGES_NETBSD),
+#    OpenBSDCatalog(None, DIR_PACKAGES_OPENBSD),
+#    RedHatCatalog(None, DIR_PACKAGES_REDHAT),
 }
 
 
diff --git a/src/config.py b/src/config.py
index 6e146ae..b7ca6ca 100644
--- a/src/config.py
+++ b/src/config.py
@@ -8,9 +8,9 @@ import os
 
 # Architectures for which we can build packages.
 ARCHITECTURES = {
-    'aarch64-unknown-cloudabi',
-    'armv6-unknown-cloudabi-eabihf',
-    'i686-unknown-cloudabi',
+#    'aarch64-unknown-cloudabi',
+#    'armv6-unknown-cloudabi-eabihf',
+#    'i686-unknown-cloudabi',
     'x86_64-unknown-cloudabi',
 }

EdSchouten · 2016-12-26T20:34:36Z

Hi there! I had to fire up the package builds after I submitted that change. That always takes some time, but should be finished now. Be sure to 'pkg upgrade'. To answer to your problem building ninja: building the cross toolchain still depends on certain tools from the host system, like Python 2. Ed

dckc · 2016-12-26T20:45:21Z

And python27-2.7.12 isn't enough Python 2 to build ninja and such?

EdSchouten · 2016-12-26T20:47:19Z

I think it uses $PATH to find it. Do you have /usr/local/bin in there?

dckc · 2016-12-26T20:49:15Z

yes:

% echo $PATH|grep local/bin
/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/home/vagrant/bin

EdSchouten · 2016-12-26T20:52:09Z

That's interesting. The idea is that this should normally work. I'll see if I can reproduce this tomorrow. Thanks for reporting this issue!

dckc · 2016-12-26T20:53:31Z

I'm past the directory issue.

I/we can open a separate issue on rebuilding FreeBSD packages from source.

vagrant@:~ % sudo pkg upgrade
...
Installed packages to be UPGRADED:
	x86_64-unknown-cloudabi-python: 3.6.0_31 -> 3.6.0_32 [CloudABI]

vagrant@:~/ZeroVault % cloudabi-run /usr/local/x86_64-unknown-cloudabi/bin/python3 < zv.yaml
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'main'

dckc · 2016-12-28T02:04:43Z

Ah... now I see why python27-2.7.12 isn't enough Python 2. It doesn't actually add anything called python to $PATH. The thing to do seems to be:

% sudo pkg install python
...
New packages to be INSTALLED:
	python: 2.7_2,2 [FreeBSD]
	python2: 2_3 [FreeBSD]

dckc · 2017-09-03T02:26:49Z

Directory fds get converted to integers... How about Path objects instead? See FdPath in https://github.com/dckc/ZeroVault/blob/cloudabi_wsgi/io_pola.py

Or at least: some flag to tell me it was an fd so I can convert to FdPath myself?

--

cloudabi-ports/packages/python/patch-tool-python

Lines 166 to 168 in 758bcea

    
           +            } else if (stat_result == 0 && S_ISDIR(sb.st_mode)) { 
        
           +                /* Directories must be expressed as raw descriptor numbers. */ 
        
           +                return PyLong_FromLong(value);

EdSchouten · 2017-09-03T13:34:15Z

As @sgielen is also using Python on CloudABI somewhat intensively, let's add him to this discussion as well.

Sjors, with the program_spawn()/program_exec() bits you added some time ago, I think we're now at the point where we can automatically convert file descriptors to any type we want, as long as we ensure they implement .fileno(), right?

EdSchouten · 2017-09-06T11:39:36Z

Hi Dan,

I had a look at the Python port with @sgielen the other day: it is indeed the case that any object that has a .fileno() operation can be passed to our program_exec()/program_spawn() bindings.

Dan, would you be interested in writing a Python class that you think would be suitable to wrap directory file descriptors? It could, for example, be based on Python 3.6's new Path abstraction. The only requirements this class should have:

A constructor that accepts a directory file descriptor,
A fileno() operation.

Apart from that, you may add any utility function you think is necessary. If you give such a class to me, I'll add it to our version of Python and extend the interpreter's startup code to construct such objects.

dckc · 2017-09-07T19:32:43Z

Ok. The FdPath from io_pola.py is close. I'll make a few tweaks.

At the language level, things like os.fdopen are powerful, so I'm avoiding referring to them as ambient authority. Is passing them in explicitly (cf. ops) OK?

EdSchouten · 2017-09-07T19:40:02Z

Thanks! Passing in operations explicitly is all right, but it would be nice if there was a wrapper function that only takes the file descriptor as an argument. That way the code in the interpreter can remain simple. Any preference for a package name?

dckc · 2017-09-08T01:56:04Z

A package is something released separately from python, installed with pip and such, yes? Is that really what you want?

Or do you just want a module with a class inside?

Should this thing go in sys?

In short: no, no preference.

I'll flip coins or something...

EdSchouten · 2017-09-08T06:35:45Z

Yes, sorry; module. :-)

EdSchouten added a commit that referenced this issue Dec 26, 2016

Don't attempt to call io.open() on directory file descriptors.

b57b98c

Previous versions of Python allowed this, but newer ones seem to throw IsADirectoryError. This should fix issue #25. Reported by: @dckc

dckc closed this as completed Dec 26, 2016

dckc reopened this Sep 3, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

how to provide directories to python programs? #25

how to provide directories to python programs? #25

dckc commented Dec 26, 2016

dckc commented Dec 26, 2016

dckc commented Dec 26, 2016 •

edited

Loading

EdSchouten commented Dec 26, 2016 via email

dckc commented Dec 26, 2016

EdSchouten commented Dec 26, 2016 via email

dckc commented Dec 26, 2016

EdSchouten commented Dec 26, 2016 via email

dckc commented Dec 26, 2016

dckc commented Dec 28, 2016

dckc commented Sep 3, 2017

EdSchouten commented Sep 3, 2017

EdSchouten commented Sep 6, 2017

dckc commented Sep 7, 2017

EdSchouten commented Sep 7, 2017

dckc commented Sep 8, 2017

EdSchouten commented Sep 8, 2017

how to provide directories to python programs? #25

how to provide directories to python programs? #25

Comments

dckc commented Dec 26, 2016

dckc commented Dec 26, 2016

dckc commented Dec 26, 2016 • edited Loading

EdSchouten commented Dec 26, 2016 via email

dckc commented Dec 26, 2016

EdSchouten commented Dec 26, 2016 via email

dckc commented Dec 26, 2016

EdSchouten commented Dec 26, 2016 via email

dckc commented Dec 26, 2016

dckc commented Dec 28, 2016

dckc commented Sep 3, 2017

EdSchouten commented Sep 3, 2017

EdSchouten commented Sep 6, 2017

dckc commented Sep 7, 2017

EdSchouten commented Sep 7, 2017

dckc commented Sep 8, 2017

EdSchouten commented Sep 8, 2017

dckc commented Dec 26, 2016 •

edited

Loading