Unicode error after installing package_python_2_7

Question: Unicode error after installing package_python_2_7_10

3.0 years ago by

Germany

Hi all,

I tried to install the newest versions of numpy, matplotlib etc. onto our local galaxy server. These need the python 2.7.10 package, which installs just fine. But the numpy, matplotlib dependencies do not install with the following error:

Traceback (most recent call last):
  File "/usr/local/bin/easy_install", line 7, in <module>
    from setuptools.command.easy_install import main
  File "/home/galaxy_production/galaxy-dist/dependency_dir/python/2.7.10/iuc/package_python_2_7_10/73631e818d3d/lib/python2.7/site-packages/setuptools/__init__.py", line 11, in <module>
    from setuptools.extension import Extension
  File "/home/galaxy_production/galaxy-dist/dependency_dir/python/2.7.10/iuc/package_python_2_7_10/73631e818d3d/lib/python2.7/site-packages/setuptools/extension.py", line 8, in <module>
    from .dist import _get_unpatched
  File "/home/galaxy_production/galaxy-dist/dependency_dir/python/2.7.10/iuc/package_python_2_7_10/73631e818d3d/lib/python2.7/site-packages/setuptools/dist.py", line 18, in <module>
    from setuptools import windows_support
  File "/home/galaxy_production/galaxy-dist/dependency_dir/python/2.7.10/iuc/package_python_2_7_10/73631e818d3d/lib/python2.7/site-packages/setuptools/windows_support.py", line 2, in <module>
    import ctypes
  File "/home/galaxy_production/galaxy-dist/dependency_dir/python/2.7.10/iuc/package_python_2_7_10/73631e818d3d/lib/python2.7/ctypes/__init__.py", line 10, in <module>
    from _ctypes import Union, Structure, Array
ImportError: /home/galaxy_production/galaxy-dist/dependency_dir/python/2.7.10/iuc/package_python_2_7_10/73631e818d3d/lib/python2.7/lib-dynload/_ctypes.so: undefined symbol: PyUnicodeUCS2_FromUnicode

So it seems that the python distribution that the installation wants to use is ucs4 coded and not ucs2. I already rebuild the systems python using the --enable-unicode=ucs2 argument. The virtual environment was still running Python 2.7.9. I build Python 2.7.10 from source and set up a new virtual environment with this version. Nothing helped.

I also updated some of the eggs today (including numpy) of which at least some are also ucs2 coded (referring from the egg names), which worked perfectly fine.

Before the update all tools worked perfectly.

Does somebody have an idea what could have gone wrong?

Thanks and best regards,

Thomas

installation python • 3.2k views

ADD COMMENT • link •

modified 3.0 years ago by Devon Ryan • 1.9k • written 3.0 years ago by thomas.draebing • 20

Can you rebuild the system wide python with --enable-unicode=ucs2. The TS packages are build with ucs4 and numpy from the TS expects this. The question we need to solve is why does the TS numpy picks up some system-wide libraries :(

ADD REPLY • link written 3.0 years ago by Bjoern Gruening ♦ 5.1k

I rebuild my system python now with --enable-unicode=ucs2. I checked whether the configuration worked with 'sys.maxunicode'. The output was correct. Trying to install numpy in galaxy gave me the same error as before. For your reference I uploaded the installation.log file in the corresponding dependency_dir path (/home/galaxy_production/galaxy-dist/dependency_dir/numpy/1.9/iuc/package_python_2_7_numpy_1_9/c1e72cd24a9c) as a google document: https://docs.google.com/document/d/15NoUCfYpYvHHpDgNT9v9cRUEqk7UV40iwVjF5w9Uxsk/edit?usp=sharing. I tried the same with the --enable-unicode=ucs4 option, but it did not make a difference. While the system python gave me the correct maxunicode value, the installation of numpy failed with the same error. The installation.log file was exactly identical.

Could it be a bug with easy_install / setuptools, since it seems to be the only path in the logs not referring to a galaxy owned directory?

ADD REPLY • link written 3.0 years ago by thomas.draebing • 20

Great troubleshooting information! This post is linked to the issue I created (here) so the team will see the comments when they review. Thanks, Jen

ADD REPLY • link written 3.0 years ago by Jennifer Hillman Jackson ♦ 25k

3.0 years ago by

Devon Ryan • 1.9k

Germany

Devon Ryan • 1.9k wrote:

I just ran into this same issue and found that it could be resolved by having package_python_2_7_10 have it's own copy of easy_install (otherwise you get the easy_install in /usr/bin, which is hard coded to use the system python). So you can:

source /galaxy-central/tool_deps/python/2.7.10/iuc/package_python_2_7_10/73631e818d3d/env.sh
wget https://bootstrap.pypa.io/ez_setup.py -O - | python

If you then try to install things again then the correct version of python will be used for everything and you should be able to get numpy/pysam/etc. installed (at least that solved things for me).

ADD COMMENT • link written 3.0 years ago by Devon Ryan • 1.9k

That worked perfectly! Thank you so much!

ADD REPLY • link written 3.0 years ago by thomas.draebing • 20

Happy to help. Let me know if you run need package_python_2_7_matplotlib_1_4 too and are running into issues with it. That one is also problematic (pkg-config --cflags is surprised because the Galaxy environment sets everything in environment variables like CPLUS_INCLUDE_PATH, but that prevents matplotlib from compiling).

ADD REPLY • link written 3.0 years ago by Devon Ryan • 1.9k

Hello Devon,

I used your solution and it works perfectly, except for matplotlib as you said. I am in the master branch, with everything updated. This is the error log:

============================================================================

BUILDING MATPLOTLIB
            matplotlib: yes [1.4.3]
                python: yes [2.7.10 (default, Nov 24 2015, 16:42:21)  [GCC
                        4.9.2]]
              platform: yes [linux2]

REQUIRED DEPENDENCIES AND EXTENSIONS
                 numpy: yes [version 1.9.2]
              dateutil: yes [using dateutil version 2.4.2]
                  pytz: yes [pytz was not found. pip will attempt to install
               tornado: yes [tornado was not found. It is required for the
                        WebAgg backend. pip/easy_install may attempt to
                        install it after matplotlib.]
             pyparsing: yes [using pyparsing version 2.0.3]
                 pycxx: yes [Couldn't import.  Using local copy.]
                libagg: yes [pkg-config information for 'libagg' could not
              freetype: no  [The C/C++ header for freetype2 (ft2build.h)
                        could not be found.  You may need to install the
                        development package.]
                   png: yes [version 1.6.7]
                 qhull: yes [pkg-config information for 'qhull' could not be
                        found. Using local copy.]

OPTIONAL SUBPACKAGES
              toolkits: yes [installing]
                 tests: yes [nose 0.11.1 or later is required to run the
                        to install it after matplotlib. / mock is required
                        may attempt to install it after matplotlib.]
        toolkits_tests: yes [nose 0.11.1 or later is required to run the
                        matplotlib test suite.  pip/easy_install may attempt
                        to install it after matplotlib. / mock is required
                        to run the matplotlib test suite.  pip/easy_install
                        may attempt to install it after matplotlib.]

OPTIONAL BACKEND EXTENSIONS
                macosx: no  [Mac OS-X only]
                qt5agg: no  [PyQt5 not found]
                pyside: no  [PySide not found]
             gtk3cairo: no  [Requires cairocffi or pycairo to be installed.]
                 tkagg: yes [installing, version 81008]
                 wxagg: no  [requires wxPython]
                   gtk: no  [Requires pygtk]
                   agg: yes [installing]
                 cairo: no  [cairocffi or pycairo not found]

OPTIONAL LATEX DEPENDENCIES
                dvipng: no
                 latex: yes [version 3.14159265]
               pdftops: yes [version 0.26.5]

                        * The following required packages can not be built:

                        * freetype

I will appreciate if you have any solution for this.

Regards, Jose

ADD REPLY • link written 3.0 years ago by jjalmagro • 0

Hi Jose, this is sort of an "in progress" solution, but this might work (I just submitted a bug report about this here). The trick is that you have to "unprepend" the freetype2 paths from the environment variables that I mentioned in my bug report. So if you look at the INSTALLATION.log and run all of the environment variables in there except make sure to modify CPLUS_INCLUDE_PATH and C_INCLUDE_PATH then you should be able to manually install it from the command line (the easy_install command is in the log file). Of course Galaxy still won't know that you have things installed and working (if only life were so simple), so you need to go into the postgresql database and modify one of the tables (I think it's tool_dependencies or something like that).

That's a horrific solution, but it seems to work (maybe, I still have things not working, but those are likely additional issues).

ADD REPLY • link written 3.0 years ago by Devon Ryan • 1.9k

Actually, I just modified /galaxy-central/tool_deps/freetype/2.5.2/iuc/package_freetype_2_5_2/3b09e7916896/env.sh to not set C_INCLUDE_PATH and CPLUS_INCLUDE_PATH and that allows Galaxy to handle the rest. That's a MUCH nicer solution.

ADD REPLY • link written 3.0 years ago by Devon Ryan • 1.9k

Yes, that worked!!. So, the problem was freetype2, not matplotlib. Therefore, I should leave like that the env.sh file for further installations that need freetype2, is it right?

Thank you!

ADD REPLY • link written 3.0 years ago by jjalmagro • 0

In my case Galaxy actually also tried to install package_freetype, although the required version was already installed. But it stuck at showing it as 'New'. Since it was displayed as missing requirement, I uninstalled the package_freetype version already existing and reinstalled it, which solved the problem.

ADD REPLY • link written 3.0 years ago by thomas.draebing • 20

package_python_2_7_matplotlib_1_4 actually installed fine for me after applying your solution. I had some undescriptive errors with the numpy and some other dependencies at first. Just some generic error that the dependency installation failed (Sorry, I cannot find the exact error message right now). The solution there was to completely uninstall all tools using this dependencies, which were basically the ones having the former problem, before installing them freshly.

ADD REPLY • link written 3.0 years ago by thomas.draebing • 20

3.0 years ago by

Jennifer Hillman Jackson ♦ 25k

United States

Jennifer Hillman Jackson ♦ 25k wrote:

Hello,

Our team is reviewing install issues with several packages, including package_python_2_7_10. Please follow this Github ticket for progress: https://github.com/galaxyproject/tools-devteam/issues/283

Thanks for reporting the problem, Jen, Galaxy team

ADD COMMENT • link written 3.0 years ago by Jennifer Hillman Jackson ♦ 25k

Please log in to add an answer.

Similar posts • Search »