The reason you're getting this error is because galaxy prevents binary files by default (for security reasons). I am aware of two ways to get around this:
You have to register a binary datatype for galaxy that allow XLS files. To do this you need to know 'magic' bits that are specific for XLS data types. You can add the datatypes either:
a. Within the galaxy code base (example: https://github.com/galaxyproject/galaxy/pull/1108/files and in particular don't forget to register it as a binary format: https://github.com/galaxyproject/galaxy/pull/1108/files#diff-409951594551183dbf886e24de6cb129R490 )
b. As part of a tool (example: https://bitbucket.org/EMCbioinf/galaxy-tool-shed-tools/src/63c7b4a6b643426d37252233a1f2e7c81014ab49/cgatools_v17/?at=master ). This involves changes in at least ./lib/galaxy/datatypes/<xlsx custom="" format="">.py ./datatypes_conf.xml . From what I have read before, this is not the ideal solution and I would suggest to go for 1a.
There are some binary data types already registered within galaxy (archive datatypes if I'm not mistaken). You could upload it as one of these and it won't give the error but neither the data file would get the XLSX data type. If you go down this way it should only be a temporary workaround to get started and I would advice you to go for solution 1.
I would advice you to use xlsx instead as this datatype is already supported within galaxy and the tool shed has a recently added python library for xlsx manipulation: https://github.com/galaxyproject/tools-iuc/pull/729