Good morning,

We are trying to deploy Galaxy for our researchers and we have quite a few errors running the tests using run_tests.sh

Our job_conf.xml looks like this:

<?xml version="1.0"?>

<job_conf>

    <plugins workers="4">

        <plugin id="pbs" type="runner" load="galaxy.jobs.runners.pbs:PBSJobRunner" workers="2">

        </plugin>

    </plugins>

    <handlers default="handlers">

        <handler id="main" tags="handlers"/>

    </handlers>

    <destinations default="pbs1">

        <destination id="pbs1" runner="pbs" tags="hpc_short">

            <param id="-q">batch</param>

            <param id="Resource_List">nodes=1:ppn=1,mem=8gb,walltime=72:00:00</param>

        </destination>

    </destinations>

    <limits>

        <limit type="registered_user_concurrent_jobs">3</limit>

        <limit type="anonymous_user_concurrent_jobs">0</limit>

        <limit type="destination_user_concurrent_jobs" tag="hpc_short">3</limit>

        <limit type="destination_total_concurrent_jobs" tag="hpc_short">40</limit>

        <limit type="walltime">72:00:00</limit>

        <limit type="output_size">10GB</limit>

    </limits>

</job_conf>

The error we are receiving looks as follows: PBS job failed: Unknown error: 1

Just in case you need the entire output here is a sample of the same error I get several times for different tests. Please help me, I am new to the Galaxy world and have been turning around this issue for far too long.

CONSOLE OUTPUT:

test_tool_000001 (functional.test_toolbox.TestForTool_trimmer)

Trim ( trimmer ) > Test-2 ... requests.packages.urllib3.connectionpool INFO 2015-08-20 10:35:08,296 Starting new HTTP connection (1): localhost

galaxy.web.framework.webapp INFO 2015-08-20 10:35:08,305 Session authenticated using Galaxy master api key

requests.packages.urllib3.connectionpool DEBUG 2015-08-20 10:35:08,326 "GET /api/users?key=TEST123 HTTP/1.1" 200 None

requests.packages.urllib3.connectionpool INFO 2015-08-20 10:35:08,332 Starting new HTTP connection (1): localhost

galaxy.web.framework.webapp INFO 2015-08-20 10:35:08,342 Session authenticated using Galaxy master api key

requests.packages.urllib3.connectionpool DEBUG 2015-08-20 10:35:08,389 "POST /api/users/d9abeb98649a6a7e/api_key HTTP/1.1" 200 None

requests.packages.urllib3.connectionpool INFO 2015-08-20 10:35:08,394 Starting new HTTP connection (1): localhost

requests.packages.urllib3.connectionpool DEBUG 2015-08-20 10:35:08,700 "POST /api/histories HTTP/1.1" 200 None

requests.packages.urllib3.connectionpool INFO 2015-08-20 10:35:08,707 Starting new HTTP connection (1): localhost

galaxy.tools.actions.upload_common INFO 2015-08-20 10:35:09,132 tool upload1 created job id 168

galaxy.tools.execute DEBUG 2015-08-20 10:35:09,226 Tool [upload1] created job [168] (402.679 ms)

requests.packages.urllib3.connectionpool DEBUG 2015-08-20 10:35:09,440 "POST /api/tools HTTP/1.1" 200 None

requests.packages.urllib3.connectionpool INFO 2015-08-20 10:35:09,446 Starting new HTTP connection (1): localhost

galaxy.jobs DEBUG 2015-08-20 10:35:09,549 (168) Working directory for job is: /tmp/tmpYoV2U1/tmp5tz7tW/job_working_directory_an5G8e/000/168

galaxy.jobs.handler DEBUG 2015-08-20 10:35:09,742 (168) Dispatching to pbs runner

requests.packages.urllib3.connectionpool DEBUG 2015-08-20 10:35:09,980 "GET /api/histories/fe8fb2641c1e0305?key=222c5f54eb594e4b055337a5ef576cbb HTTP/1.1" 200 None

requests.packages.urllib3.connectionpool INFO 2015-08-20 10:35:09,983 Starting new HTTP connection (1): localhost

galaxy.jobs DEBUG 2015-08-20 10:35:10,184 (168) Persisting job destination (destination id: pbs1)

galaxy.jobs.runners DEBUG 2015-08-20 10:35:11,225 Job [168] queued (1482.241 ms)

galaxy.jobs.handler INFO 2015-08-20 10:35:11,296 (168) Job dispatched

requests.packages.urllib3.connectionpool DEBUG 2015-08-20 10:35:11,318 "GET /api/histories/fe8fb2641c1e0305?key=222c5f54eb594e4b055337a5ef576cbb HTTP/1.1" 200 None

requests.packages.urllib3.connectionpool INFO 2015-08-20 10:35:11,324 Starting new HTTP connection (1): localhost

requests.packages.urllib3.connectionpool DEBUG 2015-08-20 10:35:11,783 "GET /api/histories/fe8fb2641c1e0305?key=222c5f54eb594e4b055337a5ef576cbb HTTP/1.1" 200 None

requests.packages.urllib3.connectionpool INFO 2015-08-20 10:35:11,786 Starting new HTTP connection (1): localhost

requests.packages.urllib3.connectionpool DEBUG 2015-08-20 10:35:12,210 "GET /api/histories/fe8fb2641c1e0305?key=222c5f54eb594e4b055337a5ef576cbb HTTP/1.1" 200 None

requests.packages.urllib3.connectionpool INFO 2015-08-20 10:35:12,213 Starting new HTTP connection (1): localhost

galaxy.jobs.runners DEBUG 2015-08-20 10:35:12,506 (168) command is: python /lustre/home/soporte/app_user/galaxy-dist/tools/data_source/upload.py /lustre/home/soporte/app_user/galaxy-dist /tmp/tmpYoV2U1/tmp5tz7tW/new_files_path_zAD4mw/tmpMNJEGb /tmp/tmpYoV2U1/tmp5tz7tW/new_files_path_zAD4mw/tmpVo0Kup 168:/tmp/tmpYoV2U1/tmp5tz7tW/job_working_directory_an5G8e/000/168/dataset_168_files:/tmp/tmpYoV2U1/tmp5tz7tW/database/files/000/dataset_168.dat; return_code=$?; python /tmp/tmpYoV2U1/tmp5tz7tW/job_working_directory_an5G8e/000/168/set_metadata_KoRBiM.py /tmp/tmpYoV2U1/tmp5tz7tW/new_files_path_zAD4mw/tmpMNJEGb /tmp/tmpYoV2U1/tmp5tz7tW/job_working_directory_an5G8e/000/168/galaxy.json /tmp/tmpYoV2U1/tmp5tz7tW/job_working_directory_an5G8e/000/168/metadata_in_HistoryDatasetAssociation_151_3jjg2M,/tmp/tmpYoV2U1/tmp5tz7tW/job_working_directory_an5G8e/000/168/metadata_kwds_HistoryDatasetAssociation_151_atIQDT,/tmp/tmpYoV2U1/tmp5tz7tW/job_working_directory_an5G8e/000/168/metadata_out_HistoryDatasetAssociation_151_7NMVIi,/tmp/tmpYoV2U1/tmp5tz7tW/job_working_directory_an5G8e/000/168/metadata_results_HistoryDatasetAssociation_151_nZAQGE,/tmp/tmpYoV2U1/tmp5tz7tW/database/files/000/dataset_168.dat,/tmp/tmpYoV2U1/tmp5tz7tW/job_working_directory_an5G8e/000/168/metadata_override_HistoryDatasetAssociation_151_N9TYm4; sh -c "exit $return_code"

galaxy.jobs.runners.pbs DEBUG 2015-08-20 10:35:12,562 (168) submitting file /lustre/home/soporte/app_user/galaxy-dist/database/pbs/168.sh

galaxy.jobs.runners.pbs DEBUG 2015-08-20 10:35:12,569 (168) queued in batch queue as 5552.clustermaster.uniandes.edu.co

galaxy.jobs DEBUG 2015-08-20 10:35:12,672 (168) Persisting job destination (destination id: pbs1)

requests.packages.urllib3.connectionpool DEBUG 2015-08-20 10:35:13,429 "GET /api/histories/fe8fb2641c1e0305?key=222c5f54eb594e4b055337a5ef576cbb HTTP/1.1" 200 None

requests.packages.urllib3.connectionpool INFO 2015-08-20 10:35:13,434 Starting new HTTP connection (1): localhost

galaxy.jobs.runners.pbs DEBUG 2015-08-20 10:35:13,543 (168/5552.clustermaster.uniandes.edu.co) PBS job state changed from N to R

requests.packages.urllib3.connectionpool DEBUG 2015-08-20 10:35:13,906 "GET /api/histories/fe8fb2641c1e0305?key=222c5f54eb594e4b055337a5ef576cbb HTTP/1.1" 200 None

requests.packages.urllib3.connectionpool INFO 2015-08-20 10:35:13,912 Starting new HTTP connection (1): localhost

requests.packages.urllib3.connectionpool DEBUG 2015-08-20 10:35:16,332 "GET /api/histories/fe8fb2641c1e0305?key=222c5f54eb594e4b055337a5ef576cbb HTTP/1.1" 200 None

requests.packages.urllib3.connectionpool INFO 2015-08-20 10:35:16,339 Starting new HTTP connection (1): localhost

requests.packages.urllib3.connectionpool DEBUG 2015-08-20 10:35:16,542 "GET /api/histories/fe8fb2641c1e0305?key=222c5f54eb594e4b055337a5ef576cbb HTTP/1.1" 200 None

requests.packages.urllib3.connectionpool INFO 2015-08-20 10:35:16,547 Starting new HTTP connection (1): localhost

requests.packages.urllib3.connectionpool DEBUG 2015-08-20 10:35:16,726 "GET /api/histories/fe8fb2641c1e0305?key=222c5f54eb594e4b055337a5ef576cbb HTTP/1.1" 200 None

requests.packages.urllib3.connectionpool INFO 2015-08-20 10:35:16,732 Starting new HTTP connection (1): localhost

requests.packages.urllib3.connectionpool DEBUG 2015-08-20 10:35:16,933 "GET /api/histories/fe8fb2641c1e0305?key=222c5f54eb594e4b055337a5ef576cbb HTTP/1.1" 200 None

requests.packages.urllib3.connectionpool INFO 2015-08-20 10:35:16,938 Starting new HTTP connection (1): localhost

requests.packages.urllib3.connectionpool DEBUG 2015-08-20 10:35:17,113 "GET /api/histories/fe8fb2641c1e0305?key=222c5f54eb594e4b055337a5ef576cbb HTTP/1.1" 200 None

requests.packages.urllib3.connectionpool INFO 2015-08-20 10:35:17,119 Starting new HTTP connection (1): localhost

requests.packages.urllib3.connectionpool DEBUG 2015-08-20 10:35:17,430 "GET /api/histories/fe8fb2641c1e0305?key=222c5f54eb594e4b055337a5ef576cbb HTTP/1.1" 200 None

requests.packages.urllib3.connectionpool INFO 2015-08-20 10:35:17,436 Starting new HTTP connection (1): localhost

requests.packages.urllib3.connectionpool DEBUG 2015-08-20 10:35:17,651 "GET /api/histories/fe8fb2641c1e0305?key=222c5f54eb594e4b055337a5ef576cbb HTTP/1.1" 200 None

requests.packages.urllib3.connectionpool INFO 2015-08-20 10:35:17,656 Starting new HTTP connection (1): localhost

requests.packages.urllib3.connectionpool DEBUG 2015-08-20 10:35:17,900 "GET /api/histories/fe8fb2641c1e0305?key=222c5f54eb594e4b055337a5ef576cbb HTTP/1.1" 200 None

requests.packages.urllib3.connectionpool INFO 2015-08-20 10:35:17,906 Starting new HTTP connection (1): localhost

requests.packages.urllib3.connectionpool DEBUG 2015-08-20 10:35:18,121 "GET /api/histories/fe8fb2641c1e0305?key=222c5f54eb594e4b055337a5ef576cbb HTTP/1.1" 200 None

requests.packages.urllib3.connectionpool INFO 2015-08-20 10:35:18,127 Starting new HTTP connection (1): localhost

requests.packages.urllib3.connectionpool DEBUG 2015-08-20 10:35:18,333 "GET /api/histories/fe8fb2641c1e0305?key=222c5f54eb594e4b055337a5ef576cbb HTTP/1.1" 200 None

requests.packages.urllib3.connectionpool INFO 2015-08-20 10:35:18,337 Starting new HTTP connection (1): localhost

requests.packages.urllib3.connectionpool DEBUG 2015-08-20 10:35:18,508 "GET /api/histories/fe8fb2641c1e0305?key=222c5f54eb594e4b055337a5ef576cbb HTTP/1.1" 200 None

requests.packages.urllib3.connectionpool INFO 2015-08-20 10:35:18,511 Starting new HTTP connection (1): localhost

requests.packages.urllib3.connectionpool DEBUG 2015-08-20 10:35:18,779 "GET /api/histories/fe8fb2641c1e0305?key=222c5f54eb594e4b055337a5ef576cbb HTTP/1.1" 200 None

requests.packages.urllib3.connectionpool INFO 2015-08-20 10:35:18,784 Starting new HTTP connection (1): localhost

requests.packages.urllib3.connectionpool DEBUG 2015-08-20 10:35:18,973 "GET /api/histories/fe8fb2641c1e0305?key=222c5f54eb594e4b055337a5ef576cbb HTTP/1.1" 200 None

requests.packages.urllib3.connectionpool INFO 2015-08-20 10:35:18,979 Starting new HTTP connection (1): localhost

requests.packages.urllib3.connectionpool DEBUG 2015-08-20 10:35:19,107 "GET /api/histories/fe8fb2641c1e0305?key=222c5f54eb594e4b055337a5ef576cbb HTTP/1.1" 200 None

requests.packages.urllib3.connectionpool INFO 2015-08-20 10:35:19,111 Starting new HTTP connection (1): localhost

requests.packages.urllib3.connectionpool DEBUG 2015-08-20 10:35:19,375 "GET /api/histories/fe8fb2641c1e0305?key=222c5f54eb594e4b055337a5ef576cbb HTTP/1.1" 200 None

requests.packages.urllib3.connectionpool INFO 2015-08-20 10:35:19,382 Starting new HTTP connection (1): localhost

requests.packages.urllib3.connectionpool DEBUG 2015-08-20 10:35:19,550 "GET /api/histories/fe8fb2641c1e0305?key=222c5f54eb594e4b055337a5ef576cbb HTTP/1.1" 200 None

requests.packages.urllib3.connectionpool INFO 2015-08-20 10:35:19,555 Starting new HTTP connection (1): localhost

galaxy.jobs.runners.pbs DEBUG 2015-08-20 10:35:19,727 (168/5552.clustermaster.uniandes.edu.co) PBS job state changed from R to C

galaxy.jobs.runners.pbs ERROR 2015-08-20 10:35:19,729 (168/5552.clustermaster.uniandes.edu.co) PBS job failed: Unknown error: 1

requests.packages.urllib3.connectionpool DEBUG 2015-08-20 10:35:20,059 "GET /api/histories/fe8fb2641c1e0305?key=222c5f54eb594e4b055337a5ef576cbb HTTP/1.1" 200 None

requests.packages.urllib3.connectionpool INFO 2015-08-20 10:35:20,062 Starting new HTTP connection (1): localhost

requests.packages.urllib3.connectionpool DEBUG 2015-08-20 10:35:20,305 "GET /api/histories/fe8fb2641c1e0305?key=222c5f54eb594e4b055337a5ef576cbb HTTP/1.1" 200 None

ERROR