Adding new DataProvider for JSON

from galaxy.datatypes import dataproviders from galaxy.datatypes.dataproviders.dataset import JsonDataProvider @dataproviders.decorators.has_dataproviders class Json( Text ): .... @dataproviders.decorators.dataprovider_factory( 'json', JsonDataProvider.settings ) def json_dataprovider( self, dataset, **settings ): dataset_source = dataproviders.dataset.DatasetDataProvider( dataset ) return JsonDataProvider( dataset_source, **settings )

class JsonDataProvider( base.DataProvider ): def __init__( self, dataset, **kwards ): self.dataset = dataset def __iter__( self ): if (self.dataset in not None): yield [list(line) for line in self.dataset] else: yield

3.4 years ago by

carlfeberhard ♦ 390

United States

carlfeberhard ♦ 390 wrote:

Hello and awesome job so far!

I believe what's happening is the 'Json' string used to request a new provider:

data = list( hda.datatype.dataprovider( hda, 'Json' ) )

does not match the 'json' string you've used to define the provider:

@dataproviders.decorators.dataprovider_factory( 'json', JsonDataProvider.settings )

ADD COMMENT • link written 3.4 years ago by carlfeberhard ♦ 390

I was able to move forward from No DataProvider issue, but ended up having an AttributeError. I have updated my original post to reflect on the stage that I am. If you could take a look and comment- that will be great!

ADD REPLY • link written 3.4 years ago by asmariyaz23 • 10

The documentation is outdated. I believe you can do that now with:

var pipeline_inspector = ${h.dumps( datalist, indent=2 )};

Just from the name pipeline_inspector, I'm assuming you're using this pattern for debugging (at least currently). If it wasn't debugging, keep in mind the thing about json in particular: if you're not filtering the results, a dataprovider may not be necessary.

Dataproviders were meant to operate on non-json files and use json as a medium of communication between a dataset and a visualization. Also, since JSON is hierarchically structured, the iterator pattern used by dataproviders may not make sense.

I'll see what I can do about making a json-specific provider available in the core - since it's a bit of odd situation.

In the meantime, another way you might make a provider directly in the datatypes/text.py file is:

from galaxy.datatypes import dataproviders

....

@dataproviders.decorators.has_dataproviders
class Json( Text ):
    edam_format = "format_3464"
    file_ext = "json"

...

    @dataproviders.decorators.dataprovider_factory( 'raw', {} )
    def raw_dataprovider( self, dataset ):
        lines = dataproviders.dataset.DatasetDataProvider( dataset )
        yield json.loads( ''.join( lines ) )

This will open the file, join the lines in it, and load it into a python dictionary.

And then, in your visualization mako file:

var pipeline_inspector = ${ h.dumps( hda.datatype.raw_dataprovider( hda ).next(), indent=2 ) };

You can use next here (and not list) since json is hierarchical and only 'has' one element (the dictionary/json-object).

ADD REPLY • link written 3.4 years ago by carlfeberhard ♦ 390

My goal is to retrieve a JSON dataset from history (which is an output of another pipeline) and feed this JSON dataset as a JSON object to the MutationInspector.js's. I will try to use the "dumps" method now.

ADD REPLY • link written 3.4 years ago by asmariyaz23 • 10

Ok your solution worked like a charm!! Thanks...

ADD REPLY • link written 3.4 years ago by asmariyaz23 • 10

Similar posts • Search »