[Bioperl-l] Next-gen modules
bix at sendu.me.uk
Thu Jun 18 02:45:17 EDT 2009
Chris Fields wrote:
> On Jun 17, 2009, at 5:10 PM, Sendu Bala wrote:
>>> I'm personally wondering if this could be done as a sequence
>>> database, something similar in theme to Lincoln's SeqFeature::Store,
>>> but sequence only, and returns quality objects in a similar manner
>>> (ala Storable)? Not sure whether that's feasible, but it's appears
>>> at least scalable.
>> I think not. Well, at least SeqFeature::Store doesn't scale. Try
>> storing millions of features in a database and watch it crawl to
>> complete unusability. I can't imagine a db scaling to holding hundreds
>> of TB of data either. I'm also not sure what the benefit is. There are
>> already high-speed ways of indexing your fastq or bam files.
> Interesting that you ran into issues with SF::Store; wonder if object
> storage is the limiting factor there, or if it is something else.
Object storage certainly was an issue, which is why I patched it to
(optionally) not store objects. That helped a great deal, but ultimately
only increased the number of features you could store before it slowed
down; it didn't solve the problem completely.
More information about the Bioperl-l