[Bioperl-l] Blast temporary open files not closed

ende mmacho at gmail.com
Sun Sep 10 08:19:15 EDT 2006





El 10/09/2006, a las 5:59, Hilmar Lapp escribió:

> Your code snippet leaves plenty of opportunities for temp files  
> accumulating, depending on what you do with the report object. So  
> long as you use the report object the associated result file will  
> remain present and open.
>


	It sounds possible.. I have thought it previously... but I returns a  
local copy of the result of $blastMachine->blastall(..);  not the  
$blastMachine at all.  How it could be possible the original object  
could get caught with that copy?






> You will have to provide the calling code as well if you want  
> someone to look at the problem.
>


	From the main object I call this DoBlast

     ########################
     #                      #
     #   ANALIZE EACH SEQ   #
     #                      #
     ########################
     #
     for (my $i=0; $i < $self->{inSeqArrTotal}; ++$i) {
         my $s = ${$self->{inSeqArr}}[$i];
         my $ID = $s->id();
         local *x = \$self->{SQ}{$ID};  # short for ref curr seq


         $x->{seq} = $s;


         # search for cloning vector seqs. in all the query seqs.
         #
         my $blastTemp = $self->DoBlastSeq($self-> 
{cloningVectorsFilename}, $s);


and then use this $blastTemp and save it in another place of the object:

         $x->{blastcv} = $blastTemp;

...



> On Sep 9, 2006, at 7:50 PM, ende wrote:
>
>>
>> 	Thank all of you for your quick and precise answer!!!
>>
>> My bioperl version must be 1.20 :(
>>
>> I can read it in the file Bio::Perl.pm  since
>>
>> 	perl -MCPAN -e shell
>> and then
>> 	i Bio::Perl
>>
>> does not inform me about any local version but says instead:
>>
>> 	trange distribution name [Bio:Perl]
>> 	Module id = Bio::Perl
>> 	    CPAN_USERID  BIRNEY (Ewan Birney <birney at sanger.ac.uk>)
>> 	    CPAN_VERSION undef
>> 	    CPAN_FILE    B/BI/BIRNEY/bioperl-1.4.tar.gz
>> 	    UPLOAD_DATE  2003-12-23
>> 	    DSLIP_STATUS       (,,,,)
>> 	    MANPAGE      Bio::Perl - Functional access to BioPerl for people
>> who don't know objects
>> 	    INST_FILE    /Library/Perl/5.8.6/Bio/Perl.pm
>> 	    INST_VERSION undef
>>
>> I read 1.20 at top of /Library/5.8.6/Bio/Perl.pm file!
>>
>> After many (many) attempts of installing Bio::Perl via -MCPAN always
>> it ends with the same message (of course using force!!):
>>
>>
>>          Failed 3/25 tests, 88.00% okay
>> t/WABA.......................ok
>> t/XEMBL_DB...................SOAP::Lite and/or XML::DOM not
>> installed. This means that Bio::DB::XEMBL module is not usable.
>> Skipping tests.
>> t/XEMBL_DB...................ok
>> Failed Test        Stat Wstat Total Fail  Failed  List of Failed
>> --------------------------------------------------------------------- 
>> ---
>> -------
>> t/BioFetch_DB.t                  27    1   3.70%  8
>> t/EMBL_DB.t                      15    3  20.00%  6 13-14
>> t/Ontology.t        255 65280    50  100 200.00%  1-50
>> t/TreeIO.t                       41    1   2.44%  42
>> t/Variation_IO.t                 25    3  12.00%  15 20 25
>> t/simpleGOparser.t  255 65280    98  196 200.00%  1-98
>> 121 subtests skipped.
>> Failed 6/179 test scripts, 96.65% okay. 154/8273 subtests failed,
>> 98.14% okay.
>> make: *** [test_dynamic] Error 2
>>    /usr/bin/make test -- NOT OK
>> Running make install
>>    make test had returned bad status, won't install without force
>> Failed during this command:
>>    BIRNEY/bioperl-1.4.tar.gz                    : make_test NO
>>
>>
>> ..and I am using "force" (exactly:)
>>
>> 	cpan> force install B/BI/BIRNEY/bioperl-1.4.tar.gz
>>
>>
>> I have installed many other modules without no problems but this
>> seems to reject me.  Finally I have dowloaded the bioperl
>> 1.5.1 .tar.gz and installed ignoring the many errors make test gave
>> me.  sudo make install and wow!!! the bioperl version changed and
>>
>> lsof +p pidofperl  | grep stat | wc -l
>>
>> again grows during the execution until reach 250 (!) and crashes.
>>
>> It was also imposible from CPAN shell to obtain the local bioperl
>> version (as was also impossible in a remote Linux installation, that
>> also said) INST_VERSION undef.
>>
>> But now the Bio/Perl.pm file heads:
>>
>> 	# Perl.pm,v 1.23.2.1 2005/10/09 15:16:18 jason Exp
>> 	#
>> 	# BioPerl module for Bio::Perl
>>
>>
>>
>>
>>
>> (I am using 10.4.7 on the MacOSX)
>>
>> the code launch blast all for each seq inside a perl object (use of
>> $blastMachine->io->_io_cleanup(); did not resolve the problem)
>>
>>> sub DoBlastSeq ($$$$) {
>>>     my ($self, $db, $seq, $outFileName) = @_;
>>>     my %params = (
>>>         program  => "blastn",
>>>         outfile  => $outFileName,
>>>         database => "$self->{path_db}/$db",
>>>         # q        =>  "-5",
>>>         G        =>  3,  # si
>>>         E        =>  3,  # si
>>>         F        =>  "\"m D\"", # "mD", # si
>>>         e        =>  700, # si
>>>         Y        =>  1.75e12,  # si
>>>         best    => 1,
>>>     );
>>>     if (!$params{outfile}) {
>>>         delete $params{outfile};
>>>     }
>>>     my $outErrs = ($outFileName || "blastErrs.err"),
>>>
>>>
>>>     my $blastMachine = Bio::Tools::Run::StandAloneBlast->new(% 
>>> params);
>>>
>>>     open(OLDSTDERR, ">&", \*STDERR) or die "Can't dup STDERR: $!";
>>>     open(STDERR, ">", $outErrs ) or die "ERROR reopening STDERR:  
>>> $!";
>>>
>>>     print "Running Blast with id: ", $seq->id(), "\n"  if $DEBUG;
>>>
>>>     my $blastResult = $blastMachine->blastall($seq);
>>>
>>>
>>>     close(STDERR);
>>>     open(STDERR, ">&", \*OLDSTDERR) or die "Can't dup OLDSTDERR:  
>>> $!";
>>>     unlink ($outErrs) if (-z $outErrs);
>>>
>>>     return $blastResult;
>>> }
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> El 09/09/2006, a las 15:45, Brian Osborne escribió:
>>
>>> Juan,
>>>
>>> I recall a bug like this was fixed a while back - what version
>>> Bioperl are
>>> you using? By the way, always give version numbers when reporting a
>>> bug, the
>>> answer "already fixed" is very common.
>>>
>>> Brian O.
>>>
>>>
>>> On 9/9/06 4:31 AM, "ende" <mmacho at gmail.com> wrote:
>>>
>>>>
>>>> Processing a fasta file with about 500 dna seqs.. my MacOSX (that
>>>> has the max number of opened files up to 512) crashes... You  
>>>> need to
>>>> divide the problem in pieces or (in bash shell, with ulimit -n  
>>>> 1024)
>>>> augment that max number of opened files.
>>>>
>>>> This has no sense for me since my perl program nor leave any open
>>>> file without its corresponding closing. On the other side, the
>>>> problem arises when the number of dnas grows _in one file_.
>>>>
>>>> In the code I run blast (StandAloneBlast... $blastMachine- 
>>>> >blastall)
>>>> for each seq.
>>>>
>>>>
>>>> Then sniffing int the perl program stopped perl program I confirmed
>>>> my suspects.  BioPerl (StandAloneBlast) does not closes temporary
>>>> opened files.  Those files seems to be created to save seqs for  
>>>> to be
>>>> then processed by blastall program...  The output of lsof indicates
>>>> (as MacOSX System Monitor) that those files are left opened but not
>>>> there (!?)
>>>>
>>>> The output of  lsof +p pidofperlprogram
>>>>
>>>> COMMAND   PID   USER   FD   TYPE DEVICE SIZE/OFF     NODE NAME
>>>> perl    21977 juanfc  cwd   VDIR   14,7      238  6835993 /Users/
>>>> juanfc/Documents/programperl    21977 juanfc  txt   VREG   14,7
>>>> 19280  1589055 /usr/bin/perl
>>>> perl    21977 juanfc  txt   VREG   14,7    23476  1580272 /System/
>>>> Library/Perl/5.8.6/darwin-thread-multi-2level/auto/IO/IO.bundle
>>>> perl    21977 juanfc  txt   VREG   14,7    17772  1580263 /System/
>>>> Library/Perl/5.8.6/darwin-thread-multi-2level/auto/Fcntl/ 
>>>> Fcntl.bundle
>>>> perl    21977 juanfc  txt   VREG   14,7   114116  1580381 /System/
>>>> Library/Perl/5.8.6/darwin-thread-multi-2level/auto/POSIX/ 
>>>> POSIX.bundle
>>>> perl    21977 juanfc  txt   VREG   14,7    23684  1580265 /System/
>>>> Library/Perl/5.8.6/darwin-thread-multi-2level/auto/File/Glob/
>>>> Glob.bundle
>>>> perl    21977 juanfc  txt   VREG   14,7  1797788  6275687 /usr/lib/
>>>> dyld
>>>> perl    21977 juanfc  txt   VREG   14,7  4379472  6276030 /usr/lib/
>>>> libSystem.B.dylib
>>>> perl    21977 juanfc  txt   VREG   14,7  1086420  6276221 /System/
>>>> Library/Perl/5.8.6/darwin-thread-multi-2level/CORE/libperl.dylib
>>>> perl    21977 juanfc    0u  VCHR    4,2   0t3748 63113092 /dev/ 
>>>> ttyp2
>>>> perl    21977 juanfc    1u  VCHR    4,2   0t3748 63113092 /dev/ 
>>>> ttyp2
>>>> perl    21977 juanfc    2u  VCHR    4,2   0t3748 63113092 /dev/ 
>>>> ttyp2
>>>> perl    21977 juanfc    3u  VCHR    4,2   0t3748 63113092 /dev/ 
>>>> ttyp2
>>>> perl    21977 juanfc    4r  VREG                          stat(/
>>>> ...
>>>> Please, help.
>>>>
>>>>
>>>> --
>>>>      Juan Falgueras
>>>>      Profesor del Depto. de Lenguajes y Ciencias de la Computación
>>>>      Universidad de Málaga
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Bioperl-l mailing list
>>>> Bioperl-l at lists.open-bio.org
>>>> http://lists.open-bio.org/mailman/listinfo/bioperl-l
>>>
>>>
>>
>>
>>
>> ---- ende
>>
>>
>>
>>
>> _______________________________________________
>> Bioperl-l mailing list
>> Bioperl-l at lists.open-bio.org
>> http://lists.open-bio.org/mailman/listinfo/bioperl-l
>>
>
> -- 
> ===========================================================
> : Hilmar Lapp  -:-  Durham, NC  -:-  hlapp at gmx dot net :
> ===========================================================
>
>
>
>
>



---- ende






More information about the Bioperl-l mailing list