[Bioperl-l] Blast temporary open files not closed

Hilmar Lapp hlapp at gmx.net
Sun Sep 10 09:40:46 EDT 2006


On Sep 10, 2006, at 8:19 AM, ende wrote:

> El 10/09/2006, a las 5:59, Hilmar Lapp escribió:
>
>> Your code snippet leaves plenty of opportunities for temp files  
>> accumulating, depending on what you do with the report object. So  
>> long as you use the report object the associated result file will  
>> remain present and open.
>>
>
> 	It sounds possible.. I have thought it previously... but I returns  
> a local copy of the result of $blastMachine->blastall(..);  not the  
> $blastMachine at all.  How it could be possible the original object  
> could get caught with that copy?

Never assume inner workings to be as you think they ought to be. In  
this case it is not the original object but the file because it is  
parsed on-demand. I.e., by the time the function returns the file has  
not been fully parsed yet and therefore remains open.

>
>> You will have to provide the calling code as well if you want  
>> someone to look at the problem.
>>
>
>
> 	From the main object I call this DoBlast
>
>     ########################
>     #                      #
>     #   ANALIZE EACH SEQ   #
>     #                      #
>     ########################
>     #
>     for (my $i=0; $i < $self->{inSeqArrTotal}; ++$i) {
>         my $s = ${$self->{inSeqArr}}[$i];
>         my $ID = $s->id();
>         local *x = \$self->{SQ}{$ID};  # short for ref curr seq
>
>
>         $x->{seq} = $s;
>
>
>         # search for cloning vector seqs. in all the query seqs.
>         #
>         my $blastTemp = $self->DoBlastSeq($self-> 
> {cloningVectorsFilename}, $s);
>
>
> and then use this $blastTemp and save it in another place of the  
> object:
>
>         $x->{blastcv} = $blastTemp;

Well this is still not the complete story but sounds like you're  
storing all the report objects which will have the files remain open.

	-hilmar


>
> ...
>
>
>
>> On Sep 9, 2006, at 7:50 PM, ende wrote:
>>
>>>
>>> 	Thank all of you for your quick and precise answer!!!
>>>
>>> My bioperl version must be 1.20 :(
>>>
>>> I can read it in the file Bio::Perl.pm  since
>>>
>>> 	perl -MCPAN -e shell
>>> and then
>>> 	i Bio::Perl
>>>
>>> does not inform me about any local version but says instead:
>>>
>>> 	trange distribution name [Bio:Perl]
>>> 	Module id = Bio::Perl
>>> 	    CPAN_USERID  BIRNEY (Ewan Birney <birney at sanger.ac.uk>)
>>> 	    CPAN_VERSION undef
>>> 	    CPAN_FILE    B/BI/BIRNEY/bioperl-1.4.tar.gz
>>> 	    UPLOAD_DATE  2003-12-23
>>> 	    DSLIP_STATUS       (,,,,)
>>> 	    MANPAGE      Bio::Perl - Functional access to BioPerl for  
>>> people
>>> who don't know objects
>>> 	    INST_FILE    /Library/Perl/5.8.6/Bio/Perl.pm
>>> 	    INST_VERSION undef
>>>
>>> I read 1.20 at top of /Library/5.8.6/Bio/Perl.pm file!
>>>
>>> After many (many) attempts of installing Bio::Perl via -MCPAN always
>>> it ends with the same message (of course using force!!):
>>>
>>>
>>>          Failed 3/25 tests, 88.00% okay
>>> t/WABA.......................ok
>>> t/XEMBL_DB...................SOAP::Lite and/or XML::DOM not
>>> installed. This means that Bio::DB::XEMBL module is not usable.
>>> Skipping tests.
>>> t/XEMBL_DB...................ok
>>> Failed Test        Stat Wstat Total Fail  Failed  List of Failed
>>> -------------------------------------------------------------------- 
>>> ----
>>> -------
>>> t/BioFetch_DB.t                  27    1   3.70%  8
>>> t/EMBL_DB.t                      15    3  20.00%  6 13-14
>>> t/Ontology.t        255 65280    50  100 200.00%  1-50
>>> t/TreeIO.t                       41    1   2.44%  42
>>> t/Variation_IO.t                 25    3  12.00%  15 20 25
>>> t/simpleGOparser.t  255 65280    98  196 200.00%  1-98
>>> 121 subtests skipped.
>>> Failed 6/179 test scripts, 96.65% okay. 154/8273 subtests failed,
>>> 98.14% okay.
>>> make: *** [test_dynamic] Error 2
>>>    /usr/bin/make test -- NOT OK
>>> Running make install
>>>    make test had returned bad status, won't install without force
>>> Failed during this command:
>>>    BIRNEY/bioperl-1.4.tar.gz                    : make_test NO
>>>
>>>
>>> ..and I am using "force" (exactly:)
>>>
>>> 	cpan> force install B/BI/BIRNEY/bioperl-1.4.tar.gz
>>>
>>>
>>> I have installed many other modules without no problems but this
>>> seems to reject me.  Finally I have dowloaded the bioperl
>>> 1.5.1 .tar.gz and installed ignoring the many errors make test gave
>>> me.  sudo make install and wow!!! the bioperl version changed and
>>>
>>> lsof +p pidofperl  | grep stat | wc -l
>>>
>>> again grows during the execution until reach 250 (!) and crashes.
>>>
>>> It was also imposible from CPAN shell to obtain the local bioperl
>>> version (as was also impossible in a remote Linux installation, that
>>> also said) INST_VERSION undef.
>>>
>>> But now the Bio/Perl.pm file heads:
>>>
>>> 	# Perl.pm,v 1.23.2.1 2005/10/09 15:16:18 jason Exp
>>> 	#
>>> 	# BioPerl module for Bio::Perl
>>>
>>>
>>>
>>>
>>>
>>> (I am using 10.4.7 on the MacOSX)
>>>
>>> the code launch blast all for each seq inside a perl object (use of
>>> $blastMachine->io->_io_cleanup(); did not resolve the problem)
>>>
>>>> sub DoBlastSeq ($$$$) {
>>>>     my ($self, $db, $seq, $outFileName) = @_;
>>>>     my %params = (
>>>>         program  => "blastn",
>>>>         outfile  => $outFileName,
>>>>         database => "$self->{path_db}/$db",
>>>>         # q        =>  "-5",
>>>>         G        =>  3,  # si
>>>>         E        =>  3,  # si
>>>>         F        =>  "\"m D\"", # "mD", # si
>>>>         e        =>  700, # si
>>>>         Y        =>  1.75e12,  # si
>>>>         best    => 1,
>>>>     );
>>>>     if (!$params{outfile}) {
>>>>         delete $params{outfile};
>>>>     }
>>>>     my $outErrs = ($outFileName || "blastErrs.err"),
>>>>
>>>>
>>>>     my $blastMachine = Bio::Tools::Run::StandAloneBlast->new(% 
>>>> params);
>>>>
>>>>     open(OLDSTDERR, ">&", \*STDERR) or die "Can't dup STDERR: $!";
>>>>     open(STDERR, ">", $outErrs ) or die "ERROR reopening STDERR:  
>>>> $!";
>>>>
>>>>     print "Running Blast with id: ", $seq->id(), "\n"  if $DEBUG;
>>>>
>>>>     my $blastResult = $blastMachine->blastall($seq);
>>>>
>>>>
>>>>     close(STDERR);
>>>>     open(STDERR, ">&", \*OLDSTDERR) or die "Can't dup OLDSTDERR:  
>>>> $!";
>>>>     unlink ($outErrs) if (-z $outErrs);
>>>>
>>>>     return $blastResult;
>>>> }
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> El 09/09/2006, a las 15:45, Brian Osborne escribió:
>>>
>>>> Juan,
>>>>
>>>> I recall a bug like this was fixed a while back - what version
>>>> Bioperl are
>>>> you using? By the way, always give version numbers when reporting a
>>>> bug, the
>>>> answer "already fixed" is very common.
>>>>
>>>> Brian O.
>>>>
>>>>
>>>> On 9/9/06 4:31 AM, "ende" <mmacho at gmail.com> wrote:
>>>>
>>>>>
>>>>> Processing a fasta file with about 500 dna seqs.. my MacOSX (that
>>>>> has the max number of opened files up to 512) crashes... You  
>>>>> need to
>>>>> divide the problem in pieces or (in bash shell, with ulimit -n  
>>>>> 1024)
>>>>> augment that max number of opened files.
>>>>>
>>>>> This has no sense for me since my perl program nor leave any open
>>>>> file without its corresponding closing. On the other side, the
>>>>> problem arises when the number of dnas grows _in one file_.
>>>>>
>>>>> In the code I run blast (StandAloneBlast... $blastMachine- 
>>>>> >blastall)
>>>>> for each seq.
>>>>>
>>>>>
>>>>> Then sniffing int the perl program stopped perl program I  
>>>>> confirmed
>>>>> my suspects.  BioPerl (StandAloneBlast) does not closes temporary
>>>>> opened files.  Those files seems to be created to save seqs for  
>>>>> to be
>>>>> then processed by blastall program...  The output of lsof  
>>>>> indicates
>>>>> (as MacOSX System Monitor) that those files are left opened but  
>>>>> not
>>>>> there (!?)
>>>>>
>>>>> The output of  lsof +p pidofperlprogram
>>>>>
>>>>> COMMAND   PID   USER   FD   TYPE DEVICE SIZE/OFF     NODE NAME
>>>>> perl    21977 juanfc  cwd   VDIR   14,7      238  6835993 /Users/
>>>>> juanfc/Documents/programperl    21977 juanfc  txt   VREG   14,7
>>>>> 19280  1589055 /usr/bin/perl
>>>>> perl    21977 juanfc  txt   VREG   14,7    23476  1580272 /System/
>>>>> Library/Perl/5.8.6/darwin-thread-multi-2level/auto/IO/IO.bundle
>>>>> perl    21977 juanfc  txt   VREG   14,7    17772  1580263 /System/
>>>>> Library/Perl/5.8.6/darwin-thread-multi-2level/auto/Fcntl/ 
>>>>> Fcntl.bundle
>>>>> perl    21977 juanfc  txt   VREG   14,7   114116  1580381 /System/
>>>>> Library/Perl/5.8.6/darwin-thread-multi-2level/auto/POSIX/ 
>>>>> POSIX.bundle
>>>>> perl    21977 juanfc  txt   VREG   14,7    23684  1580265 /System/
>>>>> Library/Perl/5.8.6/darwin-thread-multi-2level/auto/File/Glob/
>>>>> Glob.bundle
>>>>> perl    21977 juanfc  txt   VREG   14,7  1797788  6275687 /usr/ 
>>>>> lib/
>>>>> dyld
>>>>> perl    21977 juanfc  txt   VREG   14,7  4379472  6276030 /usr/ 
>>>>> lib/
>>>>> libSystem.B.dylib
>>>>> perl    21977 juanfc  txt   VREG   14,7  1086420  6276221 /System/
>>>>> Library/Perl/5.8.6/darwin-thread-multi-2level/CORE/libperl.dylib
>>>>> perl    21977 juanfc    0u  VCHR    4,2   0t3748 63113092 /dev/ 
>>>>> ttyp2
>>>>> perl    21977 juanfc    1u  VCHR    4,2   0t3748 63113092 /dev/ 
>>>>> ttyp2
>>>>> perl    21977 juanfc    2u  VCHR    4,2   0t3748 63113092 /dev/ 
>>>>> ttyp2
>>>>> perl    21977 juanfc    3u  VCHR    4,2   0t3748 63113092 /dev/ 
>>>>> ttyp2
>>>>> perl    21977 juanfc    4r  VREG                          stat(/
>>>>> ...
>>>>> Please, help.
>>>>>
>>>>>
>>>>> --
>>>>>      Juan Falgueras
>>>>>      Profesor del Depto. de Lenguajes y Ciencias de la Computación
>>>>>      Universidad de Málaga
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Bioperl-l mailing list
>>>>> Bioperl-l at lists.open-bio.org
>>>>> http://lists.open-bio.org/mailman/listinfo/bioperl-l
>>>>
>>>>
>>>
>>>
>>>
>>> ---- ende
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Bioperl-l mailing list
>>> Bioperl-l at lists.open-bio.org
>>> http://lists.open-bio.org/mailman/listinfo/bioperl-l
>>>
>>
>> -- 
>> ===========================================================
>> : Hilmar Lapp  -:-  Durham, NC  -:-  hlapp at gmx dot net :
>> ===========================================================
>>
>>
>>
>>
>>
>
>
>
> ---- ende
>
>
>

-- 
===========================================================
: Hilmar Lapp  -:-  Durham, NC  -:-  hlapp at gmx dot net :
===========================================================








More information about the Bioperl-l mailing list