ArianeSoft Forums

ArianeSoft Forums (http://forums.arianesoft.ca//index.php)
-   Support (http://forums.arianesoft.ca//forumdisplay.php?f=19)
-   -   Memory leak in ReadString? (http://forums.arianesoft.ca//showthread.php?t=2030)

Mike Halliday 11-02-2010 11:14 AM

Memory leak in ReadString?
 
I have an application that reads lines from a text file, parses the string and writes a new text file depending on the input string;

The source file is 1.813MB in size (Not massive, but big) and is a dump from Active Directory (50 lines per record, blank line as next record marker) - There are 56,335 lines total in the file.

The destination is a .CSV file.

I open both files (Source as read only) (Destination as write)

Pseudo code:

While (FEOF(input) == 0)
line = readstring(input)

if (mid(line,0,13) == "printername: ")
output = output + Mid(line,14, length(line))
end

<6 more checks for specific lines in the input file>

if (mid(line,0,1) == " ") // end of this current record
writestring(output)
end

end

My app seeks through about 13 complete records then throws an exception error. (Access Violation at 0x40e223)

I cant see why, other than PPL is running out of memory.

Is there a known memory leak with ReadString?

I have tried loading the whole file into a string to manipulate as a list, but it too runs out of memory (How big can a string be? - Not 1.8MB I guess!)

Frustrating me now, as all I am trying to do is open a file, read the lines in succession then output to another file when criteria is found. - Not asking much!

Oh, and I have tried this in 1.53 and 2.2.1 and both do the same thing.

Any suggestions guys?

<EDIT> - Ok, so I have run my code on 2 different machines and get the same problem. Machine 1 is W7 with 3GB ram and Machine 2 is XPSP3 with 1GB ram. Both fall over at roughly the same place.

Got to be memory related as the machine with 1GB warned me about Virtual Memory being low, just before it fell over.

kornalius 11-02-2010 12:30 PM

Hi Mike,

A memory leak is possible. Try free(line$); at the end of the loop, just to make sure it is a memory leak.

I would simplify the code even more to really make sure it is in ReadString.

Code:

While (FEOF(input) == 0)
  line = readstring(input)
    free(line)
end


Mike Halliday 11-02-2010 12:40 PM

Hmmm just tried that and it seems to be the same.

The first 100 or so lines whizz up the screen (in the console) then it slows to a crawl, then bombs out.

Is there any other way of reading lines from a text file?


Edit: Interestingly it seems to bomb on the readstring line.

Mike Halliday 11-02-2010 02:23 PM

As a test, I have coded a small app that reads the input file line by line, then immediately writes it to an output file

Again, the first 100 lines or so whizz by in the console output, then it slows to about 2 lines per second.

While this is going on, the PC becomes unresponsive - I can't even get task manager to load to check resources.

It then crashes :(

The output file is only 13KB in size at this stage and has only written 418 full lines and 1/2 of line 419!! :confused:

Frustrated!!!

Mike Halliday 11-02-2010 02:45 PM

Task manager is now loaded when the app is running;

CPU tops out at 10% usage
Page file use shoots up to 1.9GB

Physical Memory - Total (1039732), Available (2729 min, 4552 max)
Commit charge - 2304M/2506M

As soon as the app crashes, Page file goes back to 348MB, cpu = 2% Available memory is back to 886516 and Commit charge is 348M/2506M


:( defo looks like there is a huge memory drain when running!

As a comparison, I have run one of my previous apps (Form, text boxes , buttons and sip keyboard) and page file never rises more then 355MB and Available memory is stuck at 868312. CPU is topped out at 100% though.

Mike Halliday 11-02-2010 04:07 PM

Progress!!!

It appears to be the size of the source file that is causing the problem. I have reduced its size from 1.8mb to 146KB (By removing fluff that was not needed) and my code whizzes through it like a hot knife through butter and does not crash!

Woo hoo.

Its not sorted out the fact I cant read long files, but for what I need it for at the current time, perfect result!

Be warned anyone using READSTRING. Dont make your input file too big.

juppmaifeld 11-02-2010 05:54 PM

Hi Kornalius, Hi Mike,

I have detected the same problem using the functions ARRAYTOSTR() and CHAR() in a thread. What I discovered is, that, if you use a function that returns a string the amount of memory the system needs increases with every call of the function. For example. I use Nicknacks tcp.ppl library. If I call the tcp_recv() function in a loop you can watch the memory growth in the windows task manager. If you eliminate the "return (char(buffer$));" like I have done it in Mytcp_recv just for test reasons the problem disappears. One can argue that the problem maybe comes from not freeing the buffer$ in "tcp_recv(s$, sz$)". But that's not the case. I checked that out by successivly commenting out parts of the code of tcp_recv() to figure out which part of the code causes the problem. The same problem occured using ARRAYTOSTR() in a thread. I wrote my own ARRAYTOSTR() procedure to avoid returning a string, and it worked fine. I am not sure, but from my experience with all that testing it appears to me that there is maybe a problem that causes a memory leak if a function returns a string.

Jupp


func tcp_recv(s$, sz$)

local(buffer$, readcnt$);
// PPL 1.53, 1.62 dosen't know var(buffer$, sz$);
SDIM(buffer$, TBYTE, sz$);

if (s$ == 0)
s$ = tcp_socket$;
end;

readcnt$ = recv(s$, &buffer$, sz$, 0);

if (readcnt$ < 0)
return ( - 1);
else
setbyte(&buffer$ + readcnt$, 0);
return (char(buffer$));
end;
end;


proc Mytcp_recv(s$, sz$, strData$, res$)

local(buffer$, readcnt$);

// PPL 1.53, 1.62 dosen't know var(buffer$, sz$);
SDIM(buffer$, TBYTE, sz$);

if (s$ == 0)
s$ = tcp_socket$;
end;

readcnt$ = recv(s$, &buffer$, sz$, 0);

if (readcnt$ < 0)
res$ = -1;
else
setbyte(&buffer$ + readcnt$, 0);
if (iswide(buffer$))
strData$ = char(buffer$);
else
strData$ = buffer$;
end;
res$ = 1;
end;

free(buffer$);
end;



proc MyArrayToStr(arr$, arrlength$, string$, mode$)

local(i$, s$);

s$ = "";
string$ = "";

if (mode$ == TCP_MODE)
if (arrlength$ == 1)
sprintf(string$, "%03u", arr$[0]);
else
for (i$, 0, arrlength$ - 1, 1)
if (i$ == arrlength$ - 1)
sprintf(s$, "%03u", arr$[i$]);
else
sprintf(s$, "%03u ", arr$[i$]);
end;
string$ = string$ % s$;
// showmessage("s$ = " % s$ % "\n" %
// "string$ = " % string$);
end;
end;

else

if (arrlength$ == 1)
sprintf(string$, "%u", arr$[0]);
else
for (i$, 0, arrlength$ - 1, 1)
sprintf(s$, "%u", arr$[i$]);
if (i$ == arrlength$ - 1)
string$ = string$ % s$;
else
string$ = string$ % s$ % " ";
end;
end;
end;
end;
end;

zehlein 11-04-2010 08:05 PM

I remember I had a similar issue. While parsing a text file memory usage went sky-high and the app crashed. This happened with PPL 1.53 and 1.6.
Interesting though is the fact that there has been a PPL version without that bug, I have an early compiled version of the app that works like a charm.

kornalius 11-09-2010 05:29 PM

This is something I will have to take a look at. I just don't know when I will have the time to get back into PPL. :(


All times are GMT. The time now is 03:21 PM.

Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.
Copyright (c) ArianeSoft Inc.