Read large text files in nodejs
I've been playing around with NodeJS recently and got the following error when trying to read a very large text file:
FATAL ERROR: CALL_AND_RETRY_0 Allocation failed - process out of memoryThe following solution allows you to stream a file instead of reading it all into memory:
var fs = require('fs');
var readline = require('readline');
var stream = require('stream');
var instream = fs.createReadStream('your/file');
var outstream = new stream;
var rl = readline.createInterface(instream, outstream);
rl.on('line', function(line) {
// process line here
});
rl.on('close', function() {
// do something on finish here
});Written by Bala Clark
Related protips
11 Responses
Hi,
I tried this solution, but I still have the same exception on large files (log files ~200Mb)
The only way i found is to use line-by-line plugin : https://npmjs.org/package/line-by-line
snake51's setup must have been faulty because this code works great. I read a 7.5GB file containing over 800 million lines.
Nice and clean solution without requiring any third party modules.
For some reason, it works up until the last line.
I'm having the same issue as jake_nerdist. Thinking I'll just append a \n to the file before I start processing it.
how do you limit the number of lines to look at. I'm just trying to debug the workflow and don't want to read the whole file every time.
also how do you change the encoding to utf-8?
Is there a way to read with a different separator (',' or '#' instead of '\n')? official documentation do not specify a parameter for that.
I'm trying to build a Straming for big audio files and serve to the browser,
I'm try with this part of code:
res.setHeader("content-type", "audio/mpeg");
var stream = fs.createReadStream('record.mp3', { bufferSize: 64 * 1024 });
stream.pipe(res);
But the problem with this is that when I put the URL into the audio tag (for example).
audio>
source src="http://127.0.0.1:8080/record.mp3" />
/audio>
Node.js read the entire file and send the complete file to the browser.
I'm working with audio files that they size is biggest that 500 mb.
Thanks.
Great article. I haven't been able to find something so clear on this type of issue. Great job.
getting errors in for extended ascii chars °C. Do you have any clue why this happens. fs.createReadStream seems to work fine...but createInterface seems to give problems.
Have a fresh tip? Share with Coderwall community!
JavaScript on a page runs with the privileges of that origin. One altered library, tag manager, or snippet can read inputs, change what people see, and call your APIs as the user.
Security insights from Cside