In cases where many small writes are made to a stream
lacking _writev, the array data structure backing the
WriteReq buffer would greatly increase GC pressure.
Specifically, in the fs.WriteStream case, the
clearBuffer routine would only clear a single WriteReq
from the buffer before exiting, but would cause the
entire backing array to be GC'd. Switching to [].shift
lessened pressure, but still the bulk of the time was
spent in memcpy.
This replaces that structure with a linked list-backed
queue so that adding and removing from the queue is O(1).
In the _writev case, collecting the buffer requires an
O(N) loop over the buffer, but that was already being
performed to collect callbacks, so slowdown should be
neglible.
PR-URL: https://github.com/joyent/node/pull/8826
Reviewed-by: Timothy J Fontaine <tjfontaine@gmail.com>
Reviewed-by: Trevor Norris <trev.norris@gmail.com>
Fixes usage of global object 'Buffer' in lib/* files by ensuring that
each file does an explicit require('buffer').Buffer. Previously, when
running a repl, due to usage of global 'Buffer', any redefinition of
Buffer would cause a crash eg var Buffer = {}.
Fixes: https://github.com/joyent/node/issues/8588
PR-URL: https://github.com/joyent/node/pull/8603
Reviewed-by: Trevor Norris <trev.norris@gmail.com>
Default highWaterMark is now set properly when using stream Duplex's
writableObjectMode and readableObjectMode options.
Added condition to the already existing split objectMode test to ensure
the highWaterMark is being set to the correct default value on both the
ReadableState and WritableState for readableObjectMode and
writableObjectMode.
Signed-off-by: Fedor Indutny <fedor@indutny.com>
This commit introduces `readableObjectMode` and
`writableObjectMode` options for Duplex streams.
This can be used mostly to make parsers and
serializers with Transform streams.
Also the docs section about stream state objects
is removed, because it is not relevant anymore.
The example from the section is remade to show
new options.
fixes#6284
Signed-off-by: Timothy J Fontaine <tjfontaine@gmail.com>
If a write is above the highWaterMark, _write still manages to
fully send it synchronously, _writableState.length will be adjusted down
to 0 synchronously with the write returning false, but 'drain' will
not be emitted until process.nextTick.
If another small write which is below highWaterMark is issued before
process.nextTick happens, _writableState.needDrain will be reset to false,
and the drain event will never be fired.
So we should check needDrain before setting it up, which prevents it
from inproperly resetting to false.
Since the encoding is no longer relevant once it is decoded to a Buffer,
it is confusing and incorrect to pass the encoding as 'utf8' or whatever
in those cases.
Closes#6119
Pretty much everything assumes strings to be utf-8, but crypto
traditionally used binary strings, so we need to keep the default
that way until most users get off of that pattern.
This adds proper support for the following situation:
w.cork();
w.write(...);
w.cork();
w.write(...);
w.uncork();
w.write(...);
w.uncork();
This is relevant when you have a function (as we do in HTTP) that wants
to use cork, but in some cases, want to have a cork/uncork *around*
that function, without losing the benefits of writev.
In synchronous Writable streams (where the _write cb is called on the
current tick), the 'finish' event (and thus the end() callback) can in
some cases be called before all the write() callbacks are called.
Use a counter, and have stream.Transform rely on the 'prefinish' event
instead of the 'finish' event.
This has zero effect on most streams, but it corrects an edge case and
makes it perform more deterministically, which is a Good Thing.
Since 049903e, an end callback could be called before a write
callback if end() is called before the write is done. This patch
resolves the issue.
In collaboration with @gne
Fixes felixge/node-formidable#209
Fixes#5215
1. Get rid of unnecessary 'finishing' flag
2. Dont check both ending and ended. Extraneous.
Also: Remove extraneous 'finishing' flag, and don't check both 'ending'
and 'ended', since checking just 'ending' is sufficient.
Calling end(data) calls write(data). Doing this after end should
raise a 'write after end' error.
However, because end() calls were previously ignored on already
ended streams, this error was confusingly suppressed, even though the
data never is written, and cannot get to the other side.
This is a re-hash of 5222d19a11, but
without assuming that the data passed to end() is valid, and thus
breaking a bunch of tests.
Calling end(data) calls write(data). Doing this after end should
raise a 'write after end' error.
However, because end() calls were previously ignored on already
ended streams, this error was confusingly suppressed, even though the
data never is written, and cannot get to the other side.
The stock writable stream "write after end" message is overly vague, if
you have clearly not called end() yourself yet.
When we receive a FIN from the other side, and call destroySoon() as a
result, then generate an EPIPE error (which is what would happen if you
did actually write to the socket), with a message explaining what
actually happened.
This handles the fact that stream.Writable inherits from the Stream class,
meaning that it has the legacy pipe() method. Override that with a pipe()
method that emits an error.
Ensure that Duplex streams ARE still pipe()able, however.
Since the 'readable' flag on streams is sometimes temporary, it's probably
better not to put too much weight on that. But if something is an instanceof
Writable, rather than of Readable or Duplex, then it's safe to say that
reading from it is the wrong thing to do.
Fix#3647
The Readable and Writable classes will nextTick certain things
if in sync mode. The sync flag gets unset after a call to _read
or _write. However, most of these behaviors should also be
deferred until nextTick if no reads have been made (for example,
the automatic '_read up to hwm' behavior on Readable.push(chunk))
Set the sync flag to true in the constructor, so that it will not
trigger an immediate 'readable' event, call to _read, before the
user has had a chance to set a _read method implementation.
It seems like a good idea on the face of it, but lowWaterMarks are
actually not useful, and in practice should always be set to zero.
It would be worthwhile for writers if we actually did some kind of
writev() type of thing, but actually this just delays calling write()
and the overhead of doing a bunch of Buffer copies is not worth the
slight benefit of calling write() fewer times.
This is more backwards-compatible with stream1 streams like `fs.WriteStream`
which would allow a callback function to be passed in as the only argument.
Closes#4719.
We detect for non-string and non-buffer values in onread and
turn the stream into an "objectMode" stream.
If we are in "objectMode" mode then howMuchToRead will
always return 1, state.length will always have 1 appended
to it when there is a new item and fromList always takes
the first value from the list.
This means that for object streams, the n in read(n) is
ignored and read() will always return a single value
Fixed a bug with unpipe where the pipe would break because
the flowing state was not reset to false.
Fixed a bug with sync cb(null, null) in _read which would
forget to end the readable stream
There was previously an assert() in there, but this part of the code is
so high-volume that the added cost made a measurable dent in http_simple.
Just checking inline is fine, though, and prevents a lot of potential
hazards.
Chris Dickinson
Emmanuel Odeke
Brian White
Johnny Ray
Rod Vagg
Andrew Oppenlander
Vladimir Kurchatkin
Fedor Indutny
ayanamist
Mathias Buus
isaacs
Matthew Aynalem
Ben Noordhuis
Nathan Rajlich
Raynos