Closes#5860
In streams2, there is an "old mode" for compatibility. Once switched
into this mode, there is no going back.
With this change, there is a "flowing mode" and a "paused mode". If you
add a data listener, then this will start the flow of data. However,
hitting the `pause()` method will switch *back* into a non-flowing mode,
where the `read()` method will pull data out.
Every time `read()` returns a data chunk, it also emits a `data` event.
In this way, a passive data listener can be added, and the stream passed
off to some other reader, for use with progress bars and the like.
There is no API change beyond this added flexibility.
The Streams API doc is now broken up into 3 sections:
1. API for Consumers
2. API for Implementors
3. Details and Extras
This addresses one of the biggest points of confusion for new users who
start to consume streams, and get the impression that they have to do
lots of extra work and implement classes and such, just to get some data
out of a file.
If there is an encoding, and we do 'stream.push(chunk, enc)', and the
encoding argument matches the stated encoding, then we're converting from
a string, to a buffer, and then back to a string. Of course, this is a
completely pointless bit of work, so it's best to avoid it when we know
that we can do so safely.
Fix#5272
The consumption of a readable stream is a dance with 3 partners.
1. The specific stream Author (A)
2. The Stream Base class (B), and
3. The Consumer of the stream (C)
When B calls the _read() method that A implements, it sets a 'reading'
flag, so that parallel calls to _read() can be avoided. When A calls
stream.push(), B knows that it's safe to start calling _read() again.
If the consumer C is some kind of parser that wants in some cases to
pass the source stream off to some other party, but not before "putting
back" some bit of previously consumed data (as in the case of Node's
websocket http upgrade implementation). So, stream.unshift() will
generally *never* be called by A, but *only* called by C.
Prior to this patch, stream.unshift() *also* unset the state.reading
flag, meaning that C could indicate the end of a read, and B would
dutifully fire off another _read() call to A. This is inappropriate.
In the case of fs streams, and other variably-laggy streams that don't
tolerate overlapped _read() calls, this causes big problems.
Also, calling stream.shift() after the 'end' event did not raise any
kind of error, but would cause very strange behavior indeed. Calling it
after the EOF chunk was seen, but before the 'end' event was fired would
also cause weird behavior, and could lead to data being lost, since it
would not emit another 'readable' event.
This change makes it so that:
1. stream.unshift() does *not* set state.reading = false
2. stream.unshift() is allowed up until the 'end' event.
3. unshifting onto a EOF-encountered and zero-length (but not yet
end-emitted) stream will defer the 'end' event until the new data is
consumed.
4. pushing onto a EOF-encountered stream is now an error.
So, if you read(), you have that single tick to safely unshift() data
back into the stream, even if the null chunk was pushed, and the length
was 0.
Now that highWaterMark increases when there are large reads, this
greatly reduces the number of calls necessary to _read(size), assuming
that _read actually respects the size argument.
The first example uses Readable, and shows the use of
readable.unshift(). The second uses the Transform class, showing that
it's much simpler in this case.
It seems like a good idea on the face of it, but lowWaterMarks are
actually not useful, and in practice should always be set to zero.
It would be worthwhile for writers if we actually did some kind of
writev() type of thing, but actually this just delays calling write()
and the overhead of doing a bunch of Buffer copies is not worth the
slight benefit of calling write() fewer times.
We detect for non-string and non-buffer values in onread and
turn the stream into an "objectMode" stream.
If we are in "objectMode" mode then howMuchToRead will
always return 1, state.length will always have 1 appended
to it when there is a new item and fromList always takes
the first value from the list.
This means that for object streams, the n in read(n) is
ignored and read() will always return a single value
Fixed a bug with unpipe where the pipe would break because
the flowing state was not reset to false.
Fixed a bug with sync cb(null, null) in _read which would
forget to end the readable stream
Starting a line with `**bold**` text makes it think that it's a link,
and get confused.
This should really be fixed properly in the doc generator, but for now,
it's not a major issue. It's probably just a matter of updating marked.
Mostly quite minor edits. Those possibly of more interest are:
emitter.setMaxListeners(n)
That the limit is per event name for an emitter.
fs.readlink()
Not a path, but rather the symbolic link's string value, which
would be at best a partial path, certainly not a 'resolvedPath'
global.__filename
This may be "well-known" but this is a full path to the module
that referencing code is running in. It is not the main program's
path, unless you are in the main program. Each module knows only
its own path.
server.listen(port,...)
I actually needed this functionality... "gimme just _any_ next port"
stream.end()
stream.destroy()
Yeah, everybody knows what happens to the queued data, but let's
make it *really* explicit for the first readers.
Squashed:
* Simple change to make Stream.pipe(destination) return the destination Stream
* Test: ensure Stream.pipe(destination) returns the destination Stream
* updated Stream.pipe() documentation to reflect that it now returns the
destination stream
ChrisWren
Eivind Uggedal
Evan Solomon
isaacs
Robert Kowalski
Fedor Indutny
Iskren Ivov Chernev
Lars-Magnus Skog
Gil Pedersen
Evan Oxfeld
Nathan Rajlich
Raynos
Andreas Madsen
Ben Noordhuis
Dominic Tarr
Ben Kelly
Philip Tellis
koichik
domenic
Ryan Dahl
Thomas Shinnick
Elijah Insua
Zachary Scott
Mikeal Rogers