Add buffer.transcode(source, from, to) method. Primarily uses ICU
to transcode a buffer's content from one of Node.js' supported
encodings to another.
Originally part of a proposal to add a new unicode module. Decided
to refactor the approach towrds individual PRs without a new module.
Refs: https://github.com/nodejs/node/pull/8075
PR-URL: https://github.com/nodejs/node/pull/9038
Reviewed-By: Anna Henningsen <anna@addaleax.net>
As shown in https://github.com/nodejs/node/issues/9096, the offset and
end value of the `slice` call are coerced to numbers and then passed to
`FastBuffer`, which internally truncates the mantissa part if the number
is actually a floating point number. This actually affects the new
length of the slice calculation. For example,
> const original = Buffer.from('abcd');
undefined
> original.slice(original.length / 3).toString()
'bc'
This happens because, starting value of the slice is 4 / 3, which is
1.33 (approximately). Now, the length of the slice is calculated as
the difference between the actual length of the buffer and the starting
offset. So, it becomes 2.67 (4 - 1.33). Now, a new `FastBuffer` is
constructed, with the following values as parameters,
1. actual buffer object,
2. starting value, which is 1.33 and
3. the length 2.67.
The underlying C++ code truncates the numbers and they become 1 and 2.
That is why the result is just `bc`.
This patch makes sure that all the offsets are coerced to integers
before any calculations are done.
Fixes: https://github.com/nodejs/node/issues/9096
PR-URL: https://github.com/nodejs/node/pull/9101
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Michaël Zasso <targos@protonmail.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Franziska Hinkelmann <franziska.hinkelmann@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: Brian White <mscdex@mscdex.net>
isSharedArrayBuffer in fromObject was missing obj.buffer
moved the 'length' in obj check so that it is checked first making
the code slightly more performant and able to handle SharedArrayBuffer
without relying on an explicit check.
Ref: https://github.com/nodejs/node/pull/8510
PR-URL: https://github.com/nodejs/node/pull/8739
Reviewed-By: Ilkka Myller <ilkka.myller@nodefield.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
This makes sure that no uninitialized bytes are leaked when the specified
`totalLength` input value is greater than the actual total length of the
specified buffers array, e.g. in Buffer.concat([Buffer.alloc(0)], 100).
PR-URL: https://github.com/nodejs/node-private/pull/64
Reviewed-By: Rod Vagg <rod@vagg.org>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
V8 5.4 changed the way that the default constructor of derived classes
is called. It introduced a significant performance regression in the
buffer module for the creation of pooled buffers. This commit forces the
definition back to how it was implicitly before.
Ref: https://bugs.chromium.org/p/v8/issues/detail?id=4890
PR-URL: https://github.com/nodejs/node/pull/8754
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Trevor Norris <trev.norris@gmail.com>
Reviewed-By: Ilkka Myller <ilkka.myller@nodefield.com>
Reviewed-By: Fedor Indutny <fedor.indutny@gmail.com>
Reviewed-By: Johan Bergström <bugs@bergstroem.nu>
In Buffer.prototype.compare, the first check makes sure that target is
an instance of Buffer. The value cannot be falsy after that so we can
safely get its length.
PR-URL: https://github.com/nodejs/node/pull/8552
Reviewed-By: Rich Trott <rtrott@gmail.com>
Reviewed-By: Сковорода Никита Андреевич <chalkerx@gmail.com>
This commit fixes detection of ArrayBuffers from different V8 contexts.
This is especially a problem for environments like nw.js where the
node and browser V8 contexts are not shared.
PR-URL: https://github.com/nodejs/node/pull/8453
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Sakthipriyan Vairamani <thechargingvolcano@gmail.com>
If the Buffer object's length is zero, or equal to the underlying
buffer object's length, `parent` property returns `undefined`.
> new Buffer(0).parent
undefined
> new Buffer(Buffer.poolSize).parent
undefined
This patch makes the buffer objects to consistently expose the buffer
object via the `parent` property, always.
Fixes: https://github.com/nodejs/node/issues/8266
PR-URL: https://github.com/nodejs/node/pull/8311
Reviewed-By: Trevor Norris <trev.norris@gmail.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: James M Snell <jasnell@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/8169
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Stephen Belanger <admin@stephenbelanger.com>
Reviewed-By: Сковорода Никита Андреевич <chalkerx@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Add a `util.inspect.custom` Symbol which can be used to customize
`util.inspect()` output. Providing `obj[util.inspect.custom]`
works like providing `obj.inspect`, except that the former allows
avoiding name clashes with other `inspect()` methods.
Fixes: https://github.com/nodejs/node/issues/8071
PR-URL: https://github.com/nodejs/node/pull/8174
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Michaël Zasso <mic.besace@gmail.com>
Add a check for `size < 0` to `assertSize()`, as passing a negative
value almost certainly indicates a programming error.
This also lines up the behaviour of `.allocUnsafe()` with the ones
of `.alloc()` and `.allocUnsafeSlow()` (which previously threw errors
from the Uint8Array constructor).
Notably, this also affects `Buffer()` calls with negative arguments.
PR-URL: https://github.com/nodejs/node/pull/7079
Reviewed-By: Trevor Norris <trev.norris@gmail.com>
Reviewed-By: Сковорода Никита Андреевич <chalkerx@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Yorkie Liu <yorkiefixer@gmail.com>
* Speed up buffer.swap16 and swap32 by using builtins. Up to ~6x gain.
Drop transition point between JS and C++ implementations accordingly.
Amount of performance improvement not only depends on buffer size but
also memory alignment.
* Fix tests: C++ impl tests were testing 0-filled buffers so were
always passing.
* Add similar buffer.swap64 method.
* Make buffer-swap benchmark mirror JS impl.
doc/api/buffer.markdown has an entry of "added: REPLACEME" that should
be changed to the correct release number before tagged.
Because node is currently using a very old version of cpplint.py it
doesn't know that std::swap() has moved from <algorithm> to <utility> in
c++11. So until cpplint.py is updated simply NOLINT the line.
Technically it should be NOLINT(build/include_what_you_use), but that
puts the line over 80 characters causing another lint error.
PR-URL: https://github.com/nodejs/node/pull/7157
Reviewed-By: Trevor Norris <trev.norris@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Buffer.isEncoding and string_decoder.normalizeEncoding shared
quite a bit of logic. This moves the primary logic into
internal/util. The userland modules that monkey patch Buffer.isEncoding
should still work.
PR-URL: https://github.com/nodejs/node/pull/7207
Reviewed-By: Brian White <mscdex@mscdex.net>
Reviewed-By: Trevor Norris <trev.norris@gmail.com>
Fixes regression where creating a new Buffer from an
empty ArrayBuffer would fail.
Ref: 85ab4a5f12
PR-URL: https://github.com/nodejs/node/pull/7176
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Сковорода Никита Андреевич <chalkerx@gmail.com>
Reviewed-By: Ron Korving <ron@ronkorving.nl>
This comment applied to a line that was removed in
dd67608bfd
and is no longer relevant.
PR-URL: https://github.com/nodejs/node/pull/7264
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
When node began using the OneByte API (f150d56) it also switched to
officially supporting ISO-8859-1. Though at the time no new encoding
string was introduced.
Introduce the new encoding string 'latin1' to be more explicit. The
previous 'binary' and documented as an alias to 'latin1'. While many
tests have switched to use 'latin1', there are still plenty that do both
'binary' and 'latin1' checks side-by-side to ensure there is no
regression.
PR-URL: https://github.com/nodejs/node/pull/7111
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: James M Snell <jasnell@gmail.com>
Improves performance of allocating unsafe buffers, creating buffers from
an existing ArrayBuffer and creating .slice(...) from existing Buffer by
avoiding deoptimizing change of prototype after Uint8Array allocation
in favor of ES6 native subclassing.
This is done through an internal ES6 class that extends Uint8Array and
is used for allocations, but the regular Buffer function is exposed, so
calling Buffer(...) with or without `new` continues to work as usual
and prototype chains are also preserved.
Performance wins for .slice are +120% (2.2x), and, consequently, for
unsafe allocations up to +95% (1.9x) for small buffers, and for safe
allocations (zero-filled) up to +30% (1.3x).
PR-URL: https://github.com/nodejs/node/pull/6893
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Сковорода Никита Андреевич <chalkerx@gmail.com>
Exceptions thrown from the Uint8Array constructor would leave it
disabled.
Regression introduced in commit 27e84dd ("lib,src: clean up
ArrayBufferAllocator") from two days ago. A follow-up commit
will add a regression test.
PR-URL: https://github.com/nodejs/node/pull/7093
Reviewed-By: Сковорода Никита Андреевич <chalkerx@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Trevor Norris <trev.norris@gmail.com>
Remove the direct dependency on node::Environment (which is per-context)
from node::ArrayBufferAllocator (which is per-isolate.)
Contexts that want to toggle the zero fill flag, now do so through a
field that is owned by ArrayBufferAllocator. Better, still not great.
PR-URL: https://github.com/nodejs/node/pull/7082
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Trevor Norris <trev.norris@gmail.com>
Treat negative length arguments to `Buffer()`/`allocUnsafe()`
as if they were zero so the allocation does not affect the
pool’s offset.
Fixes: https://github.com/nodejs/node/issues/7047
PR-URL: https://github.com/nodejs/node/pull/7051
Reviewed-By: Sakthipriyan Vairamani <thechargingvolcano@gmail.com>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Reviewed-By: Сковорода Никита Андреевич <chalkerx@gmail.com>
Reviewed-By: Trevor Norris <trev.norris@gmail.com>
Reviewed-By: Rod Vagg <rod@vagg.org>
This makes sure that `kNoZeroFill` flag is not accidentally set by
moving the all the flag operations directly inside `createBuffer()`.
It safeguards against logical errors like
https://github.com/nodejs/node/issues/6006.
This also ensures that `kNoZeroFill` flag is always restored to 0 using
a try-finally block, as it could be not restored to 0 in cases of failed
or zero-size `Uint8Array` allocation.
It safeguards against errors like
https://github.com/nodejs/node/issues/2930.
It also makes the `size > 0` check not needed there.
PR-URL: https://github.com/nodejs/node-private/pull/30
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Trevor Norris <trev.norris@gmail.com>
* Remove unnecessary templating from SearchString
SearchString used to have separate PatternChar and SubjectChar template type
arguments, apparently to support things like searching for an 8-bit string
inside a 16-bit string or vice versa. However, SearchString is only used from
node_buffer.cc, where PatternChar and SubjectChar are always the same. Since
this is extra complexity that's unused and untested (simplifying to a single
Char template argument still compiles and didn't break any unit tests), I
removed it.
* Use Boyer-Hoore[-Horspool] for both indexOf and lastIndexOf
Add test cases for lastIndexOf. Test the fallback from BMH to
Boyer-Moore, which looks like it was totally untested before.
* Extra bounds checks in node_buffer.cc
* Extra asserts in string_search.h
* Buffer.lastIndexOf: clean up, enforce consistency w/ String.lastIndexOf
* Polyfill memrchr(3) for non-GNU systems
PR-URL: https://github.com/nodejs/node/pull/4846
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Trevor Norris <trev.norris@gmail.com>
Aligns the functionality of SlowBuffer with the new Buffer
constructor API. Next step is to docs-only deprecate
SlowBuffer.
Replace the internal uses of SlowBuffer with
`Buffer.allocUnsafeSlow(size)`
PR-URL: https://github.com/nodejs/node/pull/5833
Reviewed-By: Сковорода Никита Андреевич <chalkerx@gmail.com>
Reviewed-By: Trevor Norris <trev.norris@gmail.com>
Reviewed-By: Sakthipriyan Vairamani <thechargingvolcano@gmail.com>
Adds additional `targetStart`, `targetEnd`, `sourceStart,
and `sourceEnd` arguments to `Buffer.prototype.compare`
to allow comparison of sub-ranges of two Buffers without
requiring Buffer.prototype.slice()
Fixes: https://github.com/nodejs/node/issues/521
PR-URL: https://github.com/nodejs/node/pull/5880
Reviewed-By: Trevor Norris <trev.norris@gmail.com>
If `kNoZeroFill` is set here, it won't be reset in case of
pooled allocation. In case of "slow" allocation it will be
set later anyway.
Fixes: https://github.com/nodejs/node/issues/6006
PR-URL: https://github.com/nodejs/node/pull/6007
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Evan Lucas <evanlucas@me.com>
Reviewed-By: Myles Borins <myles.borins@gmail.com>
Convert anything to string, but Buffer, TypedArray and ArrayBuffer
```
var uint8 = new Uint8Array([0xf0, 0x9f, 0x90]);
Buffer.byteLength(uint8); // should be 3, but returns 11
Buffer.byteLength(uint8.buffer); // should be 3, but return 20
```
PR-URL: https://github.com/nodejs/node/pull/5255
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Reviewed-By: Trevor Norris <trev.norris@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
When create Buffer from empty string will touch
C++ binding also.
This patch can improve edge case ~70% faster.
PR-URL: https://github.com/nodejs/node/pull/4414
Reviewed-By: Trevor Norris <trev.norris@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Using let in for loops showed a regression in 4.4.0. @ofrobots
suggested that we avoid using let in for loops until TurboFan becomes
the default optimiser.
The regression that was detected was when looking at how long it took
to create a new buffer from an array of data.
When using `for (let i=0; i<length; i++) ` we saw the operation take
almost 40% longer compared to `var i=0`.
PR-URL: https://github.com/nodejs/node/pull/5819
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Michael Dawson <michael_dawson@ca.ibm.com>
Reviewed-By: Jeremiah Senkpiel <fishrock123@rocketmail.com>
Reviewed-By: Trevor Norris <trevnorris@gmail.com>
Reviewed-By: Myles Borins <myles.borins@gmail.com>
Ref: http://github.com/nodejs/benchmarking/issues/38
Several changes:
* Soft-Deprecate Buffer() constructors
* Add `Buffer.from()`, `Buffer.alloc()`, and `Buffer.allocUnsafe()`
* Add `--zero-fill-buffers` command line option
* Add byteOffset and length to `new Buffer(arrayBuffer)` constructor
* buffer.fill('') previously had no effect, now zero-fills
* Update the docs
PR-URL: https://github.com/nodejs/node/pull/4682
Reviewed-By: Сковорода Никита Андреевич <chalkerx@gmail.com>
Reviewed-By: Stephen Belanger <admin@stephenbelanger.com>
Can now call fill() using following parameters if value is a String:
fill(string[, start[, end]][, encoding])
And with the following if value is a Buffer:
fill(buffer[, start[, end]])
The encoding is ignored if value is not a String. All other non-Buffer
values are coerced to a uint32.
A multibyte strings will simply be copied into the Buffer until the
number of bytes run out. Meaning partial strings can be left behind:
Buffer(3).fill('\u0222');
// returns: <Buffer c8 a2 c8>
In some encoding cases, such as 'hex', fill() will throw if the input
string is not valid.
PR-URL: https://github.com/nodejs/node/pull/4935
Reviewed-By: James M Snell <jasnell@gmail.com>
Add fromArrayLike() to handle logic of copying in values from array-like
argument.
PR-URL: https://github.com/nodejs/node/pull/4948
Reviewed-By: Sakthipriyan Vairamani <thechargingvolcano@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Trevor Norris <trev.norris@gmail.com>
Without this change, if any of the elements in the list to be concatenated is
not a Buffer instance, the method fails with "buf.copy is not a function".
Make an isBuffer check before using the copy method so that we can throw with
a better message.
Fixes: https://github.com/nodejs/node/issues/4949
PR-URL: https://github.com/nodejs/node/pull/4951
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Sakthipriyan Vairamani <thechargingvolcano@gmail.com>
Reviewed-By: Roman Klauke <romaaan.git@gmail.com>
A handful of variable declarations in `lib/buffer.js` redeclare the same
variable in the same scope. This change removes each redeclaration by
switching to `const`, switching to `let`, or explicitly hoisting the
`var` declaration.
PR-URL: https://github.com/nodejs/node/pull/4886
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Roman Klauke <romaaan.git@gmail.com>
Currently the signature is indexOf(val[, byteOffset[, encoding]])
Instead allow indexOf(val[, byteOffset][, encoding])
so that byteOffset does not need to be passed.
PR-URL: https://github.com/nodejs/node/pull/4803
Reviewed-By: James M Snell <jasnell@gmail.com>
Make the byteLength work correctly when input is Buffer.
e.g:
```js
// The incomplete unicode string
Buffer.byteLength(new Buffer([0xe4, 0xb8, 0xad, 0xe6, 0x96]))
```
The old output: 9
The new output: 5
PR-URL: https://github.com/nodejs/node/pull/4738
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Reviewed-By: Brian White <mscdex@mscdex.net>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
These have been deprecated since Apr 27, 2013, and the plan was to
remove them in "node v0.13".
buffer.get(index) is superseded by buffer[index].
buffer.set(index, value) is superseded by buffer[index] = value.
These have never been documented at any point in node's history.
PR-URL: https://github.com/nodejs/node/pull/4594
Fixes: https://github.com/nodejs/node/issues/4587
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Trevor Norris <trev.norris@gmail.com>
Reviewed-By: Brian White <mscdex@mscdex.net>
Reviewed-By: Roman Reiss <me@silverwind.io>
When create Buffer from a Buffer will copy data
from old to new even though length is zero.
This patch can improve edge case 4x faster.
following is benchmark results.
new: buffers/buffer_zero.js n=1024: 2463.53891
old: buffers/buffer_zero.js n=1024: 618.70801
PR-URL: https://github.com/nodejs/node/pull/4326
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Add Buffer#includes() by wrapping an indexOf and performing a strict
equals check to -1.
The includes method takes the search value, byteOffset, and encoding as
arguments.
The test is a modified version of the indexOf test.
Fixes: https://github.com/nodejs/node/issues/3552
PR-URL: https://github.com/nodejs/node/pull/3567
Reviewed-By: Trevor Norris <trev.norris@gmail.com>
Reviewed-By: Evan Lucas <evanlucas@me.com>
Reviewed-By: Jeremiah Senkpiel <fishrock123@rocketmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
The Buffer constructor will generally get inlined, but any call to the Buffer
constructor for a string without encoding will cause an eager deoptimization
of any function that inlined the Buffer constructor. This is due to a an
out-of-bounds read on `arguments[1]`. This change prevents that deopt.
PR-URL: https://github.com/nodejs/node/pull/4158
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Trevor Norris <trev.norris@gmail.com>
Reviewed-By: Minwoo Jung <jmwsoft@gmail.com>
James M Snell
Sakthipriyan Vairamani (thefourtheye)
Brian White
Ojas Shirekar
Сковорода Никита Андреевич
Michaël Zasso
Anna Henningsen
Nikolai Vavilov
Zach Bjornson
Ingvar Stepanyan
Trevor Norris
Ben Noordhuis
Justin Sprigg
dcposch@dcpos.ch
Vladimir Kurchatkin
Jackson Tian
Gareth Ellis
HUANG Wei
Michaël Zasso
Rich Trott
Peter Geiss
Feross Aboukhadijeh
Mathias Buus
Alexander Martin
Matt Loring
Bryce Baril