Update cached response after revalidation

README.md

@@ -63,7 +63,7 @@ const options = {
 };
 ```
 
-If `options.shared` is true (default), then response is evaluated from perspective of a shared cache (i.e. `private` is not cacheable and `s-maxage` is respected). If `options.shared` is false, then response is evaluated from perspective of a single-user cache (i.e. `private` is cacheable and `s-maxage` is ignored).
+If `options.shared` is `true` (default), then the response is evaluated from a perspective of a shared cache (i.e. `private` is not cacheable and `s-maxage` is respected). If `options.shared` is `false`, then the response is evaluated from a perspective of a single-user cache (i.e. `private` is cacheable and `s-maxage` is ignored).
 
 `options.cacheHeuristic` is a fraction of response's age that is used as a fallback cache duration. The default is 0.1 (10%), e.g. if a file hasn't been modified for 100 days, it'll be cached for 100*0.1 = 10 days.
 
@@ -73,21 +73,21 @@ If `options.ignoreCargoCult` is true, common anti-cache directives will be compl
 
 Returns `true` if the response can be stored in a cache. If it's `false` then you MUST NOT store either the request or the response.
 
-### `satisfiesWithoutRevalidation(new_request)`
+### `satisfiesWithoutRevalidation(newRequest)`
 
 This is the most important method. Use this method to check whether the cached response is still fresh in the context of the new request.
 
 If it returns `true`, then the given `request` matches the original response this cache policy has been created with, and the response can be reused without contacting the server. Note that the old response can't be returned without being updated, see `responseHeaders()`.
 
-If it returns `false`, then the response may not be matching at all (e.g. it's for a different URL or method), or may require to be refreshed first.
+If it returns `false`, then the response may not be matching at all (e.g. it's for a different URL or method), or may require to be refreshed first (see `revalidationHeaders()`).
 
 ### `responseHeaders()`
 
-Returns updated, filtered set of response headers to return to clients receiving the cached response. This function is necessary, because proxies MUST always remove hop-by-hop headers (such as `TE` and `Connection`) and update response `Age` to avoid doubling cache time.
+Returns updated, filtered set of response headers to return to clients receiving the cached response. This function is necessary, because proxies MUST always remove hop-by-hop headers (such as `TE` and `Connection`) and update response's `Age` to avoid doubling cache time.
 
-### `revalidationHeaders(newRequest)`
+```js
-
+cachedResponse.headers = cachePolicy.responseHeaders(cachedResponse);
-Returns updated, filtered set of request headers to send to the origin server to check if the cached response can be reused. With this set of headers, the origin server may return status 304 indicating the response is still fresh.
+```
 
 ### `timeToLive()`
 
@@ -99,6 +99,55 @@ After that time (when `timeToLive() <= 0`) the response might not be usable with
 
 Chances are you'll want to store the `CachePolicy` object along with the cached response. `obj = policy.toObject()` gives a plain JSON-serializable object. `policy = CachePolicy.fromObject(obj)` creates an instance from it.
 
+### Refreshing stale cache (revalidation)
+
+When a cached response has expired, it can be made fresh again by making a request to the origin server. The server may respond with status 304 (Not Modified) without sending the response body again, saving bandwidth.
+
+The following methods help perform the update efficiently and correctly.
+
+#### `revalidationHeaders(newRequest)`
+
+Returns updated, filtered set of request headers to send to the origin server to check if the cached response can be reused. These headers allow the origin server to return status 304 indicating the response is still fresh. All headers unrelated to caching are passed through as-is.
+
+Use this method when updating cache from the origin server.
+
+```js
+updateRequest.headers = cachePolicy.revalidationHeaders(updateRequest);
+```
+
+#### `revalidatedPolicy(revalidationRequest, revalidationResponse)`
+
+Use this method to update the cache after receiving a new response from the origin server. It returns an object with two keys:
+
+* `policy` — A new `CachePolicy` with HTTP headers updated from `revalidationResponse`. You can always replace the old cached `CachePolicy` with the new one.
+* `modified` — Boolean indicating whether the response body has changed.
+   * If `false`, then a valid 304 Not Modified response has been received, and you can reuse the old cached response body.
+   * If `true`, you should use new response's body (if present), or make another request to the origin server without any conditional headers (i.e. don't use `revalidationHeaders()` this time) to get the new resource.
+
+```js
+// When serving requests from cache:
+const {oldPolicy, oldResponse} = letsPretendThisIsSomeCache.get(newRequest.url);
+
+if (!oldPolicy.satisfiesWithoutRevalidation(newRequest)) {
+    // Change the request to ask the origin server if the cached response can be used
+    newRequest.headers = oldPolicy.revalidationHeaders(newRequest);
+
+    // Send request to the origin server. The server may respond with status 304
+    const newResponse = await makeRequest(newResponse);
+
+    // Create updated policy and combined response from the old and new data
+    const {policy, modified} = oldPolicy.revalidatedPolicy(newRequest, newResponse);
+    const response = modified ? newResponse : oldResponse;
+
+    // Update the cache with the newer/fresher response
+    letsPretendThisIsSomeCache.set(newRequest.url, {policy, response}, policy.timeToLive());
+
+    // And proceed returning cached response as usual
+    response.headers = policy.responseHeaders();
+    return response;
+}
+```
+
 # Yo, FRESH
 
 ![satisfiesWithoutRevalidation](fresh.jpg)
@@ -119,4 +168,3 @@ Chances are you'll want to store the `CachePolicy` object along with the cached
 
 * Range requests, If-Range
 * Revalidation of multiple representations
-* Updating of response after revalidation

index.js

@@ -6,6 +6,12 @@ const statusCodeCacheableByDefault = [200, 203, 204, 206, 300, 301, 404, 405, 41
 const understoodStatuses = [200, 203, 204, 300, 301, 302, 303, 307, 308, 404, 405, 410, 414, 501];
 
 const hopByHopHeaders = {'connection':true, 'keep-alive':true, 'proxy-authenticate':true, 'proxy-authorization':true, 'te':true, 'trailer':true, 'transfer-encoding':true, 'upgrade':true};
+const excludedFromRevalidationUpdate = {
+    'etag': true, 'last-modified': true, // Per spec
+    'content-range': true,
+    // Since the old body is reused, it doesn't make sense to change properties of the body
+    'content-length': true, 'content-encoding': true, 'transfer-encoding': true,
+};
 
 function parseCacheControl(header) {
     const cc = {};
@@ -44,9 +50,7 @@ module.exports = class CachePolicy {
         if (!res || !res.headers) {
             throw Error("Response headers missing");
         }
-        if (!req || !req.headers) {
+        this._assertRequestHasHeaders(req);
-            throw Error("Request headers missing");
-        }
 
         this._responseTime = this.now();
         this._isShared = shared !== false;
@@ -403,4 +407,73 @@ module.exports = class CachePolicy {
 
         return headers;
     }
+
+    /**
+     * Creates new CachePolicy with information combined from the previews response,
+     * and the new revalidation response.
+     *
+     * Returns {policy, modified} where modified is a boolean indicating
+     * whether the response body has been modified, and old cached body can't be used.
+     *
+     * @return {Object} {policy: CachePolicy, modified: Boolean}
+     */
+    revalidatedPolicy(request, response) {
+        this._assertRequestHasHeaders(request);
+        if (!response || !response.headers) {
+            throw Error("Response headers missing");
+        }
+
+        // These aren't going to be supported exactly, since one CachePolicy object
+        // doesn't know about all the other cached objects.
+        let matches = false;
+        if (response.status !== undefined && response.status != 304) {
+            matches = false;
+        } else if (response.headers.etag && !/^\s*W\//.test(response.headers.etag)) {
+            // "All of the stored responses with the same strong validator are selected.
+            // If none of the stored responses contain the same strong validator,
+            // then the cache MUST NOT use the new response to update any stored responses."
+            matches = this._resHeaders.etag && this._resHeaders.etag.replace(/^\s*W\//,'') === response.headers.etag;
+        } else if (this._resHeaders.etag && response.headers.etag) {
+            // "If the new response contains a weak validator and that validator corresponds
+            // to one of the cache's stored responses,
+            // then the most recent of those matching stored responses is selected for update."
+            matches = this._resHeaders.etag.replace(/^\s*W\//,'') === response.headers.etag.replace(/^\s*W\//,'');
+        } else if (this._resHeaders['last-modified']) {
+            matches = this._resHeaders['last-modified'] === response.headers['last-modified'];
+        } else {
+            // If the new response does not include any form of validator (such as in the case where
+            // a client generates an If-Modified-Since request from a source other than the Last-Modified
+            // response header field), and there is only one stored response, and that stored response also
+            // lacks a validator, then that stored response is selected for update.
+            if (!this._resHeaders.etag && !this._resHeaders['last-modified'] &&
+                !response.headers.etag && !response.headers['last-modified']) {
+                matches = true;
+            }
+        }
+
+        if (!matches) {
+            return {
+                policy: new this.constructor(request, response),
+                modified: true,
+            }
+        }
+
+        // use other header fields provided in the 304 (Not Modified) response to replace all instances
+        // of the corresponding header fields in the stored response.
+        const headers = {};
+        for(const k in this._resHeaders) {
+            if (excludedFromRevalidationUpdate[k]) continue;
+            headers[k] = k in response.headers ? response.headers[k] : this._resHeaders[k];
+        }
+
+        const newResponse = Object.assign({}, response, {
+            status: this._status,
+            method: this._method,
+            headers,
+        });
+        return {
+            policy: new this.constructor(request, newResponse),
+            modified: false,
+        };
+    }
 };

package.json

@@ -1,6 +1,6 @@
 {
   "name": "http-cache-semantics",
-  "version": "3.5.1",
+  "version": "3.6.0",
   "description": "Parses Cache-Control and other headers. Helps building correct HTTP caches and proxies",
   "main": "index.js",
   "repository": "https://github.com/pornel/http-cache-semantics.git",

test/revalidatetest.js

@@ -42,7 +42,6 @@ describe('Can be revalidated?', function() {
     it('not if method mismatch (other than HEAD)', function(){
         const cache = new CachePolicy(simpleRequest,etaggedResponse);
         const incomingRequest = simpleRequestBut({method:'POST'});
-        // Returns the same object unmodified, which means no custom validation
         const headers = cache.revalidationHeaders(incomingRequest);
         assertHeadersPassed(headers);
         assertNoValidators(headers);

test/updatetest.js

@@ -0,0 +1,87 @@
+'use strict';
+
+const assert = require('assert');
+const CachePolicy = require('..');
+
+const simpleRequest = {
+    method:'GET',
+    headers:{
+      host:'www.w3c.org',
+      connection: 'close',
+    },
+    url:'/Protocols/rfc2616/rfc2616-sec14.html',
+};
+function withHeaders(request, headers) {
+    return Object.assign({}, request, {
+        headers: Object.assign({}, request.headers, headers),
+    });
+}
+
+const cacheableResponse = {headers:{'cache-control':'max-age=111'}};
+const etaggedResponse = {headers:Object.assign({'etag':'"123456789"'},cacheableResponse.headers)};
+const weakTaggedResponse = {headers:Object.assign({'etag':'W/"123456789"'},cacheableResponse.headers)};
+const lastModifiedResponse = {headers:Object.assign({'last-modified':'Tue, 15 Nov 1994 12:45:26 GMT'},cacheableResponse.headers)};
+const multiValidatorResponse = {headers:Object.assign({},etaggedResponse.headers,lastModifiedResponse.headers)};
+
+function notModifiedResponseHeaders(firstRequest, firstResponse, secondRequest, secondResponse) {
+    const cache = new CachePolicy(firstRequest, firstResponse);
+    const headers = cache.revalidationHeaders(secondRequest);
+    const {policy:newCache, modified} = cache.revalidatedPolicy({headers}, secondResponse);
+    if (modified) {
+        return false;
+    }
+    return newCache.responseHeaders();
+}
+
+function assertUpdates(firstRequest, firstResponse, secondRequest, secondResponse) {
+    const headers = notModifiedResponseHeaders(firstRequest, withHeaders(firstResponse, {'foo': 'original', 'x-other':'original'}),
+        secondRequest, withHeaders(secondResponse, {'foo': 'updated', 'x-ignore-new':'ignoreme'}));
+    assert(headers);
+    assert.equal(headers['foo'], 'updated');
+    assert.equal(headers['x-other'], 'original');
+    assert.strictEqual(headers['x-ignore-new'], undefined);
+}
+
+describe('Update revalidated', function() {
+    it('Matching etags are updated', function(){
+        assertUpdates(simpleRequest, etaggedResponse, simpleRequest, etaggedResponse);
+    });
+
+    it('Matching weak etags are updated', function(){
+        assertUpdates(simpleRequest, weakTaggedResponse, simpleRequest, weakTaggedResponse);
+    });
+
+    it('Matching lastmod are updated', function(){
+        assertUpdates(simpleRequest, lastModifiedResponse, simpleRequest, lastModifiedResponse);
+    });
+
+    it('Both matching are updated', function(){
+        assertUpdates(simpleRequest, multiValidatorResponse, simpleRequest, multiValidatorResponse);
+    });
+
+    it('Last-mod can vary if etag matches', function(){
+        assertUpdates(simpleRequest, multiValidatorResponse, simpleRequest, multiValidatorResponse);
+    });
+
+    it('Last-mod ignored if etag is wrong', function(){
+        assert(!notModifiedResponseHeaders(simpleRequest, multiValidatorResponse, simpleRequest, withHeaders(multiValidatorResponse, {'etag':'bad'})));
+        assert(!notModifiedResponseHeaders(simpleRequest, multiValidatorResponse, simpleRequest, withHeaders(multiValidatorResponse, {'etag':'W/bad'})));
+    });
+
+    it('Ignored if validator is missing', function(){
+        assert(!notModifiedResponseHeaders(simpleRequest, etaggedResponse, simpleRequest, cacheableResponse));
+        assert(!notModifiedResponseHeaders(simpleRequest, weakTaggedResponse, simpleRequest, cacheableResponse));
+        assert(!notModifiedResponseHeaders(simpleRequest, lastModifiedResponse, simpleRequest, cacheableResponse));
+    });
+
+    it('Ignored if validator is different', function(){
+        assert(!notModifiedResponseHeaders(simpleRequest, lastModifiedResponse, simpleRequest, etaggedResponse));
+        assert(!notModifiedResponseHeaders(simpleRequest, lastModifiedResponse, simpleRequest, weakTaggedResponse));
+        assert(!notModifiedResponseHeaders(simpleRequest, etaggedResponse, simpleRequest, lastModifiedResponse));
+    });
+
+    it('Ignored if validator doesn\'t match', function(){
+        assert(!notModifiedResponseHeaders(simpleRequest, etaggedResponse, simpleRequest, withHeaders(etaggedResponse, {etag:'"other"'})), "bad etag");
+        assert(!notModifiedResponseHeaders(simpleRequest, lastModifiedResponse, simpleRequest, withHeaders(lastModifiedResponse, {'last-modified':'dunno'})), "bad lastmod");
+    });
+});