Browse Source

[REVIEW] Gaia move to docs.blockstack (#111)

* Adding in new Gaia material
Closes #104, Closes #72, Closes #80, Closes #81,Closes #111, Closes blockstack/gaia#184, Closes blockstack/gaia#191
Entering review comments

Signed-off-by: Mary Anthony <mary@blockstack.com>

* Updating with more comments from this morning

Signed-off-by: Mary Anthony <mary@blockstack.com>

* Updating with last minute comments

Signed-off-by: Mary Anthony <mary@blockstack.com>
feat/clarity-updates
Moxiegirl 6 years ago
committed by GitHub
parent
commit
82b830240e
No known key found for this signature in database GPG Key ID: 4AEE18F83AFDEB23
  1. 2
      THEME_README.md
  2. 12
      _browser/storage-provider.md
  3. 2
      _browser/todo-list.md
  4. 156
      _common/glossary.md
  5. BIN
      _common/images/architecture.png
  6. 1
      _config.yml
  7. 6
      _core/faq_technical.md
  8. 5
      _data/glossary.csv
  9. 18
      _data/navigation_gaia.yml
  10. 2
      _data/navigation_home.yml
  11. 19
      _data/navigation_storage.yml
  12. 2
      _develop/dapp_principles.md
  13. 2
      _develop/zero_to_dapp_1.md
  14. 22
      _includes/architecture.md
  15. 148
      _includes/glossary.md
  16. 0
      _includes/related-storage.html
  17. 16
      _layouts/storage.html
  18. 480
      _storage/amazon-s3-deploy.md
  19. 34
      _storage/authentication.md
  20. 7
      _storage/cliDocs.md
  21. 106
      _storage/config-schema.md
  22. 503
      _storage/digital-ocean-deploy.md
  23. 341
      _storage/gaia-admin.md
  24. 5
      _storage/gaia_redirect.md
  25. 276
      _storage/hello-hub-choice.md
  26. 111
      _storage/hub-choice.md
  27. 102
      _storage/hub-operation.md
  28. BIN
      _storage/images/add-storage.png
  29. BIN
      _storage/images/app-sign-in.png
  30. BIN
      _storage/images/aws-console.png
  31. BIN
      _storage/images/aws-dns-fields.png
  32. BIN
      _storage/images/aws-hub.png
  33. BIN
      _storage/images/aws-launch-status.png
  34. BIN
      _storage/images/cert-check.png
  35. BIN
      _storage/images/change-root.png
  36. BIN
      _storage/images/create-space.png
  37. BIN
      _storage/images/different-provider.png
  38. BIN
      _storage/images/digital-droplet-list.png
  39. BIN
      _storage/images/digital-droplet.png
  40. BIN
      _storage/images/digital-one-click.png
  41. BIN
      _storage/images/digital-ssh-key.png
  42. BIN
      _storage/images/digital-welcome.png
  43. BIN
      _storage/images/dns-fields.png
  44. BIN
      _storage/images/docker-ps.png
  45. BIN
      _storage/images/domain-test.png
  46. BIN
      _storage/images/droplet-control.png
  47. BIN
      _storage/images/droplet-ip.png
  48. BIN
      _storage/images/ec2-dashboard.png
  49. BIN
      _storage/images/ec2-instance-details.png
  50. BIN
      _storage/images/ec2-instance.png
  51. BIN
      _storage/images/gaia-community.png
  52. BIN
      _storage/images/gaia-storage.png
  53. BIN
      _storage/images/gaia-writes.png
  54. BIN
      _storage/images/hub-running.png
  55. BIN
      _storage/images/instance-initialize.png
  56. BIN
      _storage/images/jwt-token.png
  57. BIN
      _storage/images/not-authenticated.png
  58. BIN
      _storage/images/official-provider.jpeg
  59. BIN
      _storage/images/private-connection.png
  60. BIN
      _storage/images/provider-prompt.png
  61. BIN
      _storage/images/recommended-provider.png
  62. BIN
      _storage/images/sign-in.png
  63. BIN
      _storage/images/space-access-key.png
  64. BIN
      _storage/images/space-endpoint.png
  65. BIN
      _storage/images/tag-add.png
  66. BIN
      _storage/images/tier-2-image.png
  67. BIN
      _storage/images/us-west-2.png
  68. 209
      _storage/overview.md
  69. 62
      _storage/write-to-read.md
  70. 2
      assets/css/main.scss

2
THEME_README.md

@ -322,7 +322,7 @@ To modify the primary color, open `/_sass/theme/variables.scss` and replace the
$color-main: #0F1214; $color-main: #0F1214;
``` ```
Further style customisation can be done in the following files: Further style customization can be done in the following files:
``` ```
/_sass/theme/mixins.scss /_sass/theme/mixins.scss
/_sass/theme/variables.scss /_sass/theme/variables.scss

12
_browser/storage-provider.md

@ -11,7 +11,7 @@ data _on their corporate servers_.
Blockstack DApps do not allow applications to store your data on the Blockstack DApps do not allow applications to store your data on the
application's corporate servers. Instead, you store your data with a _storage application's corporate servers. Instead, you store your data with a _storage
provider_ you choose. A storage provider gives you personal storage for your provider_ you choose. A Gaia hub provider gives you personal storage for your
application data. application data.
If you want to walk away from a particular DApp, there’s no need even to export If you want to walk away from a particular DApp, there’s no need even to export
@ -19,21 +19,21 @@ your data. You simply stop using the DApp and it no longer can read or modify
your data. And if you want to start using a different DApp with the same data, your data. And if you want to start using a different DApp with the same data,
you can. you can.
## Understand storage providers ## Understand Gaia hub providers
The Blockstack Ecosystem provides a technology for data storage called, Gaia. The Blockstack Ecosystem provides a technology for data storage called, Gaia.
Gaia is _storage hub_ technology for storing application data. A storage Gaia is _storage hub_ technology for storing application data. A storage
provider is simply an organization that implemented Gaia and is providing provider is simply an organization that implemented Gaia and is providing
storage hubs for users. Any organization or person can create a Gaia storage hub storage hubs for users. Any organization or person can create a Gaia storage
and become a storage provider. system and become a Gaia hub provider.
These providers host data in one or more existing storage systems of their These providers host data in one or more existing storage systems of their
choice. These storage systems are typically cloud storage systems such as choice. These storage systems are typically cloud storage systems such as
Amazon, Dropbox, iCloud or Azure to name a few. Amazon, Dropbox, iCloud or Azure to name a few.
Users that create an identity in the Blockstack Browser are given a choice of Users that create an identity in the Blockstack Browser are given a choice of
which data storage provider to use. Blockstack itself runs a Gaia storage hub and acts which Gaia hub provider to use. Blockstack itself runs a Gaia hub and acts
as a storage provider. The default choice is the Blockstack-run storage hub. as a hub provider. The default choice is the Blockstack-run Gaia hub.
Every provider will store your data at a particular address or URL. These Every provider will store your data at a particular address or URL. These
addresses look similar to any address you put in your web browser. When you log addresses look similar to any address you put in your web browser. When you log

2
_browser/todo-list.md

@ -129,7 +129,7 @@ In request.
A Blockstack Core node also generates a public key token which is sent to the A Blockstack Core node also generates a public key token which is sent to the
browser as an `authRequest` from the browser to your core node. The signed browser as an `authRequest` from the browser to your core node. The signed
authentication request is sent to Blockstack through a JSON Web Token (JWT). authentication request is sent to Blockstack through a JSON Web Token (JWT).
Blocktack passes the token in via a URL query string in the authRequest Blocktack passes the token in via a URL query string in the `authRequest`
parameter: parameter:
`https://browser.blockstack.org/auth?authRequest=j902120cn829n1jnvoa...` `https://browser.blockstack.org/auth?authRequest=j902120cn829n1jnvoa...`

156
_common/glossary.md

@ -1,156 +0,0 @@
# Glossary
Commonly used terms and jargon in Blockstack
## Account
A field in a profile that links the name to an existing service, like Twitter or OpenBazaar. They are listed under the `accounts` listing in a profile.
Some accounts serve as social proofs, but they can contain any data the user wants.
## Atlas
A peer-to-peer network maintained by Blockstack Core nodes that stores each
name's zone files and immutable data. See [this document]({{ site.baseurl }}/atlas/overview.html) for
details.
## Blockstack ID
(Also called a "name").
A human-readable name in Blockstack. It is comprised only of upper and lower-case ASCII characters, numbers, as well as `-`, `_`, and `.`. It must end with a `.`, followed by a namespace ID. It has at least 3 characters, and at most 37 (including the `.` and the namespace ID).
Anyone can register a Blockstack ID, such as through the [Blockstack Browser](https://github.com/blockstack/blockstack-browser)
## Blockstack Core
A server that reads a blockchain with [virtualchain](https://github.com/blockstack/blockstack-virtualchain), filters out transactions that represent name operations, and builds up a database of (name, public key, state value) triples.
## Blockstack Naming Service (BNS)
This is the naming protocol that Blockstack Core implements. See [this
document]({{ site.baseurl }}/core/naming/introduction.html) for details.
## Consensus Hash
A cryptographic hash that represents a proof-of-computation by a Blockstack Core node. Two Blockstack Core nodes have seen and processed the same name operations up to block `n` if and only if they each calculate the same consensus hash at height `n`.
A Blockstack Core node only accepts a name operation if it has a previously-calculated but recent consensus hash. Blockstack clients obtain a consensus hash from a Blockstack Core node in order to construct a name operation.
## Gaia
This is Blockstack's storage system. Gaia hosts all of your app-specific data.
## Gaia Hub
This is a publicly-routable server that serves as an entry point for Gaia data.
Anyone can stand up and run a Gaia hub by following [these
instructions](https://github.com/blockstack/gaia).
Blockstack provides a [default Gaia hub](https://gaia.blockstack.org).
## Immutable Data
This is the general term for chunks of data whose hash is cryptographically
bound to a blockchain transaction. This includes all data stored in the Atlas
network (such as your Blockstack ID's zone file),
as well as any data whose hash is stored in the Atlas network.
## mnemonic
A human-friendly representation of a seed.
## Mutable Data
This is the general term for data that is (1) signed by your Blockstack ID, and
(2) can be looked up using your Blockstack ID. This includes all your Gaia
data, as well as your profile.
## Name
See Blockstack ID.
## Name Database
The set of (name, public key, name state) triples that the Blockstack Core node generates by reading the blockchain. The name state is usually the hash of a DNS zone file stored in Atlas.
## Name Operation
A specially-crafted transaction in the underlying blockchain that, when processed, will change each Blockstack Core's name database. Examples include `NAME_PREORDER` (preorders a name), `NAME_REGISTRATION` (registers a name), `NAME_UPDATE` (changes a name's zonefile hash), `NAME_TRANSFER` (changes a name's public key), and `NAME_REVOKE` (locks everyone out of a name until it expires).
Name operations are encoded on Bitcoin as `OP_RETURN` outputs that start with `id`, followed by a 1-byte character that identifies the particular operation.
See the [wire format](wire-format.md) document for details.
## Namespace
Analogous to a DNS TLD, it represents a grouping of names. All names under the same namespace have the same pricing and lifetime rules.
Anyone can create a namespace, but doing so is expensive by design. See the
[namespace creation]({{ site.baseurl }}/core/naming/tutorial_creation.html) tutorial for details.
## Preorder
The first of two steps to acquire a name. This operation writes the hash of both the name and the address that will own it.
## Profile
A signed JSON web token that describes a [Person](https://schema.org/Person), which describes the name's owner. You can put anything you want into your profile.
Additionally, profiles hold lists of social verifications and pointers to your Gaia data.
## Register
(1) The act of acquiring a name in Blockstack.
(2) The second of two steps to create a new name entry in the name database. Reveals the name as plaintext to the world. Must match a recent preorder to be accepted.
## Registrar
An online service that lets you sign up for and manage the profiles of Blockstack IDs.
## Resolver
An online service that displays zonefile and profile data for a Blockstack ID. [The Blockstack Explorer](https://explorer.blockstack.org) is a resolver.
## Seed
A 128 or 256 bits of random data used to generate a master public and privte key pair.
## Social proof
A post in an account on an existing Web service like Twitter, Facebook, or GitHub that points back to a Blockstack ID. Used to provide some evidence that the person who owns the Blockstack ID is also the person who owns the Web service account.
Social proofs are listed in your profile.
## Storage Provider
This is any service that can serve your zone file, profile, or data. In all cases, the data is signed by one of your wallet's keys, so you can use any provider without having to worry about it trying to change the data.
Storage providers are accessed through a Gaia hub. Gaia hubs ship with drivers
that allow them to treat storage providers as dumb hard drives, which store
signed encrypted data on the hub's behalf.
Not all storage providers support writes--some of them are read-only.
Supported storage providers today include:
* Amazon S3
* Dropbox
* Your harddrive
* Any HTTP/HTTPS/FTP server (read-only)
* Any public-use Gaia hub
* IPFS
Support is being added for:
* Google Drive
* Microsoft OneDrive
* Box.com
* BitTorrent
If you have a preferred storage provider, and you're a developer, please consider sending us a pull request to add support for it!
## Zone file
A specially-formatted file that stores routing information for a Blockstack ID.
Blockstack clients use your zone file to find out where your preferred Gaia
hub(s) are. Ever Blockstack Core node stores a copy of every zone file for
every Blockstack ID by participating in the Atlas network.

BIN
_common/images/architecture.png

Binary file not shown.

After

Width:  |  Height:  |  Size: 89 KiB

1
_config.yml

@ -146,6 +146,7 @@ exclude:
- _data/*.yml - _data/*.yml
- _data/*.csv - _data/*.csv
- _data/cliRef.json - _data/cliRef.json
- _storage/digital-ocean-deploy.md
sass: sass:
style: compressed style: compressed

6
_core/faq_technical.md

@ -97,7 +97,7 @@ Blockstack, like Linux, is an open-source project with a GPLv3 license for the c
## Where is my data stored and how do I control who access it? ## Where is my data stored and how do I control who access it?
You control where your data is stored (you could run your own server, or use your own cloud storage - Dropbox, Amazon S3, and keep backups across all). You then use those places as locations pointed to by the URLs in your Blockstack ID's zone file. You control where your data is stored (you could run your own server, or use your own cloud storage - Dropbox, Amazon S3, and keep backups across all). You then use those places as locations pointed to by the URLs in your Blockstack ID's zone file.
## Why should I trust the information, like name ownership or public key mappings, read from Blockstack? ## Why should I trust the information, like name ownership or public key mappings, read from Blockstack?
@ -145,8 +145,8 @@ Yes, the Blockstack network will keep running. All of Blockstack's code is open-
As a Blockstack user, you can choose exactly where your data gets stored. As a Blockstack user, you can choose exactly where your data gets stored.
Blockstack uses a decentralized storage system called Blockstack uses a decentralized storage system called
[Gaia](https://github.com/blockstack/gaia) to host your data. Gaia is different [Gaia](https://github.com/blockstack/gaia) to host your data. Gaia is different
from other storage systems because it lets you securely host your data wherever you want---in cloud from other storage systems because it lets you securely host your data wherever you want &mdash; in cloud
storage providers, on your personal server, or in another decentralized storage software providers, on your personal server, or in another decentralized storage
system like BitTorrent or IPFS. system like BitTorrent or IPFS.
When you register, you are given a default Gaia hub that replicates your When you register, you are given a default Gaia hub that replicates your

5
_data/glossary.csv

@ -1,6 +1,7 @@
Term,Definition,Areas Term,Definition,Areas
App Review,An incentive mechanism for application developers in the early stage (first four years) of the ecosystem and helps with bootstrapping the two-sided market., App Review,An incentive mechanism for application developers in the early stage (first four years) of the ecosystem and helps with bootstrapping the two-sided market.,
App Mining,"​​A system where the Blockstack community funds decentralized app teams.", App Mining,"​​A system where the Blockstack community funds decentralized app teams.",
association token,"A whitelisted key used in a Gaia Storage system.",Blockstack Tech
Atlas,A peer network provide a global index for discovery,Blockstack Tech Atlas,A peer network provide a global index for discovery,Blockstack Tech
block,"A discrete group of records written to a blockchain that can effectively be identified and referenced by the block,"A discrete group of records written to a blockchain that can effectively be identified and referenced by the
use of headers that contain a digital fingerprint of the records each block contains.",Filing use of headers that contain a digital fingerprint of the records each block contains.",Filing
@ -46,8 +47,8 @@ replicated state machines (RSMs),"This is a model for reasoning about distribute
smart contract,"A smart contract is a computer program written to a blockchain such as the Blockstack blockchain by developers. This computer program defines the various network operations that can be performed on the digital assets on the blockchain, the computations that can be performed using the smart contract, along with defining various important properties of the respective digital assets like ownership rights and the cost in fuel required to register the digital assets.",Filing smart contract,"A smart contract is a computer program written to a blockchain such as the Blockstack blockchain by developers. This computer program defines the various network operations that can be performed on the digital assets on the blockchain, the computations that can be performed using the smart contract, along with defining various important properties of the respective digital assets like ownership rights and the cost in fuel required to register the digital assets.",Filing
soft fork,"A soft fork is a proposed update to the software governing the network that results in a post-update network that is compatible with the network as it existed prior to the update, because it restricts the network operations that can be performed after the update.",Filing soft fork,"A soft fork is a proposed update to the software governing the network that results in a post-update network that is compatible with the network as it existed prior to the update, because it restricts the network operations that can be performed after the update.",Filing
Stacks blockchain,The Stacks blockchain is the custom blockchain being developed for use by the Blockstack network that will use the Stacks token as its native currency and which will include the genesis block and subsequent blocks created by the mining activities described in The Blockstack Network—Development of the Blockstack network.,Filing Stacks blockchain,The Stacks blockchain is the custom blockchain being developed for use by the Blockstack network that will use the Stacks token as its native currency and which will include the genesis block and subsequent blocks created by the mining activities described in The Blockstack Network—Development of the Blockstack network.,Filing
storage hub,"An Gaia instance run by a storage provider or software. A storage hub stores data in separate, addressed locations within the hub.",Blockstack Tech storage hub,"An Gaia instance run by a hub provider or software. A Gaia hub stores data in separate, addressed location.",Blockstack Tech
storage provider,"A third-party company or software provider that hosts one or more Gaia storage hubs.",Blockstack Tech storage provider,"A third-party company cloud or software provider that hosts one or more Gaia storage hubs.",Blockstack Tech
transaction,"A transaction is a unit of work within a block.",General Tech transaction,"A transaction is a unit of work within a block.",General Tech
transaction fee,"Fees paid by participants in a transaction to miners.",Blockstack Tech transaction fee,"Fees paid by participants in a transaction to miners.",Blockstack Tech
two-sided market problem,Economic platforms having two distinct user groups that provide each other with network benefits. The Blockstack platform is a classic two-sided market., two-sided market problem,Economic platforms having two distinct user groups that provide each other with network benefits. The Blockstack platform is a classic two-sided market.,

Can't render this file because it contains an unexpected character in line 14 and column 0.

18
_data/navigation_gaia.yml

@ -1,18 +0,0 @@
- title: About
docs:
- gaia/overview
- gaia/control
- gaia/writeread
- gaia/access
- title: Tutorials
docs:
- gaia/hello-gaia
- gaia/media
- gaia/sharing
- title: How to
docs:
- gaia/run-hub
- gaia/configure
- gaia/test

2
_data/navigation_home.yml

@ -23,7 +23,7 @@
- title: Implement Storage with Gaia - title: Implement Storage with Gaia
desc: Backend storage drivers and interactions between developer APIs and the Gaia service. desc: Backend storage drivers and interactions between developer APIs and the Gaia service.
icon: database icon: database
doc: storage/gaia_redirect doc: storage/overview
- title: Blockstack Community - title: Blockstack Community
desc: view upcoming events, become a contributor or evangelist. desc: view upcoming events, become a contributor or evangelist.

19
_data/navigation_storage.yml

@ -0,0 +1,19 @@
- title: Introduction to Gaia storage
docs:
- storage/overview
- storage/authentication
- storage/write-to-read
- storage/hub-choice
- title: How to hub
docs:
- storage/hub-operation
- storage/amazon-s3-deploy
- storage/hello-hub-choice
- storage/gaia-admin
- title: Reference
docs:
- storage/config-schema
- storage/cliDocs
- common/javascript_ref

2
_develop/dapp_principles.md

@ -69,7 +69,7 @@ because they don't have any irreplaceable server-side logic.
In the future, Blockstack applications must continue to meet the first two In the future, Blockstack applications must continue to meet the first two
principles but need not meet this one. For example, an application could principles but need not meet this one. For example, an application could
encrypt data in-transit between the application's client and the user's chosen encrypt data in-transit between the application's client and the user's chosen
storage provider. Unless the app divulges the encryption key to the user, then Gaia hub provider. Unless the app divulges the encryption key to the user, then
the user does not have free choice of clients; they can only use clients that the user does not have free choice of clients; they can only use clients that
the app's servers choose to interact with. the app's servers choose to interact with.

2
_develop/zero_to_dapp_1.md

@ -207,7 +207,7 @@ The following table describes the key interactions and screens in the DApp.
</a> </a>
</div></td> </div></td>
<td> <td>
<p>Users can add subjects from territories in their own Animal Kingdom. The DApp updates the user's GAI hub each time the user adds a subject. Users can also visit other Animal Kingdom installations and add subjects from these as well. You'll learn how to modify the <b>Other Kingdoms</b> available in your installation. <p>Users can add subjects from territories in their own Animal Kingdom. The DApp updates the user's Gaia hub each time the user adds a subject. Users can also visit other Animal Kingdom installations and add subjects from these as well. You'll learn how to modify the <b>Other Kingdoms</b> available in your installation.
</p> </p>
</td> </td>
</tr> </tr>

22
_includes/architecture.md

@ -0,0 +1,22 @@
![Blockstack Architecture](/common/images/architecture.png)
Blockchains require consensus among large numbers of people, so they can be slow. Additionally, a blockchain is not designed to hold a lot of data. This means using a blockchain for every bit of data a user might write and store is expensive. For example, imagine if an application were storing every tweet in the chain.
Blockstack addresses blockchain performance problems using a layered approach. At the base of the system is a blockchain and the Blockstack Naming System (BNS). The blockchain governs ownership of names (identities) in the system, names such as domain names, usernames, and application names.
Names in Blockstack correspond to routing data in the OSI stack. The routing data is stored in the Atlas Peer Network, the second layer. Every core node that joins the Blockstack Network is able to obtain an entire copy of this routing data. Blockstack uses the routing data to associate names (usernames, domains, and application names) with a particular storage location.
The final layer is the Gaia Storage System. A Gaia system consists of a _hub
service_ and storage resource on a cloud software provider such as Azure,
DigitalOcean, Amazon EC2, and so forth. Typically the compute resource and the
storage resource belong to same cloud vendor. Gaia currently has
driver support for S3 and Azure Blob Storage, but the driver model allows for
other backend support as well.
Because Gaia stores application and user data off the blockchain, a Blockstack
DApp is typically more performant than DApps created on other blockchains.
Moreover, users choose where their data lives, and Gaia enables applications
to access that user data via a uniform API. When the user logs in,
the authentication process gives the application the URL of a Gaia hub, which
then writes to storage on behalf of that user.

148
_includes/glossary.md

@ -1,148 +0,0 @@
Commonly used terms and jargon in Blockstack
## Account
A field in a profile that links the name to an existing service, like Twitter or OpenBazaar. They are listed under the `accounts` listing in a profile.
Some accounts serve as social proofs, but they can contain any data the user wants.
## Atlas
A peer-to-peer network maintained by Blockstack Core nodes that stores each
name's zone files and immutable data. See [this document](atlas_network.md) for
details.
## Blockstack ID
(Also called a "name").
A human-readable name in Blockstack. It is comprised only of upper and lower-case ASCII characters, numbers, as well as `-`, `_`, and `.`. It must end with a `.`, followed by a namespace ID. It has at least 3 characters, and at most 37 (including the `.` and the namespace ID).
Anyone can register a Blockstack ID, such as through the [Blockstack Browser](https://github.com/blockstack/blockstack-browser)
## Blockstack Core
A server that reads a blockchain with [virtualchain](https://github.com/blockstack/blockstack-virtualchain), filters out transactions that represent name operations, and builds up a database of (name, public key, state value) triples.
## Blockstack Naming Service (BNS)
This is the naming protocol that Blockstack Core implements. See [this
document](blockstack_naming_service.md) for details.
## Consensus Hash
A cryptographic hash that represents a proof-of-computation by a Blockstack Core node. Two Blockstack Core nodes have seen and processed the same name operations up to block `n` if and only if they each calculate the same consensus hash at height `n`.
A Blockstack Core node only accepts a name operation if it has a previously-calculated but recent consensus hash. Blockstack clients obtain a consensus hash from a Blockstack Core node in order to construct a name operation.
## Gaia
This is Blockstack's storage system. Gaia hosts all of your app-specific data.
## Gaia Hub
This is a publicly-routable server that serves as an entry point for Gaia data.
Anyone can stand up and run a Gaia hub by following [these
instructions](https://github.com/blockstack/gaia).
Blockstack provides a [default Gaia hub](https://gaia.blockstack.org).
## Immutable Data
This is the general term for chunks of data whose hash is cryptographically
bound to a blockchain transaction. This includes all data stored in the Atlas
network (such as your Blockstack ID's zone file),
as well as any data whose hash is stored in the Atlas network.
## Mutable Data
This is the general term for data that is (1) signed by your Blockstack ID, and
(2) can be looked up using your Blockstack ID. This includes all your Gaia
data, as well as your profile.
## Name
See Blockstack ID.
## Name Database
The set of (name, public key, name state) triples that the Blockstack Core node generates by reading the blockchain. The name state is usually the hash of a DNS zone file stored in Atlas.
## Name Operation
A specially-crafted transaction in the underlying blockchain that, when processed, will change each Blockstack Core's name database. Examples include `NAME_PREORDER` (preorders a name), `NAME_REGISTRATION` (registers a name), `NAME_UPDATE` (changes a name's zonefile hash), `NAME_TRANSFER` (changes a name's public key), and `NAME_REVOKE` (locks everyone out of a name until it expires).
Name operations are encoded on Bitcoin as `OP_RETURN` outputs that start with `id`, followed by a 1-byte character that identifies the particular operation.
See the [wire format](wire-format.md) document for details.
## Namespace
Analogous to a DNS TLD, it represents a grouping of names. All names under the same namespace have the same pricing and lifetime rules.
Anyone can create a namespace, but doing so is expensive by design. See the
[namespace creation](namespace_creation.md) tutorial for details.
## Preorder
The first of two steps to acquire a name. This operation writes the hash of both the name and the address that will own it.
## Profile
A signed JSON web token that describes a [Person](https://schema.org/Person), which describes the name's owner. You can put anything you want into your profile.
Additionally, profiles hold lists of social verifications and pointers to your Gaia data.
## Register
(1) The act of acquiring a name in Blockstack.
(2) The second of two steps to create a new name entry in the name database. Reveals the name as plaintext to the world. Must match a recent preorder to be accepted.
## Registrar
An online service that lets you sign up for and manage the profiles of Blockstack IDs.
## Resolver
An online service that displays zonefile and profile data for a Blockstack ID. [The Blockstack Explorer](https://explorer.blockstack.org) is a resolver.
## Social proof
A post in an account on an existing Web service like Twitter, Facebook, or GitHub that points back to a Blockstack ID. Used to provide some evidence that the person who owns the Blockstack ID is also the person who owns the Web service account.
Social proofs are listed in your profile.
## Storage Provider
This is any service that can serve your zone file, profile, or data. In all cases, the data is signed by one of your wallet's keys, so you can use any provider without having to worry about it trying to change the data.
Storage providers are accessed through a Gaia hub. Gaia hubs ship with drivers
that allow them to treat storage providers as dumb hard drives, which store
signed encrypted data on the hub's behalf.
Not all storage providers support writes--some of them are read-only.
Supported storage providers today include:
* Amazon S3
* Dropbox
* Your harddrive
* Any HTTP/HTTPS/FTP server (read-only)
* Any public-use Gaia hub
* IPFS
Support is being added for:
* Google Drive
* Microsoft OneDrive
* Box.com
* BitTorrent
If you have a preferred storage provider, and you're a developer, please consider sending us a pull request to add support for it!
## Zone file
A specially-formatted file that stores routing information for a Blockstack ID.
Blockstack clients use your zone file to find out where your preferred Gaia
hub(s) are. Ever Blockstack Core node stores a copy of every zone file for
every Blockstack ID by participating in the Atlas network.

0
_includes/related-gaia.html → _includes/related-storage.html

16
_layouts/gaia.html → _layouts/storage.html

@ -8,11 +8,14 @@ layout: default
<div class="sidebar-fixed-width uk-visible@m"> <div class="sidebar-fixed-width uk-visible@m">
<div class="sidebar-docs uk-position-fixed"> <div class="sidebar-docs uk-position-fixed">
{% for section in site.data.navigation_gaia %} <!-- -->
<h5>{{ section.title }}</h5> {% for section in site.data.navigation_storage %}
{% if section.title %}
<h5>{{ section.title }}</h5>
{% endif %}
<ul class="uk-nav uk-nav-default doc-nav"> <ul class="uk-nav uk-nav-default doc-nav">
{% for doc in section.docs %} {% for doc in section.docs %}
{% assign doc_url = doc | prepend:"/" | append:"/" %} {% assign doc_url = doc | prepend:"/" | append:".html" %}
<!-- {% assign p = site.docs | where:"url", doc_url | first %} --> <!-- {% assign p = site.docs | where:"url", doc_url | first %} -->
{% for col in site.collections %} {% for col in site.collections %}
{% for item in col.docs %} {% for item in col.docs %}
@ -23,10 +26,11 @@ layout: default
{% endif %} {% endif %}
{% endfor %} {% endfor %}
{% endfor %} {% endfor %}
<li class="{% if doc_url == page.url %}uk-active{% endif %}"><a href="{{ doc_url }}">{{ doc_title }}</a></li> <li class="{% if doc_url == page.url %}uk-active{% endif %}"><a href="{{ doc_url }}">{{ doc_title }}</a></li>
{% endfor %} {% endfor %}
</ul> </ul>
{% endfor %} {% endfor %}
<!-- -->
</div> </div>
</div> </div>
@ -38,6 +42,7 @@ layout: default
{% if page.subtitle %}<p class="subtitle uk-text-lead uk-text-muted">{{ page.subtitle }}</p>{% endif %} {% if page.subtitle %}<p class="subtitle uk-text-lead uk-text-muted">{{ page.subtitle }}</p>{% endif %}
<div class="uk-article-meta uk-margin-top uk-margin-medium-bottom"> <div class="uk-article-meta uk-margin-top uk-margin-medium-bottom">
{% include post-meta.html %} {% include post-meta.html %}
</div> </div>
@ -52,9 +57,8 @@ layout: default
{% include paginate-doc.html %} {% include paginate-doc.html %}
{% include related-gaia.html %} <!-- {% include related-storage.html %} -->
{% if site.disqus.shortname %} {% include disqus_comments.html %} {% endif %}
</article> </article>
<script> <script>

480
_storage/amazon-s3-deploy.md

@ -0,0 +1,480 @@
---
layout: storage
permalink: /:collection/:path.html
---
# Configure a hub on Amazon EC2
{:.no_toc}
This teaches you how to run a Gaia hub on Amazon EC2. Amazon EC2 is an affordable and convenient cloud computing provider. This example uses Amazon EC2 together with an EB3 instance for file storage.
* TOC
{:toc}
<div class="uk-card uk-card-default uk-card-body">
<h5>Is this tutorial for you?</h5>
<p>This documentation is appropriate for advanced power users who are familiar with command line tools, <code>ssh</code>, and basic editing configuration files.</p>
<p>If you are planning on running an <emphasis>open-membership hub</emphasis> or an <emphasis>application-specific hub</emphasis>, see <a href="hub-operation.html">the section on Hub Operation</a>.</p>
</div>
## Prerequisites you need
This procedure uses Amazon AWS to choose and configure an Amazon Machine Image
(AMI) running a Gaia service. For this reason, you should have an AWS account
on the <a href="https://aws.amazon.com/free/" target="\_blank">Amazon AWS free
tier</a>, personal account, or corporate account. These instructions assume you
are using a free tier account.
These instructions assume you have already created a free <a
href="https://www.freenom.com" target="\_blank">domain through the freenom
service</a>. If you have another domain, you can use that instead.
Finally, setting up the SSL certificates on your EC2 instance requires you to use the terminal command line on your workstation. Make sure you have the `watch` command installed using the `which` command.
```
$ which watch
/usr/local/bin/watch
```
If `watch` is not located, install it on your workstation.
## Task 1: Launch an EC2 instance
1. Visit the <a href="https://aws.amazon.com/free/" target="\_blank">AWS Free Tier page</a> and choose **Sign in to the Console**.
![](/storage/images/aws-console.png)
2. Make sure your region is set to one close to you.
![](/storage/images/us-west-2.png)
3. Under **Build a solution** choose **Launch a virtual machine**.
The system opens the EC2 dashboard.
4. Enter `Blockstack Gaia` in the search bar.
The system finds AMIs in the Marketplace and the Community.
5. Choose **Community AMIs**.
The system displays the available Gaia hub images.
![](/storage/images/gaia-community.png)
6. Select the most recent version of the image.
Each image name has this format:
`blockstack-gaia_hub-STORAGETYPE-VERSION-hvm - ami-BUILDTAG`
So, the `blockstack-gaia_hub-ephemeral-0001.0.1-hvm - ami-0425cf8c91bb2d331` image uses ephemeral storage, is at version `0001.0.1` and has the `0425cf8c91bb2d331` tag.
You can choose an image that uses ephemeral or EBS storage. The ephemeral
storage is very small but free. Only choose this if you plan to test or use
a personal hub. Otherwise, choose the AMI for elastic block storage (EBS).
After you select an image, the system displays **Step 2: Choose an Instance Type** page.
![](/storage/images/tier-2-image.png)
7. Select **t2.micro** and choose **Next: Configure Instance Details**.
<hr>
To configure instance details, do the following:
1. Select a VPC.
A default VPC is created with a free tier account. You can use this
default VPC. Or you can choose another VPC. If you choose another VPC,
ensure the `Subnet` value is set to a subnet reachable by a public IP.
{% include important.html content="If you're using a private subnet, you
should attach an elastic IP (EIP) to the VM. This EIP allows you to
reboot the instance without worrying whether the address will reset. To
attach an IP, <strong>press allocate new address</strong> and follow the
instructions to attach the EIP to your new EC2 instance." %}
2. Set **Protect against accidental termination**.
If you terminate a Gaia instance, you lose all the data associated with it. Protection adds an extra step to terminating your Gaia instance.
3. Open the **Advanced Details**.
At this point, you are going to configure environment variables for your instance.
4. Paste the following into the **Advanced Details**.
```json
{
"ignition": { "version": "2.2.0" },
"storage": {
"files": [{
"filesystem": "root",
"path": "/etc/environment",
"mode": 420,
"contents": {
"source": "data:application/octet-stream,API_KEY%3D<KEYPHRASE>%0ADOMAIN%3D<DOMAIN_NAME_VALUE>%0ASTAGING%3D<STAGING_VALUE>"
}
}]
}
}
```
5. Replace the following values in the JSON.
<table class="uk-table uk-table-small uk-table-divider">
<tr>
<th>Value</th>
<th>Description</th>
</tr>
<tr>
<td><code>&lt;KEYPHRASE&gt;</code></td>
<td>A phrase to pass when using the hub admin. For example, <code>hubba</code> is a fun key phrase.</td>
</tr>
<tr>
<td><code>&lt;DOMAIN_NAME_VALUE&gt;</code></td>
<td>Your hub's domain name. For example, <code>maryhub.ml</code> is the domain name in this example.</td>
</tr>
<tr>
<td><code>&lt;STAGING_VALUE&gt;</code></td>
<td><p>Indicates what type of SSL to create, testing (`1`) or production (`0`). Set testing if you want to test without worrying about rate limiting. A testing cerificate is not secure.</p>
<p>For this tutorial, use production (`0`).</p>
</td>
</tr>
</table>
6. Check your **Advanced Details** they should look similar to the following:
```
{
"ignition": { "version": "2.2.0" },
"storage": {
"files": [{
"filesystem": "root",
"path": "/etc/environment",
"mode": 420,
"contents": {
"source": "data:application/octet-stream,API_KEY%3Dhubba%0ADOMAIN%3Dmaryhub.ml%0ASTAGING%3D0"
}
}]
}
}
```
At this point, you have configured your instance details.
<hr>
8. Choose **Next: Add Storage**.
![](/storage/images/add-storage.png)
The storage is set according to the AMI you selected.
9. Choose **Next: Add tags**.
10. Add a **Key** of `purpose` with the **Value** `gaia`.
![](/storage/images/tag-add.png)
11. Choose **Next: Configure Security Group**.
12. Create a security group with the following three types:
<table class="uk-table uk-table-small uk-table-divider">
<tr>
<th>Type</th>
<th>Protocol</th>
<th>Port Range</th>
<th>Source</th>
<th>Description</th>
</tr>
<tr>
<td>SSH</td>
<td>TCP</td>
<td>22</td>
<td>My IP</td>
<td>optional</td>
</tr>
<tr>
<td>HTTP</td>
<td>TCP</td>
<td>80</td>
<td>Anywhere</td>
<td>optional</td>
</tr>
<tr>
<td>HTTPS</td>
<td>TCP</td>
<td>443</td>
<td>Anywhere</td>
<td>optional</td>
</tr>
</table>
13. Choose **Review and Launch**.
The system may warn you that the selection is not free tier eligible. You can ignore this for now.
14. Press **Launch**.
The system prompts you for a key pair.
15. Select **Create a new keypair** or **Choose an existing key pair**.
16. Select **Launch Instances**.
The system launches your instance.
![](/storage/images/aws-launch-status.png)
During the launch process the machine starts and runs some initial setup processes. These processes take a few minutes depending on the network, typically launching does not take more than 10 minutes. While this is happening the instance **Status Checks** reflect the **Initializing** status.
![](/storage/images/instance-initialize.png)
## Task 2: Test your Gaia server
Now, you are ready to test your Gaia server and make sure it is up and running.
1. Visit the <a href="https://aws.amazon.com/free/" target="\_blank">AWS Free Tier page</a> and choose **Sign in to the Console**.
![](/storage/images/aws-console.png)
2. Choose **All services > EC2**.
The system displays the **EC2 Dashboard**.
![](/storage/images/ec2-dashboard.png)
3. Select **Running Instances**.
The system displays your running instances.
4. Locate your recently launched Gaia server.
Make sure the instance shows as running and **Status Checks** are complete.
Completed status checks ensures the Gaia processes and service were started.
5. Select the **Description** tab.
![](/storage/images/ec2-instance.png)
6. Locate the **IPv4 Public IP** value.
7. Copy the IP and paste it in your browser.
You should see a message that your connection is not private.
8. Press **Advanced**.
![Hub test](/storage/images/private-connection.png)
9. Choose to proceed.
10. Extend the IP with the `PUBLIC_IP/hub_info` tag like so.
You should see a response from your Gaia hub!
![Hub test](/storage/images/aws-hub.png)
At this point, you should see a **Not secure** message in the browser.
That's because you haven't yet enabled SSL certification. While `HTTPS` is
not required simple to run the hub services, Blockstack will only connect to
a hub and write to its storage over a valid `HTTPS` connection.
## Task 3: Configure a domain name
At this point, you can point a domain to your Gaia hub. Although it's not required, it is highly recommended. If you use a domain, you can migrate your instance to a different server (or even provider such as Azure or Dropbox) at any time, and still access it through the domain URL. Just point your domain at the IP address for your EC2 instance. Use an `A Record` DNS type.
These instructions assume you have already created a free <a href="https://www.freenom.com" target="\_blank">domain through the freenom service</a>. To point this freenom domain to your Gaia hub, do the following:
1. Log into your freenom account.
2. Choose the **Manage Freenom Domain** tab.
3. Add an **A** record leave the **Name** field blank.
This points your entire domain to the hub IP.
4. Save your changes.
5. Create a CNAME record.
For example, you can use the prefix `www` with your domain name. When you are done, your
6. Save your changes.
At this point, your DNS management should look similar to the following except that with your domain rather than the `maryhub.ga` domain.
![DNS fields](/storage/images/aws-dns-fields.png)
7. After your changes propagate, visit your new domain at the `hub_info` page.
![Domain test](/storage/images/domain-test.png)
If you receive another **Your connection is not private** dialogs, take the option to proceed to your domain. The *Not secure* message should no longer appear in the browser bar. If the message does appear, try waiting a few minutes for your recent changes to propagate across the net domain servers. Then, refresh the page.
9. Check the SSL certificate for your hub.
Each browser has its own check procedure, for example, Chrome:
![](/storage/images/cert-check.png)
At this point, you have the following. An EC2 instance running Gaia and a DNS
record pointing your domain to this instance.
## AWS hub tips and tricks
Once your Gaia storage hub is up and running on AWS, you may occassionally need to troubleshoot. This section contains some useful information for interacting with your EC2 instance.
### SSH into the Host
To SSH to the EC2 host directly:
```bash
ssh -t -i <your keyfile.pem> core@<public ip address>
```
### Displaying the docker services
Your EC2 instance is running several `docker` services that support the Gaia hub. You can list these services using the `docker ps` command.
{% raw %}
```bash
$ docker ps --format "table {{.ID}}\t{{.Command}}\t{{.Names}}"
CONTAINER ID COMMAND NAMES
b371234dc741 "/bin/sh -c 'trap ex…" docker_certbot_1
597866815f42 "/bin/sh -c 'envsubs…" docker_nginx_1
1d559bc51699 "npm run start" docker_admin_1
46d410a1dce5 "npm run start" docker_reader_1
f83fb8d044f5 "npm run start" docker_hub_1
```
{% endraw %}
Each service plays a particular role in running your Gaia hub.
<table class="uk-table uk-table-small uk-table-divider">
<thead>
<tr>
<th>Service</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>certbot</code></td>
<td>Service running Let's Encrypt `certbot` client to support SSL. Certbot renews your certificates and reloads Nginx to pick up the changes.</td>
</tr>
<tr>
<td><code>nginx</code></td>
<td>Runs an Nginx proxy in front of the <code>reader</code> side-car. This service does things like rate-limiting and SSL termination. Your
that nginx service relies on your hub's <code>readURL</code> to make requests. Changes to a hub's <code>readURL</code> must be reflected in the <code>nginx</code> service configuration.</td>
</tr>
<tr>
<td><code>admin</code></td>
<td>A simple administrative service that allows you to administer the Gaia hub. Use REST calls with this service to get and set hub configuration values.</td>
</tr>
<tr>
<td><code>reader</code></td>
<td>The Gaia read side-car services get file requests on URLs that start with
your Gaia hub's <code>readURL</code>. You can determine your Gaia hub's read URL by either
looking for the <code>readURL</code> key in your Gaia hub's config file. This value is or by looking for
the <code>read_url_prefix</code> field in the data returned by a <code>HUB_URL/hub_info</code> page on your
Gaia hub.</td>
</tr>
<tr>
<td><code>hub</code></td>
<td>The Gaia hub service.</td>
</tr>
</tbody>
</table>
### Locations of key files
<table>
<thead>
<tr>
<th>File or Directory</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>/etc/systemd/system</code></td>
<td>Contains services for managing your Gaia hub.</td>
</tr>
<tr>
<td><code>/etc/environment</code></td>
<td>Contains the <code>DOMAIN</code> and <code>STAGING</code> variables you entered when creating your EC2 instance.
</td>
</tr>
<tr>
<td><code>reset-ssl-certs.service</code></td>
<td>Restarts all the Gaia hub services.</td>
</tr>
<tr>
<td><code>/gaia/docker/admin-config</code></td>
<td>Configuration for the hub admin service.</td>
</tr>
</tbody>
</table>
You can `cat` the various services to see what settings they are using.
```
$ cat /etc/systemd/system/reset-ssl-certs.service
[Unit]
Description=Reset Gaia to first boot
[Service]
Type=oneshot
RemainAfterExit=no
EnvironmentFile=/gaia/docker/.env
EnvironmentFile=/etc/environment
ExecStart=/bin/bash -x /gaia/docker/nginx/certbot/reset-certs.sh
[Install]
```
### Restart services and reload certificates
This procedures requires you to interact from a workstation command line with your running EC2 instance.
1. Open a terminal on your local workstation.
2. Confirm the hub DNS is set correctly with the following command:
```bash
watch -n 2 -t -g -x host <domain>
```
Substitute your domain name for the `<domain>` variable. For example:
```bash
watch -n 2 -t -g -x host maryhub.ga
maryhub.ml has address 34.219.71.143
```
If the command returns the correct IP, the same as appears on your EC2 dashboard, stop the process with a` CTRL-C` on your keyboard.
3. Change the permissions on your downloaded `.pem` file.
For example, this
```
chmod 400 <location-of-pem>
```
4. SSH from your workstation and restart it.
This process requires that you know the location of the `.pem` file you downloaded when you created the keypair.
```
ssh -t -i <your keyfile.pem> -A core@<public ip address> "sudo systemctl restart reset-ssl-certs.service"
```
For example:
```
$ ssh -t -i /Users/manthony/gaia.pem -A core@34.219.71.143 "sudo systemctl restart reset-ssl-certs.service"
Connection to 34.219.71.143 closed.
```

34
_storage/authentication.md

@ -0,0 +1,34 @@
---
layout: storage
permalink: /:collection/:path.html
---
# Authentication and Gaia
Blockstack authentication is a bearer token-based authentication system. From an app user's perspective, login similar to third-party authentication techniques that they're familiar with. For an app developer, the flow is unlike the typical client-server flow of centralized sign-in services such as OAuth. With Blockstack the authentication flow happens entirely client-side.
In this section, you get an overview of the authentication system and learn how Gaia fits into it.
## Authentication and Gaia
A decentralized application (DApp) and the Blockstack Browser communicate during
the authentication flow by passing back and forth two tokens. The requesting
application sends the Blockstack Browser an `authRequest` token. Once a user
approves a sign-in, the Blockstack Browser responds to the application with an
`authResponse` token. These tokens are <a href="https://jwt.io/" target="\_blank">JSON Web Tokens</a>, and they are passed via
URL query strings.
When a user chooses to "Sign in with Blockstack" on your DApp, the `redirectToSignIn()` method sends the user to the Blockstack Browser. The browser responds with an authentication token and an _app private key_.
![](/storage/images/app-sign-in.png)
The app private key is application-specific. It is generated from the user's identity address private key using the `appDomain` as input. This key is deterministic, meaning that for a given Blockstack ID and domain name, the same private key is generated each time. The app private key is securely shared with the app on each authentication and encrypted by the Blockstack Browser. The key serves three functions, it:
* is used to create the credentials that give an app access to the Gaia hub storage bucket for that specific app
* is used in the end-to-end encryption of files stored for the app on the user's Gaia hub
* serves as a cryptographic secret that apps can use to perform other cryptographic functions
When an application writes to a Gaia hub, the authentication token, key, and the data are passed to the Gaia hub.
![Gaia writes](/storage/images/gaia-writes.png)
The token ensures the DApp has the authorization to write to the hub on the user's behalf.

7
_storage/cliDocs.md

@ -0,0 +1,7 @@
---
layout: storage
permalink: /:collection/:path.html
---
# blockstack_cli reference
{% include commandline.md %}

106
_storage/config-schema.md

@ -0,0 +1,106 @@
---
layout: storage
permalink: /:collection/:path.html
---
# Hub configuration parameters
```json
{
"$schema": "http://json-schema.org/draft-04/schema#",
"type": "object",
"properties": {
"servername": { "type": "string" },
"port": { "type": "integer" },
"driver": { "type": "string" },
"bucket": { "type": "string" },
"readURL": { "type": "string" },
"requireCorrectHubUrl": { "type": "string" },
"awsCredentials": {
"type": "object",
"properties": {
"endpoint": { "type": "string"},
"accessKeyId": { "type": "string"},
"secretAccessKey": { "type": "string"}
},
"required": [
"endpoint",
"accessKeyId",
"secretAccessKey"
]
},
"proofsConfig": {
"type": "object",
"properties": {
"proofsRequired": { "type": "string"}
},
"required": [
"proofsRequired"
]
},
"azCredentials": {
"type": "object",
"properties": {
"accountName": { "type": "string"},
"accountKey": { "type": "string"}
},
"required": [
"accountName",
"accountKey"
]
},
"gcCredentials": {
"type": "object",
"properties": {
"keyFilename": { "type": "string"}
},
"required": [
"keyFilename"
]
},
"diskSettings": {
"type": "object",
"properties": {
"storageRootDirectory": { "type": "string"}
},
"required": [
"storageRootDirectory"
]
},
"pageSize": { "type": "integer"},
"argsTransport": {
"type": "object",
"properties": {
"level": { "type": "string"},
"handleExceptions": { "type": "boolean" },
"stringify": { "type": "boolean" },
"timestamp": { "type": "boolean" },
"colorize": { "type": "boolean" },
"json":{ "type": "boolean" }
},
"required": [
"level",
"handleExceptions",
"stringify",
"timestamp",
"colorize",
"json"
]
}
},
"required": [
"servername",
"port",
"driver",
"bucket",
"readURL",
"requireCorrectHubUrl",
"awsCredentials",
"proofsConfig",
"azCredentials",
"gcCredentials",
"diskSettings",
"pageSize",
"argsTransport"
]
}
```

503
_storage/digital-ocean-deploy.md

@ -0,0 +1,503 @@
---
layout: storage
permalink: /:collection/:path.html
---
# Configure a hub on DigitalOcean
{:.no_toc}
This teaches you how to run a Gaia storage hub on DigitalOcean (DO). DigitalOcean is an affordable and convenient cloud computing provider. This example uses DigitalOcean Spaces for file storage. DigitalOcean Spaces is equivalent to AWS's S3 file storage solution.
DigitalOcean provides you with a compute machines known as a **Droplets** and storage called a **Spaces**. You need both to run a Gaia hub. The Gaia hub setup you create here, requires get a Digital Droplet with Docker pre-installed and a 250 GB Space. Each run for $5/month or a total of $10/month.
* TOC
{:toc}
<div class="uk-card uk-card-default uk-card-body">
<h5>Is this tutorial for you?</h5>
<p>This documentation is appropriate for advanced power users who are familiar with command line tools, editing configuration files, and basic configuration of services of DNS or Nginx. </p>
<p>If you are planning on running an <emphasis>open-membership hub</emphasis> or an <emphasis>application-specific hub</emphasis>, you'll should see <a href="hub-operation.html">the section on Hub Operation</a></p>.
</div>
## Prerequisites you need
You use DigitalOcean choose and configure assets for running droplets and spaces. To enable this, you must be sure to complete a number of prerequisites.
### Required prerequisites on DigitalOcean
You must create an account on <a href="https://digitalocean.com" target="\_blank">DigitalOcean</a>. DigitalOcean requires you to supply a credit card to create an account. You are only charged for the services you use the Gaia hub as of this writing should cost $10 USD a month.
### Optional prerequisites for SSH users
The easiest way to interact with your droplet is the DigitalOcean Console. Users who are comfortable using the secure shell (SSH) and private keys may prefer to open a local terminal on their home machine instead. To enable this, you should ensure you have the following prerequisites completed.
* Locate an existing SSH key pair on your Mac or <a href="https://help.dreamhost.com/hc/en-us/articles/115001736671-Creating-a-new-Key-pair-in-Mac-OS-X-or-Linux" target="\_blank">create a new SSH key pair</a>. Your key should have a passphrase, do not use a key pair without one.
* Add the <a href="https://www.digitalocean.com/docs/droplets/how-to/add-ssh-keys/to-account/" target="\_blank">SSH from your local machine to DigitalOcean</a>.
* Create a <a href="https://www.digitalocean.com/docs/api/create-personal-access-token/" target="\_blank">personal access token in DigitalOcean</a>.
* Install `doctl` which is the DigitalOcean command line tool. For information on installing these, see which is the DigitalOcean command line utility. Check out their [installation instructions](https://github.com/digitalocean/doctl/blob/master/README.md#installing-doctl) to see how to install it on your computer.
## Task 1: Create a DigitalOcean space
In this task you create a **Space** which is where Gaia stores your files.
1. Choose **Create > Spaces** from the dashboard menu.
![Dashboard](/storage/images/create-space.png)
2. **Choose a datacenter region** section.
{% include important.html content="Choose a region that is both
geographically close to you and that supports spaces. Currently, <strong>San
Francisco</strong>, <strong>New York</strong>, <strong>Amsterdam</strong>,
and <strong>Singapore</strong> support spaces." %}
The geographical location of your server impacts latency for storing data.
You select a region close to you so that when you use Blockstack apps,
storing data is quicker.
3. Scroll down to **Finalize and Create**.
4. **Choose a unique name**.
This name is used when reading files that you've stored through Gaia. You'll need to remember this name when you set up your Gaia server later on.
5. Click **Create a Space**.
After a moment, your Space is up and running.
## Task 2: Create a DigitalOcean droplet
In this task, you add a droplet to your account. The droplet is a droplet is a cloud-based server you can use as a compute resource. This server is where you will run the Gaia Storage System service. The droplet you create will be an Ubuntu server with Docker pre-installed.
1. Log into DigitalOcean.
2. Go to your DigitalOcean dashboard.
![Dashboard](/storage/images/digital-welcome.png)
3. Click the **Create > Droplets** button in the top right.
![Create option](/storage/images/digital-droplet.png)
4. Select the **One-click apps** tab.
![One-click app](/storage/images/digital-one-click.png)
5. Select the **Docker** app from the options presented.
This will give you
6. Scroll down to the **Choose a size** section and use the left arrow to display and select the **$5/mo** image.
This size gives you plenty of memory and disk space to run a personal hub.
7. Scroll down to the **Choose a datacenter region** section.
{% include important.html content="Choose a region that is both geographically close to you and that supports spaces. Currently, <strong>San Francisco</strong>, <strong>New York</strong>, <strong>Amsterdam</strong>, and <strong>Singapore</strong> support spaces." %}
The geographical location of your server impacts latency for storing data. You select a region close to you so that when you use Blockstack apps, storing data is quicker.
8. If you are using SSH, scroll to the **Add your SSH key** section and choose an SSH key to use. Otherwise,
9. Scroll down to the **Finalize and create** section.
10. **Choose a hostname** for you Give your Droplet such as `moxiegirl-storage-hub`.
11. Review your choices then click **Create** to start your droplet running.
At this point, your new droplet should appear in the list of resources on your DigitalOcean dashboard.
## Task 3: Open a console on your Droplet
A droplet console emulates the access you would have if you were sitting down with a keyboard and monitor attached to the actual server. In this section, you open a console on your droplet.
{% include note.html content="If you are an SSH user and have completed the prerequisites, <strong>skip this section</strong>. Instead, use <a href='https://do.co/2S4HMk1' target='\_blank'>the DigitalOcean instructions for connecting with doctl</a>." %}
1. From the DigitalOcean dashboard, select Droplets.
You should see the droplet you just created.
2. Click on the droplet name to open the control panel.
![Droplet control panel](/storage/images/droplet-control.png)
3. Choose **Access** from the control panel.
4. Select **Reset Root Password** to have DigitalOcean send you the root password.
DigitalOcean sends a temporary password to the email attached to your account. It takes a couple of minutes to reset the root password on your droplet.
5. Open your email and copy the password.
6. Switch back to the droplet control panel and choose **Launch Console**.
A new window with the console appears.
7. Enter `root` for the login.
8. Paste the password copied from your email.
The system displays a message telling you to change the `root` password.
![Droplet control panel](/storage/images/droplet-control.png)
And prompts you for the current password.
9. Past the password copied from your email again.
The system prompts you for a new password and ask you to enter it again.
10. Provide and confirm a new password.
The system logins you in and gives you a welcome message. At the conclusion of the message, you are at the console prompt.
```
Welcome to DigitalOcean's One-Click Docker Droplet.
To keep this Droplet secure, the UFW firewall is enabled.
All ports are BLOCKED except 22 (SSH), 2375 (Docker) and 2376 (Docker).
* The Docker One-Click Quickstart guide is available at:
https ://do.co/docker1804#start
* You can SSH to this Droplet in a terminal as root: ssh root@138.68.28.100
* Docker is installed and configured per Docker's recommendations:
https://docs.docker.com/install/linux/docker-ce/ubuntu/
* Docker Compose is installed and configured per Docker's recommendations:
https://docs.docker.eom/compose/install/#install-compose
For help and more information, visit http://do.co/dockerl804
To delete this message of the day: rm -rf /etc/update-motd.d/99-one-click
root@moxiegirl:~#
```
<div class="uk-card uk-card-default uk-card-body">
<h5>Useful tips for the console</h5>
<p>If you run into problems using the console, see <a href="https://www.digitalocean.com/docs/droplets/resources/console/" target="\_blank">the notes on this page in the DigitalOcean documentation</a>.</p>
<p>If you find the output from ls difficult to read, try enter the following to change the console colors from the command line: <code>LS_COLORS="di=1&semi;31"</code> You can also edit your console <code>.bashrc</code>. file permanently, of course.</p>
</div>
## Task 4: Create a space key
1. In the DigitalOcean dashboard, go to the **API** page.
2. Scroll to the **Spaces Access Keys** section.
3. Click **Generate New Key**.
The system prompts you to give the key a name.
4. Enter a name for the key.
It is helpful to choose descriptive name like `gai-hub-key`.
5. Press the check mark.
The system creates your key and displays both the key and its secret.
![Access key](/storage/images/space-access-key.png)
6. Save your secrete in a secure password manager.
You should never share your secret.
7. Leave the page up with your key and secret and go to your open console.
## Task 5: Get the Gaia code and configure your server
You should have the console open as `root` on your Droplet. In this section, you get the Gaia code and configure the Gaia service.
1. Copy the Gaia code into your droplet using the `git clone` command.
```
root@moxiegirl:~# git clone https://github.com/blockstack/gaia.git
```
Successful output from this command looks like the following.
```
Cloning into 'gaia'...
remote: Enumerating objects: 63, done.
remote: Counting objects: 100% (63/63), done.
remote: Compressing objects: 100% (46/46), done.
remote: Total 4206 (delta 27), reused 35 (delta 17), pack-reused 4143
Receiving objects: 100% (4206/4206), 17.40 MiB | 9.89 MiB/s, done.
Resolving deltas: 100% (2700/2700), done.
root@moxiegirl:~#
```
This command creates a `gaia` subdirectory.
2. Change to `hub` directory in the `gaia` code.
```
cd gaia/hub
```
3. Copy the configuration sample to a new `config.json` file.
```
cp config.sample.json config.json
```
4. Edit your new `config.json` file with `vi` or `vim`.
```
vi config.json
```
You now need to edit this JSON file to have it store files on your DigitalOcean space.
5. Set the `driver` to `aws`.
The DigitalOcean space API exactly mimics the S3 API. Since Gaia doesn't have a DigitalOcean driver, you can just use the `aws` driver with some special configuration.
6. Set the `bucket` to the name of the DigitalOcean space you just created.
If your space is called `moxiegirl-hub-space`, the `bucket` value is `moxiegirl-hub-space`.
6. Set the `readURL` to the URL of the DigitalOcean space you just created.
If your space URL called `https://moxiegirl-hub-space.sfo2.digitaloceanspaces.com `, the `readURL` name is `https://moxiegirl-hub-space.sfo2.digitaloceanspaces.com`.
7. Add an `endpoint` value to the `awsCredentials` section.
```json
"awsCredentials": {
"accessKeyId": "",
"secretAccessKey": "",
"endpoint": ""
},
```
8. Go back to your DigitalOcean dashboard open to your space key.
9. Copy the **Key** and paste it into the `accessKeyId` value in the `config.json` file.
10. Copy the **Secret** and paste it into the `secretAccessKey` value in the `config.json` file.
11. In the DigitalOcean dashboard, choose the Spaces page.
12. Copy the section of your space URL that follows the name.
![Space endpoint](/storage/images/space-endpoint.png)
In this example, you would copy the `sfo2.digitaloceanspaces.com` section.
13. Paste the string you copied into the `endpoint` value.
14. Change the `proofsRequired` value to the number `0` (zero).
This will allow Blockstack user to write to your Gaia hub, without any social proofs required. You can change this later on, and do other things to lock-down this Gaia hub to just yourself, but that is outside the scope of this document.
At this point, the json.config file should be completed and appear similar to the following &emdash; but with your values.
```json
{
"servername": "",
"port": 3000,
"driver": "aws",
"bucket": "moxiegirl-hub-space",
"readURL": "https://moxiegirl-hub-space.sfo2.digitaloceanspaces.com",
"awsCredentials": {
"accessKeyId": "W7GBQGIUWDWKA6KAGL56",
"secretAccessKey": "O6hBYRPCeRmE0d9lr3Frtc345QsWt3l+mrDgvrVT9oE",
"endpoint": "sfo2.digitaloceanspaces.com "
},
"proofsConfig": {
"proofsRequired" : 0
},
"azCredentials": {
"accountName": "",
"accountKey": ""
},
"gcCredentials": {
"projectId": ""
},
"argsTransport": {
"level": "warn",
"handleExceptions": true,
"stringify": true,
"timestamp": true,
"colorize": false,
"json": true
}
}
```
15. Save and close the `vim` editor.
The system returns you back to the prompt.
## Task 6: Run the Gaia image with Docker
While your console is still in the the `gaia/hub` folder, build the `gaia.hub` image.
1. Enter the following `docker` command at the console command line.
```
docker build -t gaia.hub .
```
This build users the `Dockerfile` already in the `gaia/hub` folder. The output of the command is similar to the following:
```
....
npm WARN gaia-hub@2.3.4 No license field.
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@1.2.4 (node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.2.4: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
added 877 packages from 540 contributors and audited 3671 packages in 38.122s
found 0 vulnerabilities
Removing intermediate container b0aef024879f
---> 5fd126019708
Step 5/5 : CMD ["npm", "run", "start"]
---> Running in ae459cc0865b
Removing intermediate container ae459cc0865b
---> b1ced6c39784
Successfully built b1ced6c39784
Successfully tagged gaia.hub:latest
```
2. Run your Gaia hub image.
```bash
docker run --restart=always -v ~/gaia/hub/config.json:/src/hub/config.json -p 3000:3000 -e CONFIG_PATH=/src/hub/config.json gaia.hub
```
This runs your Gaia hub on port `3000`. If everything runs successfully, the last line outputted from this command should be:
```bash
Successfully compiled 13 files with Babel.
{"level":"warn","message":"Listening on port 3000 in development mode","timestamp":"2019-01-23T16:35:05.216Z"}
```
3. If your command did run successfully, stop the service using the hotkey `ctrl-c`.
4. Run the the image again with this new command.
```
docker run --restart=always -v ~/gaia/hub/config.json:/src/hub/config.json -p 3000:3000 -e CONFIG_PATH=/src/hub/config.json -d gaia.hub
```
This command includes `-d` option to `docker run`. This runs Docker in detached mode, so that it runs in the background. You can run `docker ps` to see your running docker images, and get the `id` of your Gaia server.
```bash
root@moxiegirl:~/gaia/hub# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
aeca7eea4a86 gaia.hub "npm run start" 11 seconds ago Up 10 seconds 0.0.0.0:3000->3000/tcp musing_payne
```
At this point, your Gaia service is up and running. You can run `docker logs MY_CONTAINER_ID` with your running image's ID to see the logs of this server at any time.
## Task 7: Set up an Nginx reverse proxy
In this task, you set up a simple Nginx reverse proxy to serve your Docker container through a public URL. You do this from the droplet console command line.
1. Install nginx into the droplet.
```
sudo apt-get install nginx
```
2. Enter `y` to confirm the installation.
3. Edit the nginx default configuration file.
```
vi /etc/nginx/sites-available/default
```
4. Inside the `location /` block (line 48), enter the following configuration:
```nginx
location / {
proxy_pass http://localhost:3000;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
```
This simple configuration passes all requests through to your Gaia hub running at port `3000`.
5. Save and close the file.
6. Run `nginx -t` to make sure you have no syntax errors.
```
root@moxiegirl:~/gaia/hub# nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
```
7. Restart `nginx` with your new configuration.
```
systemctl restart nginx
```
8. Allow access to your Gaia hub by exposing port 80 to the public.
```
ufw allow 80
```
## Task 8: Test your Gaia server
Now, you are ready to test your Gaia server and make sure it is up and running.
1. Click on **Droplets** in the sidebar.
2. Find your Droplet running Gaia.
![Droplet IP](/storage/images/space-endpoint.png)
3. Copy the IP address for it.
4. In your browser, visit the page `MY_DROPLET_IP/hub_info`.
You should see a response from your Gaia hub!
![Hub test](/storage/images/hub-running.png)
The `read_url_prefix` should be combine from the bucket and endpoint create
in your `config.json` file, for example,
`https://moxiegirl-hub-space.s3.amazonaws.com/`.
## Task 9: Configure a domain name
At this point, you can point a domain to your Gaia hub. Although it's not required, it is highly recommended. If you use a domain, you can migrate your Droplet to a different server (or even provider such as Azure or AWS) at any time, and still access it through the domain URL. Simply point your domain at the IP address for your Droplet. Use an `A Record` DNS type.
These instructions assume you have already created a free <a href="https://www.freenom.com" target="\_blank">domain through the freenom service</a>. To point this freenom domain to your Gaia hub, do the following:
1. Log into your freenom account.
2. Choose the **Manage Freenom Domain** tab.
3. Add an **A** record leave the **Name** field blank.
This record points your entire domain to the hub IP.
4. Save your changes.
5. Create a CNAME record.
For example, you can use the prefix `www` with your domain name. When you are done, your
6. Save your changes.
At this point, your DNS management should look similar to the following except that with your domain rather than the `maryhub.ga` domain.
![DNS fields](/storage/images/dns-fields.png)
7. After your changes propagate, visit your new domain at the `hub_info` page.
![Domain test](/storage/images/domain-test.png)
## Task 10: Set up SSL
If you've configured a domain to point to your Gaia hub, then it's highly
recommended that you set up SSL to connect to your hub securely. DigitalOcean
provides <a
href="https://www.digitalocean.com/community/tutorials/how-to-secure-nginx-with-let-s-encrypt-on-ubuntu-18-04"
target="\_blank">how to setup SSL </a>. Follow those instructions, to setup SSL.
When completed, you'll be able to visit `https://mygaiadomain.com/hub_info`
securely.

341
_storage/gaia-admin.md

@ -0,0 +1,341 @@
---
layout: storage
permalink: /:collection/:path.html
---
# Use the admin service
{:.no_toc}
A Gaia service can run a simple administrative service co-located with your Gaia hub. This service allows you to administer the Gaia hub with the help of an API key. Gaia hubs installed using the Gaia Amazon Machine Image (AMI) have this service integrated automatically.
In this section, you learn how to use the Gaia admin service with your Gaia hub.
* TOC
{:toc}
{% include note.html content="
The examples in this section assume that Gaia and the admin service
were installed through the Configure a hub on Amazon EC2." %}
## Understand the configuration files
The admin service relies on two configuration files, the hub's configuration and the configuration of the admin service itself. The hub's configuration is mounted `/tmp/hub-config/config.json` in the `docker_admin_1` container. Your EC2 instance has the admin service configuration in the `/gaia/docker/admin-config/config.json` file.
The admin service needs to know the following:
* where the Gaia hub config file is located
* which API key(s) to use when authenticating administrative requests
* which command(s) to run to restart the Gaia hub on a config change
The following is the standard admin service config installed with your EC2 instance.
```json
{
"argsTransport": {
"level": "debug",
"handleExceptions": true,
"timestamp": true,
"stringify": true,
"colorize": true,
"json": true
},
"port": 8009,
"apiKeys": [ "hello" ],
"gaiaSettings": {
"configPath": "/tmp/hub-config/config.json"
},
"reloadSettings": {
"command": "/bin/sh",
"argv": [
"-c",
"docker restart docker_hub_1 &"
],
"env": {},
"setuid": 1000,
"setgid": 1000
}
}
```
The `port` is the port where Gaia is running. The `apiKeys` field is key used for making calls to the hub. The `gaiaSettings`
The `argsTransport` section configures the hub logging. The service uses the `winston` logging service. Refer to their documentation for full details on the [logging configuration options](https://github.com/winstonjs/winston).
<table class="uk-table uk-table-small uk-table-divider">
<thead>
<tr>
<th>Field</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>level</code></td>
<td>Lowest level this transport will log. (default: <code>info</code>)</td>
</tr>
<tr>
<td><code>handleException</code></td>
<td>Set to true to have this transport handle exceptions. (default: <code>false</code>)</td>
</tr>
<tr>
<td><code>timestamp</code></td>
<td>The timestamp when the message was received.</td>
</tr>
<tr>
<td><code>stringify</code></td>
<td>Converts the output to a JSON string.</td>
</tr>
<tr>
<td><code>colorize</code></td>
<td>Colorizes the standard logging level</td>
</tr>
<tr>
<td><code>json</code></td>
<td>Log format.</td>
</tr>
</tbody>
</table>
The `reloadSettings` configure the command that is used to reload your Gaia hub.
<table class="uk-table uk-table-small uk-table-divider">
<thead>
<tr>
<th>Field</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>command</code></td>
<td>A command which reloads the Gaia hub service.</td>
</tr>
<tr>
<td><code>argv</code></td>
<td>An array containing the command arguments.</td>
</tr>
<tr>
<td><code>env</code></td>
<td>This is a key/value list of any environment variables
that need to be set for the command to run. This is optional.</td>
</tr>
<tr>
<td><code>setuid</code></td>
<td>This is the UID under which the command will be run. This is optional.</td>
</tr>
<tr>
<td><code>setgid</code></td>
<td>This is the GID under which the command will run. This is optional.</td>
</tr>
</tbody>
</table>
## Using the admin service APIs
You use the admin service APIs to manage the hub. Administrating a hub requires
that you make calls from a terminal on your hub service server. To execute admin
functions on a Gaia hub created with AWS, you `ssh` into your instance as follows:
```bash
ssh -t -i <your keyfile.pem> -A core@<public ip address>
```
You must also set the `API_KEY` in your environment:
```bash
export API_KEY="hello"
```
You may find it useful to install a JSON processor such as `jq` to process the
output of the admin commands.
### Restart the Gaia Hub (`POST /v1/admin/reload`)
The admin service will make changes to the Gaia hub's config file, but the
changes will only take effect when the Gaia hub is reloaded. You can do this
as follows:
```bash
$ export API_KEY="hello"
$ curl -H "Authorization: bearer $API_KEY" -X POST http://localhost:8009/v1/admin/reload
{"result":"OK"}
```
When you `POST` to this endpoint, the admin service runs the command described
in the `reloadSettings` section of the config file. It attempts to spawn a
subprocess from the given `reloadSettings.command` binary, and pass it the
arguments given in `reloadSettings.argv`. Note that the subprocess will *NOT*
be run in a shell.
#### Errors
{:.no_toc}
If you do not supply a valid API key, this method fails with HTTP 403.
This endpoint returns HTTP 500 if the reload command fails. If this
happens, you will get back the command's exit code and possibly the signal that
killed it.
### Get the hub configuration (`GET /v1/admin/config`)
This endpoint is used to read and write a Gaia hub's non-driver-related
settings. These include the port it listens on, and its proof-checking
settings.
To read the Gaia hub settings, you would run the following:
```bash
$ export API_KEY="hello"
$ curl -H "Authorization: bearer $API_KEY" http://localhost:8009/v1/admin/config {"config":{"port":4000,"proofsConfig":{"proofsRequired":0}}}
```
### Set the hub configuration (`POST /v1/admin/config`)
To set Gaia hub settings, you simply `POST` the changed JSON fields to this
endpoint.
```bash
$ export API_KEY="hello"
$ curl -H "Authorization: bearer $API_KEY" -H 'Content-Type: application/json' -X POST --data-raw '{"port": 3001}' http://localhost:8009/v1/admin/config
{"message":"Config updated -- you should reload your Gaia hub now."}
```
If the settings were successfully applied, the method returns a message to reload your Gaia hub. You can set multiple drivers' settings with a single call. For example, you can set:
* The driver to use (`driver`)
* The Gaia's read URL endpoint (`readURL`)
* The number of items to return when listing files (`pageSize`)
* The driver-specific settings
The data accepted on `POST` must contain a valid Hub configuration, for example:
```
const GAIA_CONFIG_SCHEMA = {
type: "object",
properties: {
validHubUrls: {
type: "array",
items: { type: "string", pattern: "^http://.+|https://.+$" },
},
requireCorrectHubUrl: { type: "boolean" },
serverName: { type: "string", pattern: ".+" },
port: { type: "integer", minimum: 1024, maximum: 65534 },
proofsConfig: { type: "integer", minimum: 0 },
whitelist: {
type: "array",
items: {
type: "string",
pattern: "^[123456789ABCDEFGHJKLMNPQRSTUVWXYZabcdefghijkmnopqrstuvwxyz]+$"
}
},
driver: { type: "string", pattern: ".+" },
readURL: { type: "string", pattern: "^http://.+$|https://.+$" },
pageSize: { type: "integer", minimum: 1 },
bucket: { type: "string", pattern: ".+" },
cacheControl: { type: "string", pattern: ".+" },
azCredentials: {
accountName: { type: "string", pattern: ".+" },
accountKey: { type: "string", pattern: ".+" },
},
diskSettings: {
storageRootDirectory: { type: "string" }
},
gcCredentials: {
email: { type: "string" },
projectId: { type: "string" },
keyFilename: { type: "string" },
credentials: {
type: "object",
properties: {
client_email: { type: "string" },
private_key: { type: "string" }
}
},
},
awsCredentials: {
assessKeyId: { type: "string" },
secretAccessKey: { type: "string" },
sessionToken: { type: "string" }
}
}
}
```
The same fields are returned on `GET` within a `config` object.
#### Errors
{:.no_toc}
If you do not supply a valid API key, both the `GET` and `POST` method fail with HTTP 403.
Only relevant Gaia hub config fields are set. If you `POST` invalid settings
values, you get an HTTP 400 error.
## Example: Read and write driver settings
Use the `/v1/admin/config` endpoint to read and write storage driver settings. To get the current driver settings, you would run:
```bash
$ export API_KEY="hello"
$ curl -H "Authorization: bearer $API_KEY" http://localhost:8009/v1/admin/config
{"config":{"driver":"disk","readURL":"http://localhost:4001/","pageSize":20,"diskSettings":{"storageRootDirectory":"/tmp/gaia-disk"}}}
```
To update the driver settings, you would run:
```bash
$ export API_KEY="hello"
$ export AWS_ACCESS_KEY="<hidden>"
$ export AWS_SECRET_KEY="<hidden>"
$ curl -H "Authorization: bearer $API_KEY" -H 'Content-Type: application/json' -X POST --data-raw "{\"driver\": \"aws\", \"awsCredentials\": {\"accessKeyId\": \"$AWS_ACCESS_KEY\", \"secretAccessKey\": \"$AWS_SECRET_KEY\"}}" http://localhost:8009/v1/admin/config
{"message":"Config updated -- you should reload your Gaia hub now."}
```
## Example: Read and write the whitelist
This endpoint lets you read and write the `whitelist` section of a Gaia hub, to control who can write to it and list its files.
To get the current whitelist, you would run the following:
```bash
$ export API_KEY="hello"
$ curl -H "Authorization: bearer $API_KEY" http://localhost:8009/v1/admin/config
{"config":{"whitelist":["15hUKXg1URbQsmaEHKFV2vP9kCeCsT8gUu"]}}
```
To set the whitelist, you must set the *entire* whitelist. To set the list, pass a command such as the following:
{% raw %}
```bash
$ export API_KEY="hello"
$ curl -H "Authorization: bearer $API_KEY" -H 'Content-Type: application/json' -X POST --data-raw '{"whitelist": ["1KDcaHsYJqD7pwHtpDn6sujCVQCY2e1ktw", "15hUKXg1URbQsmaEHKFV2vP9kCeCsT8gUu"]}' http://localhost:8009/v1/admin/config
{"message":"Config updated -- you should reload your Gaia hub now."}
```
{% endraw %}
## View logs for the hub or admin service
The logs for each Gaia service are maintained by their respective Docker containers. To view the log for a particular service, use the `docker logs` command. For example, to get the logs for the hub:
```
$ docker logs docker_hub_1
> gaia-hub@2.3.4 start /src/hub
> npm run build && node lib/index.js
> gaia-hub@2.3.4 build /src/hub
> npm run lint && babel src -d lib && chmod +x lib/index.js
> gaia-hub@2.3.4 lint /src/hub
> eslint src
Successfully compiled 13 files with Babel.
{"level":"warn","message":"Listening on port 3000 in development mode","timestamp":"2019-02-14T04:00:06.071Z"}
```

5
_storage/gaia_redirect.md

@ -1,5 +0,0 @@
---
title: Storage with Gaia
layout: externalurl
redirect_url: https://github.com/blockstack/gaia
---

276
_storage/hello-hub-choice.md

@ -0,0 +1,276 @@
---
layout: storage
permalink: /:collection/:path.html
---
# Hello hub choice tutorial
{:.no_toc}
In this tutorial, you build on the <a href="{{ site.baseurl }}/browser/hello-blockstack.html" target="\_blank">Hello, Blockstack Tutorial</a>. You'll modify the authentication code so that it prompts users who have not yet created a Blockstack identity, to choose a hub URL.
* TOC
{:toc}
{% include note.html content="This tutorial was written on macOS High Sierra 10.13.4. If you use a Windows or Linux system, you can still follow along. However, you will need to \"translate\" appropriately for your operating system. Additionally, this tutorial assumes you are accessing the Blockstack Browser web application via Chrome. The application you build will also work with a local installation and/or with browsers other than Chrome. " %}
## About this tutorial and the prerequisites you need
This tutorial assumes you already set up your environment and tooling as specified in the <a href="{{ site.baseurl }}/browser/hello-blockstack.html" target="\_blank">Hello, Blockstack Tutorial</a>. You should also review that tutorial for basic information about
## Task 1: Generate an initial Blockstack application
In this section, you build an initial React.js application called `hello-hub-choice`.
1. Create the `hello-hub-choice` directory.
```bash
mkdir hello-hub-choice
```
2. Change into your new directory.
```bash
cd hello-hub-choice
```
3. Use Yeoman and the Blockstack application generator to create your initial `hello-hub-choice` application.
```bash
yo blockstack
```
You should see several interactive prompts.
{% raw %}
```bash
$ yo blockstack
...
? Are you ready to build a Blockstack app in React? (Y/n)
```
{% endraw %}
4. Respond to the prompts to populate the initial app.
After the process completes successfully, you see a prompt similar to the following:
{% raw %}
```bash
...
create public/icon-192x192.png
create public/index.html
create public/robots.txt
create public/manifest.json
Im all done. Running npm install for you to install the required dependencies. If this fails, try running the command yourself.
```
{% endraw %}
5. Verify that you have version `18.3.0` of blockstack.js or higher.
```
$ npm ls blockstack
hello-blockstack@0.0.0 /Users/manthony/sampleapps/hello-blockstack
└── blockstack@18.3.0
```
If you don't have the correct version, install it.
```
npm install blockstack@18.3.0
```
Depending on your environment you may have some problems with the `npm` packages. Go ahead and fix these before continuing to the next section.
## Task 2. Start the server and view the application
When you start the server, it will create a Node.js server, start it locally,
and open your browser `http://localhost:5000`. From the root of your new application directory:
1. Start the application server.
```bash
npm start
```
2. Choose **Allow**.
3. Open your browser to `http://localhost:8080`.
You should see a simple application:
![](images/sign-in.png)
4. Leave your new application running and move onto the next section.
## Task 3: Enable hub selection
By default, the app generator assumes you want to use the default flow `redirectToSignIn()` method. In this section, you replace that method and use the `makeAuthRequest()` method instead. The `makeAuthRequest()` method takes the following parameters:
<dl class="uk-description-list">
<dt class="uk-text-lowercase">
<code>transitPrivateKey(String = generateAndStoreTransitKey())</code>
</dt>
<dd>A HEX encoded transit private key.</dd>
<dt class="uk-text-lowercase">
<code>redirectURI(String = `${window.location.origin}/`)</code>
</dt>
<dd>Location to redirect the user to after sign in approval.</dd>
<dt class="uk-text-lowercase">
<code>manifestURI(String = `${window.location.origin}/manifest.json`)</code>
</dt>
<dd>
Location of this app's manifest file.
</dd>
<dt class="uk-text-lowercase">
<code>scopes (Array = DEFAULT_SCOPE)</code>
</dt>
<dd>The permissions this app is requesting.</dd>
<dt class="uk-text-lowercase">
<code>appDomain(String = window.location.origin)</code>
</dt>
<dd>The origin of this app.</dd>
<dt class="uk-text-lowercase">
<code>expiresAt(Number = nextHour().getTime())</code>
</dt>
<dd>The time at which this request is no longer valid.</dd>
<dt class="uk-text-lowercase">
<code>extraParams(Object = {})</code>
</dt>
<dd>Any extra parameters to pass to the authenticator. Use this to pass options that aren't part of the Blockstack authentication specification, but might be supported by special authenticators.</dd>
</dl>
To replace the default login, do the following:
1. Using your favorite editor, open the `public/app.js` file.
2. Locate the `redirectToSignIn()` method at line 4.
3. Replace `redirectToSignIn()` method with the `blockstack.redirectToSignInWithAuthRequest(authRequest)` method.
The `authRequest` is the authentication request generated by `makeAuthRequest()` method.
4. Immediately above the method you just added and below the `event.preventDefault()` method, construct a String `const` for the `authRequest`:
```
const authRequest = blockstack.makeAuthRequest(
blockstack.generateAndStoreTransitKey(),
'http://localhost:8080/',
'http://localhost:8080/manifest.json',
['store_write', 'publish_data'],
'http://localhost:8080/',
blockstack.nextHour().getTime(), {
solicitGaiaHubUrl: true
} // new options param
);
```
{% include note.html content="If your app is running a different port than <code>8080</code>, enter that port instead. " %}
The extra `solicitGaiaHubUrl` parameter of `true` will cause the Blockstack Browser to prompt new identity creators for a storage hub URL.
5. Save and close the `public/app.js` file.
6. Make sure your app rebuilds cleanly.
## Task 4: Try the new authentication flow
Try your new authentication code.
1. Refresh the client at `http://localhost:8080/`.
2. Click *Sign in with Blockstack*.
The Blockstack Browser prompts you to sign in. I you are _not already authenticated_ with the browser, you should see the following:
![](images/not-authenticated.png)
If you are already authenticated with an existing ID, you can choose that ID
or **Deny**. Choosing an existing ID, signs you into the Hello Blockstack
application, without offering a hub choice. Instead, choose **Deny**. This
ends the authentication process. Then, reset the Blockstack Browser.
3. Choose **Create new ID**.
The system prompts you to enter a username.
4. Enter a name and press **Check Availability**.
5. Press **Continue** after you find an available name you like.
6. Enter a password and chose **Register ID**.
The system prompts you for a provider.
![](images/provider-prompt.png)
The default is to user Blockstacks' Gaia hub.
7. Choose **Use a different provider**.
The system prompts you for a URL.
![](images/different-provider.png)
8. Enter a Gaia hub URL.
If you have one of your own enter it here. You can also just enter the Blockstack URL (`https://hub.blockstack.org`).
9. Press **Continue**.
The system takes a moment to check the hub is responsive and establish it in your new identity profile. Then, the system prompts you for an email.
10. Enter an email and press **Next**.
The system creates your user and prompts you to continue to your application.
11. Choose **Go to Hello, Blockstack**.
The system prompts you to allow Hello Blockstack access.
12. Grant the access to the DApp.
You user should be authenticated.
## How to recommend a Gaia hub URL for new users
If you want to create specific sign-up flows for your DApp, you can pass a preset Gaia storage hub URL also. For example, you might
do this if you have a corporate client whose employees would all like to use
your application with a company-run Gaia hub.
To suggest a Gaia hub URL, provide an additional `recommendedGaiaHubUrl` value
alongside the `solicitGaiaHubUrl`, for example:
```javascript
import {
makeAuthRequest,
redirectToSignInWithAuthRequest
} from 'blockstack';
const authRequest = makeAuthRequest(undefined, undefined, undefined, undefined, undefined, undefined, {
solicitGaiaHubUrl: true,
recommendedGaiaHubUrl: 'https://mygaiahub.com'
});
const authRequest = makeAuthRequest(
generateAndStoreTransitKey(),
'http://localhost:8080/',
'http://localhost:8080/manifest.json',
['store_write', 'publish_data'],
'http://localhost:8080/',
nextHour().getTime(), {
solicitGaiaHubUrl: true, //new options param
recommendedGaiaHubUrl: 'https://mygaiahub.com' // new options param
}
);
redirectToSignInWithAuthRequest(authRequest);
```
Passing these parameters changes the storage hub URL prompt to the following:
![Gaiastorage](/storage/images/recommended-provider.png)
## Related information
{:.no_toc}
[`makeAuthRequest()`](https://blockstack.github.io/blockstack.js/#makeauthrequest) method

111
_storage/hub-choice.md

@ -0,0 +1,111 @@
---
layout: storage
permalink: /:collection/:path.html
---
# Hubs and user choice
Blockstack operates a default Gaia storage hub at `https://hub.blockstack.org`.
Individuals or organizations may also run their own hubs, either as a for-profit
service or for other reasons.
## Background
Each user with an identity in the Blockstack Network has a Gaia hub
configured on their profile. New users that create identities with the
Blockstack Browser automatically are given storage on this default hub. For a
user to have a storage hub other than this default, the user must create an
identity through an application that enables storage hub selection.
{% include note.html content="Users with existing identities cannot yet migrate
their data from one hub to another." %}
## Enable the Gaia hub URL option for new users
A major principle of Blockstack applications is that DApps do not store or
replicated user data. Through the Gaia feature, Blockstack seeks to empower
users to choose where to store their data and how they manage it.
As a step towards fulfilling this principle, DApp developers can configure their
application to prompt new users to provide a Gaia hub URL. An application with
this feature enabled presents users creating an identity for the first time
with this prompt:
![Official provider](/storage/images/official-provider.jpeg)
Users can choose to **Use Blockstack's official provider** or **Use a different provider**. Choosing the non-default option prompts the user with the following:
![Different provider](/storage/images/different-provider.png)
Users can enter the HTTPS address of a hub hosted by an entity other than
Blockstack. Once the user submits the above form, they continue through the
standard identity creation. As they use DApps on the Blockstack Network, all
their data is stored in this storage hub and not the default Blockstack hub.
## Enabling hub selection in your DApp
To enable this feature, you must ensure your DApp is using the latest version of the blockstack.js library. Instead of the default flow `redirectToSignIn()` method, you must use the `makeAuthRequest()` method. This method takes the following parameters:
<dl class="uk-description-list">
<dt class="uk-text-lowercase">
<code>transitPrivateKey(String = generateAndStoreTransitKey())</code>
</dt>
<dd>A HEX encoded transit private key.</dd>
<dt class="uk-text-lowercase">
<code>redirectURI(String = `${window.location.origin}/`)</code>
</dt>
<dd>Location to redirect the user to after sign in approval.</dd>
<dt class="uk-text-lowercase">
<code>manifestURI(String = `${window.location.origin}/manifest.json`)</code>
</dt>
<dd>
Location of this app's manifest file.
</dd>
<dt class="uk-text-lowercase">
<code>scopes (Array = DEFAULT_SCOPE)</code>
</dt>
<dd>The permissions this app is requesting.</dd>
<dt class="uk-text-lowercase">
<code>appDomain(String = window.location.origin)</code>
</dt>
<dd>The origin of this app.</dd>
<dt class="uk-text-lowercase">
<code>expiresAt(Number = nextHour().getTime())</code>
</dt>
<dd>The time at which this request is no longer valid.</dd>
<dt class="uk-text-lowercase">
<code>extraParams(Object = {})</code>
</dt>
<dd>Any extra parameters to pass to the authenticator. Use this to pass options that aren't part of the Blockstack authentication specification, but might be supported by special authenticators.</dd>
</dl>
To ensure your DApps identity creation flow includes the Gaia URL, modify the `makeAuthRequest()` method to apply the `solicitGaiaHubUrl` parameter with value `true` when executing the method:
```javascript
import {
makeAuthRequest,
redirectToSignInWithAuthRequest
} from 'blockstack';
const authRequest = makeAuthRequest(
generateAndStoreTransitKey(),
'http://localhost:8080/',
'http://localhost:8080/manifest.json',
['store_write', 'publish_data'],
'http://localhost:8080/',
nextHour().getTime(), {
solicitGaiaHubUrl: true
} // new options param
);
redirectToSignInWithAuthRequest(authRequest);
```
You can also pass a suggested Gaia storage hub URL also. For example, you might
do this if you have a corporate client whose employees would all like to use
your application with a company-run Gaia hub. To do this, you provide an
additional `recommendedGaiaHubUrl` value alongside the `solicitGaiaHubUrl`
## Related information
{:.no_toc}
[Hello, Hub Choice Tutorial](hello-hub-choice.html) for a tutorial on using the [`makeAuthRequest()`](https://blockstack.github.io/blockstack.js/#makeauthrequest) method.

102
_storage/hub-operation.md

@ -0,0 +1,102 @@
---
layout: storage
permalink: /:collection/:path.html
---
# Understand hub operation
{:.no_toc}
This page describes the considerations hub operators must take into account when creating and operating a Gaia storage hub.
* TOC
{:toc}
## Configuration files
You should store a JSON configuration file either in the top-level directory of
the hub server. Alternatively, you can specify a file location using the
`CONFIG_PATH` environment variable. The following is an example configuration file for Amazon S3:
```json
{
"servername": "localhost",
"port": 4000,
"driver": "aws",
"readURL": "https://YOUR_BUCKET_NAME.s3.amazonaws.com/",
"pageSize": 20,
"bucket": "YOUR_BUCKET_NAME",
"awsCredentials": {
"accessKeyID": "YOUR_ACCESS_KEY",
"secretAccessKey": "YOUR_SECRET_KEY"
},
"argsTransport": {
"level": "debug",
"handleExceptions": true,
"stringify": true,
"timestamp": true,
"colorize": false,
"json": true
}
}
```
You can specify the logging level, the backend driver, the credentials
for that backend driver, and the `readURL` of the hub. Typically, this is the URL for the compute resource on the cloud computing provider &mdash; where the hub service is running.
### Require the correct hub URL
If you enable on the `requireCorrectHubUrl` option in your `config.json`
file, your Gaia hub will require that authentication requests
correctly include the `hubURL` they are trying to connect with. Use this option to prevent a malicious Gaia hub from using an authentication
token for itself on other Gaia hubs.
By default, the Gaia hub will validate that the supplied URL matches
`https://${config.servername}`, but if there are multiple valid URLs
for clients to reach the hub at, you can include a list in your `config.json`:
```javascript
{
....
servername: "normalserver.com"
validHubUrls: [ "https://specialserver.com/",
"https://legacyurl.info" ]
....
}
```
### The readURL parameter
By default, hub drivers return read URLs that point directly at the written content. For example, an S3 driver would return the URL directly to the S3 file. If you configure a CDN or domain to point at that same bucket, you can use the `readURL` parameter to tell the hub that files can be read from a given URL. For example, the `hub.blockstack.org` Gaia Hub is configured to return a read URL that looks like `https://gaia.blockstack.org/hub/`.
Unset the `readURL` parameter if you do not intend to deploy any caching.
### The proofsRequired parameter (Deprecated)
Past users could configure this setting as a crude spam-control mechanism.
However, for the smoothest operation of your Gaia hub, set the
`proofsConfig.proofsRequired` value to `0`.
## Open or private hubs
You can configure an open-membership storage hub or a private storage hub. An open-membership hub, as it sounds, allows any user to use the hub service. A private hub limits the use of the service. In this section, you learn about configuring each type.
### Open-membership hub
An open-membership storage hub permits writes for _any_ address top-level
directory. Every request is validated such that write requests must provide
valid authentication tokens for that address. Operating in this mode is
recommended for service and identity providers who wish to support many
different users.
### Private-user hub
A private-user hub receives requests for a single user. Requests are controlled
via _whitelisting_ the addresses allowed to write files. Recall that each application uses a different app- and user-specific address. It follows, to
support application storage, your configuration must add to the whitelist each application you wish to use.
Alternatively, the user's client can use the authentication scheme and generate
an association token for each app. The user should whitelist her address, and
use her associated private key to sign each app's association token. This
removes the need to whitelist each application, but with the caveat that the
user needs to take care that her association tokens do not get misused.

BIN
_storage/images/add-storage.png

Binary file not shown.

After

Width:  |  Height:  |  Size: 137 KiB

BIN
_storage/images/app-sign-in.png

Binary file not shown.

After

Width:  |  Height:  |  Size: 27 KiB

BIN
_storage/images/aws-console.png

Binary file not shown.

After

Width:  |  Height:  |  Size: 82 KiB

BIN
_storage/images/aws-dns-fields.png

Binary file not shown.

After

Width:  |  Height:  |  Size: 64 KiB

BIN
_storage/images/aws-hub.png

Binary file not shown.

After

Width:  |  Height:  |  Size: 32 KiB

BIN
_storage/images/aws-launch-status.png

Binary file not shown.

After

Width:  |  Height:  |  Size: 102 KiB

BIN
_storage/images/cert-check.png

Binary file not shown.

After

Width:  |  Height:  |  Size: 126 KiB

BIN
_storage/images/change-root.png

Binary file not shown.

After

Width:  |  Height:  |  Size: 72 KiB

BIN
_storage/images/create-space.png

Binary file not shown.

After

Width:  |  Height:  |  Size: 51 KiB

BIN
_storage/images/different-provider.png

Binary file not shown.

After

Width:  |  Height:  |  Size: 107 KiB

BIN
_storage/images/digital-droplet-list.png

Binary file not shown.

After

Width:  |  Height:  |  Size: 46 KiB

BIN
_storage/images/digital-droplet.png

Binary file not shown.

After

Width:  |  Height:  |  Size: 74 KiB

BIN
_storage/images/digital-one-click.png

Binary file not shown.

After

Width:  |  Height:  |  Size: 61 KiB

BIN
_storage/images/digital-ssh-key.png

Binary file not shown.

After

Width:  |  Height:  |  Size: 35 KiB

BIN
_storage/images/digital-welcome.png

Binary file not shown.

After

Width:  |  Height:  |  Size: 14 KiB

BIN
_storage/images/dns-fields.png

Binary file not shown.

After

Width:  |  Height:  |  Size: 65 KiB

BIN
_storage/images/docker-ps.png

Binary file not shown.

After

Width:  |  Height:  |  Size: 62 KiB

BIN
_storage/images/domain-test.png

Binary file not shown.

After

Width:  |  Height:  |  Size: 46 KiB

BIN
_storage/images/droplet-control.png

Binary file not shown.

After

Width:  |  Height:  |  Size: 60 KiB

BIN
_storage/images/droplet-ip.png

Binary file not shown.

After

Width:  |  Height:  |  Size: 74 KiB

BIN
_storage/images/ec2-dashboard.png

Binary file not shown.

After

Width:  |  Height:  |  Size: 122 KiB

BIN
_storage/images/ec2-instance-details.png

Binary file not shown.

After

Width:  |  Height:  |  Size: 212 KiB

BIN
_storage/images/ec2-instance.png

Binary file not shown.

After

Width:  |  Height:  |  Size: 147 KiB

BIN
_storage/images/gaia-community.png

Binary file not shown.

After

Width:  |  Height:  |  Size: 132 KiB

BIN
_storage/images/gaia-storage.png

Binary file not shown.

After

Width:  |  Height:  |  Size: 19 KiB

BIN
_storage/images/gaia-writes.png

Binary file not shown.

After

Width:  |  Height:  |  Size: 12 KiB

BIN
_storage/images/hub-running.png

Binary file not shown.

After

Width:  |  Height:  |  Size: 48 KiB

BIN
_storage/images/instance-initialize.png

Binary file not shown.

After

Width:  |  Height:  |  Size: 25 KiB

BIN
_storage/images/jwt-token.png

Binary file not shown.

After

Width:  |  Height:  |  Size: 138 KiB

BIN
_storage/images/not-authenticated.png

Binary file not shown.

After

Width:  |  Height:  |  Size: 103 KiB

BIN
_storage/images/official-provider.jpeg

Binary file not shown.

After

Width:  |  Height:  |  Size: 70 KiB

BIN
_storage/images/private-connection.png

Binary file not shown.

After

Width:  |  Height:  |  Size: 108 KiB

BIN
_storage/images/provider-prompt.png

Binary file not shown.

After

Width:  |  Height:  |  Size: 146 KiB

BIN
_storage/images/recommended-provider.png

Binary file not shown.

After

Width:  |  Height:  |  Size: 27 KiB

BIN
_storage/images/sign-in.png

Binary file not shown.

After

Width:  |  Height:  |  Size: 30 KiB

BIN
_storage/images/space-access-key.png

Binary file not shown.

After

Width:  |  Height:  |  Size: 83 KiB

BIN
_storage/images/space-endpoint.png

Binary file not shown.

After

Width:  |  Height:  |  Size: 72 KiB

BIN
_storage/images/tag-add.png

Binary file not shown.

After

Width:  |  Height:  |  Size: 102 KiB

BIN
_storage/images/tier-2-image.png

Binary file not shown.

After

Width:  |  Height:  |  Size: 147 KiB

BIN
_storage/images/us-west-2.png

Binary file not shown.

After

Width:  |  Height:  |  Size: 54 KiB

209
_storage/overview.md

@ -0,0 +1,209 @@
---
layout: storage
permalink: /:collection/:path.html
---
# A decentralized storage architecture
The Blockstack Network stores application data using a storage system called
Gaia. Transactional metadata is stored on the Blockstack blockchain and user
application data is stored in Gaia storage. Storing data off of the blockchain
ensures that Blockstack applications can provide users with high performance and
high availability for data reads and writes without introducing central trust
parties.
* TOC
{:toc}
## Understand Gaia in the Blockstack architecture
The following diagram depicts the Blockstack architecture and Gaia's place in it:
{% include architecture.md %}
## User control or how is Gaia decentralized?
A Gaia hub runs as a service which writes to data storage. The hub service
writes to data storage by requiring a valid authentication token from a requestor. Typically, the hub service runs on a compute resource and the storage itself on separate, dedicated storage resource. Typically, both resources belong to the same cloud computing provider.
![Gaiastorage](/storage/images/gaia-storage.png)
Gaia's approach to decentralization focuses on user control of data and its
storage. If a user can choose which Gaia hub provider to use, then that choice
is all the decentralization required to enable user-controlled applications.
The control of user data lies in the way that user data is accessed.
When an application fetches a file `data.txt` for a given user `alice.id`, the
lookup will follow these steps:
1. Fetch the `zonefile` for `alice.id`.
2. Read her profile URL from her `zonefile`.
3. Fetch Alice's profile.
4. _Verify_ that the profile is signed by `alice.id`'s key
5. Read the `gaiaHubUrl` (e.g. `https://gaia.alice.org/`) out of the profile
6. Fetch the file from `https://gaia.alice.org/data.txt`.
Because `alice.id` has access to her zonefile, she can change where her profile
is stored. For example, she may do this if the current profile's service or
storage is compromised. To change where her profile is stored, she changes her
Gaia hub URL to another Gaia hub URL from another hub provider. If Alice has
sufficient compute and storage resources herself, she may run her own Gaia
Storage System and bypass a commercial Gaia hub provider all together.
{% include note.html content="Users with existing identities cannot yet migrate
their data from one hub to another." %}
Applications writing directly on behalf of Alice do not need to perform a
lookup. Instead, the Blockstack <a
href="http://blockstack.github.io/blockstack.js/index.html"
target="\_blank">authentication flow</a> provides Alice's chosen application
root URL to the application. This authentication flow _is also_ within Alice's
control because Alice's browser _must_ generate the authentication response.
## Understand data storage
A Gaia hub stores the written data _exactly_ as given. It offers minimal
guarantees about the data. It does not ensure that data is validly formatted,
contains valid signatures, or is encrypted. Rather, the design philosophy is
that these concerns are client-side concerns.
Client libraries (such as `blockstack.js`) are capable of providing these
guarantees. Blockstack used a liberal definition of the <a href="https://en.wikipedia.org/wiki/End-to-end_principle" target="\_blank">end-to-end principle</a> to
guide this design decision.
## Gaia versus other storage systems
Here's how Gaia stacks up against other decentralized storage systems. Features
that are common to all storage systems are omitted for brevity.
<table class="uk-table uk-table-striped">
<thead>
<tr>
<th>Features</th>
<th>Gaia</th>
<th><a href="https://sia.tech/" target="\_blank">Sia</a></th>
<th><a href="https://storj.io/" target="\_blank">Storj</a></th>
<th><a href="https://ipfs.io/" target="\_blank">IPFS</a></th>
<th><a href="https://datproject.org/" target="\_blank">DAT</a></th>
<th><a href="https://www.scuttlebutt.nz/" target="\_blank">SSB</a></th>
</tr>
</thead>
<tr>
<td>User controls where data is hosted</td>
<td>X</td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
</tr>
<tr>
<td>Data can be viewed in a normal Web browser</td>
<td>X</td>
<td></td>
<td></td>
<td>X</td>
<td></td>
<td></td>
</tr>
<tr>
<td>Data is read/write</td>
<td>X</td>
<td></td>
<td></td>
<td></td>
<td>X</td>
<td>X</td>
</tr>
<tr>
<td>Data can be deleted</td>
<td>X</td>
<td></td>
<td></td>
<td></td>
<td>X</td>
<td>X</td>
</tr>
<tr>
<td>Data can be listed</td>
<td>X</td>
<td>X</td>
<td>X</td>
<td></td>
<td>X</td>
<td>X</td>
</tr>
<tr>
<td>Deleted data space is reclaimed</td>
<td>X</td>
<td>X</td>
<td>X</td>
<td>X</td>
<td></td>
<td></td>
</tr>
<tr>
<td>Data lookups have predictable performance</td>
<td>X</td>
<td></td>
<td>X</td>
<td></td>
<td></td>
<td></td>
</tr>
<tr>
<td>Writes permission can be delegated</td>
<td>X</td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
</tr>
<tr>
<td>Listing permission can be delegated</td>
<td>X</td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
</tr>
<tr>
<td>Supports multiple backends natively</td>
<td>X</td>
<td></td>
<td>X</td>
<td></td>
<td></td>
<td></td>
</tr>
<tr>
<td>Data is globally addressable</td>
<td>X</td>
<td>X</td>
<td>X</td>
<td>X</td>
<td>X</td>
<td></td>
</tr>
<tr>
<td>Needs a cryptocurrency to work</td>
<td></td>
<td>X</td>
<td>X</td>
<td></td>
<td></td>
<td></td>
</tr>
<tr>
<td>Data is content-addressed</td>
<td></td>
<td>X</td>
<td>X</td>
<td>X</td>
<td>X</td>
<td>X</td>
</tr>
</table>

62
_storage/write-to-read.md

@ -0,0 +1,62 @@
---
layout: storage
permalink: /:collection/:path.html
---
# Storage write and read
Once a user authenticates and a DApp obtains authentication, the application interacts with Gaia through the blockstack.js library. There are two simple methods for working with data in Gaia hub: the `putFile()` and `getFile()` methods. This section goes into greater detail about the methods, how they interact with a hub, and how to use them.
## Write-to and Read-from URL Guarantees
Gaia is built on a driver model that supports many storage services. So, with
very few lines of code, you can interact with providers on Amazon S3, Dropbox,
and so forth. The simple `getFile` and `putFile` interfaces are kept simple
because Blockstack assumes and wants to encourage a community of
open-source-data-management libraries.
The performance and simplicity-oriented guarantee of the Gaia specification is
that when an application submits a write-to
`https://myhub.service.org/store/foo/bar` URL, the application is guaranteed to
be able to read from the `https://myreads.com/foo/bar` URL. Note that, while the
prefix in the write-to url (for example,`myhub.service.org/store`) and the read-from URL
(`https://myreads.com`) are different, the `foo/bar` suffixes are the same.
Consistent, identical suffixes allow an application to know _exactly_ where a
written file can be read from, given the read prefix. The Gaia service defines a `hub_info` endpoint to obtain that read prefix:
```
GET /hub_info/
```
The endpoint returns a JSON object with a `read_url_prefix`, for example, if my service returns:
```javascript
{ ...,
"read_url_prefix": "https://myservice.org/read/"
}
```
The data be read with this `getFile()` and this address:
```
https://myservice.org/read/1DHvWDj834zPAkwMhpXdYbCYh4PomwQfzz/0/profile.json
```
The application is guaranteed that the profile is written with `putFile()` this request address:
```
https://myservice.org/store/1DHvWDj834zPAkwMhpXdYbCYh4PomwQfzz/0/profile.json
```
When you use the `putFile()` method it takes the user data and POSTs it to the user's Gaia storage hub. The data POSTs directly to the hub, the blockchain is not used and no data is stored there. The limit on file upload is currently 25mb.
## Address-based access-control
Access control in a Gaia storage hub is performed on a per-address basis.
Writes to URLs `/store/<address>/<file>` are allowed only if the writer can
demonstrate that they control _that_ address. This is achieved via the
authentication token which is a message _signed_ by the private key associated
with that address. The message itself is a challenge text, returned via the
`/hub_info/` endpoint.

2
assets/css/main.scss

@ -47,3 +47,5 @@ body > div.uk-lightbox.uk-overflow-hidden.uk-lightbox-panel.uk-open > div.uk-lig
.lb-data .lb-details { .lb-data .lb-details {
text-align: right; text-align: right;
} }

Loading…
Cancel
Save