Module hb_cache.erl¶
A cache of AO-Core protocol messages and compute results.
Description¶
HyperBEAM stores all paths in key value stores, abstracted by the hb_store
module. Each store has its own storage backend, but each works with simple
key-value pairs. Each store can write binary keys at paths, and link between
paths.
There are three layers to HyperBEAMs internal data representation on-disk:
- The raw binary data, written to the store at the hash of the content. Storing binary paths in this way effectively deduplicates the data.
- The hashpath-graph of all content, stored as a set of links between hashpaths, their keys, and the data that underlies them. This allows all messages to share the same hashpath space, such that all requests from users additively fill-in the hashpath space, minimizing duplicated compute.
- Messages, referrable by their IDs (committed or uncommitted). These are stored as a set of links commitment IDs and the uncommitted message.
Before writing a message to the store, we convert it to Type-Annotated Binary Messages (TABMs), such that each of the keys in the message is either a map or a direct binary.
Nested keys are lazily loaded from the stores, such that large deeply
nested messages where only a small part of the data is actually used are
not loaded into memory unnecessarily. In order to ensure that a message is
loaded from the cache after a read, we can use the ensure_loaded/1 and
ensure_all_loaded/1 functions. Ensure loaded will load the exact value
that has been requested, while ensure all loaded will load the entire
structure of the message into memory.
Lazily loadable links are expressed as a tuple of the following form:
{link, ID, LinkOpts}, where ID is the path to the data in the store,
and LinkOpts is a map of suggested options to use when loading the data.
In particular, this module ensures to stash the store option in LinkOpts,
such that the read function can use the correct store without having to
search unnecessarily. By providing an Opts argument to ensure_loaded or
ensure_all_loaded, the caller can specify additional options to use when
loading the data -- overriding the suggested options in the link.
Function Index¶
| cache_suite_test_/0* | |
| calculate_all_ids/2* | Calculate the IDs for a message. |
| commitment_path/2* | Generate the commitment path for a given base path. |
| do_write_message/3* | |
| ensure_all_loaded/1 | Ensure that all of the components of a message (whether a map, list, or immediate value) are recursively fully loaded from the stores into memory. |
| ensure_all_loaded/2 | |
| ensure_loaded/1 | Ensure that a value is loaded from the cache if it is an ID or a link. |
| ensure_loaded/2 | |
| link/3 | Make a link from one path to another in the store. |
| list/2 | List all items under a given path. |
| list_numbered/2 | List all items in a directory, assuming they are numbered. |
| prepare_commitments/2* | The structured@1.0 encoder does not typically encode commitments,
subsequently, when we encounter a commitments message we prepare its contents
separately, then write each to the store. |
| prepare_links/4* | Prepare a set of links from a listing of subpaths. |
| read/2 | Read the message at a path. |
| read_ao_types/4* | Read and parse the ao-types for a given path if it is in the supplied list of subpaths, returning a map of keys and their types. |
| read_resolved/3 | Read the output of a prior computation, given Msg1, Msg2, and some options. |
| run_test/0* | |
| store_read/3* | List all of the subpaths of a given path and return a map of keys and links to the subpaths, including their types. |
| test_deeply_nested_complex_message/1* | Test deeply nested item storage and retrieval. |
| test_device_map_cannot_be_written_test/0* | Test that message whose device is #{} cannot be written. |
| test_message_with_list/1* | |
| test_signed/1 | |
| test_signed/2* | |
| test_store_ans104_message/1* | |
| test_store_binary/1* | |
| test_store_simple_signed_message/1* | Test storing and retrieving a simple unsigned item. |
| test_store_simple_unsigned_message/1* | Test storing and retrieving a simple unsigned item. |
| test_store_unsigned_empty_message/1* | |
| test_store_unsigned_nested_empty_message/1* | |
| test_unsigned/1 | |
| to_integer/1* | |
| types_to_implicit/1* | Convert a map of ao-types to an implicit map of types. |
| write/2 | Write a message to the cache. |
| write_binary/3 | Write a raw binary keys into the store and link it at a given hashpath. |
| write_binary/4* | |
| write_hashpath/2 | Write a hashpath and its message to the store and link it. |
| write_hashpath/3* | |
| write_key/6* | Write a single key for a message into the store. |
Function Details¶
cache_suite_test_/0 *¶
cache_suite_test_() -> any()
calculate_all_ids/2 *¶
calculate_all_ids(Bin, Opts) -> any()
Calculate the IDs for a message.
commitment_path/2 *¶
commitment_path(Base, Opts) -> any()
Generate the commitment path for a given base path.
do_write_message/3 *¶
do_write_message(Bin, Store, Opts) -> any()
ensure_all_loaded/1¶
ensure_all_loaded(Msg) -> any()
Ensure that all of the components of a message (whether a map, list, or immediate value) are recursively fully loaded from the stores into memory. This is a catch-all function that is useful in situations where ensuring a message contains no links is important, but it carries potentially extreme performance costs.
ensure_all_loaded/2¶
ensure_all_loaded(Link, Opts) -> any()
ensure_loaded/1¶
ensure_loaded(Msg) -> any()
Ensure that a value is loaded from the cache if it is an ID or a link.
If it is not loadable we raise an error. If the value is a message, we will
load only the first layer of it: Representing all nested messages inside
the result as links. If the value has an associated type key in the extra
options, we apply it to the read value, 'lazily' recreating a structured@1.0
form.
ensure_loaded/2¶
ensure_loaded(Lk, RawOpts) -> any()
link/3¶
link(Existing, New, Opts) -> any()
Make a link from one path to another in the store.
Note: Argument order is link(Src, Dst, Opts).
list/2¶
list(Path, Opts) -> any()
List all items under a given path.
list_numbered/2¶
list_numbered(Path, Opts) -> any()
List all items in a directory, assuming they are numbered.
prepare_commitments/2 *¶
prepare_commitments(RawCommitments, Opts) -> any()
The structured@1.0 encoder does not typically encode commitments,
subsequently, when we encounter a commitments message we prepare its contents
separately, then write each to the store.
prepare_links/4 *¶
prepare_links(RootPath, Subpaths, Store, Opts) -> any()
Prepare a set of links from a listing of subpaths.
read/2¶
read(Path, Opts) -> any()
Read the message at a path. Returns in structured@1.0 format: Either a
richly typed map or a direct binary.
read_ao_types/4 *¶
read_ao_types(Path, Subpaths, Store, Opts) -> any()
Read and parse the ao-types for a given path if it is in the supplied list of subpaths, returning a map of keys and their types.
read_resolved/3¶
read_resolved(MsgID1, MsgID2, Opts) -> any()
Read the output of a prior computation, given Msg1, Msg2, and some options.
run_test/0 *¶
run_test() -> any()
store_read/3 *¶
store_read(Path, Store, Opts) -> any()
List all of the subpaths of a given path and return a map of keys and links to the subpaths, including their types.
test_deeply_nested_complex_message/1 *¶
test_deeply_nested_complex_message(Store) -> any()
Test deeply nested item storage and retrieval
test_device_map_cannot_be_written_test/0 *¶
test_device_map_cannot_be_written_test() -> any()
Test that message whose device is #{} cannot be written. If it were to
be written, it would cause an infinite loop.
test_message_with_list/1 *¶
test_message_with_list(Store) -> any()
test_signed/1¶
test_signed(Data) -> any()
test_signed/2 *¶
test_signed(Data, Wallet) -> any()
test_store_ans104_message/1 *¶
test_store_ans104_message(Store) -> any()
test_store_binary/1 *¶
test_store_binary(Store) -> any()
test_store_simple_signed_message/1 *¶
test_store_simple_signed_message(Store) -> any()
Test storing and retrieving a simple unsigned item
test_store_simple_unsigned_message/1 *¶
test_store_simple_unsigned_message(Store) -> any()
Test storing and retrieving a simple unsigned item
test_store_unsigned_empty_message/1 *¶
test_store_unsigned_empty_message(Store) -> any()
test_store_unsigned_nested_empty_message/1 *¶
test_store_unsigned_nested_empty_message(Store) -> any()
test_unsigned/1¶
test_unsigned(Data) -> any()
to_integer/1 *¶
to_integer(Value) -> any()
types_to_implicit/1 *¶
types_to_implicit(Types) -> any()
Convert a map of ao-types to an implicit map of types.
write/2¶
write(RawMsg, Opts) -> any()
Write a message to the cache. For raw binaries, we write the data at
the hashpath of the data (by default the SHA2-256 hash of the data). We link
the unattended ID's hashpath for the keys (including /commitments) on the
message to the underlying data and recurse. We then link each commitment ID
to the uncommitted message, such that any of the committed or uncommitted IDs
can be read, and once in memory all of the commitments are available. For
deep messages, the commitments will also be read, such that the ID of the
outer message (which does not include its commitments) will be built upon
the commitments of the inner messages. We do not, however, store the IDs from
commitments on signed inner messages. We may wish to revisit this.
write_binary/3¶
write_binary(Hashpath, Bin, Opts) -> any()
Write a raw binary keys into the store and link it at a given hashpath.
write_binary/4 *¶
write_binary(Hashpath, Bin, Store, Opts) -> any()
write_hashpath/2¶
write_hashpath(Msg, Opts) -> any()
Write a hashpath and its message to the store and link it.
write_hashpath/3 *¶
write_hashpath(HP, Msg, Opts) -> any()
write_key/6 *¶
write_key(Base, Key, HPAlg, RawCommitments, Store, Opts) -> any()
Write a single key for a message into the store.