clerify documentation for direct-read-namespaces

[eirik-andersen]

Below warning is from the documentation for direct-read-namespaces. It's not entirely clear to me in what order things happen.
https://rwynn.github.io/monstache-site/config/#direct-read-namespaces

I understand that the direct reads and processing of change events start at the same time.
But if I'm listening on change events for 2 collections how is the order of operations?

will it start direct reads and change events on the same time and then wait for the direct reads to finish before applying the change events or will it start applying the change events before the direct reads are finished.

Warning

When direct reads are enabled Monstache still processes change events while the direct reads are being performed.
It does not wait until direct reads are completed to start listening for changes. This is to ensure that any changes that occur during the direct read process get synchronized.

If using direct-read on a large collection. how is changes to the collection during the direct-read handled to ensure consistent data in the elastic index?

[rwynn]

Monstache uses document versioning (external) to maintain consistency.

https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-index_.html#index-versioning

The version number used is the based on the clusterTime field from a change document and the timestamp the document is received by monstache for direct reads. Since mongodb determines the cluster time and monstache determines the direct read timestamp this is susceptible to clock skew when both methods are involved. When only change streams are at play then all timestamps come from MongoDB.

If monstache sends documents out of logical order the stale documents should be rejected. Update should only be applied when monstache sends a version greater than or equal to the existing version. In this way monstache does not try to determine order of operations, but simply send the data with a version based on a timestamp and allow Elasticsearch to maintain the consistency.

This is the default behavior, but if you enable the setting index-as-update then monstache cannot send a version number in attempt to ensure this consistency.

https://rwynn.github.io/monstache-site/config/#index-as-update