[doc] Update, expand help and docstrings

Author: ptaoussanis

resources/docs/environmental-config.txt

@@ -0,0 +1,55 @@
+Tufte supports extensive environmental config via JVM properties,
+environment variables, or classpath resources.
+
+Environmental filter config includes:
+
+  1. Minimum level (see signal `:level`):
+    a.       JVM property: `taoensso.tufte.rt-min-level`
+    b.       Env variable: `TAOENSSO_TUFTE_RT_MIN_LEVEL`
+    c. Classpath resource: `taoensso.tufte.rt-min-level`
+
+  2. Namespace filter (see signal `:ns`):
+    a.       JVM property: `taoensso.tufte.rt-ns-filter`
+    b.       Env variable: `TAOENSSO_TUFTE_RT_NS_FILTER`
+    c. Classpath resource: `taoensso.tufte.rt-ns-filter`
+
+  3. Id filter (see signal `:id`):
+    a.       JVM property: `taoensso.tufte.rt-id-filter`
+    b.       Env variable: `TAOENSSO_TUFTE_RT_ID_FILTER`
+    c. Classpath resource: `taoensso.tufte.rt-id-filter`
+
+Config values are parsed as edn, examples:
+
+  `taoensso.tufte.rt-min-level`      => ":info"
+  `TAOENSSO_TUFTE_RT_NS_FILTER`      => "{:disallow \"taoensso.*\"}"
+  `taoensso.tufte.rt-id-filter.cljs` => "#{:my-id1 :my-id2}"
+  `TAOENSSO_TUFTE_RT_ID_FILTER_CLJ`  => "nil"
+
+Runtime vs compile-time filters
+
+  The above filters (1..4) all apply at RUNTIME ("rt").
+  This is typically what you want, since it allows you to freely adjust filtering
+  (making it less OR MORE permissive) through later API calls like `set-min-level!`.
+
+  As an advanced option, you can instead/additionally ELIDE (entirely omit) filtered
+  callsites at COMPILE-TIME ("ct") by replacing "rt"->"ct" / "RT"->"CT" in the config
+  ids above. Compile-time filters CANNOT be made MORE permissive at runtime.
+
+Tips:
+
+  - The above config ids will affect both Clj AND Cljs.
+    For platform-specific filters, use
+      ".clj"  / "_CLJ"  or
+      ".cljs" / "_CLJS" suffixes instead.
+      e.g. "taoensso.tufte.rt-min-level.cljs".
+
+  - To get the right edn syntax, first set your runtime filters using the
+    standard utils (`set-min-level!`, etc.). Then call `get-filters` and
+    serialize the relevant parts to edn with `pr-str`.
+
+  - All environmental config uses `get-env` underneath.
+    See the `get-env` docstring for more/advanced details.
+
+  - Classpath resources are files accessible on your project's
+    classpath. This usually includes files in your project's
+    `resources/` dir.

resources/docs/filters.txt

@@ -0,0 +1,77 @@
+Tufte profiling is activated using the `profiled` or  `profile` macros:
+
+  - `profiled` returns [<body-result> <?pstats>].
+    Handy when you want to consume profiling results directly at the callsite.
+
+  - `profile`  returns <body-result> and dispatches a profiling signal (map)
+    to all registered handlers. This map includes `:pstats` and other info.
+
+    Handy when you want to consume profiling results later/elsewhere.
+
+Profiling is activated only when ALL of the following are true:
+
+    1. Call filters pass (relevant for both `profile/d`):
+      a. Compile-time: sample rate, kind, ns, id, level, when form, rate limit
+      b. Runtime:      sample rate, kind, ns, id, level, when form, rate limit
+
+    2. Handler filters pass (relevant only for `profile`):
+      a. Compile-time: not applicable
+      b. Runtime:      sample rate, kind, ns, id, level, when fn, rate limit
+
+    (Relevant only for `profile`):
+    3. Call    transform (fn [signal]) => ?modified-signal returns non-nil
+    4. Handler transform (fn [signal]) => ?modified-signal returns non-nil
+
+  Transform fns provides a flexible way to modify and/or filter signals by
+  arbitrary signal data/content conditions (return nil to skip handling).
+
+  Config:
+
+    To set call filters (1a, 1b):
+
+      Use:
+        `set-kind-filter!`, `with-kind-filter`
+        `set-ns-filter!`,   `with-ns-filter`
+        `set-id-filter!`,   `with-id-filter`
+        `set-min-level!`,   `with-min-level`
+
+      or see `help:environmental-config`.
+
+    To set handler filters (2b) or transform (4):
+
+      Provide relevant opts when calling `add-handler!` or `with-handler/+`.
+      See `help:handler-dispatch-options` for details.
+
+      Note: call filters (1a, 1b) should generally be AT LEAST as permissive
+      as handler filters (2b) since they're always applied first.
+
+    To set call transform (3): use `set-xfn!`, `with-xfn`.
+
+  Compile-time vs runtime filtering:
+
+    Compile-time filters are an advanced feature that can be tricky to set
+    and use correctly. Most folks will want ONLY runtime filters.
+
+    Compile-time filters works by eliding (completely removing the code for)
+    disallowed calls. This means zero performance cost for these calls, but
+    also means that compile-time filters are PERMANENT once applied.
+
+    So if you set `:info` as the compile-time minimum level, that'll REMOVE
+    CODE for every signal call below `:info` level. To decrease that minimum
+    level, you'll need to rebuild.
+
+    Compile-time filters can be set ONLY with environmental config
+    (see `help:environmental-config` for details).
+
+  Signal and handler sampling is multiplicative:
+
+    Both calls and handlers can have independent sample rates, and these
+    MULTIPLY! If a signal is created with 20% sampling and a handler
+    handles 50% of received signals, then 10% of possible signals will be
+    handled (50% of 20%).
+
+    When sampling is active, the final (combined multiplicative) rate is
+    helpfully reflected in each signal's `:sample` rate value ∈ℝ[0,1].
+
+If anything is unclear, please ping me (@ptaoussanis) so that I can
+improve these docs!

resources/docs/p.txt

@@ -0,0 +1,17 @@
+Profiling spy.
+
+Use this macro to wrap forms that should be timed during active profiling.
+A unique form id (keyword) must be provided, e.g.:
+
+  (p      ::my-form  (do-something))
+  (p {:id ::my-form} (do-something))
+  (p {:id ::my-form
+      :level :debug} (do-something))
+
+`p` will ALWAYS execute its body and return the body's result, even when
+profiling isn't active.
+
+Options include:
+  `:id` ---- Unique id for this form in resulting `pstats` (e.g. `::my-fn-call`)
+  `:level` - Profiling level ∈ #{<int> :trace :debug :info :warn :error :fatal :report ...}
+             (default `:info`)

resources/docs/profile.txt

@@ -0,0 +1,72 @@
+Use this to conditionally activate profiling for given body:
+
+- ALWAYS executes body and returns <body-result>.
+- When filtering conditions are met (see `help:filters`), records execution times
+  of all `p` forms in body and dispatches a profiling signal map (see `help:signal-content`)
+  to any registered handlers (see `help:handlers`).
+
+Handy when you want to consume profiling results asynchronously and/or away from
+the callsite, otherwise prefer `profile`.
+
+Options include:
+
+  `:dynamic?` --- Perform multi-threaded profiling with binding conveyance? (default false)
+  `:level` ------ Profiling level (default `:info`), must be >= active minimum level to profile
+  `:id` --------- Profiling ?id for filtering, etc. (e.g. `::my-profiling-id`)
+
+  `:data` ------- Arb app-level ?data to incl. in signal: usu. a map
+  `:ctx` -------- Custom ?val to override auto (dynamic `*ctx*`) in signal, as per `with-ctx`
+  `:ctx+` ------- Custom ?val to update   auto (dynamic `*ctx*`) in signal, as per `with-ctx+`
+  `:xfn` -------- Optional       transform (fn [signal]) => ?modified-signal to apply when signal is created, as per `with-xfn`
+  `:xfn+` ------- Optional extra transform (fn [signal]) => ?modified-signal to apply when signal is created, as per `with-xfn+`
+
+  `:sample` ----- Sample rate ∈ℝ[0,1], profile only this random proportion of calls (0.75 => 75%, nil => all)
+  `:when` ------- Arb ?form; when present, form must return truthy to allow profiling
+  `:limit` ------ Rate limit ?spec given to `taoensso.tufte/rate-limiter`, see its docstring for details
+  `:limit-by` --- When present, rate limits will be enforced independently for each value (any Clojure value!)
+
+  `:nmax` ------- Max captures per `p` id before compaction (default 8e5).
+  `:elidable?` -- Should signal be subject to compile-time elision? (default true)
+
+Laziness in body:
+
+  Lazy seqs and other forms of laziness (e.g. delays) in body will only
+  contribute to profiling results if/when EVALUATION ACTUALLY OCCURS.
+  This is intentional and a useful property. Compare:
+
+    (profile {}  (delay (Thread/sleep 2000))) ; Doesn't count sleep
+    (profile {} @(delay (Thread/sleep 2000))) ; Does    count sleep
+
+Async code in body:
+
+  Execution time of any code in body that runs asynchronously on a
+  different thread will generally NOT be automatically captured by default.
+
+  `:dynamic?` can be used to support capture in cases where Clojure's
+  binding conveyance applies (e.g. futures, agents, pmap). Just make sure
+  that all work you want to capture has COMPLETED before the `profile`
+  form ends- for example, by blocking on pending futures.
+
+  In other advanced cases (notably core.async `go` blocks), please see
+  `with-profiling` and `capture-time!`.
+
+`core.async` warning:
+
+   `core.async` code can be difficult to profile correctly without a deep
+   understanding of precisely what it's doing under-the-covers.
+
+   Some general recommendations that can help keep things simple:
+
+     - Try minimize the amount of code + logic in `go` blocks. Use `go`
+       blocks for un/parking to get the data you need, then pass the data
+       to external fns. Profile these fns (or in these fns), not in your
+       `go` blocks.
+
+     - In particular: you MUST NEVER have parking calls inside
+       `(profile {:dynamic? false} ...)`.
+
+       This can lead to concurrency exceptions.
+
+       If you must profile code within a go block, and you really want to
+       include un/parking times, use `(profile {:dynamic? true} ...)`
+       instead.

resources/docs/profiled.txt

@@ -0,0 +1,65 @@
+Use this to conditionally activate profiling for given body:
+
+- ALWAYS executes body and returns [<body-result> <?pstats>].
+- When filtering conditions are met (see `help:filters`), records execution times
+  of all `p` forms in body and includes resulting `pstats` object in return value.
+
+Handy when you want to consume profiling results directly at the callsite,
+otherwise prefer `profile`.
+
+Options include:
+
+  `:dynamic?` --- Perform multi-threaded profiling with binding conveyance? (default false)
+  `:level` ------ Profiling level (default `:info`), must be >= active minimum level to profile
+  `:id` --------- Profiling ?id for filtering, etc. (e.g. `::my-profiling-id`)
+
+  `:sample` ----- Sample rate ∈ℝ[0,1], profile only this random proportion of calls (0.75 => 75%, nil => all)
+  `:when` ------- Arb ?form; when present, form must return truthy to allow profiling
+  `:limit` ------ Rate limit ?spec given to `taoensso.tufte/rate-limiter`, see its docstring for details
+  `:limit-by` --- When present, rate limits will be enforced independently for each value (any Clojure value!)
+
+  `:nmax` ------- Max captures per `p` id before compaction (default 8e5).
+  `:elidable?` -- Should profiling be subject to compile-time elision? (default true)
+
+Laziness in body:
+
+  Lazy seqs and other forms of laziness (e.g. delays) in body will only
+  contribute to profiling results if/when EVALUATION ACTUALLY OCCURS.
+  This is intentional and a useful property. Compare:
+
+    (profiled {}  (delay (Thread/sleep 2000))) ; Doesn't count sleep
+    (profiled {} @(delay (Thread/sleep 2000))) ; Does    count sleep
+
+Async code in body:
+
+  Execution time of any code in body that runs asynchronously on a
+  different thread will generally NOT be automatically captured by default.
+
+  `:dynamic?` can be used to support capture in cases where Clojure's
+  binding conveyance applies (e.g. futures, agents, pmap). Just make sure
+  that all work you want to capture has COMPLETED before the `profiled`
+  form ends- for example, by blocking on pending futures.
+
+  In other advanced cases (notably core.async `go` blocks), please see
+  `with-profiling` and `capture-time!`.
+
+`core.async` warning:
+
+   `core.async` code can be difficult to profile correctly without a deep
+   understanding of precisely what it's doing under-the-covers.
+
+   Some general recommendations that can help keep things simple:
+
+     - Try minimize the amount of code + logic in `go` blocks. Use `go`
+       blocks for un/parking to get the data you need, then pass the data
+       to external fns. Profile these fns (or in these fns), not in your
+       `go` blocks.
+
+     - In particular: you MUST NEVER have parking calls inside
+       `(profiled {:dynamic? false} ...)`.
+
+       This can lead to concurrency exceptions.
+
+       If you must profile code within a go block, and you really want to
+       include un/parking times, use `(profiled {:dynamic? true} ...)`
+       instead.

resources/docs/pstats-content.txt

@@ -0,0 +1,15 @@
+Profiling stats (`pstats`)
+
+When profiling (activated by `profile/d`) is complete, you'll get back a
+profiling stats (`pstats`) object that can be derefed and/or merged
+with other `pstats`:
+
+  - @pstats                 => {:clock {:keys [t0 t1 total]}, :stats {<id> {:keys [n sum ...]}}}
+  - @(merge-pstats ps1 ps2) => {:clock {:keys [t0 t1 total]}, :stats {<id> {:keys [n sum ...]}}}
+
+Full set of keys in the above `:stats` maps:
+  :n :min :max :mean :mad :sum :p25 :p50 :p75 :p90 :p95 :p99 :loc :last
+
+  All values are numerical (longs or doubles), except for `:loc` which is
+  a map of `p` callsite location information, or set of such maps, e.g.:
+  #{{:ns \"my-ns\", :line 122, :column 21}}

resources/docs/signal-content.txt

@@ -0,0 +1,28 @@
+Tufte profiling signals are maps with {:keys [inst id ns level data ...]},
+though they can be modified by call and/or handler transforms (xfns).
+
+Default signal keys:
+
+`:schema` ----------- Int version of signal schema (current: 1)
+`:inst` ------------- Platform instant [1] when `profile` called, monotonicity depends on system clock
+`:ns` --------------- ?str namespace of `profile` callsite
+`:coords` ----------- ?[line column] of `profile` callsite
+
+`:level` ------------ Profiling level ∈ #{<int> :trace :debug :info :warn :error :fatal :report ...}
+`:id` --------------- Profiling ?id (usu. keyword)
+
+`:data` ------------- Arb app-level data ?val (usu. a map) given to `profile` call
+`:ctx` -------------- ?val of `*ctx*` (arb app-level state) when `profile` was called
+
+`:body-result` ------ Return value of the body wrapped by `profile` call
+`:pstats` ----------- Profiling stats object that can be derefed and/or merged (see `help:pstats-content`)
+`:format-pstats-fn` - Cached (fn [pstats]) => formatted table string function
+
+`:host` ------------- (Clj only) {:keys [name ip]}       info for network host
+`:thread` ----------- (Clj only) {:keys [name id group]} info for thread that called `profile`
+
+`:sample` ----------- Sample ?rate ∈ℝ[0,1] for combined call AND handler sampling (0.75 => allow 75% of signals, nil => allow all)
+
+If anything is unclear, please ping me (@ptaoussanis) so that I can improve these docs!
+
+[1] `java.time.Instant` or `js/Date`