content + logic

Author: jagracey

.gitignore

@@ -18,3 +18,6 @@
 npm-debug.log*
 yarn-debug.log*
 yarn-error.log*
+
+# Local Netlify folder
+.netlify

docs/censorship/censorship.md

@@ -16,7 +16,7 @@ author: 'John Gracey'
 public: True
 ---
 
-Censorship is a very important concept to understand. There are two methods to limiting what data Wisdom collects available to website owners. We recommend using HTML Configuration (the first method) as much as possible. You can also enable and disable certain Wisdom [recording modules](/censorship--recording-modules) like cookie tracking and mouse tracking.
+Censorship is a very important concept to understand. There are two methods to limiting what data Wisdom collects available to website owners. We recommend using HTML Configuration (the first method) as much as possible. You can also enable and disable certain Wisdom [recording modules](/censorship/recording-modules) like cookie tracking and mouse tracking.
 
 <div class='spacer32'></div>
 
@@ -65,7 +65,7 @@ Password visibility toggles may cause Wisdom to treat password fields as normall
 ### 2. Tracking Configurations Page
 On the [Tracking Configuration](https://app.getwisdom.io/org/~/tracking) page you can specify CSS selectors to censor HTML elements, including input fields.
 
-Beyond censoring HTML, you can also enable and disable certain Wisdom "recording modules", such as Mouse movement tracking. Please see the list of [recording modules](/censorship--recording-modules) for their impact on privacy.
+Beyond censoring HTML, you can also enable and disable certain Wisdom "recording modules", such as Mouse movement tracking. Please see the list of [recording modules](recording-modules) for their impact on privacy.
 
 
 

docs/censorship/overview.md

@@ -5,7 +5,7 @@ sidebar_label: Overview
 
 ### Censorship
 
-Censorship is a very important concept to understand. There are two methods to limiting what data Wisdom collects available to website owners. We recommend using HTML Configuration (the first method) as much as possible. You can also enable and disable certain Wisdom [recording modules](/js-client-api/recording-modules/README.md) like cookie tracking and mouse tracking.
+Censorship is a very important concept to understand. There are two methods to limiting what data Wisdom collects available to website owners. We recommend using HTML Configuration (the first method) as much as possible. You can also enable and disable certain Wisdom [recording modules](recording-modules) like cookie tracking and mouse tracking.
 
 <div class='spacer32'></div>
 
@@ -53,7 +53,7 @@ Password visibility toggles may cause Wisdom to treat password fields as normall
 ### 2. Tracking Configurations Page
 On the [Tracking Configuration](https://app.getwisdom.io/org/~/tracking) page you can specify CSS selectors to censor HTML elements, including input fields.
 
-Beyond censoring HTML, you can also enable and disable certain Wisdom "recording modules", such as Mouse movement tracking. Please see the list of [recording modules](js-client-api/recording-modules/README.md) for their impact on privacy.
+Beyond censoring HTML, you can also enable and disable certain Wisdom "recording modules", such as Mouse movement tracking. Please see the list of [recording modules](recording-modules) for their impact on privacy.
 
 
 

docs/data-access/cube-js.md

@@ -9,6 +9,8 @@ import TabItem from '@theme/TabItem';
 import Code from '../../src/common/code/'
 
 
+## Intro
+
 <Image img={require('./cubejs.png')} className='noShadow'/>
 
 [Visit Cubejs](https://cube.dev/)
@@ -17,20 +19,21 @@ import Code from '../../src/common/code/'
 Cubejs is an open source analytics framework that greatly simplifies running analytics queries. Cubejs comprises of a frontend library and backend analytical server infrastructure that Wisdom fully manages for you. Like many modern analytics frameworks, Cubejs differentiates between data dimensions (time, browser) and data measures (user count, unique page views).
 
 
-### Cubes
-
-atomic.person
-atomic.session
-atomic.events
-atomic.mail_inbox
-
-virtual.pageviews
-
+## Cubes
 
-rollup
+| Cube Name  | Cube Description   |
+|---|---|
+| atomic.person |  |
+| atomic.session |  |
+| atomic.events |  |
+| atomic.mail_inbox |  |
+| |  |
+| virtual.pageviews |  |
+| |  |
+| rollup |  |
 
 
-### Example Code
+## Example Code
 
 
 <Tabs

docs/data-access/management.md

@@ -5,4 +5,30 @@ sidebar_label: Lifecycle Management:preview
 
 
 
-hello world
+### Data Retention
+
+Wisdom utilizes pass-through (cost plus) pricing ontop of AWS infrastructure allowing our customers to fine-tune their storage requirements- and in many cases receive substantial cost savings.
+
+1. Duration: Increase Data Retention from zero days to an indefinitely timeline (billing per gigabyte/hour).
+2. Durability: Reducing data replication factor to cut cost by ~20%, reducing data availability and data safety in the case of data center destruction.
+
+
+| Type                                  | Description                      |
+|---------------------------------------|----------------------------------|
+| S3 Objects: Recorded Sessions         |                                  |
+| S3 Objects: (everything else)         |                                  |
+| SQL Tables: <br/> - atomic.sessions <br/> - atomic.events table|         |
+| SQL Tables: (everything else)         |                                  |
+
+
+### Data Durability
+
+| Type                 | Description   |
+|----------------------|---------------|
+| S3 Recorded Sessions |  Configurable |
+| PSQL                 |  Configurable |
+
+
+## Data Deprecation
+
+Transfering out old SQL data to S3 on a weekly basis for data continuity multiple years out.

docs/data-access/overview.md

@@ -1,4 +1,19 @@
 ---
-title: Overview
+title: Data Access Overview
 sidebar_label: Overview
 ---
+
+
+
+
+Wisdom is your modern web analytics data lake, home to an exciting community of SaaS product builders who share and contribute custom reports. This is what makes Wisdom such a compelling solution- that Wisdom allows customers to directly access and query their SQL data from their own provisioned PostgreSQL database, and their own AWS S3 bucket.
+
+Here we discuss how to access and manage this data.
+
+| Section                     |
+|-----------------------------|
+| [Data Lifecycle Management](/data-access/management) |
+| [Object Storage (AWS S3)](/data-access/blob) |
+| [SQL Tables Schemas](/data-access/sql-tables/overview) |
+| [HTTP REST API: Overview](/data-access/api/overview) |
+| [Ecosystem Overview](/ecosystem/overview) |

docs/data-access/sql-reports/@wisdom_developing_all-together.md

@@ -7,8 +7,8 @@ sidebar_label: '@wisdom/developing/all-together'
 ## Example Usage
 
 ```sql
-SELECT * FROM community.report(NULL::"@wisdom/developing/all-together", 
-	_project_id = ?
+SELECT * FROM community.report(NULL::community_types."@wisdom/developing/all-together", 
+	_project_id => ?
 );
 ```
 

docs/data-access/sql-reports/@wisdom_developing_building-on-other-queries.md

@@ -7,7 +7,7 @@ sidebar_label: '@wisdom/developing/building-on-other-queries'
 ## Example Usage
 
 ```sql
-SELECT * FROM community.report(NULL::"@wisdom/developing/building-on-other-queries");
+SELECT * FROM community.report(NULL::community_types."@wisdom/developing/building-on-other-queries");
 ```
 
 

docs/data-access/sql-reports/@wisdom_developing_hello-world.md

@@ -7,7 +7,7 @@ sidebar_label: '@wisdom/developing/hello-world'
 ## Example Usage
 
 ```sql
-SELECT * FROM community.report(NULL::"@wisdom/developing/hello-world");
+SELECT * FROM community.report(NULL::community_types."@wisdom/developing/hello-world");
 ```
 
 

docs/data-access/sql-reports/@wisdom_developing_passing-parameters.md

@@ -7,8 +7,8 @@ sidebar_label: '@wisdom/developing/passing-parameters'
 ## Example Usage
 
 ```sql
-SELECT * FROM community.report(NULL::"@wisdom/developing/passing-parameters", 
-	_project_id = ?
+SELECT * FROM community.report(NULL::community_types."@wisdom/developing/passing-parameters", 
+	_project_id => ?
 );
 ```
 

docs/data-access/sql-reports/overview.md

@@ -3,24 +3,28 @@ title: Community SQL Reports Overview
 sidebar_label: Overview
 ---
 
-## Running Community Queries
+## About
 
-A Few Sample Queries to get you started.
+Like most things in life, someone's already thought of it, and very likely already done it. Just try registering a `.com` domain name or registering a username on Reddit. With Relational Databases being around since the 1970s thanks to IBM's Edgar Codd, analytics and reporting ideas have been around for over 50 years. Building on this body of knowledge, we've put together a SQL library built by the Wisdom community to easily share analytics reports- collaboration taking place from [Github](https://github.com/Wisdom/community-sql).
 
+<br/>
 
+Getting started is easy:
+
+```sql
+-- 1. Update the Library stored in your PostgreSQL database:
+SELECT community.update_reports();
+
+-- 2. Pick a query from the library and run it.
+SELECT * FROM community.report(NULL::community_types."@wisdom/developing/hello-world");
+```
 
-### A Safe, Simple and Collaborative system.
-From an open community, reports are created 
 
+### How it Works
+When the `community.report()` function is run, the SQL code of the desired report is looked-up within the `community.queries` table- and this table is updated with the `community.update_reports()` function that pulls in queries from the [Github Repo](https://github.com/Wisdom/community-sql). On Github, SQL reports are defined with two files: 1) the SQL code stored in `query.sql`, 2) Metadata stored in a `manifest.json` file. Within the manifest file all mandatory SQL query parameters are defined. 
 
-- Only Variable literal values (single quote vars). Passing idents are not supported (double quote variables).
-- Function is executed as current user?????
-- During a `report_update` function call, all views within the `community_types` schema will be deleted, cascading to all dependents, and then replaced. Therefore, if you are building queries ontop of community queries, they too should be created as a community query to ensure they are always updated and not deleted.
-- Undercovers, PLPSQL uses `EXECUTE` from a `format()`. Var markers are only injected if they are non-null.
+You may note the strange syntax when running a community query (eg. `null::community_types."@example/test"). PostgreSQL's procedural language ("PL/pgSQL") has a strongly-typed type system that requires all queries have a predefined return type. This strange syntax converts null to the proper return type of the SQL query, ultimately defined within the manifest.json file.
 
-SQL:
-    format('SELECT $$_session_id$$ LIMIT 5;', 
-    REPLACE(__sql, '$$_session_id$$', _session_id)
 
 
 A Few Example Queries to get you started developing
@@ -31,55 +35,6 @@ A Few Example Queries to get you started developing
 
 | Report Query Path                                  | Title                     | Requires Params |
 |----------------------------------------------------|---------------------------|-----------------|
-| [@wisdom/developing/all-together](./@wisdom/developing/all-together) | HelloworldQuery           | ✓               |
-| [@wisdom/developing/building-on-other-queries](./@wisdom/developing/building-on-other-queries) | HelloworldQuery           | ✘               |
-| [@wisdom/developing/hello-world](./@wisdom/developing/hello-world) | HelloWorldQuery           | ✘               |
-| [@wisdom/developing/passing-parameters](./@wisdom/developing/passing-parameters) | HelloworldQuery           | ✓               |
-
-
-
-
-
-CORE:
-    - list all pages.
-    - get users with first + last visit time + activity trend???
-
-@wisdom/components/page-engagement`
-
-
-PERIOD_SUM
-PERIOD TREND:
-    Activity Time (avg, median, stddev)
-    Page Views
-    Proportional
-
-Questions To ask:
-    - Do users know this feature exists.
-            -- Have they visited, AND interacted on it?
-    - Do users like this feature?
-            -- 
-    
-
-    -- When do users interact with features?  1. Time to Activation overlayed with 2. Median activity time per user
-        (cohort by calendar date (days), or activeTime sum (15 mins))
-
-FEATURE SUCCESS:
-    - number of users as percent of app total? (shift delay days???)
-    - median activity time.
-    - 
-
-Page Engagement
-Page Engagement by Person:
-    TREND
-
-Histogram cohorted?
-
-
-
-
-The query will not be executed if two dollar signs exist in the query: `$$`, indicating that a param has not been substituted.
-
-
 
 
 
@@ -114,16 +69,20 @@ A few example community queries to help you get started developing SQL reports.
 ## Semantic Requirements And Conventions
 
 When namimg files and folders, keep in mind:
-- Root directory folders must start with '@', as Github name handle.
-- Files and folders must be lowercased, and only allow a-z, 0-9 and dash.
+- Root directory folders must start with '@', as your own Github name handle- so everyone knows who the author/owner is.
+- Files and folders must be lowercased, and only allow the following characters: a-z, 0-9 and dash.
 - Three files are required per report: query.sql, manifest.json, and README.md.
+- To contribute, submit a pull request and accept the MIT contributor license assignment.
 
 
-### Strict Type Checking
--- views created from JSON type
 
+### A Safe, Simple and Collaborative system.
+From an open community, reports are created 
 
 
+- Only literal value variables (single quote variables) are possible. Passing idents are not supported (double quote variables).
+- The community report is executed as the current user, with all permissions they have.
+- During a `community.report_update` function call, all views within the `community_types` schema will be deleted, cascading to all dependents, and then replaced. Therefore, if you are building queries ontop of community queries, they too should be created as a community query to ensure they are always updated and not deleted.
+- Undercovers, PLPSQL uses `EXECUTE` from a `format()`. Variables are only injected if they are non-null.
+- You can access variables from the $1 JSONB parameter with type casting:  `($1->>'_project_id')::INTEGER`
 
-PR Process:
-    - must accept contributor license MIT contributor guide as part of a PR checklist

docs/data-access/sql-tables/atomic.session.md

@@ -76,6 +76,7 @@ Column          |           Type           | Nullable | Default | Storage  |
 `"wisdomVersion"`           | character varying(20)    |          |         | extended | 
 `"pageCreatedAt"`           | timestamp with time zone |          |         | plain    | 
 `"softDeletedAt"`           | timestamp with time zone |          |         | plain    | 
+s3Bytes                 | integer                  |          |         | plain    | 
 ### Indexes:
 ```
 "Session_pkey" PRIMARY KEY, btree ("projectId", "sessionStart", "sessionId")

docs/data-access/sql-tables/overview.md

@@ -86,19 +86,19 @@ select nspname from pg_catalog.pg_namespace;
 
 | Schemas                        |
 |--------------------------------|
-| atomic                         |
-| rollup                         |
-| team                           |
-| enrichment                     |
-| trained                        |
+| admin                          |
 | alpha                          |
-| beta                           |
-| normalized_vendor              |
 | app                            |
-| sandbox                        |
-| admin                          |
+| atomic                         |
+| beta                           |
 | community                      |
 | community_types                |
+| enrichment                     |
+| normalized_vendor              |
+| rollup                         |
+| sandbox                        |
+| team                           |
+| trained                        |
 
 
 
@@ -107,17 +107,12 @@ select nspname from pg_catalog.pg_namespace;
 
 | Tables with Schema             |
 |--------------------------------|
-| [alpha.aggregate_mouse_position](./alpha.aggregate_mouse_position.md) |
-| [alpha.aggregate_scroll_abandonment](./alpha.aggregate_scroll_abandonment.md) |
-| [alpha.aggregate_scroll_position](./alpha.aggregate_scroll_position.md) |
-| [alpha.url_category](./alpha.url_category.md) |
 | [atomic.company](./atomic.company.md) |
 | [atomic.employee](./atomic.employee.md) |
 | [atomic.event](./atomic.event.md) |
 | [atomic.mail_inbox](./atomic.mail_inbox.md) |
 | [atomic.person](./atomic.person.md) |
 | [atomic.session](./atomic.session.md) |
-| [community.queries](./community.queries.md) |
 | [enrichment.domain_rank](./enrichment.domain_rank.md) |
 | [enrichment.person_enrichment](./enrichment.person_enrichment.md) |
 | [enrichment.spam_email_service](./enrichment.spam_email_service.md) |
@@ -135,7 +130,7 @@ select nspname from pg_catalog.pg_namespace;
 
 
 ### Data Syncing SaaS Vendors
-Note: Requires setup of vendor secret API keys (auth tokens). Since this functionality is realtively speaking new, you should reach out to your assigned Wisdom account manager to discuess getting started with normalized vendor data.
+Note: Requires setup of vendor secret API keys (auth tokens). Since this functionality is relatively new, you should reach out to your assigned Wisdom account manager to discuss getting started with normalized vendor data.
 
 | Category | Vendor Name         | SQL Schema                 
 |----------|---------------------|----------------------------
@@ -164,7 +159,7 @@ Note: Requires setup of vendor secret API keys (auth tokens). Since this functio
 
 
 
-### Reserved for administrative and future use
+### Tables Reserved for Administrative and for Future Use
 
    Schema   |               Name               | Note |
 ------------|----------------------------------|------|
@@ -183,11 +178,11 @@ Note: Requires setup of vendor secret API keys (auth tokens). Since this functio
 
 #### Triggers
 
-1. On record insert into table `person`, a trigger will broadcast a notification identified as `project_${projectId}`. This will help you implement real-time data processing within your application, for example, new user signup notifications.
+On record insert into table `person`, a trigger will broadcast a notification identified as `project_${projectId}`. This will help you implement real-time data processing within your application, for example, new user signup notifications.
 
 #### Functions
 
-2. NOTE: See [Community Reports](#) for how to use and run community report functions.
+NOTE: See [Community Reports](#) for how to use and run community report functions.
    - `community.report();`
    - `community.update_reports();`