2.4 release notes, changelog, upgrade guide (#4658) (#4671)

* update changelog, release notes, and upgrade guide for 2.4

* tweak release notes a little

* make path relative for kafka config

* update upgrade guide

* more docs
fix the cache size not applied to results cache.

* fix prefix name in ring_config

(cherry picked from commit 3fef08c)

Co-authored-by: Ed Welch <[email protected]>

Author: grafanabot

CHANGELOG.md

@@ -951,7 +951,6 @@ Displays a web page with the ruler hash ring status, including the state, health
 
 ### List rule groups
 
-<span style="background-color:#f3f973;">This experimental endpoint is disabled by default and can be enabled via the -experimental.ruler.enable-api CLI flag or the YAML config option.</span>
 
 ```
 GET /loki/api/v1/rules
@@ -999,8 +998,6 @@ List all rules configured for the authenticated tenant. This endpoint returns a
 
 ### Get rule groups by namespace
 
-<span style="background-color:#f3f973;">This experimental endpoint is disabled by default and can be enabled via the -experimental.ruler.enable-api CLI flag or the YAML config option.</span>
-
 ```
 GET /loki/api/v1/rules/{namespace}
 ```
@@ -1024,8 +1021,6 @@ rules:
 
 ### Get rule group
 
-<span style="background-color:#f3f973;">This experimental endpoint is disabled by default and can be enabled via the -experimental.ruler.enable-api CLI flag or the YAML config option.</span>
-
 ```
 GET /loki/api/v1/rules/{namespace}/{groupName}
 ```
@@ -1034,8 +1029,6 @@ Returns the rule group matching the request namespace and group name.
 
 ### Set rule group
 
-<span style="background-color:#f3f973;">This experimental endpoint is disabled by default and can be enabled via the -experimental.ruler.enable-api CLI flag or the YAML config option.</span>
-
 ```
 POST /loki/api/v1/rules/{namespace}
 ```
@@ -1073,8 +1066,6 @@ Deletes a rule group by namespace and group name. This endpoints returns `202` o
 
 ### Delete namespace
 
-<span style="background-color:#f3f973;">This experimental endpoint is disabled by default and can be enabled via the -experimental.ruler.enable-api CLI flag or the YAML config option.</span>
-
 ```
 DELETE /loki/api/v1/rules/{namespace}
 ```
@@ -1083,8 +1074,6 @@ Deletes all the rule groups in a namespace (including the namespace itself). Thi
 
 ### List rules
 
-<span style="background-color:#f3f973;">This experimental endpoint is disabled by default and can be enabled via the -experimental.ruler.enable-api CLI flag or the YAML config option.</span>
-
 ```
 GET /prometheus/api/v1/rules
 ```
@@ -1095,8 +1084,6 @@ For more information, refer to the [Prometheus rules](https://prometheus.io/docs
 
 ### List alerts
 
-<span style="background-color:#f3f973;">This experimental endpoint is disabled by default and can be enabled via the -experimental.ruler.enable-api CLI flag or the YAML config option.</span>
-
 ```
 GET /prometheus/api/v1/alerts
 ```

docs/sources/api/_index.md

@@ -301,7 +301,7 @@ scrape_configs:
 ```
 
 Only the `brokers` and `topics` is required.
-see the [configuration](https://grafana.com/docs/loki/latest/clients/promtail/configuration/#kafka) section for more information.
+see the [configuration](../../configuration/#kafka) section for more information.
 
 ## Relabeling
 

docs/sources/clients/promtail/scraping.md

@@ -1842,8 +1842,6 @@ chunks:
 The `compactor_config` block configures the compactor component. This component periodically
 compacts index shards to more performant forms.
 
-<span style="background-color:#f3f973;">Retention through the Compactor is experimental.</span>
-
 ```yaml
 # Directory where files can be downloaded for compaction.
 # CLI flag: -boltdb.shipper.compactor.working-directory
@@ -2364,7 +2362,7 @@ kvstore:
 
   # The prefix for the keys in the store. Should end with a /.
   # CLI flag: -<prefix>.prefix
-  [prefix: <string> | default = "schedulers/"]
+  [prefix: <string> | default = "collectors/"]
 
   # The consul_config configures the consul client.
   [consul: <consul_config>]

docs/sources/configuration/_index.md

@@ -5,8 +5,6 @@ weight: 40
 
 # Matching IP addresses
 
-<span style="background-color:#f3f973;">Matching IP addresses is an experimental feature.</span>
-
 LogQL supports matching IP addresses.
 
 With logs such as

docs/sources/logql/ip.md

@@ -389,8 +389,6 @@ will get those labels extracted:
 
 #### Pattern
 
-<span style="background-color:#f3f973;">The pattern parser is a beta feature.</span>
-
 The pattern parser allows the explicit extraction of fields from log lines by defining a pattern expression (`| pattern "<pattern-expression>"`). The expression matches the structure of a log line.
 
 Consider this NGINX log line.

docs/sources/logql/log_queries.md

@@ -8,7 +8,6 @@ Retention in Grafana Loki is achieved either through the [Table Manager](#table-
 
 Retention through the [Table Manager](../table-manager/) is achieved by relying on the object store TTL feature, and will work for both [boltdb-shipper](../boltdb-shipper) store and chunk/index store. However retention through the [Compactor](../boltdb-shipper#compactor) is supported only with the [boltdb-shipper](../boltdb-shipper) store.
 
-<span style="background-color:#f3f973;">Retention through the [Compactor](#Compactor) is experimental.</span>
 The Compactor retention will become the default and have long term support. It supports more granular retention policies on per tenant and per stream use cases.
 
 ## Compactor

docs/sources/operations/storage/retention.md

@@ -4,16 +4,70 @@ title: V2.4
 
 # Version 2.4 release notes
 
-The Loki team is excited to announce the release of Loki 2.4! 
+With Loki 2.4 we’ve focused on 2 main things:
+
+1. Making it easier than ever to get your logs into Loki. 
+2. Making it easier to run and operate, anyone not using Kubernetes or allergic to microservices should pay close attention, this release has a lot for you.
 
 ## Features and enhancements
 
-* Blurb about allowing out-of-order log entries. Configured an a per-tenant basis.  Refer to the configuration section's [Accept out-of-order writes](../../configuration/#accept-of-out-of-order-writes) for details.
+* [**Loki no longer requires logs to be sent in perfect chronological order.**](../../configuration/#accept-out-of-order-writes) Support for out of order logs is one of the most highly requested features for Loki and in 2.4 the strict ordering constraint has been removed.
+* Scaling Loki is now much easier with a hybrid mode between a single binary and microservices we are calling the [**Simple Scalable Deployment**](../../fundamentals/architecture/#modes-of-operation). It’s now possible to scale Loki with new `read` and `write` targets. Where previously you would have needed Kubernetes and the microservices approach to start tapping into Loki’s potential, it’s now possible to do this in a much simpler way.
+* We introduced a [common config section](../../configuration/#common_config) and updated and updated a lot of defaults. This results in a **70% smaller Loki config** and comes out of the box with more appropriate defaults and limits. [Check out the example local config](https://github.com/grafana/loki/blob/main/cmd/loki/loki-local-config.yaml) as the new reference for running Loki.
+* **[Recording rules](../../rules/#recording-rules) are no longer marked experimental** and have a much more resilient implementation which leverages the existing write ahead log code in Prometheus.
+* Do you use Kafka for your logs? You should check out the new [**Promtail Kafka Consumer**](../../clients/promtail/scraping/#kafka) which can easily get your logs out of Kafka and into Loki.
+* **LogQL has received some nice enhancements** thanks to the amazing Loki community in the form of [group_left and group_right](../../logql/#many-to-one-and-one-to-many-vector-matches) as well as  the `label_format` and `line_format`functions now support [working with dates and times](../../logql/template_functions/#now)
+* Another great community contribution to Promtail allows it to [**accept ndjson and plaintext log files over HTTP**](../../clients/promtail/configuration/#loki_push_api).
+
+All in all about 260 PR’s went into Loki 2.4 and we thank everyone for helping us make the best Loki yet.
+
+For a full list of all changes please look at the [CHANGELOG](https://github.com/grafana/loki/blob/main/CHANGELOG.md#240-20211105)
+
+## Upgrade Considerations
+
+Please read the [upgrade guide](../../upgrading/#240) before updating Loki.
+
+We made a lot of changes to Loki’s configuration as part of this release, we have tried our best to make sure changes are compatible with existing configurations however some changes to default limits may impact users who didn't have values explicitly set for these limits in their config files.
+
+Here is a list of limits changed:
+
+| config | new default | old default |
+| --- | --- | --- |
+| ingestion_rate_strategy | "global" | "local" |
+| max_global_streams_per_user | 5000 | 0 (no limit) |
+| max_query_length | "721h" | "0h" (no limit) |
+| max_query_parallelism | 32 | 14 |
+| max_streams_per_user | 0 (no limit) | 10000 |
+| reject_old_samples | true | false |
+| reject_old_samples_max_age | "168h" | "336h" |
+| per_stream_rate_limit | 3MB | - |
+| per_stream_rate_limit_burst | 15MB | - |
+
+Also we have enabled the [in-memory FIFO cache by default](https://github.com/grafana/loki/pull/4519) when no existing external cache is configured. This helps improve Loki's performance however it will also allow Loki to consume more memory.
 
 ## Bug fixes
 
 Lists of bug fixes for 2.4.x.
 
 ### 2.4.0 bug fixes
 
-* Bug fix here.
+Here is a list of some of the more important fixes in 2.4.0:
+
+* [4598](https://github.com/grafana/loki/pull/4598) **kavirajk**: Fix `ip` matcher lexer to differentiate filter from identifier
+* [4563](https://github.com/grafana/loki/pull/4563) **cyriltovena**: Fixes the Series function to handle properly sharding.
+* [4518](https://github.com/grafana/loki/pull/4518) **slim-bean**: Loki: Fix bug where items are returned to a sync.Pool incorrectly
+* [4411](https://github.com/grafana/loki/pull/4411) **slim-bean**: Loki: Bug: frontend waiting on results which would never come
+* [4238](https://github.com/grafana/loki/pull/4238) **liguozhong**: [fix] distributor: fix goroutine leak
+
+## Security fixes
+
+List of security fixes for 2.4.x
+
+### 2.4.0 security fixes
+
+2.4.0 contains one fix which is security related
+
+[4627](https://github.com/grafana/loki/pull/4627) Loki: Explicitly define allowed HTTP methods on HTTP endpoints
+
+A community user noticed that all Loki endpoints would respond to HTTP OPTIONS requests, and the proxy they placed in front of Loki responsible for HTTP authentication would pass OPTIONS requests unauthenticated to Loki allowing users to make unauthenticated/unauthorized queries to Loki. This PR restricts the type of HTTP methods allowed on each endpoint and disallows OPTIONS requests.
+