escape what RemoteSyslogAppender cannot send instead of deleting it #315

RFC 3164 allows only visible ASCII and space in the message, and everything else
fell through the loop unwritten. "Schoenwetter <CJK>" reached the collector as
"Schnwetter ", and a tab vanished from between its neighbours, with no marker and
no error.

Such characters are now written as a \uXXXX escape, which stays inside the allowed
range. Encoding still cannot make the message body non-ASCII, which the appender
page now says.

Author: FreeAndNil

src/changelog/3.5.0/315-syslog-non-ascii.xml

@@ -0,0 +1,14 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<entry xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+       xmlns="https://logging.apache.org/xml/ns"
+       xsi:schemaLocation="https://logging.apache.org/xml/ns https://logging.apache.org/xml/ns/log4j-changelog-0.xsd"
+       type="fixed">
+  <issue id="315" link="https://github.com/apache/logging-log4net/pull/315"/>
+  <description format="asciidoc">
+    escape the characters `RemoteSyslogAppender` cannot send instead of deleting them. RFC 3164
+ allows only the visible ASCII characters and space, and everything else was dropped silently, so
+ `Sch&#246;nwetter &#20320;&#22909;` reached the collector as `Schnwetter ` and a tab disappeared
+ from between its neighbours. Such characters are now written as a `\uXXXX` escape, which keeps
+ the record inside the allowed range and readable (audit da18b6fd-f035)
+  </description>
+</entry>

src/log4net.Tests/Appender/RemoteSyslogAppenderTest.cs

@@ -69,6 +69,30 @@ private sealed class RecordingErrorHandler : IErrorHandler
 
   private const int FlushTimeoutMillis = 30_000;
 
+  /// <summary>
+  /// Content outside the RFC 3164 range used to be deleted with no marker, so a message written
+  /// in a non-Latin script reached the audit trail empty.
+  /// </summary>
+  [Test]
+  public void NonAsciiContentIsEscapedAndNotDeleted()
+  {
+    List<byte[]> sentBytes = ExecuteAppend("Sch\u00f6nwetter \u4f60\u597d");
+
+    Assert.That(sentBytes, Has.Count.EqualTo(1));
+    Assert.That(Encoding.ASCII.GetString(sentBytes[0]),
+      Is.EqualTo(@"<14>TestDomain: INFO  - Sch\u00f6nwetter \u4f60\u597d"));
+  }
+
+  /// <summary>A control character other than CR or LF was dropped as well.</summary>
+  [Test]
+  public void OtherControlCharactersAreEscaped()
+  {
+    List<byte[]> sentBytes = ExecuteAppend("a\tb");
+
+    Assert.That(sentBytes, Has.Count.EqualTo(1));
+    Assert.That(Encoding.ASCII.GetString(sentBytes[0]), Is.EqualTo(@"<14>TestDomain: INFO  - a\u0009b"));
+  }
+
   /// <summary>Bounded, so an unreachable server cannot grow the queue without limit.</summary>
   [Test]
   public void SendQueueSizeDefaultsTo500()

src/log4net/Appender/RemoteSyslogAppender.cs

@@ -18,6 +18,7 @@
 #endregion
 
 using System;
+using System.Globalization;
 using System.Text;
 using System.Threading;
 using log4net.Appender.Internal;
@@ -484,6 +485,12 @@ protected virtual void AppendMessage(string message, ref int characterIndex, Str
           break;
         }
       }
+      else
+      {
+        // Escaped, not dropped: content is masked visibly rather than deleted. RFC 3164 allows
+        // only 0x20 to 0x7E here, so the escape itself stays inside that range.
+        builder.Append("\\u").Append(((int)c).ToString("x4", CultureInfo.InvariantCulture));
+      }
     }
   }
 

src/site/antora/modules/ROOT/pages/manual/configuration/appenders/remotesyslogappender.adoc

@@ -45,6 +45,10 @@ You can also specify:
 * SendQueueSize (default: 500), how many datagrams may wait to be sent
 * EnqueueTimeoutMillis (default: 5000), how long a logging call waits for room in a full queue
 
+RFC 3164 allows only the visible ASCII characters and space in the message, so anything else is
+written as a `\uXXXX` escape rather than dropped: a message in a non-Latin script reaches the
+collector readable instead of empty. `Encoding` therefore does not make the message body non-ASCII.
+
 Datagrams are handed to a background thread, so a slow or unreachable syslog server does not hold
 up logging. Once the queue is full, a logging call waits `EnqueueTimeoutMillis` for room and the
 datagram is then discarded and counted, rather than growing the queue without limit. `Flush` waits