m2.1 : introduce configurable fsync strategies - #4
Conversation
| if err != nil { | ||
| log.Printf("wal fsync failure: %s, stopping the server", err) | ||
| w.fatalErrChan <- err | ||
| } |
There was a problem hiding this comment.
Ig this will cause a race condition , i am not sure here but i had a scenario in my head
lets say
thread 1 - calls append get error in fsync its holding the mutex tries to send to fatalErrChan and blocked here
thread 2 - cant aquire lock blocked here
fsync error 1 → goes into channel (buffered, doesn't block)
handleFatalErrors reads it → calls cancel() → exits
fsync error 2 → channel is empty, goes in (buffered, doesn't block)
fsync error 3 → channel is full (error 2 still in it)
send BLOCKS while holding mutex → deadlock
Like what if this error piles up in fatalErrChan isnt that a deadlock help me out here?
There was a problem hiding this comment.
thread 1 returns after sending to fatalErrChan and unlocks the mutex. fatalErrChan is buffered so it won't block.
fatalErrChan will trigger the server to close the wal file. There can be two cases here for thread 2 on acquiring the mutex lock -
- case1 wal is closed -> thread2 simply returns and unlocks the mutex.
- case2 wal is not closed yet -> thread2 tries to write and gets error. Sends it to fatalErrChan and returns unlocking the mutex. The previous error wouldn't be there in FatalErrChan as there is nothing blocking the handle interrupt processing. And subsequent calls to cancel function do nothing.
| if w.fsyncStrategy != ALWAYS { | ||
| bytes, err := fmt.Fprintf(w.asyncBuffer, "%s\n", strings.Join(cmd, " ")) | ||
| return bytes, err | ||
| } |
There was a problem hiding this comment.
this never goes to the os only stays in the buffer and fills it up shouldnt we flush this to buffer?
There was a problem hiding this comment.
There is a cron scheduled which does the flushing. But yes the cron is not started in NEVER mode. Good point.
2 participants
This PR introduces 3 fsync strategies for the WAL operations -
EVERY_SEC- If this is selected, a cron is scheduled to run every second which Fsyncs the wal file (balanced performance, at most 1 second of data loss). If fsync fails in this mode, server is stopped.ALWAYS- Fsync happens on each append call synchronously (Least performant but no data loss)NEVER- No explicit Fsync. Let the operating system semantics handle it. (Most performant, susceptible to data loss)It also adds a memory buffer for the WAL contents. All WAL writes are appended to the buffer first (except in ALWAYS mode) and then flushed based on the strategy.