os: file write blocks subsequent Open() with "sharing violation" in OneDrive

[networkimprov]

What version of Go are you using (go version)?

$ go version
go version go1.13.3 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

Windows 10, amd64 (go env not available on that machine)

What did you do?

EDIT: updated reproducer

package main

import ( "io/ioutil"; "time" )

func main() {
   aFile := "z35219.txt"
   for {
      err := ioutil.WriteFile(aFile, []byte("abcdefghijklmnopqrstuvwxyz"), 0600)
      if err != nil { panic(err) }
      aTmr := time.NewTimer(10*time.Second)
      for aRead := true; aRead; {
         select {
         case <-aTmr.C:
            aRead = false
         default:
            _, err := ioutil.ReadFile(aFile)
            if err != nil { panic(err) }
         }
      }
   }
}

What did you expect to see?

No error; this is the case on Win7.

What did you see instead?

On Win10:

panic: open z35219.txt: The process cannot access the file because
 it is being used by another process.

goroutine 1 [running]: main.main() /.../z35219.go:17 +0x10a

Discussion

I see this intermittently in a real app.

EDIT: the following is related to the initial reproducer
I wondered whether GetFileAttributesEx() changed in Windows 10 (or 8) to disallow concurrent read access. But the above program still fails (maybe less often?) on Win10 if I patch stat() to only call CreateFile(), like so:

const READ_CONTROL, FILE_READ_ATTRIBUTES uint32 = 0x20000, 0x80 // from WinAPI docs
h, err = CreateFile(namep, READ_CONTROL|FILE_READ_ATTRIBUTES, FILE_SHARE_READ, nil,
                    OPEN_EXISTING, createFileAttrs, 0)

Those arguments are suggested here: https://stackoverflow.com/questions/52813820/createfile-fails-because-of-sharin-violation-because-some-other-os-process-is-us

Stat(): https://golang.org/src/os/stat_windows.go#L76
Open(): https://golang.org/src/syscall/syscall_windows.go#L266

@jstarks any ideas?

cc @alexbrainman @zx2c4 @bcmills
@gopherbot add release-blocker OS-Windows

[ianlancetaylor]

Is this problem new in 1.13?

[jstarks]

I'm unable to reproduce this on the latest version of Windows 10 with Go 1.13.1 (I should probably update...).

@networkimprov, do you have third-party antivirus software installed?

[networkimprov]

Hi John, how long did you let the program run? It could take several minutes to show up

The machine seeing these failures is not mine (I only have Win7) but I've forwarded your Q.

Note Go 1.13.3 has a significant change to the Windows runtime, tho it's prob not a factor here.

[jstarks]

I let it run overnight with no repro.

[slonopotamus]

@networkimprov

You only get SHARING_VIOLATION from CreateFile if someone has the same file already open in a such access mode that is not listed in your sharing modes.

For example, I hit SHARING_VIOLATION in git-lfs/git-lfs#3880:
one process calls os.Rename while another tries to os.Open destination file.

os.Stat doesn't do CreateFile, so I think your problem is caused by something different. You could try using Process Monitor to record kernel calls related to your file to find out who is opening in a mode that interferes with your app.

[networkimprov]

Thanks for suggestion re Process Monitor.

Assuming this is due to a legitimate system config (ie not malware) any suggestions for fixes? Patching syscall.Open() to retry N times on sharing-violation in a loop with runtime.Gosched()?

one process calls os.Rename while another tries to os.Open destination file.

I tried really hard to get this fixed... #32088 :-(

[slonopotamus]

From my current knowledge, retrying is the only working option. You may want to look at robustio.go file from Go, it has related stuff, but not exposed for public usage.

[rsc]

But the above program still fails (maybe less often?) on Win10 if I patch stat() to only call CreateFile(),

Did you mean patch open? What business would stat have calling CreateFile?

[jstarks]

stat calls CreateFile in a fallback path since apparently there are cases where GetFileAttributesEx fails.

[networkimprov]

@rsc WinAPI CreateFile() doesn't create files, it opens them :-)

[networkimprov]

The box showing this reportedly runs Norton Lifelock.
From systeminfo command: OS Version: 10.0.18362 N/A Build 18362

This is a better reproducer. The problem is triggered by ioutil.WriteFile().

package main

import ( "io/ioutil"; "time" )

func main() {
   aFile := "z35219.txt"
   for {
      err := ioutil.WriteFile(aFile, []byte("abcdefghijklmnopqrstuvwxyz"), 0600)
      if err != nil { panic(err) }
      aTmr := time.NewTimer(10*time.Second)
      for aRead := true; aRead; {
         select {
         case <-aTmr.C:
            aRead = false
         default:
            _, err := ioutil.ReadFile(aFile)
            if err != nil { panic(err) }
         }
      }
   }
}