Looking at the dircache debug output, we see
that a "git status" workload has a very bad
cache hit rate because the entries expire or
get evicted before they can be reused.
Increase both cache size and lifetime for
a 4x speedup:
Before: 75s
After: 17s
https://github.com/rfjakob/gocryptfs/issues/410
We need
fd7328faf9
to fix a crash reported in https://github.com/rfjakob/gocryptfs/issues/430 :
2019/10/30 17:14:16 Unknown opcode 2016
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x508d38]
This patch is only in the v2.x.x branch. Upgrade to v2, as the
old API is also supported there.
Running
git grep hanwen/go-fuse | grep -v hanwen/go-fuse/v2
to check for forgotten references comes back clean.
Adding flags allows to use inomap in reverse mode,
replacing the clunky inoBaseDirIV/inoBaseNameFile
logic that causes problems with high underlying
inode numbers ( https://github.com/rfjakob/gocryptfs/issues/457 )
Microbenchmarks (values below) show that the "SingleDev"
case is now much slower due to an extra map lookup,
but this has no visible effects in ./test.bash results,
so there was no time spent optimizing the case further.
$ go test -bench=.
goos: linux
goarch: amd64
pkg: github.com/rfjakob/gocryptfs/internal/inomap
BenchmarkTranslateSingleDev-4 18757510 61.5 ns/op
BenchmarkTranslateManyDevs-4 18061515 64.5 ns/op
PASS
ok github.com/rfjakob/gocryptfs/internal/inomap 2.467s
We used to restrict setting xattrs to the "user."
namespace. I don't see a real reason for this
anymore, and it causes trouble for users who are using
acls.
Tests will be added in the next commit.
https://github.com/rfjakob/gocryptfs/issues/453
Error from Travis CI was:
+GOOS=darwin
+GOARCH=amd64
+go build -tags without_openssl
# github.com/rfjakob/gocryptfs/internal/fusefrontend
internal/fusefrontend/fs.go:88:45: cannot use st.Dev (type int32) as type uint64 in argument to openfiletable.NewInumMap
Add uint64 to fix it.
The comment is outdated, at this point, we should
really not get any errors from ReadDirIVAt.
The change is best seen when running the fsck tests. Before:
fsck: error opening dir "missing_diriv": 2=no such file or directory
After:
OpenDir "K2m0E6qzIfoLkVZJanoUiQ": could not read gocryptfs.diriv: no such file or directory
fsck: error opening dir "missing_diriv": 5=input/output error
See https://github.com/rfjakob/gocryptfs/issues/403 , where
the extra info would have been helpful.
This was meant as a way to inform the user that
something is very wrong, however, users are hitting
the condition on MacOS due to ".DS_Store" files, and
also on NFS due to ".nfsXXX" files.
Drop the whole thing as it seems to cause more pain
than gain.
Fixes https://github.com/rfjakob/gocryptfs/issues/431
Closing the fd means the inode number may be reused immediately
by a new file, so we have to get the old fileID out of the table
beforehand!
Hopefully fixes https://github.com/rfjakob/gocryptfs/issues/363
This adds support for gitignore-like wildcards and exclude patters in
reverse mode. It (somewhat) fixes#273: no regexp support, but the
syntax should be powerful enough to satisfy most needs.
Also, since adding a lot of --exclude options can be tedious, it adds
the --exclude-from option to read patterns from a file (or files).
This fixed the "Permission denied" bug, but still has the problem that
the directory may be replaced behind our back. Mitigated by the fact
that we skip the workaround when running as root with -allow_other.
https://github.com/rfjakob/gocryptfs/issues/354
For Linux, everything effectively stays the same. For both path-based and
fd-based Utimens() calls, we use unix.UtimesNanoAt(). To avoid introducing
a separate syscall wrapper for futimens() (as done in go-fuse, for example),
we instead use the /proc/self/fd - trick.
On macOS, this changes quite a lot:
* Path-based Utimens() calls were previously completely broken, since
unix.UtimensNanoAt() ignores the passed file descriptor. Note that this
cannot be fixed easily since there IS no appropriate syscall available on
macOS prior to High Sierra (10.13). We emulate this case by using
Fchdir() + setattrlist().
* Fd-based Utimens() calls were previously translated to f.GetAttr() (to
fill any empty parameters) and syscall.Futimes(), which does not does
support nanosecond precision. Both issues can be fixed by switching to
fsetattrlist().
Fixes https://github.com/rfjakob/gocryptfs/issues/350
The only call forwarded to loopbackFileSystem was Statfs,
which is trivial to implement.
Implement it and drop loopbackFileSystem, as having it carries the
risk that a coding error bypasses the usual encryption/decryption
chain.
Instead of manually adjusting the user after creating the symlink,
adjust effective permissions and let the kernel deal with it.
Related to https://github.com/rfjakob/gocryptfs/issues/338.
Instead of manually adjusting the user and mode after creating the
device file, adjust effective permissions and let the kernel deal
with it.
Related to https://github.com/rfjakob/gocryptfs/issues/338.
The current code has a risk of race-conditions, since we pass a path
containing "/" to Fchownat. We could fix this by opening a file descriptor,
however, this does not seem worth the effort. We also don't chown *.name files.
Make sure that the directory belongs to the correct owner before users
can access it. For directories with SUID/SGID mode, there is a risk of
race-conditions when files are created before the correct owner is set.
They will then inherit the wrong user and/or group.
See https://github.com/rfjakob/gocryptfs/issues/327 for more details.
Reported by @slackner at https://github.com/rfjakob/gocryptfs/issues/327 :
Possible race-conditions between file creation and Fchownat
* Assume a system contains a gocryptfs mount as root user
with -allow_other
* As a regular user create a new file with mode containing
the SUID flag and write access for other users
* Before gocryptfs executes the Fchownat call, try to open
the file again, write some exploit code to it, and try to run it.
For a short time, the file is owned by root and has the SUID flag, so
this is pretty dangerous.
The files are apparently processed in alphabetic order, so cli_args.go is
processed before main.go. In order to run before the go-fuse imports, put
the 'ensure fds' code in a separate package. Debug messages are omitted
to avoid additional imports (that might contain other code messing up our
file descriptors).
Setting/removing extended attributes on directories was partially fixed with
commit eff35e60b6. However, on most file systems
it is also possible to do these operations without read access (see tests).
Since we cannot open a write-access fd to a directory, we have to use the
/proc/self/fd trick (already used for ListXAttr) for the other operations aswell.
For simplicity, let's separate the Linux and Darwin code again (basically revert
commit f320b76fd1), and always use the
/proc/self/fd trick on Linux. On Darwin we use the best-effort approach with
openBackingFile() as a fallback.
More discussion about the available options is available in
https://github.com/rfjakob/gocryptfs/issues/308.
We alread have this warning in Open(), but xfstests generic/488
causes "too many open files" via Create. Add the same message so
the user sees what is going on.
When the old size is zero, there are no existing blocks to merge the
new data with. Directly use Ftruncate if the size is block-aligned.
Fixes https://github.com/rfjakob/gocryptfs/issues/305
Bug looked like this:
$ ls -l .
total 0
drwxrwxr-x. 2 jakob jakob 60 Jan 3 15:42 foo
-rw-rw-r--. 1 jakob jakob 0 Jan 3 15:46 x
$ ls -l .
ls: cannot access '.': No such file or directory
(only happened when "" was in the dirCache)
This function is in all fastpaths, will get a cache, and needs
its own file.
renamed: internal/fusefrontend/names.go -> internal/fusefrontend/openbackingdir.go
renamed: internal/fusefrontend/names_test.go -> internal/fusefrontend/openbackingdir_test.go
An Open() a fifo blocks until it is opened for writing.
This meant that xattr operations on FIFOs would block.
Pass O_NONBLOCK to fix that, and add a test.
Failure was:
+ GOOS=darwin
+ GOARCH=amd64
+ go build -tags without_openssl
# github.com/rfjakob/gocryptfs/internal/fusefrontend
internal/fusefrontend/fs_dir.go:159:60: cannot use origMode | 448 (type uint16) as type uint32 in argument to syscallcompat.Fchmodat
internal/fusefrontend/fs_dir.go:170:33: cannot use origMode (type uint16) as type uint32 in argument to syscallcompat.Fchmodat
Use openBackingDir() and Fstatat().
High performance impact, though part of it should be
mitigated by adding DirIV caching to the new code paths.
$ ./benchmark.bash
Testing gocryptfs at /tmp/benchmark.bash.Eou: gocryptfs v1.6-37-ge3914b3-dirty; go-fuse v20170619-66-g6df8ddc; 2018-10-14 go1.11
WRITE: 262144000 bytes (262 MB, 250 MiB) copied, 1.2289 s, 213 MB/s
READ: 262144000 bytes (262 MB, 250 MiB) copied, 1.02616 s, 255 MB/s
UNTAR: 24.490
MD5: 13.120
LS: 3.368
RM: 9.232
The directory was already created, so return success even if Fchownat fails.
The same error handling is already used if fs.args.PlaintextNames is false.
Even though filesystem notifications aren't implemented for FUSE, I decided to
try my hand at implementing the autounmount feature (#128). I based it on the
EncFS autounmount code, which records filesystem accesses and checks every X
seconds whether it's idled long enough to unmount.
I've tested the feature locally, but I haven't added any tests for this flag.
I also haven't worked with Go before. So please let me know if there's
anything that should be done differently.
One particular concern: I worked from the assumption that the open files table
is unique per-filesystem. If that's not true, I'll need to add an open file
count and associated lock to the Filesystem type instead.
https://github.com/rfjakob/gocryptfs/pull/265
Error was:
# github.com/rfjakob/gocryptfs/internal/fusefrontend
internal/fusefrontend/fs.go:179: cannot use perms | 256 (type uint16) as type uint32 in argument to syscall.Fchmod
internal/fusefrontend/fs.go:185: cannot use perms (type uint16) as type uint32 in argument to syscall.Fchmod
Rename openBackingPath to openBackingDir and use OpenDirNofollow
to be safe against symlink races. Note that openBackingDir is
not used in several important code paths like Create().
But it is used in Unlink, and the performance impact in the RM benchmark
to be acceptable:
Before
$ ./benchmark.bash
Testing gocryptfs at /tmp/benchmark.bash.bYO: gocryptfs v1.6-12-g930c37e-dirty; go-fuse v20170619-49-gb11e293; 2018-09-08 go1.10.3
WRITE: 262144000 bytes (262 MB, 250 MiB) copied, 1.07979 s, 243 MB/s
READ: 262144000 bytes (262 MB, 250 MiB) copied, 0.882413 s, 297 MB/s
UNTAR: 16.703
MD5: 7.606
LS: 1.349
RM: 3.237
After
$ ./benchmark.bash
Testing gocryptfs at /tmp/benchmark.bash.jK3: gocryptfs v1.6-13-g84d6faf-dirty; go-fuse v20170619-49-gb11e293; 2018-09-08 go1.10.3
WRITE: 262144000 bytes (262 MB, 250 MiB) copied, 1.06261 s, 247 MB/s
READ: 262144000 bytes (262 MB, 250 MiB) copied, 0.947228 s, 277 MB/s
UNTAR: 17.197
MD5: 7.540
LS: 1.364
RM: 3.410
As uncovered by xfstests generic/465, concurrent reads and writes
could lead to this,
doRead 3015532: corrupt block #1039: stupidgcm: message authentication failed,
as the read could pick up a block that has not yet been completely written -
write() is not atomic!
Now writes take ContentLock exclusively, while reads take it shared,
meaning that multiple reads can run in parallel with each other, but
not with a write.
This also simplifies the file header locking.
xfstests generic/083 fills the filesystem almost completely while
running fsstress in parallel. In fsck, these would show up:
readFileID 2580: incomplete file, got 18 instead of 19 bytes
This could happen when writing the file header works, but writing
the actual data fails.
Now we kill the header again by truncating the file to zero.
If the underlying filesystem is full, it is normal get ENOSPC here.
Log at Info level instead of Warning.
Fixes xfstests generic/015 and generic/027, which complained about
the extra output.
O_DIRECT accesses must be aligned in both offset and length. Due to our
crypto header, alignment will be off, even if userspace makes aligned
accesses. Running xfstests generic/013 on ext4 used to trigger lots of
EINVAL errors due to missing alignment. Just fall back to buffered IO.
"gocryptfs -fsck" will need access to helper functions,
and to get that, it will need to cast a gofuse.File to a
fusefrontend.File. Make fusefrontend.File exported to make
this work.
We are clean again.
Warnings were:
internal/fusefrontend/fs.go:443:14: should omit type string from declaration
of var cTarget; it will be inferred from the right-hand side
internal/fusefrontend/xattr.go:26:1: comment on exported method FS.GetXAttr
should be of the form "GetXAttr ..."
internal/syscallcompat/sys_common.go:9:7: exported const PATH_MAX should have
comment or be unexported
Reading system.posix_acl_access and system.posix_acl_default
should return EOPNOTSUPP to inform user-space that we do not
support ACLs.
xftestest essientially does
chacl -l | grep "Operation not supported"
to determine if the filesystem supports ACLs, and used to
wrongly believe that gocryptfs does.
mv is unhappy when we return EPERM when it tries to set
system.posix_acl_access:
mv: preserving permissions for ‘b/x’: Operation not permitted
Now we return EOPNOTSUPP like tmpfs does and mv seems happy.
Values a binary-safe, there is no need to base64-encode them.
Old, base64-encoded values are supported transparently
on reading. Writing xattr values now always writes them binary.
We previously returned EPERM to prevent the kernel from
blacklisting our xattr support once we get an unsupported
flag, but this causes lots of trouble on MacOS:
Cannot save files from GUI apps, see
https://github.com/rfjakob/gocryptfs/issues/229
Returning ENOSYS triggers the dotfiles fallback on MacOS
and fixes the issue.
OpenDir and ListXAttr skip over corrupt entries,
readFileID treats files the are too small as empty.
This improves usability in the face of corruption,
but hides the problem in a log message instead of
putting it in the return code.
Create a channel to report these corruptions to fsck
so it can report them to the user.
Also update the manpage and the changelog with the -fsck option.
Closes https://github.com/rfjakob/gocryptfs/issues/191
This should not happen via FUSE as the kernel caps the size,
but with fsck we have the first user that calls Read directly.
For symmetry, check it for Write as well.
A few places have called tlog.Warn.Print, which directly
calls into log.Logger due to embedding, losing all features
of tlog.
Stop embedding log.Logger to make sure the internal functions
cannot be called accidentially and fix (several!) instances
that did.
Both fusefrontend and fusefrontend_reverse were doing
essentially the same thing, move it into main's
initFuseFrontend.
A side-effect is that we have a reference to cryptocore
in main, which will help with wiping the keys on exit
(https://github.com/rfjakob/gocryptfs/issues/211).
Now that we have Fstatat we can use it in Getdents to
get rid of the path name.
Also, add an emulated version of getdents for MacOS. This allows
to drop the !HaveGetdents special cases from fusefrontend.
Modify the getdents test to test both native getdents and the emulated
version.
In PlaintextNames mode the "gocryptfs.longname." prefix does not have any
special meaning. We should not attempt to delete any .name files.
Partially fixes https://github.com/rfjakob/gocryptfs/issues/174
