...to account for unaligned reads.
I have not seen this happen in the wild because the kernel
always seems to issue 4k-aligned requests. But the cost
of the additional block in the pool is low and prevents
a buffer overrun panic when an unaligned read does happen.
Our byte cache pools are sized acc. to MAX_KERNEL_WRITE, but the
running kernel may have a higher limit set. Clamp to what we can
handle.
Fixes a panic on a Synology NAS reported at
https://github.com/rfjakob/gocryptfs/issues/145
The extended TestLongnameStat() exposes a pathological case
when run on ext4, as ext4 reuses inode numbers immediately.
This change modifies the test to not delete the files immediately,
so the inode numbers cannot be reused immediately.
Fix for the underlying issue is a TODO.
A file with a name of exactly 176 bytes length caused this error:
ls: cannot access ./tmp/dsg/sXSGJLTuZuW1FarwIkJs0w/b6mGjdxIRpaeanTo0rbh0A/QjMRrQZC_4WLhmHI1UOBcA/gocryptfs.longname.QV-UipdDXeUVdl05WruoEzBNPrQCfpu6OzJL0_QnDKY: No such file or directory
ls: cannot access ./tmp/dsg/sXSGJLTuZuW1FarwIkJs0w/b6mGjdxIRpaeanTo0rbh0A/QjMRrQZC_4WLhmHI1UOBcA/gocryptfs.longname.QV-UipdDXeUVdl05WruoEzBNPrQCfpu6OzJL0_QnDKY.name: No such file or directory
-????????? ? ? ? ? ? gocryptfs.longname.QV-UipdDXeUVdl05WruoEzBNPrQCfpu6OzJL0_QnDKY
-????????? ? ? ? ? ? gocryptfs.longname.QV-UipdDXeUVdl05WruoEzBNPrQCfpu6OzJL0_QnDKY.name
Root cause was a wrong shortNameMax constant that failed to
account for the obligatory padding byte.
Fix the constant and also expand the TestLongnameStat test case
to test ALL file name lengths from 1-255 bytes.
Fixes https://github.com/rfjakob/gocryptfs/issues/143 .
The encrypt and decrypt path both had a copy that were equivalent
but ordered differently, which was confusing.
Consolidate it in a new dedicated function.
MacOS creates lots of these files, and if the directory is otherwise
empty, we would throw an IO error to the unsuspecting user.
With this patch, we log a warning, but otherwise pretend we did not
see it.
Mitigates https://github.com/rfjakob/gocryptfs/issues/140
...and if Getdents is not available at all.
Due to this warning I now know that SSHFS always returns DT_UNKNOWN:
gocryptfs[8129]: Getdents: convertDType: received DT_UNKNOWN, falling back to Lstat
This behavoir is confirmed at http://ahefner.livejournal.com/16875.html:
"With sshfs, I finally found that obscure case. The dtype is always set to DT_UNKNOWN [...]"
The exit codes have been documented in CLI_ABI.md for a while,
but they should also be listed in the man page.
Also fix the rendering of "[-o COMMA-SEPARATED-OPTIONS]", where
the square brackets where interpreted as something. Escape all
square brackets to be safe.
The local user id of the packager is not interesting for users who
download the tarball.
Also it will cause the gocryptfs binary to have an unintended owner
when the tarball is extraced as root.
Fix the issue by using "tar --owner=root --group=root" which
overwrites user and group id with zero.
The benchmark that supported the decision for 512-byte
prefetching previously lived outside the repo.
Let's add it where it belongs so it cannot get lost.
The Readdir function provided by os is inherently slow because
it calls Lstat on all files.
Getdents gives us all the information we need, but does not have
a proper wrapper in the stdlib.
Implement the "Getdents()" wrapper function that calls
syscall.Getdents() and parses the returned byte blob to a
fuse.DirEntry slice.
When you run "go vet" explicitely against go1.4.go, it ignores
the "+build !go1.5" tag and, of course, throws a syntax error:
$ go vet go1.4.go
can't load package: package main:
go1.4.go:5:1: expected 'package', found 'STRING' "You need Go 1.5 or higher to compile gocryptfs!"
Unfortunatey, this is how https://goreportcard.com/ seems to call
"go vet", and means we get 0% on the "go vet" test and see this
error:
An error occurred while running this test (strconv.Atoi: parsing " go1.4.go": invalid syntax)
By reworking the logic to use a non-existant package we get an
uglier error
$ GOROOT=/opt/go1.4.3 /opt/go1.4.3/bin/go build
go1.4.go:7:8: cannot find package "You_need_Go_1.5_or_higher_to_compile_gocryptfs" in any of:
/opt/go1.4.3/src/You_need_Go_1.5_or_higher_to_compile_gocryptfs (from $GOROOT)
/home/jakob/go/src/You_need_Go_1.5_or_higher_to_compile_gocryptfs (from $GOPATH)
profiling.go:6:2: cannot find package "runtime/trace" in any of:
/opt/go1.4.3/src/runtime/trace (from $GOROOT)
/home/jakob/go/src/runtime/trace (from $GOPATH)
but make "go vet" happy.
Remove the "Masterkey" field from fusefrontend.Args because it
should not be stored longer than neccessary. Instead pass the
masterkey as a separate argument to the filesystem initializers.
Then overwrite it with zeros immediately so we don't have
to wait for garbage collection.
Note that the crypto implementation still stores at least a
masterkey-derived value, so this change makes it harder, but not
impossible, to extract the encryption keys from memory.
Suggested at https://github.com/rfjakob/gocryptfs/issues/137
* extend the diriv cache to 100 entries
* add special handling for the immutable root diriv
The better cache allows to shed some complexity from the path
encryption logic (parent-of-parent check).
Mitigates https://github.com/rfjakob/gocryptfs/issues/127
Dir is like filepath.Dir but returns "" instead of ".".
This was already implemented in fusefrontend_reverse as saneDir().
We will need it in nametransform for the improved diriv caching.
As noticed by @riking, the logic in the bash script will break
when Go 1 version numbers reach double-digits.
Instead, use a build tag "!go1.5" to cause a syntax error:
$ /opt/go1.4.3/bin/go build
can't load package: package github.com/rfjakob/gocryptfs:
go1.4.go:5:1: expected 'package', found 'STRING' "You need Go 1.5 or higher to compile gocryptfs!"
Fixes https://github.com/rfjakob/gocryptfs/issues/133
...and move all profiling functionality to its own file, as
the main function is already long enough.
Periodically saving the memory profile allows capturing the used
memory during normal operation, as opposed to on exit, where the
kernel has already issued FORGETs for all inodes.
This functionality has been used to create the memory profile shown
in https://github.com/rfjakob/gocryptfs/issues/132 .
scrypt (used during masterkey decryption) allocates a lot of memory.
Go only returns memory to the OS after 5 minutes, which looks like
a waste. Call FreeOSMemory() to return it immediately.
Looking a fresh mount:
before: VmRSS: 73556 kB
after: VmRSS: 8568 kB
A directory with a long name has two associated virtual files:
the .name file and the .diriv files.
These used to get the same inode number:
$ ls -di1 * */*
33313535 gocryptfs.longname.2togDFouca9mrTwtfF1RNW5DZRAQY8alaR7wO_Xd5Zw
1000000000033313535 gocryptfs.longname.2togDFouca9mrTwtfF1RNW5DZRAQY8alaR7wO_Xd5Zw/gocryptfs.diriv
1000000000033313535 gocryptfs.longname.2togDFouca9mrTwtfF1RNW5DZRAQY8alaR7wO_Xd5Zw.name
With this change we use another prefix (2 instead of 1) for .name files.
$ ls -di1 * */*
33313535 gocryptfs.longname.2togDFouca9mrTwtfF1RNW5DZRAQY8alaR7wO_Xd5Zw
1000000000033313535 gocryptfs.longname.2togDFouca9mrTwtfF1RNW5DZRAQY8alaR7wO_Xd5Zw/gocryptfs.diriv
2000000000033313535 gocryptfs.longname.2togDFouca9mrTwtfF1RNW5DZRAQY8alaR7wO_Xd5Zw.name
We passed our stdout and stderr to the new logger instance,
which makes sense to see any error message, but also means that
the fd is kept open even when we close it.
Fixes the new TestMountBackground test and
https://github.com/rfjakob/gocryptfs/issues/130 .
I have added a subset of fsstress-gocryptfs.bash to EncFS as
fsstress-encfs.sh, improving the code a bit.
This change forward-ports these improvements to
fsstress-gocryptfs.bash.
