The Readdir function provided by os is inherently slow because
it calls Lstat on all files.
Getdents gives us all the information we need, but does not have
a proper wrapper in the stdlib.
Implement the "Getdents()" wrapper function that calls
syscall.Getdents() and parses the returned byte blob to a
fuse.DirEntry slice.
When you run "go vet" explicitely against go1.4.go, it ignores
the "+build !go1.5" tag and, of course, throws a syntax error:
$ go vet go1.4.go
can't load package: package main:
go1.4.go:5:1: expected 'package', found 'STRING' "You need Go 1.5 or higher to compile gocryptfs!"
Unfortunatey, this is how https://goreportcard.com/ seems to call
"go vet", and means we get 0% on the "go vet" test and see this
error:
An error occurred while running this test (strconv.Atoi: parsing " go1.4.go": invalid syntax)
By reworking the logic to use a non-existant package we get an
uglier error
$ GOROOT=/opt/go1.4.3 /opt/go1.4.3/bin/go build
go1.4.go:7:8: cannot find package "You_need_Go_1.5_or_higher_to_compile_gocryptfs" in any of:
/opt/go1.4.3/src/You_need_Go_1.5_or_higher_to_compile_gocryptfs (from $GOROOT)
/home/jakob/go/src/You_need_Go_1.5_or_higher_to_compile_gocryptfs (from $GOPATH)
profiling.go:6:2: cannot find package "runtime/trace" in any of:
/opt/go1.4.3/src/runtime/trace (from $GOROOT)
/home/jakob/go/src/runtime/trace (from $GOPATH)
but make "go vet" happy.
Remove the "Masterkey" field from fusefrontend.Args because it
should not be stored longer than neccessary. Instead pass the
masterkey as a separate argument to the filesystem initializers.
Then overwrite it with zeros immediately so we don't have
to wait for garbage collection.
Note that the crypto implementation still stores at least a
masterkey-derived value, so this change makes it harder, but not
impossible, to extract the encryption keys from memory.
Suggested at https://github.com/rfjakob/gocryptfs/issues/137
* extend the diriv cache to 100 entries
* add special handling for the immutable root diriv
The better cache allows to shed some complexity from the path
encryption logic (parent-of-parent check).
Mitigates https://github.com/rfjakob/gocryptfs/issues/127
Dir is like filepath.Dir but returns "" instead of ".".
This was already implemented in fusefrontend_reverse as saneDir().
We will need it in nametransform for the improved diriv caching.
As noticed by @riking, the logic in the bash script will break
when Go 1 version numbers reach double-digits.
Instead, use a build tag "!go1.5" to cause a syntax error:
$ /opt/go1.4.3/bin/go build
can't load package: package github.com/rfjakob/gocryptfs:
go1.4.go:5:1: expected 'package', found 'STRING' "You need Go 1.5 or higher to compile gocryptfs!"
Fixes https://github.com/rfjakob/gocryptfs/issues/133
...and move all profiling functionality to its own file, as
the main function is already long enough.
Periodically saving the memory profile allows capturing the used
memory during normal operation, as opposed to on exit, where the
kernel has already issued FORGETs for all inodes.
This functionality has been used to create the memory profile shown
in https://github.com/rfjakob/gocryptfs/issues/132 .
scrypt (used during masterkey decryption) allocates a lot of memory.
Go only returns memory to the OS after 5 minutes, which looks like
a waste. Call FreeOSMemory() to return it immediately.
Looking a fresh mount:
before: VmRSS: 73556 kB
after: VmRSS: 8568 kB
A directory with a long name has two associated virtual files:
the .name file and the .diriv files.
These used to get the same inode number:
$ ls -di1 * */*
33313535 gocryptfs.longname.2togDFouca9mrTwtfF1RNW5DZRAQY8alaR7wO_Xd5Zw
1000000000033313535 gocryptfs.longname.2togDFouca9mrTwtfF1RNW5DZRAQY8alaR7wO_Xd5Zw/gocryptfs.diriv
1000000000033313535 gocryptfs.longname.2togDFouca9mrTwtfF1RNW5DZRAQY8alaR7wO_Xd5Zw.name
With this change we use another prefix (2 instead of 1) for .name files.
$ ls -di1 * */*
33313535 gocryptfs.longname.2togDFouca9mrTwtfF1RNW5DZRAQY8alaR7wO_Xd5Zw
1000000000033313535 gocryptfs.longname.2togDFouca9mrTwtfF1RNW5DZRAQY8alaR7wO_Xd5Zw/gocryptfs.diriv
2000000000033313535 gocryptfs.longname.2togDFouca9mrTwtfF1RNW5DZRAQY8alaR7wO_Xd5Zw.name
We passed our stdout and stderr to the new logger instance,
which makes sense to see any error message, but also means that
the fd is kept open even when we close it.
Fixes the new TestMountBackground test and
https://github.com/rfjakob/gocryptfs/issues/130 .
I have added a subset of fsstress-gocryptfs.bash to EncFS as
fsstress-encfs.sh, improving the code a bit.
This change forward-ports these improvements to
fsstress-gocryptfs.bash.
On MacOS, building and testing without openssl is much easier.
The tests should skip tests that fail because of missing openssl
instead of aborting.
Fixes https://github.com/rfjakob/gocryptfs/issues/123
Fixed by including the correct header. Should work on older openssl
versions as well.
Error was:
locking.go:21: undefined reference to `CRYPTO_set_locking_callback'
Due to RMW, we always need read permissions on the backing file. This is a
problem if the file permissions do not allow reading (i.e. 0200 permissions).
This patch works around that problem by chmod'ing the file, obtaining a fd,
and chmod'ing it back.
Test included.
Issue reported at: https://github.com/rfjakob/gocryptfs/issues/125
Currently neither gocryptfs nor go-fuse automatically call load_osxfuse
if the /dev/osxfuse* device(s) do not exist. At least tell the user
what to do.
See https://github.com/rfjakob/gocryptfs/issues/124 for user pain.
Previously we ran through the decryption steps even for an empty
ciphertext slice. The functions handle it correctly, but returning
early skips all the extra calls.
Speeds up the tar extract benchmark by about 4%.
go-fuse caps MaxWrite at MAX_KERNEL_WRITE anyway, and we
actually depend on this behavoir now as the byte pools
are sized according to MAX_KERNEL_WRITE.
So let's use MAX_KERNEL_WRITE explicitely.
We use two levels of buffers:
1) 4kiB+overhead for each ciphertext block
2) 128kiB+overhead for each FUSE write (32 ciphertext blocks)
This commit adds a sync.Pool for both levels.
The memory-efficiency for small writes could be improved,
as we now always use a 128kiB buffer.
