Setting/removing extended attributes on directories was partially fixed with
commit eff35e60b6. However, on most file systems
it is also possible to do these operations without read access (see tests).
Since we cannot open a write-access fd to a directory, we have to use the
/proc/self/fd trick (already used for ListXAttr) for the other operations aswell.
For simplicity, let's separate the Linux and Darwin code again (basically revert
commit f320b76fd1), and always use the
/proc/self/fd trick on Linux. On Darwin we use the best-effort approach with
openBackingFile() as a fallback.
More discussion about the available options is available in
https://github.com/rfjakob/gocryptfs/issues/308.
Otherwise this can happen, as triggered by xfstests generic/011:
go-fuse: can't convert error type: openat failed: too many open files
The app then gets a misleading "Function not implemented" error.
We alread have this warning in Open(), but xfstests generic/488
causes "too many open files" via Create. Add the same message so
the user sees what is going on.
Go version go1.10.7 linux/amd64 complains with:
internal/fusefrontend_reverse/rfs.go:333: declaration of "entries" shadows
declaration at internal/fusefrontend_reverse/rfs.go:327
Fixes https://github.com/rfjakob/gocryptfs/issues/286 :
While the actual file is properly excluded, the * .name file is still leaked in the directory listing:
```
drwxr-xr-x 2 sebastian sebastian 4,0K Dez 17 14:58 .
drwxr-xr-x 7 sebastian sebastian 4,0K Dez 17 14:45 ..
-r-------- 1 sebastian sebastian 408 Dez 17 14:56 gocryptfs.conf
-r--r--r-- 1 sebastian sebastian 16 Dez 17 14:58 gocryptfs.diriv
-r--r--r-- 1 sebastian sebastian 320 Dez 17 14:58 gocryptfs.longname.3vZ_r3eDPb1_fL3j5VA4rd_bcKWLKT9eaxOVIGK5HFA.name
```
Excluded files showed up in directory listing like this:
drwxr-xr-x 2 sebastian sebastian 4,0K Dez 17 14:48 .
drwxr-xr-x 7 sebastian sebastian 4,0K Dez 17 14:45 ..
-????????? ? ? ? ? ? abcd
-r-------- 1 sebastian sebastian 366 Dez 17 14:45 gocryptfs.conf
Fixes https://github.com/rfjakob/gocryptfs/issues/285
When the old size is zero, there are no existing blocks to merge the
new data with. Directly use Ftruncate if the size is block-aligned.
Fixes https://github.com/rfjakob/gocryptfs/issues/305
Bug looked like this:
$ ls -l .
total 0
drwxrwxr-x. 2 jakob jakob 60 Jan 3 15:42 foo
-rw-rw-r--. 1 jakob jakob 0 Jan 3 15:46 x
$ ls -l .
ls: cannot access '.': No such file or directory
(only happened when "" was in the dirCache)
This function is in all fastpaths, will get a cache, and needs
its own file.
renamed: internal/fusefrontend/names.go -> internal/fusefrontend/openbackingdir.go
renamed: internal/fusefrontend/names_test.go -> internal/fusefrontend/openbackingdir_test.go
An Open() a fifo blocks until it is opened for writing.
This meant that xattr operations on FIFOs would block.
Pass O_NONBLOCK to fix that, and add a test.
Failure was:
+ GOOS=darwin
+ GOARCH=amd64
+ go build -tags without_openssl
# github.com/rfjakob/gocryptfs/internal/fusefrontend
internal/fusefrontend/fs_dir.go:159:60: cannot use origMode | 448 (type uint16) as type uint32 in argument to syscallcompat.Fchmodat
internal/fusefrontend/fs_dir.go:170:33: cannot use origMode (type uint16) as type uint32 in argument to syscallcompat.Fchmodat
Use openBackingDir() and Fstatat().
High performance impact, though part of it should be
mitigated by adding DirIV caching to the new code paths.
$ ./benchmark.bash
Testing gocryptfs at /tmp/benchmark.bash.Eou: gocryptfs v1.6-37-ge3914b3-dirty; go-fuse v20170619-66-g6df8ddc; 2018-10-14 go1.11
WRITE: 262144000 bytes (262 MB, 250 MiB) copied, 1.2289 s, 213 MB/s
READ: 262144000 bytes (262 MB, 250 MiB) copied, 1.02616 s, 255 MB/s
UNTAR: 24.490
MD5: 13.120
LS: 3.368
RM: 9.232
The directory was already created, so return success even if Fchownat fails.
The same error handling is already used if fs.args.PlaintextNames is false.
Old XFS filesystems always return DT_UNKNOWN. Downgrade the message
to "info" level if we are on XFS.
Using the "warning" level means that users on old XFS filesystems
cannot run the test suite as it intentionally aborts on any
warnings.
Fixes https://github.com/rfjakob/gocryptfs/issues/267
When gocryptfs was started on a terminal and later
daemonized, the color codes stayed active in the syslog
output.
The codes are not visible in "journalctl -f", which is why
I have not noticed it yet, but they do show up in normal
syslog as the usual "#033[33m" crap.
Even though filesystem notifications aren't implemented for FUSE, I decided to
try my hand at implementing the autounmount feature (#128). I based it on the
EncFS autounmount code, which records filesystem accesses and checks every X
seconds whether it's idled long enough to unmount.
I've tested the feature locally, but I haven't added any tests for this flag.
I also haven't worked with Go before. So please let me know if there's
anything that should be done differently.
One particular concern: I worked from the assumption that the open files table
is unique per-filesystem. If that's not true, I'll need to add an open file
count and associated lock to the Filesystem type instead.
https://github.com/rfjakob/gocryptfs/pull/265
Error was:
# github.com/rfjakob/gocryptfs/internal/fusefrontend
internal/fusefrontend/fs.go:179: cannot use perms | 256 (type uint16) as type uint32 in argument to syscall.Fchmod
internal/fusefrontend/fs.go:185: cannot use perms (type uint16) as type uint32 in argument to syscall.Fchmod
Rename openBackingPath to openBackingDir and use OpenDirNofollow
to be safe against symlink races. Note that openBackingDir is
not used in several important code paths like Create().
But it is used in Unlink, and the performance impact in the RM benchmark
to be acceptable:
Before
$ ./benchmark.bash
Testing gocryptfs at /tmp/benchmark.bash.bYO: gocryptfs v1.6-12-g930c37e-dirty; go-fuse v20170619-49-gb11e293; 2018-09-08 go1.10.3
WRITE: 262144000 bytes (262 MB, 250 MiB) copied, 1.07979 s, 243 MB/s
READ: 262144000 bytes (262 MB, 250 MiB) copied, 0.882413 s, 297 MB/s
UNTAR: 16.703
MD5: 7.606
LS: 1.349
RM: 3.237
After
$ ./benchmark.bash
Testing gocryptfs at /tmp/benchmark.bash.jK3: gocryptfs v1.6-13-g84d6faf-dirty; go-fuse v20170619-49-gb11e293; 2018-09-08 go1.10.3
WRITE: 262144000 bytes (262 MB, 250 MiB) copied, 1.06261 s, 247 MB/s
READ: 262144000 bytes (262 MB, 250 MiB) copied, 0.947228 s, 277 MB/s
UNTAR: 17.197
MD5: 7.540
LS: 1.364
RM: 3.410
The function used to do two things:
1) Walk the directory tree in a manner safe from symlink attacks
2) Open the final component in the mode requested by the caller
This change drops (2), which was only used once, and lets the caller
handle it. This simplifies the function and makes it fit for reuse in
forward mode in openBackingPath(), and for using O_PATH on Linux.
These were silently ignored until now (!) but
are rejected by Go 1.11 stdlib.
Drop the flags so the tests work again, until
we figure out a better solution.
https://github.com/golang/go/issues/20130
Show enable_trezor in the version string if we were compiled
with `-tags enable_trezor`. And hide the `-trezor` flag from
the help output if we were not.
As uncovered by xfstests generic/465, concurrent reads and writes
could lead to this,
doRead 3015532: corrupt block #1039: stupidgcm: message authentication failed,
as the read could pick up a block that has not yet been completely written -
write() is not atomic!
Now writes take ContentLock exclusively, while reads take it shared,
meaning that multiple reads can run in parallel with each other, but
not with a write.
This also simplifies the file header locking.
xfstests generic/083 fills the filesystem almost completely while
running fsstress in parallel. In fsck, these would show up:
readFileID 2580: incomplete file, got 18 instead of 19 bytes
This could happen when writing the file header works, but writing
the actual data fails.
Now we kill the header again by truncating the file to zero.
If the underlying filesystem is full, writing to gocryptfs.diriv may
fail, and later fsck show this:
OpenDir "xyz": could not read gocryptfs.diriv: wanted 16 bytes, got 0
Uncovered by xfstests generic/083.
Also fixes a fd leak in the error path.
If the underlying filesystem is full, it is normal get ENOSPC here.
Log at Info level instead of Warning.
Fixes xfstests generic/015 and generic/027, which complained about
the extra output.
O_DIRECT accesses must be aligned in both offset and length. Due to our
crypto header, alignment will be off, even if userspace makes aligned
accesses. Running xfstests generic/013 on ext4 used to trigger lots of
EINVAL errors due to missing alignment. Just fall back to buffered IO.
The trezor libraries are not yet stable enough to build
gocryptfs with trezor support by default.
It does not even compile at the moment:
$ ./build.bash -tags enable_trezor
# github.com/conejoninja/tesoro/vendor/github.com/trezor/usbhid
../../conejoninja/tesoro/vendor/github.com/trezor/usbhid/hid.go:32:11: fatal error: os/threads_posix.c: No such file or directory
#include "os/threads_posix.c"
^~~~~~~~~~~~~~~~~~~~
compilation terminated.
https://github.com/conejoninja/tesoro/issues/9
"gocryptfs -fsck" will need access to helper functions,
and to get that, it will need to cast a gofuse.File to a
fusefrontend.File. Make fusefrontend.File exported to make
this work.
TrezorPayload stores 32 random bytes used for unlocking
the master key using a Trezor security module. The randomness makes sure
that a unique unlock value is used for each gocryptfs filesystem.
We are clean again.
Warnings were:
internal/fusefrontend/fs.go:443:14: should omit type string from declaration
of var cTarget; it will be inferred from the right-hand side
internal/fusefrontend/xattr.go:26:1: comment on exported method FS.GetXAttr
should be of the form "GetXAttr ..."
internal/syscallcompat/sys_common.go:9:7: exported const PATH_MAX should have
comment or be unexported
Reading system.posix_acl_access and system.posix_acl_default
should return EOPNOTSUPP to inform user-space that we do not
support ACLs.
xftestest essientially does
chacl -l | grep "Operation not supported"
to determine if the filesystem supports ACLs, and used to
wrongly believe that gocryptfs does.
mv is unhappy when we return EPERM when it tries to set
system.posix_acl_access:
mv: preserving permissions for ‘b/x’: Operation not permitted
Now we return EOPNOTSUPP like tmpfs does and mv seems happy.
Values a binary-safe, there is no need to base64-encode them.
Old, base64-encoded values are supported transparently
on reading. Writing xattr values now always writes them binary.
We previously returned EPERM to prevent the kernel from
blacklisting our xattr support once we get an unsupported
flag, but this causes lots of trouble on MacOS:
Cannot save files from GUI apps, see
https://github.com/rfjakob/gocryptfs/issues/229
Returning ENOSYS triggers the dotfiles fallback on MacOS
and fixes the issue.
OpenDir and ListXAttr skip over corrupt entries,
readFileID treats files the are too small as empty.
This improves usability in the face of corruption,
but hides the problem in a log message instead of
putting it in the return code.
Create a channel to report these corruptions to fsck
so it can report them to the user.
Also update the manpage and the changelog with the -fsck option.
Closes https://github.com/rfjakob/gocryptfs/issues/191
Most corruption cases except xattr should be covered.
With test filesystem.
The output is still pretty ugly. xattr support will
be added in the next commits.
This should not happen via FUSE as the kernel caps the size,
but with fsck we have the first user that calls Read directly.
For symmetry, check it for Write as well.
A few places have called tlog.Warn.Print, which directly
calls into log.Logger due to embedding, losing all features
of tlog.
Stop embedding log.Logger to make sure the internal functions
cannot be called accidentially and fix (several!) instances
that did.
We now print the number in a debug message, so define
the numeric values explicitely instead of using iota.
This way you don't have to understand how iota works
to find out what the number means. Lack of understanding
of how iota works is also the reason why the numbers
start at 3 (to keep the current behavoir).
Overwrite the masterkey with zeros once we
have encrypted it, and let it run out of scope.
Also get rid of the password duplicate in
readpassword.Twice.
As soon as we don't need them anymore, overwrite
keys with zeros and make sure they run out of scope
so we don't create a risk of inadvertedly using all-zero
keys for encryption.
https://github.com/rfjakob/gocryptfs/issues/211
Both fusefrontend and fusefrontend_reverse were doing
essentially the same thing, move it into main's
initFuseFrontend.
A side-effect is that we have a reference to cryptocore
in main, which will help with wiping the keys on exit
(https://github.com/rfjakob/gocryptfs/issues/211).
$ go.gcc build
# github.com/rfjakob/gocryptfs/internal/syscallcompat
internal/syscallcompat/unix2syscall_linux.go:32:13: error: incompatible types in assignment (cannot use type int64 as type syscall.Timespec_sec_t)
s.Atim.Sec = u.Atim.Sec
^
On mips64le, syscall.Getdents() and struct syscall.Dirent do
not fit together, causing our Getdents implementation to
return garbage ( https://github.com/rfjakob/gocryptfs/issues/200
and https://github.com/golang/go/issues/23624 ).
Switch to unix.Getdents which does not have this problem -
the next Go release with the syscall package fixes is too
far away, and will take time to trickle into distros.
Steps to reproduce:
* Create a regular reverse mount point
* Create a file "test" in the original directory
* Access the corresponding encrypted directory in the mount point (ls <encrypted dir>)
* Quickly delete the file in the original data - instead create a device node
* Access the file again, it will access the device node and attempt to read from it
Fixes https://github.com/rfjakob/gocryptfs/issues/187
Unfortunately, faccessat in Linux ignores AT_SYMLINK_NOFOLLOW,
so this is not completely atomic.
Given that the information you get from access is not very
interesting, it seems good enough.
https://github.com/rfjakob/gocryptfs/issues/165
Add faccessat(2) with a hack for symlink, because the
kernel does not actually looks at the passed flags.
From man 2 faccessat:
C library/kernel differences
The raw faccessat() system call takes only the first three argu‐
ments. The AT_EACCESS and AT_SYMLINK_NOFOLLOW flags are actually
implemented within the glibc wrapper function for faccessat().
...when opening intermedia directories to give us an
extra layer of safety.
From the FreeBSD man page:
This flag can be used to prevent applications with elevated
privileges from opening files which are even unsafe to open with O_RDONLY,
such as device nodes.
...by using the new OpenNofollow helper.
The benchmark shows a small but acceptable performance loss:
$ ./benchmark-reverse.bash
LS: 2.182
CAT: 18.221
Tracking ticket: https://github.com/rfjakob/gocryptfs/issues/165
Now that we have Fstatat we can use it in Getdents to
get rid of the path name.
Also, add an emulated version of getdents for MacOS. This allows
to drop the !HaveGetdents special cases from fusefrontend.
Modify the getdents test to test both native getdents and the emulated
version.
