This adds support for gitignore-like wildcards and exclude patters in
reverse mode. It (somewhat) fixes#273: no regexp support, but the
syntax should be powerful enough to satisfy most needs.
Also, since adding a lot of --exclude options can be tedious, it adds
the --exclude-from option to read patterns from a file (or files).
Trezor support has been broken since Sept 2018
( https://github.com/rfjakob/gocryptfs/issues/261 ).
Disable trezor.go by renaming to trezor.go.broken.
This keeps "dep" from having to pull in A LOT OF dependencies:
Before:
$ du -sh vendor/
49M vendor/
After:
$ du -sh vendor/
16M vendor/
This fixed the "Permission denied" bug, but still has the problem that
the directory may be replaced behind our back. Mitigated by the fact
that we skip the workaround when running as root with -allow_other.
https://github.com/rfjakob/gocryptfs/issues/354
In the error case, buf was not restored to the original
capacity. Instead of truncating "buf" and restoring (or forgetting to restore)
later, introduce the "data" slice.
Fixes https://github.com/rfjakob/gocryptfs/issues/356
For Linux, everything effectively stays the same. For both path-based and
fd-based Utimens() calls, we use unix.UtimesNanoAt(). To avoid introducing
a separate syscall wrapper for futimens() (as done in go-fuse, for example),
we instead use the /proc/self/fd - trick.
On macOS, this changes quite a lot:
* Path-based Utimens() calls were previously completely broken, since
unix.UtimensNanoAt() ignores the passed file descriptor. Note that this
cannot be fixed easily since there IS no appropriate syscall available on
macOS prior to High Sierra (10.13). We emulate this case by using
Fchdir() + setattrlist().
* Fd-based Utimens() calls were previously translated to f.GetAttr() (to
fill any empty parameters) and syscall.Futimes(), which does not does
support nanosecond precision. Both issues can be fixed by switching to
fsetattrlist().
Fixes https://github.com/rfjakob/gocryptfs/issues/350
This should get rid of
Openat: O_NOFOLLOW missing: flags = 0x0
Fchmodat: adding missing AT_SYMLINK_NOFOLLOW flag
sys_common_test.go:203: chmod on symlink should have failed, but did not. New mode=0333
UnmountErr: "[...]/057376762.mnt" was not found in MountInfo, cannot check for FD leak
and add some context to
--- FAIL: TestUtimesNano (0.00s)
matrix_test.go:628: no such file or directory
See https://github.com/rfjakob/gocryptfs/pull/343#issuecomment-453888006
for full test output
The only call forwarded to loopbackFileSystem was Statfs,
which is trivial to implement.
Implement it and drop loopbackFileSystem, as having it carries the
risk that a coding error bypasses the usual encryption/decryption
chain.
Instead of manually adjusting the user after creating the symlink,
adjust effective permissions and let the kernel deal with it.
Related to https://github.com/rfjakob/gocryptfs/issues/338.
Instead of manually adjusting the user and mode after creating the
device file, adjust effective permissions and let the kernel deal
with it.
Related to https://github.com/rfjakob/gocryptfs/issues/338.
The current code has a risk of race-conditions, since we pass a path
containing "/" to Fchownat. We could fix this by opening a file descriptor,
however, this does not seem worth the effort. We also don't chown *.name files.
Make sure that the directory belongs to the correct owner before users
can access it. For directories with SUID/SGID mode, there is a risk of
race-conditions when files are created before the correct owner is set.
They will then inherit the wrong user and/or group.
See https://github.com/rfjakob/gocryptfs/issues/327 for more details.
Reported by @slackner at https://github.com/rfjakob/gocryptfs/issues/327 :
Possible race-conditions between file creation and Fchownat
* Assume a system contains a gocryptfs mount as root user
with -allow_other
* As a regular user create a new file with mode containing
the SUID flag and write access for other users
* Before gocryptfs executes the Fchownat call, try to open
the file again, write some exploit code to it, and try to run it.
For a short time, the file is owned by root and has the SUID flag, so
this is pretty dangerous.
The files are apparently processed in alphabetic order, so cli_args.go is
processed before main.go. In order to run before the go-fuse imports, put
the 'ensure fds' code in a separate package. Debug messages are omitted
to avoid additional imports (that might contain other code messing up our
file descriptors).
Setting/removing extended attributes on directories was partially fixed with
commit eff35e60b6. However, on most file systems
it is also possible to do these operations without read access (see tests).
Since we cannot open a write-access fd to a directory, we have to use the
/proc/self/fd trick (already used for ListXAttr) for the other operations aswell.
For simplicity, let's separate the Linux and Darwin code again (basically revert
commit f320b76fd1), and always use the
/proc/self/fd trick on Linux. On Darwin we use the best-effort approach with
openBackingFile() as a fallback.
More discussion about the available options is available in
https://github.com/rfjakob/gocryptfs/issues/308.
Otherwise this can happen, as triggered by xfstests generic/011:
go-fuse: can't convert error type: openat failed: too many open files
The app then gets a misleading "Function not implemented" error.
We alread have this warning in Open(), but xfstests generic/488
causes "too many open files" via Create. Add the same message so
the user sees what is going on.
Go version go1.10.7 linux/amd64 complains with:
internal/fusefrontend_reverse/rfs.go:333: declaration of "entries" shadows
declaration at internal/fusefrontend_reverse/rfs.go:327
Fixes https://github.com/rfjakob/gocryptfs/issues/286 :
While the actual file is properly excluded, the * .name file is still leaked in the directory listing:
```
drwxr-xr-x 2 sebastian sebastian 4,0K Dez 17 14:58 .
drwxr-xr-x 7 sebastian sebastian 4,0K Dez 17 14:45 ..
-r-------- 1 sebastian sebastian 408 Dez 17 14:56 gocryptfs.conf
-r--r--r-- 1 sebastian sebastian 16 Dez 17 14:58 gocryptfs.diriv
-r--r--r-- 1 sebastian sebastian 320 Dez 17 14:58 gocryptfs.longname.3vZ_r3eDPb1_fL3j5VA4rd_bcKWLKT9eaxOVIGK5HFA.name
```
Excluded files showed up in directory listing like this:
drwxr-xr-x 2 sebastian sebastian 4,0K Dez 17 14:48 .
drwxr-xr-x 7 sebastian sebastian 4,0K Dez 17 14:45 ..
-????????? ? ? ? ? ? abcd
-r-------- 1 sebastian sebastian 366 Dez 17 14:45 gocryptfs.conf
Fixes https://github.com/rfjakob/gocryptfs/issues/285
When the old size is zero, there are no existing blocks to merge the
new data with. Directly use Ftruncate if the size is block-aligned.
Fixes https://github.com/rfjakob/gocryptfs/issues/305
Bug looked like this:
$ ls -l .
total 0
drwxrwxr-x. 2 jakob jakob 60 Jan 3 15:42 foo
-rw-rw-r--. 1 jakob jakob 0 Jan 3 15:46 x
$ ls -l .
ls: cannot access '.': No such file or directory
(only happened when "" was in the dirCache)
This function is in all fastpaths, will get a cache, and needs
its own file.
renamed: internal/fusefrontend/names.go -> internal/fusefrontend/openbackingdir.go
renamed: internal/fusefrontend/names_test.go -> internal/fusefrontend/openbackingdir_test.go
An Open() a fifo blocks until it is opened for writing.
This meant that xattr operations on FIFOs would block.
Pass O_NONBLOCK to fix that, and add a test.
Failure was:
+ GOOS=darwin
+ GOARCH=amd64
+ go build -tags without_openssl
# github.com/rfjakob/gocryptfs/internal/fusefrontend
internal/fusefrontend/fs_dir.go:159:60: cannot use origMode | 448 (type uint16) as type uint32 in argument to syscallcompat.Fchmodat
internal/fusefrontend/fs_dir.go:170:33: cannot use origMode (type uint16) as type uint32 in argument to syscallcompat.Fchmodat
Use openBackingDir() and Fstatat().
High performance impact, though part of it should be
mitigated by adding DirIV caching to the new code paths.
$ ./benchmark.bash
Testing gocryptfs at /tmp/benchmark.bash.Eou: gocryptfs v1.6-37-ge3914b3-dirty; go-fuse v20170619-66-g6df8ddc; 2018-10-14 go1.11
WRITE: 262144000 bytes (262 MB, 250 MiB) copied, 1.2289 s, 213 MB/s
READ: 262144000 bytes (262 MB, 250 MiB) copied, 1.02616 s, 255 MB/s
UNTAR: 24.490
MD5: 13.120
LS: 3.368
RM: 9.232
The directory was already created, so return success even if Fchownat fails.
The same error handling is already used if fs.args.PlaintextNames is false.
Old XFS filesystems always return DT_UNKNOWN. Downgrade the message
to "info" level if we are on XFS.
Using the "warning" level means that users on old XFS filesystems
cannot run the test suite as it intentionally aborts on any
warnings.
Fixes https://github.com/rfjakob/gocryptfs/issues/267
When gocryptfs was started on a terminal and later
daemonized, the color codes stayed active in the syslog
output.
The codes are not visible in "journalctl -f", which is why
I have not noticed it yet, but they do show up in normal
syslog as the usual "#033[33m" crap.
Even though filesystem notifications aren't implemented for FUSE, I decided to
try my hand at implementing the autounmount feature (#128). I based it on the
EncFS autounmount code, which records filesystem accesses and checks every X
seconds whether it's idled long enough to unmount.
I've tested the feature locally, but I haven't added any tests for this flag.
I also haven't worked with Go before. So please let me know if there's
anything that should be done differently.
One particular concern: I worked from the assumption that the open files table
is unique per-filesystem. If that's not true, I'll need to add an open file
count and associated lock to the Filesystem type instead.
https://github.com/rfjakob/gocryptfs/pull/265
Error was:
# github.com/rfjakob/gocryptfs/internal/fusefrontend
internal/fusefrontend/fs.go:179: cannot use perms | 256 (type uint16) as type uint32 in argument to syscall.Fchmod
internal/fusefrontend/fs.go:185: cannot use perms (type uint16) as type uint32 in argument to syscall.Fchmod
Rename openBackingPath to openBackingDir and use OpenDirNofollow
to be safe against symlink races. Note that openBackingDir is
not used in several important code paths like Create().
But it is used in Unlink, and the performance impact in the RM benchmark
to be acceptable:
Before
$ ./benchmark.bash
Testing gocryptfs at /tmp/benchmark.bash.bYO: gocryptfs v1.6-12-g930c37e-dirty; go-fuse v20170619-49-gb11e293; 2018-09-08 go1.10.3
WRITE: 262144000 bytes (262 MB, 250 MiB) copied, 1.07979 s, 243 MB/s
READ: 262144000 bytes (262 MB, 250 MiB) copied, 0.882413 s, 297 MB/s
UNTAR: 16.703
MD5: 7.606
LS: 1.349
RM: 3.237
After
$ ./benchmark.bash
Testing gocryptfs at /tmp/benchmark.bash.jK3: gocryptfs v1.6-13-g84d6faf-dirty; go-fuse v20170619-49-gb11e293; 2018-09-08 go1.10.3
WRITE: 262144000 bytes (262 MB, 250 MiB) copied, 1.06261 s, 247 MB/s
READ: 262144000 bytes (262 MB, 250 MiB) copied, 0.947228 s, 277 MB/s
UNTAR: 17.197
MD5: 7.540
LS: 1.364
RM: 3.410
The function used to do two things:
1) Walk the directory tree in a manner safe from symlink attacks
2) Open the final component in the mode requested by the caller
This change drops (2), which was only used once, and lets the caller
handle it. This simplifies the function and makes it fit for reuse in
forward mode in openBackingPath(), and for using O_PATH on Linux.
These were silently ignored until now (!) but
are rejected by Go 1.11 stdlib.
Drop the flags so the tests work again, until
we figure out a better solution.
https://github.com/golang/go/issues/20130
Show enable_trezor in the version string if we were compiled
with `-tags enable_trezor`. And hide the `-trezor` flag from
the help output if we were not.
As uncovered by xfstests generic/465, concurrent reads and writes
could lead to this,
doRead 3015532: corrupt block #1039: stupidgcm: message authentication failed,
as the read could pick up a block that has not yet been completely written -
write() is not atomic!
Now writes take ContentLock exclusively, while reads take it shared,
meaning that multiple reads can run in parallel with each other, but
not with a write.
This also simplifies the file header locking.
xfstests generic/083 fills the filesystem almost completely while
running fsstress in parallel. In fsck, these would show up:
readFileID 2580: incomplete file, got 18 instead of 19 bytes
This could happen when writing the file header works, but writing
the actual data fails.
Now we kill the header again by truncating the file to zero.
If the underlying filesystem is full, writing to gocryptfs.diriv may
fail, and later fsck show this:
OpenDir "xyz": could not read gocryptfs.diriv: wanted 16 bytes, got 0
Uncovered by xfstests generic/083.
Also fixes a fd leak in the error path.
If the underlying filesystem is full, it is normal get ENOSPC here.
Log at Info level instead of Warning.
Fixes xfstests generic/015 and generic/027, which complained about
the extra output.
O_DIRECT accesses must be aligned in both offset and length. Due to our
crypto header, alignment will be off, even if userspace makes aligned
accesses. Running xfstests generic/013 on ext4 used to trigger lots of
EINVAL errors due to missing alignment. Just fall back to buffered IO.
The trezor libraries are not yet stable enough to build
gocryptfs with trezor support by default.
It does not even compile at the moment:
$ ./build.bash -tags enable_trezor
# github.com/conejoninja/tesoro/vendor/github.com/trezor/usbhid
../../conejoninja/tesoro/vendor/github.com/trezor/usbhid/hid.go:32:11: fatal error: os/threads_posix.c: No such file or directory
#include "os/threads_posix.c"
^~~~~~~~~~~~~~~~~~~~
compilation terminated.
https://github.com/conejoninja/tesoro/issues/9
"gocryptfs -fsck" will need access to helper functions,
and to get that, it will need to cast a gofuse.File to a
fusefrontend.File. Make fusefrontend.File exported to make
this work.
TrezorPayload stores 32 random bytes used for unlocking
the master key using a Trezor security module. The randomness makes sure
that a unique unlock value is used for each gocryptfs filesystem.
We are clean again.
Warnings were:
internal/fusefrontend/fs.go:443:14: should omit type string from declaration
of var cTarget; it will be inferred from the right-hand side
internal/fusefrontend/xattr.go:26:1: comment on exported method FS.GetXAttr
should be of the form "GetXAttr ..."
internal/syscallcompat/sys_common.go:9:7: exported const PATH_MAX should have
comment or be unexported
Reading system.posix_acl_access and system.posix_acl_default
should return EOPNOTSUPP to inform user-space that we do not
support ACLs.
xftestest essientially does
chacl -l | grep "Operation not supported"
to determine if the filesystem supports ACLs, and used to
wrongly believe that gocryptfs does.
mv is unhappy when we return EPERM when it tries to set
system.posix_acl_access:
mv: preserving permissions for ‘b/x’: Operation not permitted
Now we return EOPNOTSUPP like tmpfs does and mv seems happy.
Values a binary-safe, there is no need to base64-encode them.
Old, base64-encoded values are supported transparently
on reading. Writing xattr values now always writes them binary.
We previously returned EPERM to prevent the kernel from
blacklisting our xattr support once we get an unsupported
flag, but this causes lots of trouble on MacOS:
Cannot save files from GUI apps, see
https://github.com/rfjakob/gocryptfs/issues/229
Returning ENOSYS triggers the dotfiles fallback on MacOS
and fixes the issue.
OpenDir and ListXAttr skip over corrupt entries,
readFileID treats files the are too small as empty.
This improves usability in the face of corruption,
but hides the problem in a log message instead of
putting it in the return code.
Create a channel to report these corruptions to fsck
so it can report them to the user.
Also update the manpage and the changelog with the -fsck option.
Closes https://github.com/rfjakob/gocryptfs/issues/191
Most corruption cases except xattr should be covered.
With test filesystem.
The output is still pretty ugly. xattr support will
be added in the next commits.
This should not happen via FUSE as the kernel caps the size,
but with fsck we have the first user that calls Read directly.
For symmetry, check it for Write as well.
A few places have called tlog.Warn.Print, which directly
calls into log.Logger due to embedding, losing all features
of tlog.
Stop embedding log.Logger to make sure the internal functions
cannot be called accidentially and fix (several!) instances
that did.
We now print the number in a debug message, so define
the numeric values explicitely instead of using iota.
This way you don't have to understand how iota works
to find out what the number means. Lack of understanding
of how iota works is also the reason why the numbers
start at 3 (to keep the current behavoir).
Overwrite the masterkey with zeros once we
have encrypted it, and let it run out of scope.
Also get rid of the password duplicate in
readpassword.Twice.
As soon as we don't need them anymore, overwrite
keys with zeros and make sure they run out of scope
so we don't create a risk of inadvertedly using all-zero
keys for encryption.
https://github.com/rfjakob/gocryptfs/issues/211
Both fusefrontend and fusefrontend_reverse were doing
essentially the same thing, move it into main's
initFuseFrontend.
A side-effect is that we have a reference to cryptocore
in main, which will help with wiping the keys on exit
(https://github.com/rfjakob/gocryptfs/issues/211).
$ go.gcc build
# github.com/rfjakob/gocryptfs/internal/syscallcompat
internal/syscallcompat/unix2syscall_linux.go:32:13: error: incompatible types in assignment (cannot use type int64 as type syscall.Timespec_sec_t)
s.Atim.Sec = u.Atim.Sec
^
On mips64le, syscall.Getdents() and struct syscall.Dirent do
not fit together, causing our Getdents implementation to
return garbage ( https://github.com/rfjakob/gocryptfs/issues/200
and https://github.com/golang/go/issues/23624 ).
Switch to unix.Getdents which does not have this problem -
the next Go release with the syscall package fixes is too
far away, and will take time to trickle into distros.
Steps to reproduce:
* Create a regular reverse mount point
* Create a file "test" in the original directory
* Access the corresponding encrypted directory in the mount point (ls <encrypted dir>)
* Quickly delete the file in the original data - instead create a device node
* Access the file again, it will access the device node and attempt to read from it
Fixes https://github.com/rfjakob/gocryptfs/issues/187
Unfortunately, faccessat in Linux ignores AT_SYMLINK_NOFOLLOW,
so this is not completely atomic.
Given that the information you get from access is not very
interesting, it seems good enough.
https://github.com/rfjakob/gocryptfs/issues/165
Add faccessat(2) with a hack for symlink, because the
kernel does not actually looks at the passed flags.
From man 2 faccessat:
C library/kernel differences
The raw faccessat() system call takes only the first three argu‐
ments. The AT_EACCESS and AT_SYMLINK_NOFOLLOW flags are actually
implemented within the glibc wrapper function for faccessat().
...when opening intermedia directories to give us an
extra layer of safety.
From the FreeBSD man page:
This flag can be used to prevent applications with elevated
privileges from opening files which are even unsafe to open with O_RDONLY,
such as device nodes.
...by using the new OpenNofollow helper.
The benchmark shows a small but acceptable performance loss:
$ ./benchmark-reverse.bash
LS: 2.182
CAT: 18.221
Tracking ticket: https://github.com/rfjakob/gocryptfs/issues/165
Now that we have Fstatat we can use it in Getdents to
get rid of the path name.
Also, add an emulated version of getdents for MacOS. This allows
to drop the !HaveGetdents special cases from fusefrontend.
Modify the getdents test to test both native getdents and the emulated
version.
