Changed invalid file decoding and decryption. Function
DecryptName now shortens the filename until the filename is
decodable and decryptable. Will work with valid **and**
invalid Base64URL delimiter (valid delimiter [0-9a-zA-z_\\-].
If the filename is not decryptable at all, it returns the
original cipher name with flag suffix Changed cli tests to
generate decryptable and undecryptable file names with correct
encrypted content. Replacing #474, extends #393
On CIFS mounts, unix.Getdents can return sudden ENOENT
in the middle of data. This will not be reported as an error
by user space tools, so return EIO instead.
Also log it as a warning.
https://github.com/rfjakob/gocryptfs/issues/483
When filename encryption is on, we do know when we
overwrite a directory, and can clear only in this case.
sshfs-benchmark.bash: sshfs gocryptfs-on-sshfs
git init 1.74 7.80
rsync 6.19 11.63
Mkdir can not cause existing entries in the cache to go
stale. So don't clear it. Benchmark results:
sshfs-benchmark.bash: sshfs gocryptfs-on-sshfs
git init 1.65 8.74
rsync 6.09 17.54
Looking at the dircache debug output, we see
that a "git status" workload has a very bad
cache hit rate because the entries expire or
get evicted before they can be reused.
Increase both cache size and lifetime for
a 4x speedup:
Before: 75s
After: 17s
https://github.com/rfjakob/gocryptfs/issues/410
We need
fd7328faf9
to fix a crash reported in https://github.com/rfjakob/gocryptfs/issues/430 :
2019/10/30 17:14:16 Unknown opcode 2016
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x508d38]
This patch is only in the v2.x.x branch. Upgrade to v2, as the
old API is also supported there.
Running
git grep hanwen/go-fuse | grep -v hanwen/go-fuse/v2
to check for forgotten references comes back clean.
Adding flags allows to use inomap in reverse mode,
replacing the clunky inoBaseDirIV/inoBaseNameFile
logic that causes problems with high underlying
inode numbers ( https://github.com/rfjakob/gocryptfs/issues/457 )
Microbenchmarks (values below) show that the "SingleDev"
case is now much slower due to an extra map lookup,
but this has no visible effects in ./test.bash results,
so there was no time spent optimizing the case further.
$ go test -bench=.
goos: linux
goarch: amd64
pkg: github.com/rfjakob/gocryptfs/internal/inomap
BenchmarkTranslateSingleDev-4 18757510 61.5 ns/op
BenchmarkTranslateManyDevs-4 18061515 64.5 ns/op
PASS
ok github.com/rfjakob/gocryptfs/internal/inomap 2.467s
The comment still mentioned CBC, which has been removed
a long time ago.
The test definition can be rewritten using slice literals,
saving sume stuttering.
We used to prefer openssl in this situation, which
used to make sense, but now Go gained an optimized
assembly implementation for aes-gcm on arm64 with
aes instructions:
root@q1:~/go/src/github.com/rfjakob/gocryptfs# ./gocryptfs -speed
gocryptfs v1.7.1-46-g73436d9; go-fuse v1.0.1-0.20190319092520-161a16484456; 2020-04-13 go1.14.2 linux/arm64
AES-GCM-256-OpenSSL 212.30 MB/s (selected in auto mode)
AES-GCM-256-Go 452.30 MB/s
AES-SIV-512-Go 100.25 MB/s
XChaCha20-Poly1305-Go 137.35 MB/s
https://github.com/rfjakob/gocryptfs/issues/452
We used to restrict setting xattrs to the "user."
namespace. I don't see a real reason for this
anymore, and it causes trouble for users who are using
acls.
Tests will be added in the next commit.
https://github.com/rfjakob/gocryptfs/issues/453
Now that I have discovered golang.org/x/sys/cpu and that Go
versions below 1.6 are uncommon, there was not much useful
code left in prefer_openssl.
Merge the remains into stupidgcm.
Error from Travis CI was:
+GOOS=darwin
+GOARCH=amd64
+go build -tags without_openssl
# github.com/rfjakob/gocryptfs/internal/fusefrontend
internal/fusefrontend/fs.go:88:45: cannot use st.Dev (type int32) as type uint64 in argument to openfiletable.NewInumMap
Add uint64 to fix it.
The comment is outdated, at this point, we should
really not get any errors from ReadDirIVAt.
The change is best seen when running the fsck tests. Before:
fsck: error opening dir "missing_diriv": 2=no such file or directory
After:
OpenDir "K2m0E6qzIfoLkVZJanoUiQ": could not read gocryptfs.diriv: no such file or directory
fsck: error opening dir "missing_diriv": 5=input/output error
See https://github.com/rfjakob/gocryptfs/issues/403 , where
the extra info would have been helpful.
This was meant as a way to inform the user that
something is very wrong, however, users are hitting
the condition on MacOS due to ".DS_Store" files, and
also on NFS due to ".nfsXXX" files.
Drop the whole thing as it seems to cause more pain
than gain.
Fixes https://github.com/rfjakob/gocryptfs/issues/431
Closing the fd means the inode number may be reused immediately
by a new file, so we have to get the old fileID out of the table
beforehand!
Hopefully fixes https://github.com/rfjakob/gocryptfs/issues/363
Error was
+GOOS=darwin
+GOARCH=amd64
+go build -tags without_openssl
# github.com/rfjakob/gocryptfs/internal/syscallcompat
internal/syscallcompat/unix2syscall_darwin.go:22:32: u.Atimespec undefined (type unix.Stat_t has no field or method Atimespec)
internal/syscallcompat/unix2syscall_darwin.go:23:32: u.Mtimespec undefined (type unix.Stat_t has no field or method Mtimespec)
internal/syscallcompat/unix2syscall_darwin.go:24:32: u.Ctimespec undefined (type unix.Stat_t has no field or method Ctimespec)
caused by 87c872767d (diff-4913a9178621eadcdf191db17915fbcb)
When we run tests as root, they will leave root-owned files
in testParentDir, which causes trouble when we run tests as
a normal user later on. Split by UID.
CheckTrailingGarbage was called even when "-passfile" was
used, which is stupid, and causes false positives:
https://github.com/rfjakob/gocryptfs/issues/391
(false error "Received trailing garbage after the password"
when using -passfile in .bash_profile)
Instead of trying to improve the logic to handle that case
and make everything even more complicated, delete the function.
It is unclear if actually helps in some cases, and it definitely
harms as shown by the above bug report.
This adds support for gitignore-like wildcards and exclude patters in
reverse mode. It (somewhat) fixes#273: no regexp support, but the
syntax should be powerful enough to satisfy most needs.
Also, since adding a lot of --exclude options can be tedious, it adds
the --exclude-from option to read patterns from a file (or files).
Trezor support has been broken since Sept 2018
( https://github.com/rfjakob/gocryptfs/issues/261 ).
Disable trezor.go by renaming to trezor.go.broken.
This keeps "dep" from having to pull in A LOT OF dependencies:
Before:
$ du -sh vendor/
49M vendor/
After:
$ du -sh vendor/
16M vendor/
This fixed the "Permission denied" bug, but still has the problem that
the directory may be replaced behind our back. Mitigated by the fact
that we skip the workaround when running as root with -allow_other.
https://github.com/rfjakob/gocryptfs/issues/354
In the error case, buf was not restored to the original
capacity. Instead of truncating "buf" and restoring (or forgetting to restore)
later, introduce the "data" slice.
Fixes https://github.com/rfjakob/gocryptfs/issues/356
For Linux, everything effectively stays the same. For both path-based and
fd-based Utimens() calls, we use unix.UtimesNanoAt(). To avoid introducing
a separate syscall wrapper for futimens() (as done in go-fuse, for example),
we instead use the /proc/self/fd - trick.
On macOS, this changes quite a lot:
* Path-based Utimens() calls were previously completely broken, since
unix.UtimensNanoAt() ignores the passed file descriptor. Note that this
cannot be fixed easily since there IS no appropriate syscall available on
macOS prior to High Sierra (10.13). We emulate this case by using
Fchdir() + setattrlist().
* Fd-based Utimens() calls were previously translated to f.GetAttr() (to
fill any empty parameters) and syscall.Futimes(), which does not does
support nanosecond precision. Both issues can be fixed by switching to
fsetattrlist().
Fixes https://github.com/rfjakob/gocryptfs/issues/350
This should get rid of
Openat: O_NOFOLLOW missing: flags = 0x0
Fchmodat: adding missing AT_SYMLINK_NOFOLLOW flag
sys_common_test.go:203: chmod on symlink should have failed, but did not. New mode=0333
UnmountErr: "[...]/057376762.mnt" was not found in MountInfo, cannot check for FD leak
and add some context to
--- FAIL: TestUtimesNano (0.00s)
matrix_test.go:628: no such file or directory
See https://github.com/rfjakob/gocryptfs/pull/343#issuecomment-453888006
for full test output
The only call forwarded to loopbackFileSystem was Statfs,
which is trivial to implement.
Implement it and drop loopbackFileSystem, as having it carries the
risk that a coding error bypasses the usual encryption/decryption
chain.
Instead of manually adjusting the user after creating the symlink,
adjust effective permissions and let the kernel deal with it.
Related to https://github.com/rfjakob/gocryptfs/issues/338.
Instead of manually adjusting the user and mode after creating the
device file, adjust effective permissions and let the kernel deal
with it.
Related to https://github.com/rfjakob/gocryptfs/issues/338.
The current code has a risk of race-conditions, since we pass a path
containing "/" to Fchownat. We could fix this by opening a file descriptor,
however, this does not seem worth the effort. We also don't chown *.name files.
Make sure that the directory belongs to the correct owner before users
can access it. For directories with SUID/SGID mode, there is a risk of
race-conditions when files are created before the correct owner is set.
They will then inherit the wrong user and/or group.
See https://github.com/rfjakob/gocryptfs/issues/327 for more details.
Reported by @slackner at https://github.com/rfjakob/gocryptfs/issues/327 :
Possible race-conditions between file creation and Fchownat
* Assume a system contains a gocryptfs mount as root user
with -allow_other
* As a regular user create a new file with mode containing
the SUID flag and write access for other users
* Before gocryptfs executes the Fchownat call, try to open
the file again, write some exploit code to it, and try to run it.
For a short time, the file is owned by root and has the SUID flag, so
this is pretty dangerous.
The files are apparently processed in alphabetic order, so cli_args.go is
processed before main.go. In order to run before the go-fuse imports, put
the 'ensure fds' code in a separate package. Debug messages are omitted
to avoid additional imports (that might contain other code messing up our
file descriptors).
Setting/removing extended attributes on directories was partially fixed with
commit eff35e60b6. However, on most file systems
it is also possible to do these operations without read access (see tests).
Since we cannot open a write-access fd to a directory, we have to use the
/proc/self/fd trick (already used for ListXAttr) for the other operations aswell.
For simplicity, let's separate the Linux and Darwin code again (basically revert
commit f320b76fd1), and always use the
/proc/self/fd trick on Linux. On Darwin we use the best-effort approach with
openBackingFile() as a fallback.
More discussion about the available options is available in
https://github.com/rfjakob/gocryptfs/issues/308.
Otherwise this can happen, as triggered by xfstests generic/011:
go-fuse: can't convert error type: openat failed: too many open files
The app then gets a misleading "Function not implemented" error.
We alread have this warning in Open(), but xfstests generic/488
causes "too many open files" via Create. Add the same message so
the user sees what is going on.
Go version go1.10.7 linux/amd64 complains with:
internal/fusefrontend_reverse/rfs.go:333: declaration of "entries" shadows
declaration at internal/fusefrontend_reverse/rfs.go:327
Fixes https://github.com/rfjakob/gocryptfs/issues/286 :
While the actual file is properly excluded, the * .name file is still leaked in the directory listing:
```
drwxr-xr-x 2 sebastian sebastian 4,0K Dez 17 14:58 .
drwxr-xr-x 7 sebastian sebastian 4,0K Dez 17 14:45 ..
-r-------- 1 sebastian sebastian 408 Dez 17 14:56 gocryptfs.conf
-r--r--r-- 1 sebastian sebastian 16 Dez 17 14:58 gocryptfs.diriv
-r--r--r-- 1 sebastian sebastian 320 Dez 17 14:58 gocryptfs.longname.3vZ_r3eDPb1_fL3j5VA4rd_bcKWLKT9eaxOVIGK5HFA.name
```
Excluded files showed up in directory listing like this:
drwxr-xr-x 2 sebastian sebastian 4,0K Dez 17 14:48 .
drwxr-xr-x 7 sebastian sebastian 4,0K Dez 17 14:45 ..
-????????? ? ? ? ? ? abcd
-r-------- 1 sebastian sebastian 366 Dez 17 14:45 gocryptfs.conf
Fixes https://github.com/rfjakob/gocryptfs/issues/285
When the old size is zero, there are no existing blocks to merge the
new data with. Directly use Ftruncate if the size is block-aligned.
Fixes https://github.com/rfjakob/gocryptfs/issues/305
Bug looked like this:
$ ls -l .
total 0
drwxrwxr-x. 2 jakob jakob 60 Jan 3 15:42 foo
-rw-rw-r--. 1 jakob jakob 0 Jan 3 15:46 x
$ ls -l .
ls: cannot access '.': No such file or directory
(only happened when "" was in the dirCache)
This function is in all fastpaths, will get a cache, and needs
its own file.
renamed: internal/fusefrontend/names.go -> internal/fusefrontend/openbackingdir.go
renamed: internal/fusefrontend/names_test.go -> internal/fusefrontend/openbackingdir_test.go
An Open() a fifo blocks until it is opened for writing.
This meant that xattr operations on FIFOs would block.
Pass O_NONBLOCK to fix that, and add a test.
Failure was:
+ GOOS=darwin
+ GOARCH=amd64
+ go build -tags without_openssl
# github.com/rfjakob/gocryptfs/internal/fusefrontend
internal/fusefrontend/fs_dir.go:159:60: cannot use origMode | 448 (type uint16) as type uint32 in argument to syscallcompat.Fchmodat
internal/fusefrontend/fs_dir.go:170:33: cannot use origMode (type uint16) as type uint32 in argument to syscallcompat.Fchmodat
Use openBackingDir() and Fstatat().
High performance impact, though part of it should be
mitigated by adding DirIV caching to the new code paths.
$ ./benchmark.bash
Testing gocryptfs at /tmp/benchmark.bash.Eou: gocryptfs v1.6-37-ge3914b3-dirty; go-fuse v20170619-66-g6df8ddc; 2018-10-14 go1.11
WRITE: 262144000 bytes (262 MB, 250 MiB) copied, 1.2289 s, 213 MB/s
READ: 262144000 bytes (262 MB, 250 MiB) copied, 1.02616 s, 255 MB/s
UNTAR: 24.490
MD5: 13.120
LS: 3.368
RM: 9.232
The directory was already created, so return success even if Fchownat fails.
The same error handling is already used if fs.args.PlaintextNames is false.
Old XFS filesystems always return DT_UNKNOWN. Downgrade the message
to "info" level if we are on XFS.
Using the "warning" level means that users on old XFS filesystems
cannot run the test suite as it intentionally aborts on any
warnings.
Fixes https://github.com/rfjakob/gocryptfs/issues/267
When gocryptfs was started on a terminal and later
daemonized, the color codes stayed active in the syslog
output.
The codes are not visible in "journalctl -f", which is why
I have not noticed it yet, but they do show up in normal
syslog as the usual "#033[33m" crap.
Even though filesystem notifications aren't implemented for FUSE, I decided to
try my hand at implementing the autounmount feature (#128). I based it on the
EncFS autounmount code, which records filesystem accesses and checks every X
seconds whether it's idled long enough to unmount.
I've tested the feature locally, but I haven't added any tests for this flag.
I also haven't worked with Go before. So please let me know if there's
anything that should be done differently.
One particular concern: I worked from the assumption that the open files table
is unique per-filesystem. If that's not true, I'll need to add an open file
count and associated lock to the Filesystem type instead.
https://github.com/rfjakob/gocryptfs/pull/265
Error was:
# github.com/rfjakob/gocryptfs/internal/fusefrontend
internal/fusefrontend/fs.go:179: cannot use perms | 256 (type uint16) as type uint32 in argument to syscall.Fchmod
internal/fusefrontend/fs.go:185: cannot use perms (type uint16) as type uint32 in argument to syscall.Fchmod
Rename openBackingPath to openBackingDir and use OpenDirNofollow
to be safe against symlink races. Note that openBackingDir is
not used in several important code paths like Create().
But it is used in Unlink, and the performance impact in the RM benchmark
to be acceptable:
Before
$ ./benchmark.bash
Testing gocryptfs at /tmp/benchmark.bash.bYO: gocryptfs v1.6-12-g930c37e-dirty; go-fuse v20170619-49-gb11e293; 2018-09-08 go1.10.3
WRITE: 262144000 bytes (262 MB, 250 MiB) copied, 1.07979 s, 243 MB/s
READ: 262144000 bytes (262 MB, 250 MiB) copied, 0.882413 s, 297 MB/s
UNTAR: 16.703
MD5: 7.606
LS: 1.349
RM: 3.237
After
$ ./benchmark.bash
Testing gocryptfs at /tmp/benchmark.bash.jK3: gocryptfs v1.6-13-g84d6faf-dirty; go-fuse v20170619-49-gb11e293; 2018-09-08 go1.10.3
WRITE: 262144000 bytes (262 MB, 250 MiB) copied, 1.06261 s, 247 MB/s
READ: 262144000 bytes (262 MB, 250 MiB) copied, 0.947228 s, 277 MB/s
UNTAR: 17.197
MD5: 7.540
LS: 1.364
RM: 3.410
The function used to do two things:
1) Walk the directory tree in a manner safe from symlink attacks
2) Open the final component in the mode requested by the caller
This change drops (2), which was only used once, and lets the caller
handle it. This simplifies the function and makes it fit for reuse in
forward mode in openBackingPath(), and for using O_PATH on Linux.
These were silently ignored until now (!) but
are rejected by Go 1.11 stdlib.
Drop the flags so the tests work again, until
we figure out a better solution.
https://github.com/golang/go/issues/20130
Show enable_trezor in the version string if we were compiled
with `-tags enable_trezor`. And hide the `-trezor` flag from
the help output if we were not.
As uncovered by xfstests generic/465, concurrent reads and writes
could lead to this,
doRead 3015532: corrupt block #1039: stupidgcm: message authentication failed,
as the read could pick up a block that has not yet been completely written -
write() is not atomic!
Now writes take ContentLock exclusively, while reads take it shared,
meaning that multiple reads can run in parallel with each other, but
not with a write.
This also simplifies the file header locking.
xfstests generic/083 fills the filesystem almost completely while
running fsstress in parallel. In fsck, these would show up:
readFileID 2580: incomplete file, got 18 instead of 19 bytes
This could happen when writing the file header works, but writing
the actual data fails.
Now we kill the header again by truncating the file to zero.
If the underlying filesystem is full, writing to gocryptfs.diriv may
fail, and later fsck show this:
OpenDir "xyz": could not read gocryptfs.diriv: wanted 16 bytes, got 0
Uncovered by xfstests generic/083.
Also fixes a fd leak in the error path.
If the underlying filesystem is full, it is normal get ENOSPC here.
Log at Info level instead of Warning.
Fixes xfstests generic/015 and generic/027, which complained about
the extra output.
O_DIRECT accesses must be aligned in both offset and length. Due to our
crypto header, alignment will be off, even if userspace makes aligned
accesses. Running xfstests generic/013 on ext4 used to trigger lots of
EINVAL errors due to missing alignment. Just fall back to buffered IO.
The trezor libraries are not yet stable enough to build
gocryptfs with trezor support by default.
It does not even compile at the moment:
$ ./build.bash -tags enable_trezor
# github.com/conejoninja/tesoro/vendor/github.com/trezor/usbhid
../../conejoninja/tesoro/vendor/github.com/trezor/usbhid/hid.go:32:11: fatal error: os/threads_posix.c: No such file or directory
#include "os/threads_posix.c"
^~~~~~~~~~~~~~~~~~~~
compilation terminated.
https://github.com/conejoninja/tesoro/issues/9
"gocryptfs -fsck" will need access to helper functions,
and to get that, it will need to cast a gofuse.File to a
fusefrontend.File. Make fusefrontend.File exported to make
this work.
TrezorPayload stores 32 random bytes used for unlocking
the master key using a Trezor security module. The randomness makes sure
that a unique unlock value is used for each gocryptfs filesystem.
We are clean again.
Warnings were:
internal/fusefrontend/fs.go:443:14: should omit type string from declaration
of var cTarget; it will be inferred from the right-hand side
internal/fusefrontend/xattr.go:26:1: comment on exported method FS.GetXAttr
should be of the form "GetXAttr ..."
internal/syscallcompat/sys_common.go:9:7: exported const PATH_MAX should have
comment or be unexported
Reading system.posix_acl_access and system.posix_acl_default
should return EOPNOTSUPP to inform user-space that we do not
support ACLs.
xftestest essientially does
chacl -l | grep "Operation not supported"
to determine if the filesystem supports ACLs, and used to
wrongly believe that gocryptfs does.
mv is unhappy when we return EPERM when it tries to set
system.posix_acl_access:
mv: preserving permissions for ‘b/x’: Operation not permitted
Now we return EOPNOTSUPP like tmpfs does and mv seems happy.
Values a binary-safe, there is no need to base64-encode them.
Old, base64-encoded values are supported transparently
on reading. Writing xattr values now always writes them binary.
We previously returned EPERM to prevent the kernel from
blacklisting our xattr support once we get an unsupported
flag, but this causes lots of trouble on MacOS:
Cannot save files from GUI apps, see
https://github.com/rfjakob/gocryptfs/issues/229
Returning ENOSYS triggers the dotfiles fallback on MacOS
and fixes the issue.
OpenDir and ListXAttr skip over corrupt entries,
readFileID treats files the are too small as empty.
This improves usability in the face of corruption,
but hides the problem in a log message instead of
putting it in the return code.
Create a channel to report these corruptions to fsck
so it can report them to the user.
Also update the manpage and the changelog with the -fsck option.
Closes https://github.com/rfjakob/gocryptfs/issues/191
Most corruption cases except xattr should be covered.
With test filesystem.
The output is still pretty ugly. xattr support will
be added in the next commits.
This should not happen via FUSE as the kernel caps the size,
but with fsck we have the first user that calls Read directly.
For symmetry, check it for Write as well.
A few places have called tlog.Warn.Print, which directly
calls into log.Logger due to embedding, losing all features
of tlog.
Stop embedding log.Logger to make sure the internal functions
cannot be called accidentially and fix (several!) instances
that did.
We now print the number in a debug message, so define
the numeric values explicitely instead of using iota.
This way you don't have to understand how iota works
to find out what the number means. Lack of understanding
of how iota works is also the reason why the numbers
start at 3 (to keep the current behavoir).
Jakob Unterwurzacher