hardcoresushi/libgocryptfs
1
0
Fork 0
Code Issues Pull Requests Releases Activity
1,568 Commits 2 Branches 0 Tags 6 MiB
22e3eec153
Commit Graph

1568 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Jakob Unterwurzacher
22e3eec153 getdents-debug: loop and stop on first error 
Also try to improve and unify output a little.

$ ./getdents /usr/share/man/man1
  1: unix.Getdents: n=9984; n=9984; n=9968; n=9976; n=9984; n=9968; n=10000; n=9976; n=9992; n=10000; n=9976; n=9992; n=2312; n=0; err=<nil>; total 122112 bytes
  2: unix.Getdents: n=9984; n=48; n=9976; n=9968; n=9976; n=9976; n=9992; n=9984; n=9992; n=10000; n=9976; n=9968; n=10000; n=2272; n=0; err=<nil>; total 122112 bytes
  3: unix.Getdents: n=9984; n=9984; n=9968; n=704; n=10000; n=10000; n=9968; n=9968; n=9992; n=10000; n=9960; n=9992; n=9992; n=1600; n=0; err=<nil>; total 122112 bytes
  4: unix.Getdents: n=9984; n=9984; n=9968; n=9976; n=9984; n=32; n=9992; n=9984; n=9992; n=10000; n=9976; n=9968; n=10000; n=2272; n=0; err=<nil>; total 122112 bytes

$ ./getdents_c /usr/share/man/man1
  1: getdents64: n=9984; n=9984; n=9968; n=9976; n=9984; n=9968; n=10000; n=9976; n=9992; n=10000; n=9976; n=9992; n=2312; n=0; errno=0 total 122112 bytes
  2: getdents64: n=9984; n=9984; n=9968; n=9976; n=9984; n=9968; n=10000; n=9976; n=9992; n=10000; n=9976; n=9992; n=2312; n=0; errno=0 total 122112 bytes
  3: getdents64: n=9984; n=9984; n=9968; n=9976; n=9984; n=9968; n=10000; n=9976; n=9992; n=10000; n=9976; n=9992; n=2312; n=0; errno=0 total 122112 bytes
  4: getdents64: n=9984; n=9984; n=9968; n=9976; n=9984; n=9968; n=10000; n=9976; n=9992; n=10000; n=9976; n=9992; n=2312; n=0; errno=0 total 122112 bytes
 2020-05-28 23:23:53 +02:00
Jakob Unterwurzacher
b275c53fa7 contrib/getdents-debug: implement getdents -loop 
$ ./getdents -loop /mnt/synology/public/tmp/g1
unix.Getdents: n=4176; n=4176; n=4176; n=4176; n=4176; n=3192; n=0; err=<nil>; total 24072 bytes
unix.Getdents: n=4176; n=4176; n=4176; n=4176; n=4176; n=3192; n=0; err=<nil>; total 24072 bytes
unix.Getdents: n=4176; n=-1; err=no such file or directory; total 4176 bytes
 2020-05-25 23:32:11 +02:00
Jakob Unterwurzacher
0d522e0d3b Add contrib/getdents-debug/readdirnames 
Another way to repro the problem in
https://github.com/rfjakob/gocryptfs/issues/483
 2020-05-24 23:51:38 +02:00
Jakob Unterwurzacher
6019598fdb contrib: collect getdents stuff in getdents-debug folder 2020-05-24 23:46:41 +02:00
Jakob Unterwurzacher
71c0481f0e Revert "fusefrontend: don't always clear the dircache in Rename" 
As noticed by @slackner in
cb8872577d (commitcomment-39405233) ,
this is not safe.

This reverts commit cb8872577d.
 2020-05-24 23:36:11 +02:00
Jakob Unterwurzacher
15ff79bf14 syscallcompat: warn when Getdents truncates data 
On CIFS mounts, unix.Getdents can return sudden ENOENT
in the middle of data. This will not be reported as an error
by user space tools, so return EIO instead.

Also log it as a warning.

https://github.com/rfjakob/gocryptfs/issues/483
 2020-05-24 23:30:25 +02:00
Jakob Unterwurzacher
b3350f0ebb contrib: add getdents_c 
Same thing like contrib/getdents, but written in C.
 2020-05-24 23:29:59 +02:00
Jakob Unterwurzacher
c7a9425e1b Add contrib/getdents 
Small tool to try to debug unix.Getdents problems on CIFS mounts
https://github.com/rfjakob/gocryptfs/issues/483
 2020-05-24 22:54:58 +02:00
Jakob Unterwurzacher
1a91a11e00 contrib/sshfs-benchmark.bash: add rmdir, mkdir, touch 
And also, stop using the wrong directory for sshfs git init.

sshfs-benchmark.bash:    sshfs  gocryptfs-on-sshfs
git init                  4.35                7.82
rsync                     7.72               11.66
rm -R                     2.71               11.04
mkdir                     1.33                4.15
rmdir                     0.47                3.97
touch                     2.32                2.85
rm                        0.45                0.45
 2020-05-24 22:03:40 +02:00
Jakob Unterwurzacher
cb8872577d fusefrontend: don't always clear the dircache in Rename 
When filename encryption is on, we do know when we
overwrite a directory, and can clear only in this case.

sshfs-benchmark.bash:    sshfs  gocryptfs-on-sshfs
git init                  1.74                7.80
rsync                     6.19               11.63
 2020-05-24 20:19:27 +02:00
Jakob Unterwurzacher
2a9b99a0ef fusefrontend: don't clear dircache on Mkdir 
Mkdir can not cause existing entries in the cache to go
stale. So don't clear it. Benchmark results:

sshfs-benchmark.bash:    sshfs  gocryptfs-on-sshfs
git init                  1.65                8.74
rsync                     6.09               17.54
 2020-05-24 15:30:14 +02:00
Jakob Unterwurzacher
11dfcfd6c0 contrib: add sshfs-benchmark.bash 
Let's get some reproducible numbers for
https://github.com/rfjakob/gocryptfs/issues/481
and
https://github.com/rfjakob/gocryptfs/issues/410

Example run:

$ ./sshfs-benchmark.bash nuetzlich.net
working directory: /tmp/sshfs-benchmark.bash.vu4
sshfs mounted: nuetzlich.net:/tmp -> sshfs.mnt
gocryptfs mounted: sshfs.mnt/sshfs-benchmark.bash.KM9/gocryptfs.crypt -> gocryptfs.mnt

                         sshfs  gocryptfs-on-sshfs
git init                  1.68               11.23
rsync                     6.07               20.35
 2020-05-24 15:30:14 +02:00
Jakob Unterwurzacher
25f1727de9 syscallcompat: getdents: retry on EINTR 
Fixes: https://github.com/rfjakob/gocryptfs/issues/483
Related: https://github.com/golang/go/issues/38836
 2020-05-23 22:54:23 +02:00
Jakob Unterwurzacher
f8ad2ac3e2 dircache: increase cache size & lifetime 
Looking at the dircache debug output, we see
that a "git status" workload has a very bad
cache hit rate because the entries expire or
get evicted before they can be reused.

Increase both cache size and lifetime for
a 4x speedup:

Before: 75s
After:  17s

https://github.com/rfjakob/gocryptfs/issues/410
 2020-05-17 21:37:36 +02:00
Jakob Unterwurzacher
f6088e5008 dircache: improve debug messages 
Before:

Lookup "errno.html/1/2/3/4/5": miss
Store: "errno.html/1/2/3/4/5" fd=26 iv=21be6e083d60dcabfe7368264d5082b7
Lookup "errno.html": hit 25 6d68a16d217978915036a3bd55428ae7
Lookup "errno.html/1": hit 25 932a464c299b3430c5e55c924f98ac4d
Lookup "errno.html/1/2": hit 25 7d53348b1692d537f017bf86b3cf5feb
Lookup "errno.html/1/2/3": hit 25 2aef1c9d1ab2b55b163215053fefe703
Lookup "errno.html/1/2/3/4": hit 25 cb802be53721c46a46247c5e4e0f4ce6
Lookup "errno.html/1/2/3/4": hit 25 cb802be53721c46a46247c5e4e0f4ce6
Lookup "errno.html": hit 25 6d68a16d217978915036a3bd55428ae7

After:

Lookup "earlyoom/.git/refs"                     hit fd=10 dup=17 iv=6ae2cecd269a25e8d946aff6afe9b8b8
Lookup "earlyoom/.git/refs/remotes"             hit fd=19 dup=17 iv=f04c2d2a5bcc33ebdeaca664859c980d
Lookup "earlyoom/.git/refs/remotes/origin"      miss
Store  "earlyoom/.git/refs/remotes/origin"      fd=17 iv=834a64a1697c9f5705455ba6dbed22b5
Lookup "earlyoom"                               hit fd=7 dup=25 iv=2303a892d6e2357c483574a8070b7679
Lookup "earlyoom/.git"                          hit fd=11 dup=25 iv=d43ca4aff23720c57789c9f62f0aee00
Lookup "earlyoom/.git"                          hit fd=11 dup=25 iv=d43ca4aff23720c57789c9f62f0aee00
Lookup "earlyoom/.git/refs"                     hit fd=10 dup=25 iv=6ae2cecd269a25e8d946aff6afe9b8b8
Lookup "earlyoom/.git/refs/heads"               hit fd=13 dup=25 iv=f9245e7c066b9adc768a1a666da9fbc8
 2020-05-17 21:26:56 +02:00
Jakob Unterwurzacher
bf66da6880 Update changelog 2020-05-17 19:40:30 +02:00
Jakob Unterwurzacher
416080203b main: accept multiple -passfile options 
Each file will be read and then concatenated
for the effictive password. This can be used as a
kind of multi-factor authenticiton.

Fixes https://github.com/rfjakob/gocryptfs/issues/288
 2020-05-17 19:31:04 +02:00
Jakob Unterwurzacher
ded4bbe645 go mod tidy 
The go-fuse v1 dependency is spurious. Will be fixed by
https://github.com/hanwen/go-fuse/pull/360
 2020-05-17 14:25:30 +02:00
Jakob Unterwurzacher
ec74d1d2f4 Update go-fuse import path to github.com/hanwen/go-fuse/v2 
We need
fd7328faf9
to fix a crash reported in https://github.com/rfjakob/gocryptfs/issues/430 :

  2019/10/30 17:14:16 Unknown opcode 2016
  panic: runtime error: invalid memory address or nil pointer dereference
  [signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x508d38]

This patch is only in the v2.x.x branch. Upgrade to v2, as the
old API is also supported there.

Running

  git grep hanwen/go-fuse | grep -v hanwen/go-fuse/v2

to check for forgotten references comes back clean.
 2020-05-17 14:23:47 +02:00
Jakob Unterwurzacher
ead7008a08 Fix spelling mistakes found by misspell 
https://github.com/client9/misspell
 2020-05-10 00:25:49 +02:00
Jakob Unterwurzacher
1d145be5a1 contrib: delete ctlsock-encrypt.bash 
Closes https://github.com/rfjakob/gocryptfs/issues/416
 2020-05-10 00:15:06 +02:00
Jakob Unterwurzacher
5af7d3c699 gocryptfs-xray: document -encrypt-paths / -decrypt-paths 2020-05-10 00:14:03 +02:00
Jakob Unterwurzacher
a9895b3487 gocryptfs-xray: add -0 flag, add tests 
The -0 flags works like xargs -0.
 2020-05-10 00:04:14 +02:00
Jakob Unterwurzacher
f2e8b776f8 main: add "go doc" package comment 
Should show up on https://pkg.go.dev/github.com/rfjakob/gocryptfs?tab=doc
which currently reads "No documentation available for this package!"
 2020-05-09 19:18:53 +02:00
Jakob Unterwurzacher
24554b11f7 gocryptfs-xray: integrate ctlsock path encryption/decryption 
Implementation seems to work ok, but is missing tests and
documentation for now.
I will only delete ctlsock-encrypt.bash when both are
done.

https://github.com/rfjakob/gocryptfs/issues/416
 2020-05-09 19:11:06 +02:00
Jakob Unterwurzacher
171b1eac91 test_helpers: use new ctlsock.CtlSock API 2020-05-09 19:09:33 +02:00
Jakob Unterwurzacher
7e51073400 ctlsock: add CtlSock API 2020-05-09 19:09:09 +02:00
Jakob Unterwurzacher
f0184804f4 test_helper: kill lsof after 1 second 
lsof may get stuck when gocryptfs itself is stuck.
 2020-05-09 19:03:16 +02:00
Jakob Unterwurzacher
16221facb9 ctlsock: create exported ctlsock client library 
The former interal ctlsock server package is renamed
to ctlsocksrv.
 2020-05-09 17:36:41 +02:00
Jakob Unterwurzacher
3ef563493a tests: add TestPasswdMasterkeyStdin 
Tests that `gocryptfs -passwd -masterkey=stdin` works.
This was fixed by ff04b1d83a.

Fixes https://github.com/rfjakob/gocryptfs/issues/461
 2020-05-09 16:53:25 +02:00
Jakob Unterwurzacher
ff04b1d83a main: untangle -masterkey handling and config loading 
This was handled both in getMasterKey(). Split it apart.
 2020-05-09 16:53:12 +02:00
Jakob Unterwurzacher
7622c9f538 main: rename parseMasterKey() -> unhexMasterKey() 
Make it clear that function does NOT parse the "-masterkey"
command line argument, it just unhexes the payload.
 2020-05-09 16:10:22 +02:00
Jakob Unterwurzacher
c19baa10f8 tests: use t.Name() 
We did not use t.Name() as it was not available
before Go 1.8. Now the oldest Go version we support is
Go 1.11, so we can use it.
 2020-05-09 15:42:57 +02:00
Jakob Unterwurzacher
5dbf376860 tests: cli: rename TestBypass -> TestBadname 
The command line option is now called `-badname`,
so adjust the test name to match.
 2020-05-09 15:29:21 +02:00
Jakob Unterwurzacher
a6f515008f Update README for v1.8.0 release 2020-05-09 14:20:21 +02:00
Jakob Unterwurzacher
d612ee5d91 Documentation: update performance.txt 
Bisecting shows that the performance drop is caused by
this commit:

commit ca9e912a28 (refs/bisect/bad)
Author: Jakob Unterwurzacher <jakobunt@gmail.com>
Date:   Sat Feb 29 19:58:08 2020 +0100

    fusefrontend: drop xattr user namespace restriction
 2020-05-03 21:20:30 +02:00
Jakob Unterwurzacher
b23f77c8ea go mod: update dependencies 
Updated using

    go get -t -u ./...
 2020-05-03 20:49:18 +02:00
Jakob Unterwurzacher
3e4545bbac build-without-openssl.bash: suppress "not a dynamic executable" message 
We redirected the wrong ldd fd to /dev/null. Fix it.
 2020-05-03 20:47:39 +02:00
Jakob Unterwurzacher
feaeee90e2 inomap: fix TestSpill bit check 
Wrong bit operator was used.
 2020-05-03 20:28:26 +02:00
Jakob Unterwurzacher
8c9c68fb72 inomap: fix spillBit not set on 2nd hit 
Also add a test for this.

Thanks @slackner for the comment.
 2020-05-03 20:21:11 +02:00
Jakob Unterwurzacher
91f5c242a8 inomap: remove leftover debug output 
This was committed by accident.
 2020-05-03 20:01:12 +02:00
Jakob Unterwurzacher
518771e4e2 fusefrontend_reverse: use inomap for inode number translation 
Gets rid of static inode number value limitations.

Fixes https://github.com/rfjakob/gocryptfs/issues/457
 2020-05-03 15:22:10 +02:00
Jakob Unterwurzacher
db93a6c54c tests: reverse: add inode mapping test (TestVirtualFileIno) 
Verify that virtual files get assigned inode numbers
we expect.
 2020-05-03 14:49:32 +02:00
Jakob Unterwurzacher
483054efaa inomap: comment constants 2020-04-19 22:09:21 +02:00
Jakob Unterwurzacher
9f9d59ded9 inomap: rework logic to efficiently support flags 
Adding flags allows to use inomap in reverse mode,
replacing the clunky inoBaseDirIV/inoBaseNameFile
logic that causes problems with high underlying
inode numbers ( https://github.com/rfjakob/gocryptfs/issues/457 )

Microbenchmarks (values below) show that the "SingleDev"
case is now much slower due to an extra map lookup,
but this has no visible effects in ./test.bash results,
so there was no time spent optimizing the case further.

$ go test -bench=.
goos: linux
goarch: amd64
pkg: github.com/rfjakob/gocryptfs/internal/inomap
BenchmarkTranslateSingleDev-4   	18757510	        61.5 ns/op
BenchmarkTranslateManyDevs-4    	18061515	        64.5 ns/op
PASS
ok  	github.com/rfjakob/gocryptfs/internal/inomap	2.467s
 2020-04-19 22:00:56 +02:00
Jakob Unterwurzacher
fcdeb52390 inomap: add benchmark 
$ go test -bench=.
goos: linux
goarch: amd64
pkg: github.com/rfjakob/gocryptfs/internal/inomap
BenchmarkTranslateSingleDev-4   	202479382	         5.88 ns/op
BenchmarkTranslateManyDevs-4    	16095795	        71.9 ns/op
PASS
ok  	github.com/rfjakob/gocryptfs/internal/inomap	3.039s
 2020-04-19 21:35:06 +02:00
Jakob Unterwurzacher
1c169ac55e build.bash: handle missing git tags 
The case of a git repo without any tags used to fail
with:

  fatal: No names found, cannot describe anything.

Now we continue, using "[no_tags_found]" as the
version string.
 2020-04-18 17:44:51 +02:00
Jakob Unterwurzacher
5da5e9fdf2 build.bash: don't enable -buildmode=pie for static builds 
Causes warnings:

  $ ./build-without-openssl.bash
  # github.com/rfjakob/gocryptfs
  loadinternal: cannot find runtime/cgo
  # github.com/rfjakob/gocryptfs/gocryptfs-xray
  loadinternal: cannot find runtime/cgo
  # github.com/rfjakob/gocryptfs/contrib/statfs
  loadinternal: cannot find runtime/cgo
  gocryptfs v1.7.1-48-gf6b1c68 without_openssl; go-fuse v1.0.1-0.20190319092520-161a16484456; 2020-04-18 go1.13.6 linux/amd64

https://github.com/golang/go/issues/30986
 2020-04-18 17:09:25 +02:00
Jakob Unterwurzacher
f6b1c680b3 nametransform: update comment & simplify tests 
The comment still mentioned CBC, which has been removed
a long time ago.

The test definition can be rewritten using slice literals,
saving sume stuttering.
 2020-04-18 16:14:48 +02:00
Jakob Unterwurzacher
8f5c2a613d Prefer Go stdlib aes-gcm on arm64 with aes instructions 
We used to prefer openssl in this situation, which
used to make sense, but now Go gained an optimized
assembly implementation for aes-gcm on arm64 with
aes instructions:

  root@q1:~/go/src/github.com/rfjakob/gocryptfs# ./gocryptfs -speed
  gocryptfs v1.7.1-46-g73436d9; go-fuse v1.0.1-0.20190319092520-161a16484456; 2020-04-13 go1.14.2 linux/arm64
  AES-GCM-256-OpenSSL      212.30 MB/s    (selected in auto mode)
  AES-GCM-256-Go           452.30 MB/s
  AES-SIV-512-Go           100.25 MB/s
  XChaCha20-Poly1305-Go    137.35 MB/s

https://github.com/rfjakob/gocryptfs/issues/452
 2020-04-13 22:34:07 +02:00