Mismatching GPG keys:
An extreamly frustraiting problem I had when setting up multiple servers is mismatching keys. ie you setup server AA and server BB with camlistore then try to upload files from AA to BB and notice that they aren't showing in BB's webui properly. Thats because your gpg keys don't match. You either need to use the same keystore for both or use --vivify when uploading. (Vivify tells the server to create a permanode for the file and sign it itself, but your not allowed to tag it or anything else)
I didn't discover why this is the case; It lets you upload files and create permanodes and mutations, then just ignores them. Maybe multi key/user support will be fleshed out in the future but for now its something to be mindful of.
An extreamly frustraiting problem I had when setting up multiple servers is mismatching keys. ie you setup server AA and server BB with camlistore then try to upload files from AA to BB and notice that they aren't showing in BB's webui properly. Thats because your gpg keys don't match. You either need to use the same keystore for both or use --vivify when uploading. (Vivify tells the server to create a permanode for the file and sign it itself, but your not allowed to tag it or anything else)
I didn't discover why this is the case; It lets you upload files and create permanodes and mutations, then just ignores them. Maybe multi key/user support will be fleshed out in the future but for now its something to be mindful of.
Camput:
camput file:
--filenodes only creates a permanode if the file doesn't already exist
--permanode always creates a peramanode so if adding duplicate files you can end up with 2 permanodes for the same file.
--vivify does same as filenodes
--filenodes with directory creates permanodes for all files but not the directory
--permanode with directory creates permanode for directory but not files
to see if file or directory belongs to any permanode
The indexer is what extracts metadata from files (exif, id3, etc) I don't think that info is stored in the blobstore.
Incrementals:
Lets start with a test
While uploading a lot of identical pictures contained in different directories I noticed camlistore was creating a new file meta blob every time it encountered the same file.
camput file:
--filenodes only creates a permanode if the file doesn't already exist
--permanode always creates a peramanode so if adding duplicate files you can end up with 2 permanodes for the same file.
--vivify does same as filenodes
--filenodes with directory creates permanodes for all files but not the directory
--permanode with directory creates permanode for directory but not files
to see if file or directory belongs to any permanode
➜ camlistore.org git:(master) ✗ devcam tool search --rawquery '{"constraint":{"permanode":{"attr":"camliContent","valueMatches":{"equals":"sha1-ca438fc23405c98cda406bfced5bea79dac9eac3"}}}}'
{
"blobs": [
{
"blob": "sha1-1b762ddc7f6cf6444eaef226930ddec9fa913dfa"
}
],
"description": null
}The indexer is what extracts metadata from files (exif, id3, etc) I don't think that info is stored in the blobstore.
Incrementals:
Lets start with a test
➜ camlistore.org git:(master) ✗ dd if=/dev/urandom bs=1024 count=512000 > urandom.test
512000+0 records in
512000+0 records out
524288000 bytes (524 MB) copied, 27.5681 s, 19.0 MB/s
➜ camlistore.org git:(master) ✗ devcam put file --filenodes --tag="urandom" urandom.test
sha1-3d1e7ed90d4b351893f3db47a0a9c909b9b70138
sha1-9cdb32d52ff577dee28f2fca2e41e5f6980a7626
sha1-4363c216e04601f8ea6809e42f7bc2a883bceb28
sha1-820516b8a785800dcae7a9d4c27c20b794712265
➜ camlistore.org git:(master) ✗ dd if=/dev/urandom bs=1024 count=25600 >> urandom.test
25600+0 records in
25600+0 records out
26214400 bytes (26 MB) copied, 1.32075 s, 19.8 MB/s
➜ camlistore.org git:(master) ✗ devcam put file --filenodes --tag="urandom" urandom.test
sha1-d2980f0702301604de2187abb2dff65dcccf3fe9
sha1-03c8aee6e6bb141aafa2dde46b05748f3c3603b5
sha1-c68b5d6088206393993489643a7d6614e33df3b0
sha1-d8a494b69330144834a1690e4259931a4dd60f3e
Obviously it created a new file and permanode, but if you pull the files and compare the parts; the majority of the original blobs were reused for the second file. For me only four parts changed totaling 29Mib so theres a 4Mib overlap. Pretty nice.
Though my knowledge of how the checksum and blobstorage work is very rudamentary, it appears camlistore is smart enough to find identical blobs in unrelated files to further reduce duplication.
Ctimes:Though my knowledge of how the checksum and blobstorage work is very rudamentary, it appears camlistore is smart enough to find identical blobs in unrelated files to further reduce duplication.
While uploading a lot of identical pictures contained in different directories I noticed camlistore was creating a new file meta blob every time it encountered the same file.
➜ camlistore.org git:(master) ✗ devcam put file --filenodes -exiftime -tag=exiftest ~/Android/backups/sdcard-incremental-backups/mako/2013-07-08/DCIM/Camera/IMG_20121227_124124.jpg
sha1-604519893399d329f53171fa595c22f11c959da9
sha1-cac227ddfebc4f2d7d2afddd4e269837a85009a5
sha1-9f50d69fd09d0a5813f44466408479d0127a9b51
sha1-c3c54d53ac1d9ce87c801bb7c04ff65c4ff3c2d6
➜ camlistore.org git:(master) ✗ devcam put file --filenodes -exiftime -tag=exiftest ~/Android/backups/sdcard-incremental-backups/mako/2013-07-27/DCIM/Camera/IMG_20121227_124124.jpg
2015/04/02 16:37:04 Found dup of contents sha1-7e5fb0359dfe07c7a8489a24ee23e3f4b043d20e in file schema sha1-c3c54d53ac1d9ce87c801bb7c04ff65c4ff3c2d6
sha1-604519893399d329f53171fa595c22f11c959da9
sha1-9b7acbc6c459d9d4f7699923eba490cc12b55ead
sha1-9f50d69fd09d0a5813f44466408479d0127a9b51
sha1-24f572ded13a811d4c8e2b67fbf8779b1c116947
So what it did here was reuse the same permanode, the tag claim, it also noticed the files were identical and reused all the part blobs. But it also created a new file meta blob for the identical file and assigned that blob to the permanodes camliContent. This probably isn't a big deal but i didn't want it to do that. So i did some digging and found the --capctime flag that prevents this, by setting the ctime attr to the mtime value.
➜ camlistore.org git:(master) ✗ CAMLI_DEBUG=true devcam put file -capctime --filenodes -exiftime -tag=exiftest ~/Android/backups/sdcard-incremental-backups/mako/2013-07-08/DCIM/Camera/IMG_20121227_124125.jpg
sha1-1e962013748495b794c5735c8e7ad6d33fec44a6
sha1-4f6096763a69c5c63ca45cd9a5bd4bd741a95e78
sha1-2150648b8b0699fd137afb94ab0e5810125cc212
sha1-304fb30134546aa62eea3eddddbbb7b014300713
➜ camlistore.org git:(master) ✗ CAMLI_DEBUG=true devcam put file -capctime --filenodes -exiftime -tag=exiftest ~/Android/backups/sdcard-incremental-backups/mako/2013-07-27/DCIM/Camera/IMG_20121227_124125.jpg
2015/04/02 16:44:51 Found dup of contents sha1-f726f6b478119623cca4bf5b536fe9dea6165a75 in file schema sha1-304fb30134546aa62eea3eddddbbb7b014300713
sha1-1e962013748495b794c5735c8e7ad6d33fec44a6
sha1-4f6096763a69c5c63ca45cd9a5bd4bd741a95e78
sha1-2150648b8b0699fd137afb94ab0e5810125cc212
sha1-304fb30134546aa62eea3eddddbbb7b014300713I should also mention the --exiftime is helpful when the mtime of the file is newer that the date the pic was taken.