Wednesday, April 8, 2015

More Camlistore thoughts

Mismatching GPG keys:

An extreamly frustraiting problem I had when setting up multiple servers is mismatching keys. ie you setup server AA and server BB with camlistore then try to upload files from AA to BB and notice that they aren't showing in BB's webui properly. Thats because your gpg keys don't match. You either need to use the same keystore for both or use --vivify when uploading. (Vivify tells the server to create a permanode for the file and sign it itself, but your not allowed to tag it or anything else)
I didn't discover why this is the case; It lets you upload files and create permanodes and mutations, then just ignores them. Maybe multi key/user support will be fleshed out in the future but for now its something to be mindful of.

Camput:

camput file:
--filenodes only creates a permanode if the file doesn't already exist
--permanode always creates a peramanode so if adding duplicate files you can end up with 2 permanodes for the same file.
--vivify does same as filenodes
--filenodes with directory creates permanodes for all files but not the directory
--permanode with directory creates permanode for directory but not files

to see if file or directory belongs to any permanode

➜  camlistore.org git:(master) ✗ devcam tool search --rawquery '{"constraint":{"permanode":{"attr":"camliContent","valueMatches":{"equals":"sha1-ca438fc23405c98cda406bfced5bea79dac9eac3"}}}}'
{
  "blobs": [
    {
      "blob": "sha1-1b762ddc7f6cf6444eaef226930ddec9fa913dfa"
    }
  ],
  "description": null
}


The indexer is what extracts metadata from files (exif, id3, etc) I don't think that info is stored in the blobstore.

Incrementals:

Lets start with a test

➜  camlistore.org git:(master) ✗ dd if=/dev/urandom bs=1024 count=512000 > urandom.test
512000+0 records in
512000+0 records out
524288000 bytes (524 MB) copied, 27.5681 s, 19.0 MB/s
➜  camlistore.org git:(master) ✗ devcam put file --filenodes --tag="urandom" urandom.test
sha1-3d1e7ed90d4b351893f3db47a0a9c909b9b70138
sha1-9cdb32d52ff577dee28f2fca2e41e5f6980a7626
sha1-4363c216e04601f8ea6809e42f7bc2a883bceb28
sha1-820516b8a785800dcae7a9d4c27c20b794712265
➜  camlistore.org git:(master) ✗ dd if=/dev/urandom bs=1024 count=25600 >> urandom.test
25600+0 records in
25600+0 records out
26214400 bytes (26 MB) copied, 1.32075 s, 19.8 MB/s
➜  camlistore.org git:(master) ✗ devcam put file --filenodes --tag="urandom" urandom.test
sha1-d2980f0702301604de2187abb2dff65dcccf3fe9
sha1-03c8aee6e6bb141aafa2dde46b05748f3c3603b5
sha1-c68b5d6088206393993489643a7d6614e33df3b0
sha1-d8a494b69330144834a1690e4259931a4dd60f3e


Obviously it created a new file and permanode, but if you pull the files and compare the parts; the majority of the original blobs were reused for the second file. For me only four parts changed totaling 29Mib so theres a 4Mib overlap. Pretty nice.

Though my knowledge of how the checksum and blobstorage work is very rudamentary, it appears camlistore is smart enough to find identical blobs in unrelated files to further reduce duplication.

Ctimes:

While uploading a lot of identical pictures contained in different directories I noticed camlistore was creating a new file meta blob every time it encountered the same file.

➜  camlistore.org git:(master) ✗ devcam put file --filenodes -exiftime -tag=exiftest ~/Android/backups/sdcard-incremental-backups/mako/2013-07-08/DCIM/Camera/IMG_20121227_124124.jpg
sha1-604519893399d329f53171fa595c22f11c959da9
sha1-cac227ddfebc4f2d7d2afddd4e269837a85009a5
sha1-9f50d69fd09d0a5813f44466408479d0127a9b51
sha1-c3c54d53ac1d9ce87c801bb7c04ff65c4ff3c2d6
➜  camlistore.org git:(master) ✗ devcam put file --filenodes -exiftime -tag=exiftest ~/Android/backups/sdcard-incremental-backups/mako/2013-07-27/DCIM/Camera/IMG_20121227_124124.jpg
2015/04/02 16:37:04 Found dup of contents sha1-7e5fb0359dfe07c7a8489a24ee23e3f4b043d20e in file schema sha1-c3c54d53ac1d9ce87c801bb7c04ff65c4ff3c2d6
sha1-604519893399d329f53171fa595c22f11c959da9
sha1-9b7acbc6c459d9d4f7699923eba490cc12b55ead
sha1-9f50d69fd09d0a5813f44466408479d0127a9b51
sha1-24f572ded13a811d4c8e2b67fbf8779b1c116947

So what it did here was reuse the same permanode, the tag claim, it also noticed the files were identical and reused all the part blobs. But it also created a new file meta blob for the identical file and assigned that blob to the permanodes camliContent. This probably isn't a big deal but i didn't want it to do that. So i did some digging and found the --capctime flag that prevents this, by setting the ctime attr to the mtime value.

➜  camlistore.org git:(master) ✗ CAMLI_DEBUG=true devcam put file -capctime --filenodes -exiftime -tag=exiftest ~/Android/backups/sdcard-incremental-backups/mako/2013-07-08/DCIM/Camera/IMG_20121227_124125.jpg
sha1-1e962013748495b794c5735c8e7ad6d33fec44a6
sha1-4f6096763a69c5c63ca45cd9a5bd4bd741a95e78
sha1-2150648b8b0699fd137afb94ab0e5810125cc212
sha1-304fb30134546aa62eea3eddddbbb7b014300713
➜  camlistore.org git:(master) ✗ CAMLI_DEBUG=true devcam put file -capctime --filenodes -exiftime -tag=exiftest ~/Android/backups/sdcard-incremental-backups/mako/2013-07-27/DCIM/Camera/IMG_20121227_124125.jpg
2015/04/02 16:44:51 Found dup of contents sha1-f726f6b478119623cca4bf5b536fe9dea6165a75 in file schema sha1-304fb30134546aa62eea3eddddbbb7b014300713
sha1-1e962013748495b794c5735c8e7ad6d33fec44a6
sha1-4f6096763a69c5c63ca45cd9a5bd4bd741a95e78
sha1-2150648b8b0699fd137afb94ab0e5810125cc212
sha1-304fb30134546aa62eea3eddddbbb7b014300713

CAMLI_DEBUG=true is required to gain acess to the --capctime flag.
I should also mention the --exiftime is helpful when the mtime of the file is newer that the date the pic was taken.

A day with Camlistore

I've known about camlistore for at least a year, and even tried to use it once before but gave up pretty quickly. It wasn't exactly user friendly. I gave it another look, but this time stuck to it. Here's some notes on what I learned.

All examples will be done with devcam, cause I didn't want to screw with my real instance. Devcam is just a wrapper around camlistored that creates a temporary blobstore and index so you can experiment and stuff.

Uploading files

From what I can understand. Camlistore is all about the blobs, everything you do in camlistore is essentially a blob stored in the blobstore as its sha1-. Files are actually several blobs. The file part is usually split into several blobs (parts). Then there is a metadata blob with camliType "file" that stores the meta necessary to restore the file when you want to get it out of camlistore.

➜  camlistore.org git:(master) devcam put file ~/Pictures/mako-screens/Screenshot_2012-12-27-16-00-50.png
sha1-8d87e15e54b9da2cf2b9e65e483107e03b8dfbc7


The hash you get back is the metadata for the file, we can pull it like so.

➜  camlistore.org git:(master) devcam tool describe sha1-8d87e15e54b9da2cf2b9e65e483107e03b8dfbc7
{
  "meta": {
    "sha1-8d87e15e54b9da2cf2b9e65e483107e03b8dfbc7": {
      "blobRef": "sha1-8d87e15e54b9da2cf2b9e65e483107e03b8dfbc7",
      "camliType": "file",
      "size": 380,
      "file": {
        "fileName": "Screenshot_2012-12-27-16-00-50.png",
        "size": 62675,
        "mimeType": "image/png",
        "time": "2014-03-14T04:09:46.825709033Z",
        "wholeRef": "sha1-cc9fb21e63446b5e6056942b520898c9afa065d5"
      },
      "image": {
        "width": 768,
        "height": 1280
      }
    }
  }
}


camtool describe just asks the indexer what it knows about the ref. To view the raw blob use camget.

➜  camlistore.org git:(master) devcam get sha1-8d87e15e54b9da2cf2b9e65e483107e03b8dfbc7
{"camliVersion": 1,
  "camliType": "file",
  "fileName": "Screenshot_2012-12-27-16-00-50.png",
  "parts": [
    {
      "blobRef": "sha1-cc9fb21e63446b5e6056942b520898c9afa065d5",
      "size": 62675
    }
  ],
  "unixGroup": "users",
  "unixGroupId": 100,
  "unixMtime": "2014-03-14T04:09:46.825709033Z",
  "unixOwner": "drew",
  "unixOwnerId": 1000,
  "unixPermission": "0644"
}


to retrieve the actual file use -o

➜  camlistore.org git:(master) devcam get -- -o . sha1-8d87e15e54b9da2cf2b9e65e483107e03b8dfbc7
2015/04/01 12:03:51 Writing sha1-8d87e15e54b9da2cf2b9e65e483107e03b8dfbc7 to Screenshot_2012-12-27-16-00-50.png ...


-o just fetches the meta then loops through and concats all the parts.

also note that the file doesn't show in the webgui this is because it isn't associated with a permanode. I'll get to that later. But you can still find it if you know exactly what your looking for (im not sure about future behavior here, camlistore doesn't have gc yet but it will and my understanding is that blobs not 'claimed' can be gc'd)

➜  camlistore.org git:(master) ✗ devcam tool search --rawquery '{"constraint":{"file":{"fileName":{"equals":"Screenshot_2012-12-27-16-00-50.png"}}}}'
{
  "blobs": [
    {
      "blob": "sha1-8d87e15e54b9da2cf2b9e65e483107e03b8dfbc7"
    }
  ],
  "description": null
}


--rawquery is pretty flexible. you can checkout all the possible constraints at https://camlistore.googlesource.com/camlistore/+/master/pkg/search/query.go

Uploading Files with Filenodes

My screenshot is still unclaimed, we can do this manually but i'd rather not (more on this below). So I'll use camput do do it for me.

➜  camlistore.org git:(master) ✗ devcam put file --filenodes --tag="Screen Shot","mako","Nexus 4" ~/Pictures/mako-screens/Screenshot_2014-02-28-11-38-33.png
sha1-65095c3dc3c69d05be69eac0a6c705e9ba8e77c6
sha1-1d6e7cf65556f235becd8660b2d4280454b38e48
sha1-b281f844a329cc4b874e4855754091185fd69b40
sha1-5dabe73fdcb2ef72306bb82bdc285472cfd95142
sha1-42911aa3019889b90a88e9d5bd95652c19beae2a
sha1-05135e0472e025c3ff4937589d199e3ed1ac99db




Notice we get back 6 hashes. The first is usually the created permanode


➜  camlistore.org git:(master) ✗ devcam tool describe sha1-65095c3dc3c69d05be69eac0a6c705e9ba8e77c6
{
  "meta": {
    "sha1-65095c3dc3c69d05be69eac0a6c705e9ba8e77c6": {
      "blobRef": "sha1-65095c3dc3c69d05be69eac0a6c705e9ba8e77c6",
      "camliType": "permanode",
      "size": 576,
      "permanode": {
        "attr": {
          "camliContent": [
            "sha1-05135e0472e025c3ff4937589d199e3ed1ac99db"
          ],
          "tag": [
            "Screen Shot",
            "mako",
            "Nexus 4"
          ]
        },
        "modtime": "2014-03-14T04:09:57.255844581Z"
      }
    }
  }
}




The next four are mutation claims, one assigning the file as the camliContent. The other three add the tags. The permanode and claims are all just blobs and immutable, its the indexer that puts everything together to make the describe output above. You can mutate the permanode by adding any arbitrary attribute you want to it, but there are several reserved attributes namely; tag, title, and anything starting with camli. All the currently used attrs are listed in the link above.

Permanodes

Now lets pretend like i uploaded several other screenshots using --filenodes and i want to group them into a collection.

Since i tagged them i can easily locate their permanodes


➜  camlistore.org git:(master) ✗ devcam tool search 'tag:mako'
{
  "blobs": [
    {
      "blob": "sha1-7a6d438affd32d701e153fc2e2b1dfd48cb7cd25"
    },
    {
      "blob": "sha1-eb1e40d5aa8f758439ca1cc62e9ef35fa2c1a50a"
    },
    {
      "blob": "sha1-093e4aa28f33b822b21a75614311d6a1a3b43aeb"
    },
    {
      "blob": "sha1-65095c3dc3c69d05be69eac0a6c705e9ba8e77c6"
    }
  ],
  "description": null
}



Now create the permanode under which to group them


➜  camlistore.org git:(master) ✗ devcam put permanode --title="Mako Screens"
sha1-cf968b27d78f4b12438b3fb8d1fe4b26b73e269b
sha1-ea94b63a4c683a94bc26340c23a58e31fabd0860



And start adding them


➜  camlistore.org git:(master) ✗ devcam put attr --add sha1-cf968b27d78f4b12438b3fb8d1fe4b26b73e269b camliMember sha1-7a6d438affd32d701e153fc2e2b1dfd48cb7cd25
sha1-f8679b4937703bdaac03c890c282ee4ec1783f23



This says add the sha1-f867 permanode to the camliMember attribute in the sha1-cf96 (Mako Screens) permanode.

Then pretend i added all the remaining screens to the "Mako Screens" permanode


➜  camlistore.org git:(master) ✗ devcam tool describe sha1-cf968b27d78f4b12438b3fb8d1fe4b26b73e269b
{
  "meta": {
    "sha1-cf968b27d78f4b12438b3fb8d1fe4b26b73e269b": {
      "blobRef": "sha1-cf968b27d78f4b12438b3fb8d1fe4b26b73e269b",
      "camliType": "permanode",
      "size": 562,
      "permanode": {
        "attr": {
          "camliMember": [
            "sha1-7a6d438affd32d701e153fc2e2b1dfd48cb7cd25",
            "sha1-eb1e40d5aa8f758439ca1cc62e9ef35fa2c1a50a",
            "sha1-093e4aa28f33b822b21a75614311d6a1a3b43aeb",
            "sha1-65095c3dc3c69d05be69eac0a6c705e9ba8e77c6"
          ],
          "title": [
            "Mako Screens"
          ]
        },
        "modtime": "2015-04-01T18:14:53.742708604Z"
      }
    }
  }
}



And witness that there is a new "Mako Screens" folder in the webui, containing the four screens

Permanode summary

What I've gleened so far is that there are (at least) two kinds of permanodes.

The first, filenode, is a permanode with a "camliContent" attribute that points either to a file or a set (i haven't discussed sets and won't but it essentially a directory). The "camliContent" attribute cannot be another permanode. I should also point out that althought the "camliContent" attribute is an array it only wants one value.

The second, I'll call it a collection, is a permanode with a "camliMember" attribute, all hashes in the "camliMember" array must be other permanodes.

I haven't confirmed this but my reasoning is that the "camliContent" and "camliMember" are mutually exclusive.

Deleting

Now lets delete something. Delete is a pretty strong word here. Because nothing in camlistore is ever deleted. (...yet we'll see when gc is implemented). The best we can do is 'hide' a permanode by issuing a delete mutation.

➜  camlistore.org git:(master) ✗ devcam put delete sha1-cf968b27d78f4b12438b3fb8d1fe4b26b73e269b
sha1-775ee5a2bedbf98e21398cce277f6aa8bd95ef3f




And witness it removed from the webui.

This is what our delete claim (mutation) looks like (using camget since describe doesn't tell us what type of claim it is)


➜  camlistore.org git:(master) ✗ devcam get sha1-775ee5a2bedbf98e21398cce277f6aa8bd95ef3f
{"camliVersion": 1,
  "camliSigner": "",
  "camliType": "claim",
  "claimDate": "2015-04-01T18:28:06.028725401Z",
  "claimType": "delete",
  "target": "sha1-cf968b27d78f4b12438b3fb8d1fe4b26b73e269b"
,"camliSig":""}




But wait... shit. that's not the right one! I meant to delete another permanode. Well thats not a problem. All we have to do is delete the delete mutation. (Its hash was output above)


➜  camlistore.org git:(master) ✗ devcam put delete sha1-775ee5a2bedbf98e21398cce277f6aa8bd95ef3f
sha1-f226e2400266ab6240a2825f8ecb8df7381f523b




And Mako Screens is back in the webui. Again notice, it wasn't actually deleted, we just mutated the delete claim. Essentially telling the indexer to ignore it.

Additional note about claims, claim search isn't implemented yet. But you can get a list of claims assossiated with a permanode by pinging the api. (Theres probably another way but I haven't found it yet)


➜  camlistore.org git:(master) ✗ curl localhost:3179/my-search/camli/search/claims\?permanode=sha1-cf968b27d78f4b12438b3fb8d1fe4b26b73e269b




More about claims:

A delete claim can only be issues on a permanode or another claim, it won't work on files (immutable remeber?)

I intermingled the words claim and mutation a bit. In this context they mean the same thing, a mutation is anything with the camliType "claim". A mutation essentially changes a blobref. I'm pretty sure you can mutate any blob but I'm not sure how useful it is. I've only mutated permanodes so far. The most useful mutations I've used are: adding/removing tags, changing title, changing camliMember, changing camliContent, deleting, undeleting, also removing/adding arbitrary attributes to the permanode. Though I'm sure there are other kinds of mutations I haven't used.

When mutating permanode attributes be aware that camlistore has several reserved attributes, namely; tag, title, and anything starting with camli. Further doc at https://camlistore.googlesource.com/camlistore/+/master/doc/schema/claims/attributes.txt

Summary:

At present camlistore is very image oriented which is why my examples used images. It also has the ability to parse mp3s (if you have good id3 tags). But wasn't able to pull meta from videos I added.

Camlistore is all about archival. Everything you add and every change made (mutation) is an immutable blob that's there for life!

The current release is 0.8 but thats really old, building from master is a better option and 0.9 shouldn't be too far away. And it should have importers (there's some code in master but afaik they aren't functional, or at least I didn't see a way to configure them but I also didn't look into it that much).

Sorry if this post seems clumsily put together, camlistore is emmensly complex and trying to write a coherent "intro to camlistore" turned out to be difficult as hell. More posts to come as I did deeper.