Preparing Plone to start the WebDAV service and setting the permissions to allow the users to make use of it is only half the battle, actually using it, especially from automated systems like build servers is another struggle.

Using the Cadaver WebDAV client

Although WebDAV is currently well integrated in modern desktop environments, a CLI alternative is useful for automation, like Jenkins build scripts.

Automatic Login

Ideally we want password-less operation from script, both from a usability as from a security standpoint.

Cadaver supports automatically logging in to servers requiring authentication via a .netrc file, like the ftp client. The syntax is such that the file can be shared for both tools.

The file ~/.netrc may be used to automatically login to a server requiring authentication. The following tokens (separated by spaces, tabs or newlines) may be used:

machine host

Identify a remote machine host which is compared with the hostname given on the command line or as an argument to the open command. Any subsequent tokens up to the end of file or the next machine or default token are associated with this entry.

default

This is equivalent to the machine token but matches any hostname. Only one default token may be used and it must be after all machine tokens.

login username

Specifies the username to use when logging in to the remote machine.

password secret

Specifies the password to use when logging in to the remote machine. (Alternatively the keyword passwd may be used)

Example ~/.netrc file

default
login jenkins
passwd secret

Example Session

Troubleshooting

Error 409 Conflict

This can mean a lot of things, like a real conflict. However most of the time it means that the folder where the stuff is uploaded does not exist:

dav:/cmdb/Members/pti/> mput Dropbox/Apps/Byword/plone_webdav.md
Uploading Dropbox/Apps/Byword/plone_webdav.md to `/cmdb/...':
Progress: [=============================>] 100,0% of 1488 bytes failed:
409 Conflict

This actually tries to upload the file to a subfolder Dropbox/Apps/Bywordwhich does not exist, causing this confusing error.

Simply changing the local directory solves the issue:

dav:/cmdb/Members/pti/> lcd Dropbox/Apps/Byword
dav:/cmdb/Members/pti/> mput plone_webdav.md
Uploading plone_webdav.md to `/cmdb/Members/pti/plone_webdav.md':
Progress: [=============================>] 100,0% of 1488 bytes succeeded.

Cannot create folders using WebDAV

Problem: The WebDAV “make folder” method, MKCOL, requires the “Add Folders” permission. This is not normally granted to Members or Owners on the site.

dav:/cmdb/Members/jenkins/> mkcol test2
Creating `test2': Authentication required for Zope on server `cmdb-uat.elex.be':
Username: 
Password: 
Retrying: Authentication required for Zope on server `cmdb-uat.elex.be':
Username: Terminated by signal 2.

Plone asks to login again because the current user has insufficient rights.

Workaround: In the Zope Management Interface, under the “Security” tab for the Plone root, check the “Owners” and “Managers” box for the “Add Folders” permission setting.

~  ᐅ cadaver dav://cmdb-uat.elex.be:1980/cmdb/Members/jenkins
dav:/cmdb/Members/jenkins/> mkcol test2
Creating `test2': succeeded.
dav:/cmdb/Members/jenkins/> 

Source: Members Can’t Create Folders Through WebDAV

Automating Cadaver with davpush.pl

Cadaver uses an ftp like command language to interact with the WebDAV server. This is very flexible, but impractical when a large number of files and folders must be uploaded. This happens often when the documentation for a new release must replace the previous version.

Cadaver accepts its input on the stdin stream, which allows us to pipe a script of commands to it. Since it is non-trivial to create and maintain such a script by hand, a script generator is needed. The generator presented here is meant to be simple and easy to use and modify. No attempt was made to made to add advanced syncing (like removing deleted files), handle exceptions gracefully or ‘do the right thing’.

With that in mind, organize the docs in such a way that it is easy to delete the target folder and push a fresh copy to clean everything up. This is common (and good) practice anyway in order to effectively use relative links within a subsite.

The principle is to cd to the root directory of the documentation root and run the script there and point it to the target.

Usage

davpush.pl dav://_hostname_:_port_/_upload path_

Uploads all files and folders recursively to the WebDAV folder passed in the url.

Code

#!/usr/bin/perl
use File::Find;

my $script = "";

sub wanted() {
  my $f = $File::Find::name;
  if (-f $f) {
    $script .= "put $f\n";
  } else {
    $script .= "cd $target_dir\n";
    $script .= "mkdir $f\n";
    $script .= "cd $f\n"
  }
}

my $url = $ARGV[0];
print "URL: $url";

if ($url =~ m#dav://.*?(/\S*)#) {

  my $target_url = "$0";
  my $target_dir = "$1";

  find({'wanted'=>\&wanted, 'no_chdir' => 1},   ".");


  $pid = open(POUT, "| cadaver $url");
  print POUT $script;
  print POUT "bye\n";
  close POUT;

} else {
  print "Usage: davpush.pl dav://<hostname>:<port>/<upload path>\n";
  print "\n";
  print "Uploads all files and folders recursively to the WebDAV folder passed in the url.";
}

Code Notes

The standard perl File::Find module traverses the folder tree in the right order to make sure all folders are created before other files or folders are created in them. Default behavior is to chdir to the directory, but then we lose the nice paths relative from the root, which would require additional administration entering and leaving the directory. Setting the no_chdir flag in the options keeps the paths like we want them in the script. (Look at the preprocess and postprocess options to help with the directory admin, but I think the added complexity will outweigh the gains for small to moderate trees)

For every file or folder, the wanted subroutine is called. For files we just add a mput command to copy the file over, because it keeps the path intact. If there is a file already (and the permissions are not screwed up) then it is overwritten. When we enter a new folder then we create the folder. If the folder already exists we get a (harmless) 405 Method Not Allowed error. Here we make another offer to the God of Simplicity, and ignore it.

After walking the tree, we have the script in the $script variable. It is unceremoniously piped as input for cadaver. We add the bye command to close the session, and we’re done. The output of cadaver appears on the stdout for easy verification using a MkI Eyeball check or by piping it to grep.

Sample session

~/Dropbox/Apps/Byword  ᐅ perl ~/tmp/davpush.pl dav://cmdb-uat.elex.be:1980/cmdb/Members/pti
URL: dav://cmdb-uat.elex.be:1980/cmdb/Members/ptiCreating `.': failed:
405 Method Not Allowed
Uploading ./plone_webdav.html to `/cmdb/Members/pti/plone_webdav.html':
Progress: [=============================>] 100,0% of 4007 bytes succeeded.
Uploading ./plone_webdav.md to `/cmdb/Members/pti/plone_webdav.md':
Progress: [=============================>] 100,0% of 6369 bytes succeeded.
Uploading ./Untitled.txt to `/cmdb/Members/pti/Untitled.txt':
Progress: [=============================>] 100,0% of 203 bytes succeeded.
Uploading ./Uploading to `/cmdb/Members/pti/Uploading': Could not open file: No such file or directory
Uploading documents to `/cmdb/Members/pti/documents': Could not open file: No such file or directory
Uploading to to `/cmdb/Members/pti/to': Could not open file: No such file or directory
Uploading Plone to `/cmdb/Members/pti/Plone': Could not open file: No such file or directory
Uploading with to `/cmdb/Members/pti/with': Could not open file: No such file or directory
Uploading WebDAV.md to `/cmdb/Members/pti/WebDAV.md': Could not open file: No such file or directory
Creating `./foo': failed:
405 Method Not Allowed
Creating `./foo/bar': failed:
405 Method Not Allowed
Creating `./foo/bar/baz': failed:
405 Method Not Allowed
Uploading ./foo/bar/baz/plone_webdav.md to `/cmdb/Members/pti/foo/bar/baz/plone_webdav.md':
Progress: [=============================>] 100,0% of 3380 bytes succeeded.
Creating `./images': failed:
405 Method Not Allowed
Uploading ./images/SJ09_1.jpg to `/cmdb/Members/pti/images/SJ09_1.jpg':
Progress: [=============================>] 100,0% of 31637 bytes succeeded.
Uploading ./images/SJ09_2.jpg to `/cmdb/Members/pti/images/SJ09_2.jpg':
Progress: [=============================>] 100,0% of 29182 bytes succeeded.
Uploading ./images/SJ09_3.jpg to `/cmdb/Members/pti/images/SJ09_3.jpg':
Progress: [=============================>] 100,0% of 31296 bytes succeeded.
Uploading ./images/SJ09_4.jpg to `/cmdb/Members/pti/images/SJ09_4.jpg':
Progress: [=============================>] 100,0% of 31094 bytes succeeded.
Uploading ./images/SJ09_5.jpg to `/cmdb/Members/pti/images/SJ09_5.jpg':
Progress: [=============================>] 100,0% of 26886 bytes succeeded.
Uploading ./images/SJ09_6.jpg to `/cmdb/Members/pti/images/SJ09_6.jpg':
Progress: [=============================>] 100,0% of 29373 bytes succeeded.
Uploading ./images/SJ09_7.jpg to `/cmdb/Members/pti/images/SJ09_7.jpg':
Progress: [=============================>] 100,0% of 34486 bytes succeeded.
Uploading ./images/SJ09_8.jpg to `/cmdb/Members/pti/images/SJ09_8.jpg':
Progress: [=============================>] 100,0% of 28561 bytes succeeded.
Uploading ./images/SJ09_9.jpg to `/cmdb/Members/pti/images/SJ09_9.jpg':
Progress: [=============================>] 100,0% of 27381 bytes succeeded.
Connection to `cmdb-uat.elex.be' closed.

2011-07-22 Fri 16:33
Figure out proxy settings Linux

Ubuntu has a Network Proxy chooser which allows you to select a location (a la MacOSX). This works well enough except that the UI is a bit counter-intuitive (in my humble opinion)which causes me to regularly nuke some predefined setting inadvertently. This is not a big deal though.

However for update manager (and several other tools) to pick up the new proxy settings you need to push the settings down to the system level. This takes 2 times typing your password. Now, this IS a big deal.

When I go back and forth between work and home I have to change this at least 2 times per day. Also it irks me that a detail setting like the proxy is not auto-detected and I need to login to change this ‘system’ setting. My laptop is essentially a single user system and I do not see switching the proxy as a serious security issue, even with 3 kids running around the home.

To come back to auto-detection, while this works fine at work, it fails to figure out that at home that there is a direct connection to the Internet. I can probably fix this by replacing my aging wireless router with my Time Capsule as the Internet gateway router, but I prefer to have the Time Capsule close to my desk.

In any case the Network proxy shows 2 times the authentication dialog box. A particularly nice feature (Is this new in Natty?) is that the dialog shows for which DBUS setting access is being asked.

The first dialog asks access to com.ubuntu.systemservice.setProxy. This response is configured in the file /usr/share/polkit-1/actions/com.ubuntu.systemservice.policy. This is a very readable XML file which contains a section for the setProxy action. I feel no reservation in allowing unchecked access to the setProxy. Although this might make a man-in-the-middle attack easier someone with the sophistication to pull this off, does not need to doctor my PC to do it.

<action id="com.ubuntu.systemservice.setproxy">
  <description>Set current global proxy</description>
  <message>System policy prevents setting proxy settings</message>
  <defaults>
    <!-- PTI : original settings
    <allow_inactive>no</allow_inactive>
    <allow_active>auth_admin_keep</allow_active>
    -->
    <allow_inactive>yes</allow_inactive>
    <allow_active>yes</allow_active>
  </defaults>
</action>

Retrying, and indeed one of the authentication requests dropped.

Note that the action was configured with auth_admin_keep which according to the docs would mean we should be authenticated for some time,so I would not expect the second authentication I am getting. Must be a subtlety which escapes me at the moment.

The second action is more problematic since the set-system om the system gconf settings is much less fine-grained than setProxy and can potentially cause more damage to the system.

<action id="org.gnome.gconf.defaults.set-system">
  <description gettext-domain="GConf2">Change GConf system values</description>
  <message gettext-domain="GConf2">Privileges are required to change GConf system values</message>
  <defaults>
    <allow_inactive>no</allow_inactive>
    <!-- PTI: Original setting
    <allow_active>auth_admin</allow_active>
    -->
    <allow_active>yes</allow_active>
  </defaults>
</action>

After relaxing this second method, I can finally easily switch proxies between Locations.

There are several things bugging me:

1. The set-system method really is too wide in scope.
2. There should be a more elegant way that modifying files under /usr/share
3. My system should actually switch location unaided.

For the time being, I fixed a frustration and learned something in the process. The result is not yet fully satisfactory, but that will improve over time. **

• Tutorial explaining editing static files
• pklocalauthority man page
• Install Updates as Unprivileged User

Of course there is a more elegant solution than editing files under /usr/share folder.

Everything is explained in the man pages pklocalauthority and PolicyKit.conf (among others).

But that’s for another day…

Dotfiles, can’t live with ‘em, can’t live without ‘em.

Dot files are the term for folders and files starting with a ‘.’ so they do not show upwhen using plain ls.

Tha Mac has a cool keycode to toggle visibility of dotfiles in the File Open/Save dialog, but this does not work in the finder for one reason or another.

In practice this meant I had to deal with dotFiles and dotDriectories I found on the net some incantation to force the setting for the finder to show/hide the dotfiles. Upon restarting the finder the widows will reopen with the updated setting.

I found on the net some snippets of sh script (but I forgot where and cannot immediately retrieve it), and I immediately dumped them in my ~/bin folder.

~/bin/hide-dotfiles :

#!/bin/sh
defaults write com.apple.finder AppleShowAllFiles -bool FALSE
killall Finder

~/bin/show-dotfile :

#!/bin/sh
defaults write com.apple.finder AppleShowAllFiles -bool TRUE
killall Finder

So in order to toggle the setting in the graphical finderyou drop to the commandline set the flag in the desired state and restart the finders. Works nice…

One remark : if you suddenly see strange icons on your desktop, then you probably still have the dotFiles visible selected. hide-dotfiles will clean your desktop up again.

*