Fix subverison error: Valid UTF-8 data followed by invalid UTF-8 sequence

This is a solution to fix problems with SVN when you can't update your working copy for some rather odd reason. Everything you get is an obscure error message like this:

svn: Valid UTF-8 data
(hex: 65 64 69 74 65 64)
followed by invalid UTF-8 sequence
(hex: ad 6c 69 73)

This does not only appear when doing svn update but even pops up while svn status.

Since Subverion can handle binary files this is quite confusing. At luck, after some googling I found out that these errors are caused by file names with e.g. Chinese characters.

Unfortunately the error message can't display the corrupt file name because it contains non-UTF-8 data. So I figured that the „Valid UTF-8 data“ (In this case the hex sequence 0x65, 0x64, 0x69, 0x74, 0x65, 0x64) translates to the string „edited“ using some UTF-8 table.

Since there were way too many files with this string I had to look for the 0xAD 0x6C sequence. This could be the asian symbol 구, but you can't grep for this, because it is not UTF-8 encoded. However we can look for the byte sequence using some perl magic:

find /path/to/workingcopy | perl -n -e "print if /\xAD\x6C/" | less

Note the hex-regexp used here to scan binary content in file names. It outputs a nice (and in this case rather short) list like:

/path/to/workingcopy/folder/of/colleque/edited<AD>list.txt

Interestingly, less tries to expose the binary data. Now you just have to rename the file and you're good to go and able to update your working copy again.

Howto: Start using Subversion

Sometimes you want to start using Subversion code control with an existing project. This tutorial explains the steps required to create a repository and add the files of the existing project.

Start by creating a repository:

svnadmin create /dir/to/store/repo/repository-name

This will create a directory „repository-name“ containing the database of files and revisions.

Then let's add some internal directories to our repository:

svn mkdir file:///dir/to/store/repo/repository-name/trunk \
 -m "Creating repo directory structure"

Now that you have created the repo-directory that should contain all the files, import the existing project files. Start by checking out the empty repo-dir to into your project dir. This will make your project dir a working copy (that is: create a dir named „.svn“ containig some internal info) but won't change anything else.

cd /existing/poject/dir
svn checkout file:///dir/to/store/repo/repository-name/trunk .

Then go on adding (or planing to add) all the files:

svn add *

This command will list all the files that will be loaded into the repository. You can always look at the planned changes with svn st.

You probably want to exclude some files such as configuration files, runtime data. You just have to revert the add-command again:

svn revert runtimedata # exclude whole dir
svn revert config/my.ini # exclude single file

Then apply the changes to your repository (commit it):

svn commit \
-m "initial import"

And that's it. You project is added to the repository without extra files. That can be crucial if your runtime data has a huge file size. If you're paranoid or just curious check the repository for success:

svnlook tree /dir/to/store/repo/repository-name

This should show the imported directory tree. Now you can start making changes to your files and commit them as usual.

vim app.cpp
svn commit -m "typo"
svn log -r 1:HEAD #show full revision history

If you ever got stuck, you may use the built in help:

svn help #list commands
svn help commit #list options
svnlook help tree #works too

I will start using subversion for everything... soon.

SVN mit SVG und XSLT gibt Visualisierung

Warning: filesize() [function.filesize]: stat failed for /home/jail/bxt/bxt/de.bernhardhaeussner/htdocs/upl/svn+svg+xslt_rockz.tar.gz in /home/jail/bxt/bxt/de.bernhardhaeussner/lib/php/engine.classes.php on line 229

Als ich zum ersten Mal über XSL bzw. XSLT stolperte interessierte es mich sofort. Im Gegensatz zu der von PHP und MySQL gewohnten, mehr oder weniger prozeduralen, Verarbeitung und dem Datensatz-Prinzip werden nun XML-Bäume mit XPath-Templates transformiert. Nun habe ich erstmals eine sinnvollere Anwendung gefunden: Das Transformieren eines XML-Subversion-Logs in eine Visualisierung als SVG.

Vorallem im Web-Bereich liegen fast alle Daten (wenn nicht in einer SQL-DB) in XML-Dateien vor. Ein Beispiel sind AJAX-Anwendungen. Auch viele APIs benutzen XML zur Informationsübermittlung. Natürlich kann auch PHP mit XML umgehen, doch mit XSLT findet sich eine einfachere Möglichkeit, da man im XML-Bereich bleibt und nicht so viel PHP-Syntax und anderes hineinmischen muss.

Natürlich ist XSLT zu PHP grundsätzlich verschieden, da die Scripte nicht „live“ auf einem Server laufen, sondern die Transformation entweder ganz beim Client erfolgt oder das einmal Transformierte Resultat gesendet wird. Und XSLT liegt eigentlich auch fernab von jeder Webanwendung, ganz allgemein eben transformiert es Daten zwischen verschiedenen XML-Formaten.

Nun wird XSLT im Webbereich meist verwendet, um Daten in XHTML-Seiten umzuwandeln, ich verwende allerdings ein anderes Format der XML-Syntax: SVG.

Download

Es ist noch ein kleines Schell-Script dabei, um das Log zu erstellen und den XLST-Prozessor zu starten.