Paul Ivanov’s Journal

Code generation tools: a degenerative disease for free and open source software

2025-05-08T00:00:00-07:00

There have been some conversations in the Scientific Python community over the past several months about what, if anything, should be done regarding contributions that utilize code generation tools. In preparation for next week's Developer Summit, Matthew Brett opened an issue to begin drafting a SPEC to help the community navigate this terrain. What follows is my contribution on that thread.

Any project that accepts code that was generated by tools violating license terms dilute and invalidate their own license.

These include approximately all of the automated tools in use today, as even training sets that exclude GPL nevertheless violate the terms of BSD and MIT licenses which require attribution.

I think a way to address this would be to have attestations in the form of checkboxes and fill-in-the-blank, akin to those in Contributor License Agreements (CLAs), that states either that "I did not use code-generation tools in the production of the code submitted here", or that "The code generation tools I used was trained only on public domain, no attribution such as zero-clause BSD, zero-clause MIT, or code within this project" with a fill-in-the-blank requirement to specify the models that were used. In particular, in no uncertain terms it should reject contribution that utilized any of the popular tools in use today, which includes GitHub CoPilot, ChatGPT, Claude, Llama, Mistral, Replit Code, StarCoder, CodeLLama.

Projects without such a direct approach are open to drifting and sliding into a post-legal world. It doesn't matter what license they use, as they lack the means to assert their own license due to a lack of provenance of their incoming contributions.

You can find this post on LinkedIn, Bsky, Twitter, or Mastodon.

diff-jfk

2025-03-28T00:00:00-07:00

I did some data munging to match the latest release of JFK assassination records to their previously released version to produce a pop-out effect of the newly declassified parts. The resulting over two thousand documents, spanning eighty thousand pages are now all posted on github.

As of March 20th, there is a total of 2,343 documents released by the U.S. National Archives and Records Administration (NARA) in JFK Assassination Records - 2025 Documents Release.

The total page count that NARA indicates that there were a total of 77,100 pages released, though in downloading all 2,343 files and summing their page numbers, I get a higher total of 82,864 pages.

Wouldn't it be nice if someone went through to highlight the newly un-redacted parts of previous reports? Well, dear reader, I got you!

But first let me help you get oriented.

`_fresh` vs `_highlighted`

There are two flavors of PDFs that I've generated - _fresh and _highlighted. The _fresh versions contain only the pages where there is something different, and each page has some text at the top indicate what the corresponding page number was in the original released document. There's also a _highlighted version, which contains whole document, with the highlighting of differences in their entire context. The notes at the top of the highlighted document (and top of each fresh page) are in light blue so as not to be too intrusive, and you can highlight them to read or copy when you end up seeking out the original, because in cases where the diff-jfk version is not legible.

Now you have the necessary information, if you're ready to dive in, just click below:

>> Happy Sleuthing <<

Otherwise, stick around for some more information and examples.

Unredacted diffs

Turns out that Mary Ferrell Foundation is an excellent resource, see their 2025 JFK Records Releases. Unlike the MFF, which is choosing to redact the social security numbers, diff-jfk is not.

The only visual differences in the diff-jfk generated documents that are omitted from counting toward fresh pages are the very top of the first page document. This is where there would be an expected difference in the banner indicating the year of the release. And also the bottom of each page that was used to keep track of document ID and page number but ended up reproducing slightly different in some of the documents, so I discarded that to get fewer false-positives.

These trivial changes are not informative, so they has been explicitly omitted from "fresh" consideration. So a document that was initially released in 2023, and again released in 2025, would have a red mark in the last digit of the year in the banner on top of its first page, but such a page would not be considered "fresh" if there are no other changes on that first page. You will still see that red mark, however, if there were other changes to the page.

Some samples

Here are some examples of the new information that's been released:

2025/104-10301-10001_fresh.pdf - fresh page 1 has previously censored details of CIA's 1961 budget: $41.5 million spent on Cuban operations, $11 million on the rest of Latin America

2025/124-10114-10010_highlighted.pdf shows you the one previously redacted box, which now reveals the word "Embassy" in between "Paris" and "CAS". CAS is one of the terms of art that I learned in preparing these documents - it stands for Controlled American Source - CIA.

When it works, it works brilliantly. 2025/177-10002-10105_fresh.pdf has just the two pages that have newly declassified information which concerns understanding of Soviet submarine fleet in 1966-68, but if you wish you can peruse the entire 14 page document which now highlights the declassification.

Take for example, 2025/104-10307-10006_highlighted.pdf, where you can learn, among other things about how CIA set up a special facility in Miami to intercept electronically internal communications within Cuba,

details of the 14 countries' spy agencies which "have been most cooperative and helpful" (page 12),

where one of the sources was a former protege of Raul CASTRO , one was and ambassador of a North African country in Cuba, and another was a Cuban ambassador in Western Europe (page 5).

Document 104-10178-10000 contains, among other things, the tales of WKSCARLET-3, with some juicy bits about plans for that Venezuelan to bug the Colombian delegation in Rome during Colombian-Venezuelan border talks in 1973.

The full-context highlighted document has 310 pages, whereas the freshly declassified pages add up to 86. The image above is page 72 in the original document, and appears on page 9 of the fresh-only version.

Limitations of Minimal Magic

I didn't do anything fancy to try to get the pages to line up properly. Also, the approach I took with highlighting works pretty well for pages that were previously redacted in white, but isn't as legible for documents which were previously blacked out. So take this with a grain of salt. Hopefully it helps you identify where to look in the original document if you can't make it out in the diff-jfk version.

Slight rotations, rescans, change to the cropping, and so on, make the previously unredacted parts of the document either hard to read or completely illegible. Nevertheless, despite this limitation, it's still makes it easier to scan the highlighted document to identify which pages to look at in the newly unredacted document. For example, scanning through the otherwise atrocious mess in the diff-jfk version of 2025/124-10179-10220_highlighted.pdf, I was quickly able to come to page 37 that previously had censorship boxes, and looking on the same page of the original, I can see that the August 20, 1968 (Warsaw Pact invasion of Czechoslovakia) example of positive intelligence involved the wiretapping of, previously redacted, "the Soviet Mission to the United Nations". On the next page, phrase "the Romanian Embassy" was omitted from a description of another wiretap, and further down the page the previously censored phrase of "intercepts on various missions to the United Nations and embassies" providing many indicators during Arab-Israeli Six Day War and the India-Pakistan War.

There's more to be done, but I wanted to get this out there and not let perfect be the enemy of good enough. Here's an example of what's supposed to be the same document but having different page numbers. On the left hand side is Page 57 of 2023/docid-32404520.pdf and on the right is page 61 of 2025/104-10332-10007.pdf

By the way, that's definitely one of the recurring themes for what I've come across being previously censored but now reveled in these documents: the use of State department cover. It also came up in, to name just a few: 20225/104-10330-10102.pdf and 2025/104-10330-10083.pdf

How `diff-jfk` came to be

The first thing we can do is just hope for the best and assume that the filename hasn't changed and exactly matches the previous release (2023).

That gets us 849 files, 36.24% of the total file count with exact matches that were previously released in 2023. Not bad for a first pass, and this haul actually gets us the bulk of the raw-page count, because these 849 files contain 44770 pages, so we're already more than half way there!

To continue on, it's a bit of a slog, because whereas in 2025 all files started off with the 3-5-5 pattern, that's not the case for earlier years. We luckily have a spreadsheet from 2023 which gives us a mapping for record number to filename. There are also such spreadsheets for 2022, 2021, and the 2017-2018 releases.

Duplicate ambiguity

Ok, we also have the problem of ambiguity where from the filename, we have multiple files for the same record number. For example, we have ./2025/104-10105-10290 (C06932214).pdf and also ./2025/104-10105-10290.pdf, there's also 2025/124-10273-10007_multirif.pdf and 2025/124-10273-10007.pdf and so on.

Anyway, I'll spare you the rest of the details, except to say that I got pretty close to finding previous versions of all of the files.

2023:       matching files: 873,    left: 1456
2022:       matching files: 1349,   left: 107
2021:       matching files: 0,      left: 107
2017_2018:  matching files: 13,     left: 94

There are 2,221 files from the 2025 release that I've managed to match to a previous version with the same page count.

Another 26 files where the previous and 2025 release have a different number of pages. A handful of those just lost or gained a page or two, I was able to accommodate and fix, but there's more work to be done to completely close those out.

Table of 26 files where previous version does not have same page number (click here toggle visibility)

File Name	Num Pages	Previous File Name	Previous Num Pages	Page Difference	Notes
`2025/104-10302-10028.pdf`	133	2023/docid-32401336.pdf	140	-7	The President’s Intelligence Checklist. 2025 shorter by 7 pages, it starts at 21 Nov 1963, whereas in 2023 there was also 20 Nov 1963 – page scrabmled issues
`2025/104-10330-10044.pdf`	5	2023/docid-32404031.pdf	6	-1	Page 1 from 2023 omitted in 2025, scanned in a slightly rotated orientation
`2025/104-10330-10083.pdf`	13	2023/docid-32404069.pdf	14	-1	Page 1 from 2023 omitted in 2025, scanned in a slightly rotated orientation
`2025/104-10332-10007.pdf`	81	2023/docid-32404520.pdf	77	4	4 pages inserted somewhere, order of pages different after page 35
`2025/104-10332-10008.pdf`	22	2023/docid-32404521.pdf	23	-1	Page 1 from 2023 omitted in 2025, scanned in a slightly rotated orientation
`2025/104-10332-10009.pdf`	76	2023/docid-32404522.pdf	77	-1	None
`2025/104-10332-10021.pdf`	22	2023/docid-32404534.pdf	23	-1	Page 1 from 2023 omitted in 2025 and order of pages is different
`2025/124-10179-10217.pdf`	2	2022/docid-32989640.pdf	370	-368	None
`2025/124-10179-10219_multirif.pdf`	370	2023/124-10179-10219.pdf	2	368	None
`2025/124-10179-10221_multirif.pdf`	370	2022/docid-32989574.pdf	640	-270	should actually be 2022/docid-32989640.pdf
`2025/124-10185-10099 (c06716626).pdf`	8	2023/docid-32989624.pdf	288	-280	None
`2025/124-10185-10099.pdf`	170	2023/docid-32989624.pdf	288	-118	2025/124-10186-10041.pdf is 118 pages long, does it fit here?, also with 8 pages from c0671 below, there’s 2025/104-10210-10037.pdf that’s 110 pages long, as is 2025/104-10164-10110.pdf
`2025/124-10185-10214.pdf`	11	2022/docid-32989599.pdf	126	-115	None
`2025/124-10204-10101.pdf`	10	2018/docid-32989602.pdf	147	-137	None
`2025/124-10206-10180.pdf`	40	2022/docid-32290904.pdf	12	28	None
`2025/124-10222-10171.pdf`	3	2022/docid-32298395.pdf	2	1	Page 3 new in 2025
`2025/124-10226-10305.pdf`	165	2022/docid-32300025.pdf	35	130	None
`2025/124-10271-10299.pdf`	14	2023/docid-32989588.pdf	270	-256	None
`2025/124-10271-10301.pdf`	1	2022/docid-32989609.pdf	223	-222	None
`2025/124-10273-10086_multirif.pdf`	440	2018/docid-32989602.pdf	147	293	None
`2025/124-10273-10289_redacted.pdf`	3057	2023/docid-32989646.pdf	1417	1640	None
`2025/124-10274-10029.pdf`	3	2022/docid-32989627.pdf	304	-301	None
`2025/124-90073-10039.pdf`	28	2022/docid-32328335.pdf	3	25	None
`2025/157-10014-10005.pdf`	112	2023/docid-32423393.pdf	113	-1	Page 61 from 2023 omitted in 2025 (as evidenced by page count at the bottom that goes 60 to 62.
`2025/177-10002-10092.pdf`	7	2023/177-10002-10092.pdf	6	1	Page 2 new in 2025
`2025/180-10147-10176.pdf`	24	2023/180-10147-10176.pdf	25	-1	Page 1 from 2023 omitted in 2025, scanned in a slightly rotated orientation, size mismatch

It's not perfect, but it's a pretty damn good start.

Previously unreleased files

Finally, here's a list of files that I believe are entirely new in the 2025 releases. As there is nothing to diff against, I did not generate fresh or highlight files for them. You can these new files here in their own folder.

Table of 94 files and their page counts, that were first released in 2025. (click here toggle visibility)

File Name	Num Pages
`2025/124-10203-10246.pdf`	6
`2025/124-10203-10276.pdf`	19
`2025/124-10203-10292.pdf`	3
`2025/124-10203-10294.pdf`	21
`2025/124-10203-10464.pdf`	6
`2025/124-10203-10465.pdf`	2
`2025/124-10203-10466.pdf`	3
`2025/124-10203-10468.pdf`	2
`2025/124-10203-10482.pdf`	3
`2025/124-10203-10485.pdf`	2
`2025/124-10203-10489.pdf`	4
`2025/124-10203-10494.pdf`	2
`2025/124-10203-10497.pdf`	2
`2025/124-10204-10004.pdf`	4
`2025/124-10204-10146.pdf`	2
`2025/124-10204-10147.pdf`	3
`2025/124-10204-10154.pdf`	2
`2025/124-10204-10157.pdf`	3
`2025/124-10204-10203.pdf`	4
`2025/124-10204-10204.pdf`	4
`2025/124-10204-10206.pdf`	3
`2025/124-10204-10209.pdf`	2
`2025/124-10204-10211.pdf`	4
`2025/124-10204-10212.pdf`	2
`2025/124-10204-10213.pdf`	3
`2025/124-10204-10215.pdf`	2
`2025/124-10204-10278.pdf`	3
`2025/124-10204-10297.pdf`	4
`2025/124-10204-10322.pdf`	4
`2025/124-10204-10495.pdf`	46
`2025/124-10223-10008.pdf`	3
`2025/124-10223-10015.pdf`	2
`2025/124-10223-10017.pdf`	3
`2025/124-10223-10042.pdf`	2
`2025/124-10223-10064.pdf`	3
`2025/124-10223-10067.pdf`	2
`2025/124-10223-10068.pdf`	3
`2025/124-10223-10070.pdf`	3
`2025/124-10223-10071.pdf`	2
`2025/124-10223-10073.pdf`	3
`2025/124-10223-10074.pdf`	2
`2025/124-10223-10076.pdf`	2
`2025/124-10223-10077.pdf`	2
`2025/124-10223-10079.pdf`	3
`2025/124-10223-10082.pdf`	3
`2025/124-10223-10083.pdf`	2
`2025/124-10223-10085.pdf`	2
`2025/124-10223-10086.pdf`	2
`2025/124-10223-10087.pdf`	2
`2025/124-10223-10088.pdf`	3
`2025/124-10223-10101.pdf`	3
`2025/124-10223-10121.pdf`	2
`2025/124-10223-10159.pdf`	4
`2025/124-10223-10164.pdf`	3
`2025/124-10223-10169.pdf`	3
`2025/124-10223-10171.pdf`	3
`2025/124-10223-10173.pdf`	4
`2025/124-10223-10304.pdf`	3
`2025/135-10001-10237.pdf`	3
`2025/135-10001-10238.pdf`	3
`2025/135-10001-10239.pdf`	2
`2025/135-10001-10243.pdf`	11
`2025/135-10001-10245.pdf`	5
`2025/135-10001-10246.pdf`	28
`2025/135-10001-10253.pdf`	23
`2025/135-10001-10255.pdf`	19
`2025/135-10001-10276.pdf`	31
`2025/135-10001-10279.pdf`	4
`2025/135-10001-10283.pdf`	7
`2025/135-10001-10287.pdf`	9
`2025/135-10001-10288.pdf`	57
`2025/135-10001-10297.pdf`	215
`2025/144-10001-10213.pdf`	5
`2025/144-10001-10218.pdf`	3
`2025/180-10143-10222.pdf`	6
`2025/180-10143-10384.pdf`	10
`2025/206-10001-10000.pdf`	22
`2025/206-10001-10001.pdf`	34
`2025/206-10001-10002.pdf`	68
`2025/206-10001-10003.pdf`	23
`2025/206-10001-10004.pdf`	10
`2025/206-10001-10005.pdf`	18
`2025/206-10001-10006.pdf`	9
`2025/206-10001-10007.pdf`	13
`2025/206-10001-10008.pdf`	10
`2025/206-10001-10009.pdf`	11
`2025/206-10001-10010.pdf`	11
`2025/206-10001-10011.pdf`	89
`2025/206-10001-10012.pdf`	29
`2025/206-10001-10013.pdf`	30
`2025/206-10001-10014.pdf`	7
`2025/206-10001-10015.pdf`	15
`2025/206-10001-10016.pdf`	8
`2025/206-10001-10017.pdf`	26

Issues and Discussion

If you want to leave a comment, or just come across something cool or interesting that you want to share with others, please open a new thread or join one over in the discussions.

Please report any issues you come across in the diff-jfk data. In particular, I'd like to know if I have mislabelled some of the 2025 releases which I marked as new, so do let me know if there's a prior version of one of those documents, so that I can generate a proper diff of those. I'd also like to know if some of the file contents from NARA have changed. This last part might seem like an odd, but here's are some examples of document id confusion from 2018: multiple record identifiers mapping to the same file name, and the same filename having different version depending on when it was downloaded.

Parting thoughts

It took more time than you might expect to tinker around and connect all of the dots to produce diff-jfk. I hope you enjoy perusing it as much as I do. If you end up finding something interesting, why not share it with others by posting in the discussions. For attribution, please link to either this post or the diff-jfk repository, which you can also star if you've got a github account and are feeling generous. I also enabled sponsorship on that repo if you want to support my work financially.

>> Happy Sleuthing <<

Paul Ivanov

Current Challenges in Free Software and Open Source Development

2025-01-24T00:00:00-08:00

This is a lightly edited transcript of a talk I gave at Bay Area Drupal Camp in Oakland on October 24th, 2024 to an audience of about 40 people. I will be inlining images, but you can also peruse the slides directly. There's a recording available that has audio synchronized with the slides, but this post leaves out my stumbles, uhs, ums, and other vocal pauses. (Additional context will appear like this)

Thank you all for coming. I just want to get started by asking you a little bit about yourself. So how many of you identify as developers? ( 3/4 of audience raises their hands ) How many of you have transcended developers? (a fifth of the room) Still alive, still doing okay? Okay. How about open source developers? (half the room) Commercial developers? (a fifth) Okay, so some of the same hands are going up. Great and competing sessions are also great, so we won't say anything bad about them. I don't even use Playwright or what was the other one, Cypress — I use TestCafe I'm, like, weird.

There we go, a little bit about me — I'm... weird.

I've been in the Python ecosystem for a very long time. I started off in graduate school (2006) at UC Berkeley working on matplotlib. I got my commit rights there, so that's the community I came from, and then I became one of the leaders in IPython and the Jupyter world, a project that's been rewarded by the ACM with the Software System Award (2017).

Now, I don't normally start off my talks with like "oh look at me — I'm a big shot! Look at this big award I got" but this will be relevant, very relevant, to the talk because the ACM is the Association for Computing Machinery, as you may know. It's the Professional Organization for computer scientists it was founded in 1947, so 77 years ago.

Since most of the room raised their hands and said they're developers, I'm going to get super nerdy because we're all among friends here, and talk history and things like that.

The award is a glass trapezoid sitting on its side, so it doesn't actually film very well. You can kind of maybe see that there's this reflection, anyway, but the books in the background (Frank Herbert's Dune) are foreshadowing. So you can kind of mentally place a 🌶️ chili emoji somewhere on there, if you're into that sort of thing, because the spice must flow. (audience chuckles)

The other things on the slide is that I used to work in academia at UC Berkeley, that's where I stuck around basically to work on Jupyter, and then I worked at a couple of startups and a couple of finance firms, and currently working on a Jupyter notebook search engine over at spines.dev. So that's enough about me.

Contributors to this talk ( Itay Dafna, Matt Turk, Juan Nunez-Iglesias, David Nicholson, Carol Willing )

I want to thank some of my friends that helped me reason through some of these things talking about them in particular Matt Turk who spent a lot of time back and forth IM-ing ideas and what to say and how to say it. So this is the first attempt at capturing this sort of thing.

But enough about them let's talk about me again. (chuckles)

I was born 1984, which makes me 40, and the same year the Macintosh was introduced. You remember Apple's smashing the screen Big Brother commercial?

So that was 1984 and that was also the year of the first hackers conference which was just across the Bay and that happened because Steven Levy wrote a book Hackers: Heroes of the computer Revolution

How many have read that book? (no one raises a hand) Oh that's such a good book, it's great. This is "hackers" before that word was associated with breaking into computers because people weren't even breaking into computers yet, people were busy trying to build computers to begin with.

So at that hacker conference, Richard Stallman first publicly explicitly stated the idea that all software should be free and makes it clear that free refers to freedom, not price, and saying that software should be freely accessible to everyone.

It was the first time he did that publicly, okay and so what's interesting about that being 40 years ago, is that if you go back 40 years in the other direction.

(I spread out my arms, turning them into a timeline, each arm measuring 40 years)

# 1944 - - - - - - 1984  - - - - - - 2024
#   \                \                 \
#    no computers    free software     today

With time flowing left to right, my nose is where I was born, 1984. And all the way on your right, this is where we are today. At the tip of my opposite hand, there were no computers, basically. 1944 was when the first computer, the Harvard Mark 1, was built, and if you were interested in computing Bessel functions you had a computer for that. You didn't really have a generic computer yet, so this (my armspan) is all of computing history in the modern sense. This is not talking about just calculators but computers, right. And so half of the time we've spent with this idea of free software and building it and sharing it, explicitly.

Okay, so the oldest programming language that's still in use today, FORTRAN was specified in 1954 and it took 'em another 3 years to build a compiler for it. COBOL was finished in 1959, LISP in 1960, so that's all back here. But what I'm concerned about is the future, because a tremendous amount of value has been unlocked from all of this creative tinkering that people were doing trying to scratch their own itch. The previous talk(CTO of The Drupal Association Tim Lehnen) mentioned the LAMP stack right: Linux — okay that's free software, Apache — yeah that's free software, MySQL / Maria DB whatever it was, that was free software too, and P stood for PHP in this world, it's Python where I'm from, all of that is free software. Linux kernel developer and maintainer Greg Kroah-Hartman in a talk just about a month ago said that Debian runs 70% of the world. He talks to a lot of cloud vendors. He's paid by the Linux Foundation to work on Linux and he says at least 70% of cloud workloads run Debian, which is a non-commercial distribution, if you're not familiar, and 80% or more than 80% he estimates run non-commercial distributions in general. So he says that Red Hat, SUSE, Ubuntu are great distros, but they're not what the world is using.

(I neglected to explicitly assert here that there would be no cloud computing without open source and free software )

And the concern is that this world that is being used right now is going to shrink and shrivel and die on the vine.

Have I painted a dark enough picture?

So the challenges that I want to talk about today are the three up here. We're going to talk about LLMs for code completion. We're going to talk about mixing paid and volunteer labor and how inherently you can't get away from it like that's that's a struggle that will always be there provided that you want to have hobbyists around. And then the last one is this relicensing of open source software that been happening with a lot of venture backed companies, which people have been referring to's a "rug pull" of taking away a license that was previously "I can do anything with the software" to now "I can't" and what to do about that.

So let me just start with that first one and say that

GitHub Copilot, and other large language models for code completion trained on publicly available software with no regard for the licenses of that software, acts as a license laundering cudgel that denigrates the work of open source developers who have contributed their code under a legal framework for how their code can be used. If their code was contributed under a copyleft license, like the GPL which Richard Stallman introduced a couple of years after that 1984 meeting, their expectation is that no user of their code will ever lose the ability to modify the software. If it was contributed under a permissive license, like the BSD or MIT licenses, the expectation was that inclusion of their code would result in an attribution to the project they contributed to. Tools like GitHub Copilot makes all of those expectation of the original authors null and void, enabling theft of their work, all in the name of making it "easier" for other programmers.

Okay so I called this a pyramid scheme in the abstract and here's what I mean: normalizing plagiarism. It's also a heist. This is a picture of the Louvre, if you're not familiar with it, and it's on the scale of taking everything from the Louvre. That's what I think. Talk about "value capture", right from the previous talk.

And for a pyramid scheme, you need two things. Okay, it's loosely defined as a pyramid scheme, it's just an effective metaphor, I think. You need to have some schemers that want to corrupt other parties. They want to to get more people into the game, "oh yeah, let's get them in there" and then you need somebody profiting at the top. Sort of, the higher up in the pyramid you are, the more profit you'll get okay.

So these are the two things that I'll need to show, or that I will be end up showing.

Hopefully it's self-evident but let's roll with it anyway.

So let's start with the base and work our way up.

If enough people participate in a plagiarism, we expand the world of plagiarist. We normalize it. I'm just repeating myself here. Let's take a look at how the corruption pyramid looks like at its base.

Fundamentally, no one using it, or rather the system is built in such a way that you are meant to not care where the code came from and I think that matters and I think we should care where the code came from.

So GitHub does their little meeting or big meeting, GitHub Universe they call it, I think it's this week or next week across the bay in San Francisco. And every year they release the State of Open Source as they see it. This year's survey hasn't yet come out, but in last year's survey there was a nice little banner the in the middle of it telling you: "GitHub Copilot: don't fly solo. Try it, it's free. First one's free."

Okay, in the middle of their report they say "speaking of generative AI, almost a third of Open Source projects use GitHub Copilot." Well, that sounds like a lot, although you know "at least one star"? I can star my own projects. Nevermind.

Then not five sentences later they repeat the same claim: "open source maintainers are adopting generative of AI" and then let's take a look here oh "this follows our program to offer a GitHub Copilot free to open source maintainers"

In 2023 there were "301 million total contributions to open source projects across GitHub". Wow, that's great, and "commercially backed projects continued to attract most of the open source contributions" — that's also interesting.

Okay, so, hmmm... I mean I know I'm a little conspiratorial here, but this is what this sounds like to me.

(laughter, Tim Lehnen sitting near the front says: "But I like cookies?!") You do like cookies? I like cookies, too. I have coffee, I don't have a cookie, right now. Wouldn't it be nice to have a cookie? I'd like a cookie...

I'm not going to convince them, the makers of these tools how to do it, but I think that there's a different way to live.

So hopefully I've made the case for sort of the base of the pyramid and the pyramid keeps wanting to perpetuate itself in the hype cycle that wishes to make the whole world feel it's normal to steal other people's work without attribution.

Okay so where's the top of the pyramid? Somebody has to be profiting from this.

Well in the Q2 earnings report, I mean you can you can read faster than I can but basically, GitHub Copilot as of July of this year makes more money than all of the GitHub business did when Microsoft acquire it acquired it 6 years ago.

So Microsoft acquired GitHub for for $7.5 billion 6 years ago. Today, GitHub Copilot makes more money for GitHub than GitHub did back then.

Moreover, that's not all. If you license Copilot business or Copilot Enterprise directly from GitHub this document applies to you.

The first one's okay you get to keep your own code. That's that sounds great until you read the next one which tells you that, oh by the way, you're on the hook if you're doing something wrong. If you're violating somebody's rights, that's on you, that's not on us. We're just mAkInG sUgGeStIoNs (chuckles) "I'm not stealing other people's work without giving them credit, I'm making a suggestion." So they got users' money so that the users themselves can potentially infringe others rights.

I think there's also a secondary pyramid scheme, one that might work out for those of us in the room that are already good at developing software.

The case is that we have been, for 40 years, writing lots of code most code is buggy so we've got lots of work to do. It's job security for those of us that know how to code.

I would claim that these tools that make it easier to learn how to code, they help to make it easier for you to get fish, they don't teach you how to fish. You get the end product faster and there's evidence for this I don't have the citation handy (Here the CACM article: "The Impact of AI on Computer Science Education " about Professor Eric Klopfer's experiment in his undergraduate computer science class at MIT), but there was a study done with three groups of students to complete a programming project. One of the groups got to use something like Chat-GPT, then the second group it was something more like Copilot where they weren't talking back and forth with getting code suggestions but it was just inline code generation. And then the last group got to use a search engine and had to do it.

So guess who completed the task fastest? ChatGPT, right. Guess who was unable to, or who was the worst at figuring out how to change that same code for a follow-up task? ChatGPT.

So the search engine User Group took the longest to finish their task but they were they all understood what that code did, and they were able to to change their code for followup task.

On "Productivity"

The Communications of the ACM is a journal that gets published every month from the ACM and in March of this year (2024) they released a little video previewing the front cover article on "Measuring GitHub's Copilot's Impact on Productivity" and I was enraged as I hopefully demonstrated earlier on. (chuckles)

I wrote a letter to the editor the contents of which I read to you up front and I the subject of that line was "Measuring GitHub's Integrity?" or it could have also been "Measuring GitHub's impact on the erosion of trust" or "the erosion of professional ethics"

We're sitting here, I'm supposed to be an ethical professional, and we're talking about productivity.

I mean, listen, we're talking about... productivity?

Not ethics.

Productivity.

We're talking... about... productivity.

Not ethics... that we go out there and project into the world, we're talking about productivity.

So they posted this video and I was fuming and I decided I was going to do something about it I was going to leave a comment on the video. (slight chuckle)

And something happened, not two minutes later that comment was gone and I was like well what's going on here? Okay so clearly there was some glitch in The Matrix let me repost again. I repost it again and it was gone two minutes later and I'm like what's going on? Is the ACM really unable to handle any kind of criticism? And if you go to this link right now you will see that there's three comments on the video but you have to kind of click around in order to see them because I've been shadow banned. When I go as a logged in user I see my comment posted on there but none of the rest of the world gets to see it.

So what's nefarious about this well I made the classic now mistake of including "as a large language model" in my commentary because YouTube now is used to having bots writing answers to them and you know how bots are? They always write "As a large language model I can do this but I can't do that" so I, I basically became a bot, as far as YouTube is concerned. (slight chuckles) So I was censored by trying to talk about this problem

Okay, so how comfortable are we with licenses in this if I if I throw up this picture are we good?

(Note this chart contains an oversimplification, as there isn't one GPL license, but two versions in use, and they can be incompatible with one another, per Rob Landley's OhioLinuxFest 2013 talk, where he uses the concrete example of Linux and Samba being both GPL licensed, but not being able to share code, because one is GPLv2, and the other is GPLv3))

We've got BSD up at the top that's a permissive license.

We got GPL, a copyleft license that's in the middle: it's it can receive works from other permissive or copyleft licenses but the produced work must remain GPL. Some people call that the viral nature of GPL other people call it the security of their users.

Proprietary software is out there — there's a lot of it. It can only take from the permissive licensed works, and can't take from GPL. Anything that takes GPL licensed work has to be GPL.

So what is the most popular successful open source project? It's Linux kernel. And what license does it use? GPLv2, which is the same license that Drupal uses.

Okay, so Linux had its 91st birthday, oh-uh sorry 33rd birthday, (chuckles, someone, probably Tim again, says "You came from the future?") Yeah, hehe... Come with me if you want to live. (laughter) Yes, thank you for that.

I found out recently (via Rob Landley's Ohio LinuxFest 2013 talk) that actually the first couple of versions of Linux weren't GPL licensed. Linux was GPLed in 1992, so there was some interim where the license was just "no commercial use." It was actually explicitly against commercial use.

So the question is: did GitHub Copilot exclude GPL code from the training of the code assistant?

I don't see any head nods and I see some shaking heads. Okay so I don't know, but the world is bigger than just GitHub, right, and so there's another project that is called BigCode project and luckily they did exclude GPL code from their training set.

But it's not all good and gravy, right. The spice must flow, right? I'm not going to be nice to everyone or anyone.

Bigcode is an open scientific collaboration. It's supported by Huggingface and Service Now Research. That's who funds them and that's where the development comes from. It trained this StarCoder — StarCoder — there's a lot of stars today, right I guess we're going to the stars, maybe? We're all made of stars? I'm not sure. Stardust, okay? Ziggy, no? Is this thing on? (one chuckle) okay, okay good.

So the Stack V2 is a data set that has over 600 programming languages. That sounds impressive. It's 32 terabytes of code. Okay and then they cite, because they're responsible, you see, they said that they're going to be "working on the responsible development," they're going to follow these principles from the Software Heritage statement on LLMs, which amongst other things says that they're here to "democratize the software creation process" okay

Now, hmmm, be wary of people that say "democratize" anything, okay. Some have already inferred by my last name that I come from the former Soviet Union and you just, just beware, okay.

So let's take a look at the aggregate licenses in this data set

So the vast majority of the data set has no license on it, whatsoever. And of the little slice of pie, the less than a quarter of the rest of it, the majority of that is MIT.

Okay, so they did do something right, they left out GPL up front. That's great, but what's wrong is that there are other conditions outside of the kind of work that the derivative works may be combined with that apply, namely, that the licenses stipulate attribution for the original authors of the work.

So the question is, are you in the Stack? is your code in the Stack? They have an app for that.

So you can go to this URL and click on that and you're welcome to type in your GitHub username. The point is that I'm, in the Stack. 83 of my repositories are in the Stack.

And have they considered that not everyone's code was written on GitHub? Like, there's a bunch of code that pre-dates GitHub. What happens to that code? And like— what the hell, man. Why is it that the authors have to opt out? Why is opt-in the default?

Like, if only there was some way that we as developers could indicate to other people how we want our code to be used — (laughter)

oh! Oh! We do have this mechanism — it's called a license.

Alright, so Gabe Newell, is the main guy behind Steam. He's a former Microsoft executive that started Valve Software and he runs it. And he had this statement which is that "if property rights are provisional then your value of property changes accordingly" and I think the value of our property has dropped precipitously when it is no longer even attributable. Like, good job, good job, Tech World. You're managing to cannibalize your own roots, basically. we gave this thing away for free, all we asked is, most of us asked just attribute us, right, and you managed to still pirate that. Cool. That's going to be a wonderful future to live in.

Okay and so ignoring licenses I argue could be seen as a license to kill the community (Tearing up the social contract we operated under for the past 40 years. The erosion loop has begun. We will become part of the sediment)

Others might say "well, that's just like, your opinion, man" because some developers are embracing this new reality.

And Rob Landley is one of them. He was actually involved in the GPL lawsuits because he was the maintainer of BusyBox, which is like a standalone micro Linux distribution that lots of embedded systems use, and he was interested in getting code back from these vendors that were shipping products, routers and things like that, without providing the code. And he's like "I want to see the code" and he's seen the code and he wanted nothing to do with it. He wanted the lawyers involved (Software Freedom Law Center) to stop doing the licensing wars and they wouldn't and so he decided to rewrite BusyBox from scratch, which he called Toy Box. Using a BSD license, he actually created this Zero-Clause BSD license so you don't even need the attribution part of it, so it's completely kind of like free-for-all: "Just use it. Don't sue me." And ToyBox is now shipped in every Android device and continues to be used and he continues to work on it. I've left out some of the details of this story, there's more in the Toybox vs BusyBox talk

There are also commercial companies that are kind of routing around the GPL in a different way by just supporting projects like LLVM that aren't GPL licensed, and that allow them to do the things that they want to do.

A friend of mine (Matt Turk) also told me that I should include the Santa Cruz Operation trial in here.

These are quotes from Jonathan Corbet's article about it. The SCO lawsuit was filed 21 years ago, but all of this is to say that this is nothing new: there's always somebody knocking, knocking on our door, trying to destroy what we have.

Okay, so what are the paths forward here?

I think it's clear that we need to provide data sets and models that train only on attribution-free licenses: like MIT Zero, like Zero-Clause BSD, or public domain. I already mentioned Toybox, SQLite is another example of a project that has one of these licenses, it's public domain. We have these projects, just don't take everything.

Gentoo Linux put forth a policy of not allowing any AI generated code in their distribution in March of 2024. NetBSD Foundation followed suit in May. There's hope on that front, that we can continue to work in an ethical manner.

And there's a lawsuit that was filed.

I didn't know about that, my friend David Nicholson told me about it. So I don't know how this will turn out, but I'm glad. I'm glad somebody's doing this because this isn't right.

Okay, moving on:

Oil and water

The effort and expectation differences of mixing paid and volunteer work.

Popular collaboratively developed software can attract a mixture of monetarily compensated development as well as the hobbyist unpaid labor that nourish the software and its nascency. Business critical gift economies don't exist, but projects and their participants vary on the spectrum, from those two opposite poles and positions shift over time. Paradoxically, this tension can be both productive and detrimental at the same time, depending on the perspective of those involved.

So let's make this a bit darker — you're used to me by now, (laughter) we're friends.

So the paid workers, I'm going to call them as oil. Because, you know oil is dirty, right, it's dark. Water's abundant, it's pristine and clean. Yeah, I'm a hippie and things are more complicated. but let's do a little story of Keith Packard.

Keith Packard is one of the main developers of the X Window System. If you've ever used Linux and or the BSDs, and had a graphical user interface, you have Keith Packard, partially, to thank for that. He's also behind Fontconfig and Cairo. So the story here is in 1988, there's a problem with a bunch of the Unix vendors. Linux doesn't exist yet, right, there's a problem that they have in that Sun Microsystems has kind of the only working graphical system that is worth anything and so the rest of the vendors decide to get together and work in the open - to fund this effort that will allow them to leap frog Sun View and capture the users. This is the origin of the MIT license, it comes from this project and I linked here the talk itself.

Here's what Keith had to say is that he would answer email from people that were paying him first, right? Makes sense, so even though it was developed out in the open, and you know if I was able to type back then, and had access to the computer, and could speak English, and lots of things - but I could have sent him packages but they would have just sat in his queue unanswered.

So that's an early example, right, it predates Linux, an early example of a success story of commercial entities working together, but even though the work was out in the open not everyone could participate in it equally and not just because they didn't have the time, but because the attention of those involved was focused on where the money was, what their job was.

Okay, so I think this tension is always going to be there. So, there's a spectrum of being a paid or a volunteer and you know you can you can think of yourself like students don't get paid grad students do get paid a little bit as I can tell you... (Tim: "Students pay") That's true, students do pay. Yeah, I didn't think of that, that's true. Yeah, I guess yeah I was thinking of students like a high school student in public school, but yeah you're right. a college student does end up paying, thank you for that. yeah good contribution. Uh, you can heckle me, too you guys, (laughter) This is fine, we're all friends, it's a small crowd.

Okay, so there's also on the spectrum of, well you can be paid, but you could be paid a lot more if you didn't do open source, for example. But if you want to do open source, maybe part of your sliding on that scale of paid to volunteer. you say to yourself "okay I really care about this stuff and I'd rather just make less money and continue to be able to do it" Yes, maximally, it would be great to make tons of money doing this stuff, but realistically maybe I'm okay with just, you know, paying the bills while while doing something that I care about.

There are different kinds of projects that get started, right. You can have academic projects or you can have industry-led projects, like the story I started with in this section. And then there's another project that Matt Turk told me about which is that "prove yourself" project, you got to make a splash.

Talking about the technology that insists on monopolizing your attention, from a previous talk, you have projects that insist on "here's a cool thing, like it might not be useful, but it's cool" There's lots of projects out there that now we're used to that operate under the premise of "oh, you have to make a splash in order to make a contribution".

Okay, but I argue that, other than just this volunteer-paid axis there are other axes.

self <---> society

There's an axis of self versus society: is it is it about me or is it about what I'm taking on into the world. Some people care about intellectual curiosity, "I just want to get something out there and it happens to be that I'm doing it out in the open". Other people are like "No, I really want to make an impact in the world".

journey <---> destination

Okay then there's the journey and destination. Do I care about the experience of what it feels like to be doing this thing or where we end up, in the product.

Athenian <---> Spartan

Do we care about egalitarianism or do we care about hierarchy? Do we take marching orders or are we all voting? Do we all have an equal voting share, right, what does that look like?

dogmatic <---> pragmatic

Are we going to be stand for ideals or are we going to get something done? Is it about how quickly we're going to do it, or does it not matter to us how quickly we do it because we care about what the thing is going to be and how it's going to get to get done.

silly <---> serious

Do we like to have fun? Well, what kind of fun are we talking about? Is it okay to be silly or do we prefer things to be serious, so that we can have efficiency?

courteous <---> abrasive

Are we nice or is it not about being nice? OpenBSD and the 9front communities are shining examples of this abrasive culture that nevertheless is able to stand up and get lots of things done. And I wish that people didn't take credit away from them when they assumed that you have to be courteous, right. This is controversial, yes but abrasive can be effective. You don't have to be nice. This is a very American notion that everybody has to be nice. In other cultures, they're much more comfortable with being in open conflict, and it's productive that way. You can be productive that way.

pushing <---> steering

Are we pushing or are we steering? Some people don't want to do the work, some people don't care what kind of work they're do they just want to work out.

emotional <---> intellectual

And then there's emotional and intellectual. Is this work or is this a diversion from work?

I have this tweet from 2011 when I was working on Jupyter, on a grant for Jupyter. So I was being paid out of a grant and there were lots of flyby contributors coming in, and they were donating code left and right and I was like: "If pyramids had a commit log, they could have been built by volunteers." And you might be thinking, "well, they were paid by slave labor." No, they weren't paid by slave labor. They were paid laborers, right. This was established in the 1990s, they found the graves of the builders of the pyramids, and they were well compensated for their work.

So it's a nice notion that maybe pyramids could have been built by volunteers, and I still think maybe they could be. It depends on what else is involved. It depends on the other axes in the project.

So, would you pitch in and help build the pyramids, if you knew that others are being paid for the work? Again, I think it depends. I don't think it's clear, because if I get to do the code stuff that I like to do, and somebody's doing the project management stuff that I hate doing, then yeah, that that seems like a fair trade. Somebody else is organizing the conference? Yeah, I can pitch in and give a talk. So it depends.

That wasn't dig at this conference, this is a great conference. This is great, you guys are great.(chuckles) I'm just saying.

So how do you balance this paid and volunteer work axis? Sometimes, you might identify a great volunteer, and you want to bring them into the fold, right, you want to compensate them for their work. How do the other volunteers feel about that? When the money gets a little bit tighter, what happens when you kind of have to let them go? You know that they were already great to begin with, maybe they'll go away, maybe they'll stick around. It's a tough problem.

I already mentioned about this game company that I'm sure if you've done any PC gaming, you're well aware of who they are (Valve). But they're now supporting, as of just a couple couple weeks ago, Arch Linux, a distribution that they use for their little Steam Box, err, so Steam, what is it? (audience: "Steam Deck!") Yes, the gamer nerds are helping me out, thank you so much, of which I'm one, but I'm just not a very eloquent one.

Then, there's also a recent effort that's led by the folks at Sentry called The Open Source Pledge, which is asking companies to pledge to donate at least $2,000 per one of their internal developers per year to open source. I have links here to Chad Whitacre's Blog where he writes about this, and he's been trying to advocate for this since 2017 Interestingly, the number of $2K hasn't changed (audience chuckles), but anyway ...

So, I think the tension is always there. I think the tension is there, personally for me, the tension is there, because Jupyter recently formed a 501C6 Foundation, explicitly trying to seek more money from big companies. I think that this is a case of hobbyists being squeezed out. I was one of the people that voted against this.

Now, I don't want to, you know, bring my dirty Jupyter laundry over to you guys and dump it on you, but it's just just an aside. Matt also pointed out to me that "given Copilot's voracious maw, it seems like it's harder to say what's a hobbyist project and what isn't since the usage of hobbyist code in commercial products is basically unknowable". I might start off as a hobbyist, do something clever, get a job offer out of it, okay, and then I get some financial support or a grant, great.

Innovation happens at the edge. Innovation happens with individual users. This is Eric von Hippel's work that Carol Willing was the one that pointed me to it. The stuff that I do on my computer is where the magic happens, right. The stuff that you all do on your computers is where the magic happens. It's not organizations that make that magic happen. Organizations provide the fuel to make that magic happen. Producers have to worry about the size of the market. When you did something cool, it's not enough. When you did something cool, it's cool enough for you, it might be interesting, but it might not be profitable or it might not be worth pursuing as a commercial enterprise.

It can be lonely to tinker and do things by yourself and no talk about open source would be complete without a slide about burnout. (excerpt from page 39 of the 2024 Tidelift State of the Open Source Maintainer Report)

Recall that property of oil to hold more heat. There's a lot of things that I put up with when I'm being compensated for it, monetarily. There's a four-letter word that comes to mind, and it's JIRA. (laughter).

Money can be a salve to things that we don't like doing but... you don't want to end up in a Dune world, where all the water evaporates. "Try our one size fits all miracle cure"

Okay, so what is this formula?

(1/3)Bh = V

Math Pop Quiz! ...Anyone?

(audience member: one third of the business-hour is the velocity of project?).

Okay, you're close, I'll give you a hint.

B is the size of the user base.

h is the height of entitlement (audience begins cracking chuckles)

V is the volume of outrage (laughter)

This is the formula for the rage pyramid. It's a problem in open source, it's a challenge.

All right, last thing to talk about.

Silver spoons lead to later forks. Pay the piper: relicensing trends in commercially supported source-available projects.

With increasing frequency, previously open source software projects backed by a commercial entity are shifting away from traditional licenses. Recent examples include Sentry, Terraform, Redis, and CockroachDB. Why are they doing that and what are affected users and developers doing in response?

So why are they doing that? This is a shorter section. To make money. They fear hyperscalers. They've received insufficient return on investment from trying to foster contributors.

What are we doing in response? Largely, we're forking. Note that the forks already happened when they changed their license. That was the original fork, they just happened to take the original name along with that fork.

Here's 's where the talk, from being spicy turns into a bit of self flagellation.

Philanthropy. This is how you should think of open source software, regardless of who is making it.

You can't be mad at a donor, that you relied on for a long time, that stops giving to your cause. I mean, you can get mad about anything, you can be Hulk, right? But you wouldn't begrudge a past-time top contributor from not being around anymore, because they wish to spend their time elsewhere. You shouldn't begrudge a business that's choosing to spend its money elsewhere.

Okay, we're all trying our best. I mean, not me, I'm not, but other people, right?

Fair Source is an alternative to closed source. Again, this is a recent development, also from Chad Whitacre and the other folks at Sentry, leading the charge here. I don't know if this will work out, but there's already eight companies adopting it.

The idea is that it transitions to being a traditional open source license after some time. I don't know if it'll work out, but people are allowed to make decisions, and make changes in how they they live in the world.

And it's not all bad, because, for example just recently, I think in August of this year, after 4 years of not being open source, Elasticsearch is back to being open source again. So sometimes they change their mind and they come back and you can read about how how it is that they did that (Elastic Blog; Techcrunch)

Then there's other disruptive changes.

So the left-pad incident: the thing that doesn't get talked about the left-pad incident enough is it's sort of been painted as like "oh one developer got mad and decided to just screw everybody". And no, that's not what happened. npm decided to side with one of their commercial users. Kik was a company and they wanted the name that this developer, who happened to also develop left-pad, had been using. And when they took away his kik... (ahem) When they kicked him out... No? Too much? Yeah it's good? Okay, I'll leave it in. Okay, and he says that this is not driven by logic — I'm sorry it was driven by logic. (I got some help from the audience here as I confused myself) It's not driven by logic, anger, or greed. It's an exclusive nor? (audience chuckles) Thank you for the laugh, appreciate that. So he says that " Left-pad was like a 'death' and 're-birth' moment for me. The part of me passionate about open-source was dead, and something new took over. Now, I'm passionate about business, marketing, running companies / teams in different ways, as much as I'm about programming

Okay, and then the other disruptive challenge are the ongoing WordPress Saga. But let's remind ourselves of the rage pyramid.

Rage still applies here, because previously I used the rage pyramid in defense of maintainers, and this time it's worth mentioning that commercial developers are allowed to do what they want to do. The open source licenses allow them to do it and we're allowed to fork.

Okay so we're almost home.

The three things that I talked about is that I tried to give you the kinds of flavors of challenges that exists in developing open source software today. The big alarm bell is this attribution theft. Then business cycles come and go, have their booms and busts, and there should always be tension between free and open source projects and the paid contributors inside of them. And finally forking is a right: you have to adjust your expectations accordingly.

Thank you for your time and attention. [Applause]

NumFOCUS concerns

2024-04-18T00:00:00-07:00

I want to shed some light and raise awareness about what's been going on with NumFOCUS.

This is a long post that I started writing in late February. The basic gist that I am trying to get across is that

From the perspective of many sponsored and affiliated projects, NumFOCUS has been struggling to meet their needs in an adequate or timely manner. At a Town Hall meeting in February, NumFOCUS announced it is moving in a new direction, which caught many by surprise.
Over the years, NumFOCUS has lacked transparency in how its board is selected, had a mixture of early-resignations as well board members staying well beyond term limits, has struggled to incorporate a European affiliate for 9 years, and is currently overdue in providing minutes for the second half of 2023, or any part of 2024.
NumFOCUS initiated an effort to run an election for open board seats and proposed changing its governance structure.
Related to 1 and 2, some projects are considering and actively pursuing alternative venues for fiscal sponsorship.

I have been in touch with many members of the community, including a half-dozen former and one current NumFOCUS board members, to get their perspective, which I include below. I welcome your corrections, clarifications, and thoughts.

You can join the discussion on LinkedIn, Mastodon, Twitter (most of the responses are on the LinkedIn post).

Outline:

Brief overview of concerns stemming from the NumFOCUS Town Hall meeting

Feedback from former board members and community members

Responses from NumFOCUS

501c6

NumFOCUS origin story

Linux Foundation: Voltron of Bureaucracy

How a 501c6 differs from a 501c3

NumFOCUS Europe: still not a thing 9 years later

Elections

Term limits — what are they good for? Absolutely nothing. Say it again.

Election committee

Previous "Election" non-transparency

Resignations

Paths forward and alternatives to NumFOCUS

Brief overview of concerns stemming from the NumFOCUS Town Hall meeting

Several weeks ago (February 12, 2024), a new direction was announced at a town hall meeting.

This public development unfolded over the background of several years of private discussions among NumFOCUS affiliated projects about the efficacy of NF in serving the needs of constituent projects.

In the Town Hall meeting, there was some acknowledgement of pains in the way NF currently operates, but not specifics as to what those pain points are nor what the plan is to mitigate them. It was therefore surprising to learn of the plan to form a 501c6 in addition to the 501c3 while navigating these (underspecified?) headwinds. >> skip to 501c6 section >>

NumFOCUS Executive Director Leah Silen outlined some of the challenges related to forming a European entity, though I was alarmed to hear about 17 accounting firms not being sufficient to arrive at a suitable accommodation. >> skip to NF Europe section >>

There was also a theme of AI, which is not representative of interest or relevance to every project. "AI" was mentioned 37 times, according to the transcript. It wasn't clear if this is a new direction for NumFOCUS as a whole, if it is related to and somehow bundled with the 501c6 move, or if it's just an announcement of direction for the Open-Source Science initiative that's a joint partnership between NF and IBM. (This does not have its own section, but does get mentioned in the feedback from former board members and community members)

Finally, the last concern raised for me was around a proposed election and change in governance structure of NumFOCUS. >> skip to the Elections section >>

At the conclusion of this post, I link to some alternatives to NumFOCUS >> skip to the paths forward section >>

Feedback from former board members and community members

I posted on mastodon, linkedin, and twitter:

Anyone else have concerns about the proposed new direction of NumFOCUS? I finally watched the recording from Monday’s Town Hall and I’d love to talk / read your thoughts, be it in public or in private.

In response to my feeler about concerns, here's some of what I heard back.

One former board member wrote this:

I guess it would come as no surprise that I have some concerns. A few include

Misdirected AI thrust.

Repeated claims of funding the projects, without mentioning overhead taken from project grants.

Emphasis on organization & personnel growth, instead of focusing on existing service delivery.

Complex organizational restructure, when services are not being rendered effectively. New structure separates current board from technical board; risks isolating projects further.

501c6: new opportunities for funding (potentially good), but NF has no current demonstrated route other than SDG[¹] to get funding to projects, and no related policies in place.

Board elections: the committee solicited will not control next election of board? Board should have more community representation.

Board is in contravention of bylaws (president, e.g., has been on the board for a year longer than allowed).

Another former board member said:

I do have concerns. Unfortunately from my past and current experience with NumFOCUS, I don’t see a lot of ability to influence change

A third former board member wrote:

I have a confession. I never ran any grants through NF because I didn't trust them.

A fourth former board member wrote:

It has been many years since I have been on the NumFOCUS board...When I was on the board - strong personalities generally drove much of the direction. ... At that point there could have been more engagement with community and projects, but I am saying this with the wisdom of looking back (ah hindsight, if only we had you sooner).

A community project leader wrote:

The new proposed governance structure seems to come out of the current board/staff without any input from anyone from the projects (that I know of) and as [redacted] mentioned the new structure may not really help with getting the community together

Another community project leader wrote:

I also found it underwhelming to say the least… It even annoyed me on the AI take. I thought we wanted to push science and open source, not please one specific board member and the investors. Yes, to some extent, our projects are the backbone of AI, but they presented this initiative as if the AI business was going to drive what we do in our projects. And the bot thing… come on.

Instead I would have liked a back to basics. Science and open source. Focus on that, providing the services they should be already providing for all the money taken. And instead of randomly saying AI, talk about what real science and expertise we bring to the table vs all the BS models out there.

A third community project leader wrote:

I personally have a bias against 501(c)(6) NGOs. Seems to be inappropriate for publicly funded and public-interest scientific software. I understand why they are doing this experiment, and it may "work", but it will have real governance consequences.

A fourth wrote:

Within all the buzzwords and stuff, it took them about 70 minutes to mention scientific computing or science (and even that was a read out loud one of the zoom comments) I think that summarises it all where the actual priorities lie and that "projects first" is a well sounding but absolutely empty phrase without any actions to support it.

A fifth wrote:

it's clear to me that NumFOCUS really signed its own death warrant with that town hall. So the question is where to next. Things that have been spitballed in our various conversations:

start a new non-profit

start a co-op

find a different sponsor

take over NumFOCUS

More private communication:

I have so many thoughts on NF, but so far I have been sharing them in small support groups only. Basically after this Monday mess I think it's hopeless to fix it, and we may better look for another nonprofit who actually does the accounting and paying project bills on time.

currently the projects seem to serve pydata not the other way around even though the marketing says the opposite.

Anyway, after the townhall I kind of lost all the last drops of hope I had that everything will work out eventually as there was not a single point that showed a positive change from the current status quo.

Responses from NumFOCUS

One of the people first people who reached out to me was Executive Director Leah Silen, but I was trying to piece something together in writing (this post) and concentrated my line of inquiry with her around the election and election committee, so her responses are quoted in the election section.

Current board member Logan Kilpatrick, also reached out on February 20th, 2024 saying "Happy to chat about your NumFOCUS feedback!" but did not respond after I wrote about my concerns. Logan holds the position of Secretary on the NF board, so had he responded to me, I could have followed up to ask about why the board has not released meeting minutes for the past 11 months, but I did not get this opportunity.

I also emailed Sylvain Corlay, a current board member I know personally from our long time shared involvement with Jupyter, and he did respond, I am including some of what he shared with me in this post, starting with clarification about what I referred to as "underspecified headwinds":

Regarding the pain points that were encountered, I believe that Leah was referring to complaints from projects regarding the slow processing of financial service requests by NumFOCUS. I think that a lot of these issues have been addressed, especially after the recruitment of Miriam, our new finance director.

In my opinion, the growth of NumFOCUS in recent years has presented us with the typical challenges that any organization encounters during growth. NumFOCUS deals with a high level of detail on the accounting of projects (down to the individual travel expense, and regular financial reporting on grants etc), and the number of affiliated projects has grown from a dozen to over sixty. The way in which you can operate when you have a dozen sponsored project and when you get closer to a hundred is not the same - and the way to address this question required systemic changes - such as adopting better tools and processes - which understandably took time to implement. I think that we are better equipped now, and Leah presented plans around adopting new tools, notably a ticketing system for financial requests.

(A side note: While some frustration was understandable, I think that some of the criticism received was unfair. The consolidated accounting that NumFOCUS provides for all projects is more complex than it may initially appear, as I can attest as a business owner.)

Because I also asked about the pervading AI preoccupation at the Town Hall, Sylvain responded to that:

I don't think this is very central in our plans for NumFOCUS, but several people have said that they felt that the PyData ecosystem and NumFOCUS had "missed" AI.

There's more from Sylvain in the 501c6 and NumFOCUS Europe sections.

501c6

NumFOCUS origin story

NumFOCUS started as an organization in 2012. Here's a snippet from the NumFOCUS history page where I just correct one inaccuracy, the part about Anthony coming up with the NumFOCUS name, by crossing it off:

In the early Oughts, the open source scientific computing community was reaching a critical point of maturity for Numeric/Numpy, IPython, Matplotlib, and SciPy. By 2010, a critical mass of projects were in need of a more formal structure to provide support and help to organize the community.

…enter NumFOCUS!

Travis Oliphant (author of NumPy), Fernando Pérez (author of IPython), Perry Greenfield (author of Numarray and Astropy), John Hunter (author of Matplotlib), Jarrod Millman (release manager for SciPy), and Anthony Scopatz ~~(who came up with the name "NumFOCUS")~~ became the founding board of NumFOCUS

Have you ever wondered why FOCUS is capitalized? The name came from Travis Oliphant in January 2012, in a thread he started with a dozen people[2] whose subject was "New Foundation Name?" After a few of us tossed in some initial ideas as well as desirable properties of the new name, Travis pitched the following :

Foundation for Open and Usable Code for Science -- or

Foundation for Open Code Usable for Science (FOCUS)

which Anthony Scopatz suggested be modified:

+1 with the following modification "Foundation for Open Code for Usable Science." There are many codes "usable for science" but very few that make science more transparent and usable.

A few days later, Travis followed up

So, what about NumFOCUS for the name of the foundation? Initially it will represent "NumPy Foundation for Open Code for Usable Science" But, later the Num can just represent anything relating to Numerical Computing.

Unless there are strong objections (or better alternatives that emerge in the next few days), I think I will go ahead and start the legal process with this name. I will give another 3 days for other ideas to emerge, then with the help of legal zoom start things going.

There was some assent, and no alternatives or opposition, so Travis moved forward with the plan, reporting that "NumFocus, Inc." was formed as a Texas non-profit corporation a week later (January 14th, 2012).

Travis Oliphant and Peter Wang also founded a company at the same time, Continuum Analytics, which later renamed to Anaconda to reflect and capitalize on the recoginition of its popular Python distribution. Again back to the NumFOCUS history page

Continuum Analytics (now Anaconda) provided integral support on many levels during the early years of NumFOCUS. This included space for the first NumFOCUS office, support staff, and salary of the Executive Director. They also launched the first PyData and contributed many resources and funds to the early events which made it possible for event proceeds to directly support NumFOCUS.

So NumFOCUS, from inception, has had some form of corporate sponsorship. The executive directory mentioned is Leah Silen, who has been the executive directory of NumFOCUS for as long as NF has had an executive director. Being a 501c3 non-profit allows individuals and corporations which pay taxes in the United States to donate to NumFOCUS and deduct those donations on their tax returns. There was a period of time where there was some notion of "becoming a member" of NumFOCUS though donation as an individual, but that has gone away, though there is still a way for corporations to get perks for donating, for example, if they donate $50,000 dollars or more, they get a seat on the Advisory Council.

When I first found out that NF proposed forming a 501c6, alarm bells started going off in my head, because of the reputation of and my experiences with another 501c6: The Linux Foundation. And by a happy coincidence, I had just stumbled across a pithy and apt description of what's wrong with the Linux Foundation.

Linux Foundation: Voltron of Bureaucracy

For the impatient, I include Rob Landley's words from the clip above, but I recommend the original source video for its superior delivery:

The Linux Foundation was formed when two black holes of bureaucracy combined to form Voltron of Bureaucracy. ...and when they combined, the focus shifted from maintaining this standard, to asking donors for money.

Linux Foundation has a well deserved reputation for being pay-to-play. It's where vendors with big pocket books send money. Rob Landley has a more a more meaty post explaining the LF situation in 2010

...the Linux Foundation has set itself up to translate between Fortune 500 companies and individual hobbyist developers. And it's very good at talking to Fortune 500 companies, because that's where its money comes from. But it's not very good at talking to hobbyists, because it's a giant bureaucracy, which is the antithesis of hobbyist "playing around with stuff".

...The Linux Foundation tries to influence and monitor Linux development, to provide results for the corporations that fund it. It sponsors conferences, writes white papers, maintains a technical advisory board... It's a bit like a modern Usenix, really.

Around that same time, I was a member of the Linux Foundation (2010-2012) - I was young and naive. I thought it was amazing to have an organization that includes individuals: a grad student tinkerer like me, along with professional developers who get to work on the Linux kernel full-time. After seeing Jim Zemlin, the Linux Foundation Executive Director, give a talk at Xerox PARC in 2010 using a Mac and making excuses how Linux isn't quite there for doing presentations, which clearly demonstrated what total corporate shill he was, especially given his "vision" for Linux was that a) it was already everywhere, and b) in the future, it will be more everywhere. I thought that things could turn for the better, it was a temporary accident that someone both dismissive and vacuously bombastic about Linux was leading the Linux Foundation. I was wrong. Zemlin is still the LF Executive director now.

Rob Landley had a much closer view of the Linux Foundation transformation:

Oh I worked for them back in 2007. (The Documentation fellowship.) There was some definite frog boiling going on there over the years, they got more bureaucratic as time went on. The purge of all hobbyists from Linux development didn't happen until... it wasn't until 2013 they started asking "do we still have any hobbyists in this community, what IS a hobbyist anyway?"

And then ignoring the replies

Michael Crusoe pointed to a post by Matthew Garrett from 2016 titled Linux Foundation quietly drops community representation

How a 501c6 differs from a 501c3

A 2016 post by Eben Moglen (that Rob Landley referred me to) contains this explanation:

The Linux Foundation, which was formed by merging OSDL and the Free Standards Group, is a trade association, which is a special form of legal entity under US tax law. It is not a charity, or a public service organization. It’s an association of businesses, paying dues, in order to achieve common business purposes, which in the case of LF results in a wide range of activities, all centered around promoting the success of Linux. Because it does not pay federal income taxes, and its exemption—like those of charitable or public benefit organizations—comes from section 501c of the Internal Revenue Code, lay observers might suppose that it has some “community benefit” function, but the precise opposite is true. The tax-exemption rationale for trade associations is that their primary income is the dues paid to them for achievement of business purposes. Since that expenditure would be a business expense netted against revenue for the member businesses if they performed the same tasks individually, there is a sharp disincentive to group together for common business purposes unless the dues are also untaxed in the hands of the recipient. It is axiomatic to their identity that trade associations aren’t charities, that they only benefit their donors (as opposed to 501c3 tax exempt charities, which must not benefit their donors specially), and that they pursue business goals, not social interests.

NumFOCUS Board member Sylvain Corlay explained this about the 501c6

On the subject of the 501c6, some projects expressed disappointment that NumFOCUS was not actively engaged in fundraising on their behalf. While we offer assistance with grant proposals and fund small development grants, there is no dedicated effort to secure additional funding for projects such as Jupyter. This is something that we want to be more deliberate about.

A successful funding model employed by other open-source foundations, and which requires using a 501c6, is a membership model for companies, as exemplified by the Linux Foundation. This model provides revenue recurrence and aligns well with building a relationship with regular donors. (It also matches well with how large corporations are used to engage with such open-source foundations.) Finally, such a membership model scales really well compared to a grant-based funding model.

I think that the 501c6 direction was the wrong one for NumFOCUS to pursue. The reason is that we have a nonprofit, that was created to serve the public good, now transforming itself to also serve the needs of the business community. I'm not against someone wanting to set up a 501c6 for scientific open source projects, I just think that NF doing this, without consulting projects, given how much dissatisfaction there has been with NF on their core services they're supposed to provide projects, is particularly treacherous[³].

To project the LF situation into our ecosystem, the hobbyists there are the academics here, if you squint. And if scikit-learn, one of the breakout superstar projects, can't get help from NF to get a credit card so they can run CI with GPUs for months at a time, it'll only get worse when NF starts to consider businesses who pay NF real money and expect real service as a 501c6. Like it'll be Nvidia, Microsoft, Google, Facebook, etc, as "members", and sure, you and me will be "members", too, I'm sure we'll all be treated equally.

You can click on the image above, but I also reproduce the text from a scikit-learn developer posting on the NumFOCUS slack #general channel in October 2023:

So from the scikit-learn side, we started the process of trying to get a PEX credit card in May so that we can pay for GPUs in our CI.

After some discussions, we filed the form in June for the card, and I put my info there as for the name on the card.

Now, being October, apparently the account requested for me is on hold or blocked (talking to @Mir Mueller about it), and my guess is that it's because my name sounds too middle eastern for the bank. My nationality on the request is not even Iran, I'm using my German nationality to request it, and it's still blocked (for now at least). With conversations with many members of the staff, including @Leah Silen I've been asking for a European entity since 2019 at least, and I think NumFocus now does have a European entity, which AFAIK is not used. The reason I've asked for it has been two folds:

donors can deduct donations from their taxes if they're in Europe and they donate to a European NGO kind of entity.

we can fund and financially deal with people who come from countries against whom US is very hostile, like me.

In the context of diversity and getting people from the global south, it's also important that we could actually do that, and it's not just all talks.

Believe if or not, it's really not fun to have to deal with such issues and after 4 months still not be able to have a simple PEX card. For people like me, it would be really nice if we could actually deal with the European entity, and have our dealings with that entity instead of having to deal with the US system. So the question is, how can we get to actually do this and what's needed for it to happen?

The mention of "asking for a European entity since 2019 at least" is a good segue to the next concern.

NumFOCUS Europe: still not a thing 9 years later

From the board meeting notes for December 1, 2014 (page 50 of NumFOCUS Meeting Minutes 2014-2015)

Moving forward to establish NumFOCUS as a charitable organization in Europe as a Dutch “foundation” to receive donations from European entities at a cost of $1,500.

And a note on the same page:

France has more potential members? - However France is worst place to set up

Fast forward to 2024, and the "What is our status in Europe?" comes up again at the Town Hall, at 1 hour and 2 minutes into the recording (passcode is NFtownhall2024-12-02 )

These are Leah Silen's quotes from the Town Hall:

Boy. This has been so much more of a challenge than we ever could imagine. this, as soon we had the bank account. Everything set up, we decided to go the Volunteer Association in France that was established over 2 years ago.

Then mentions difficulties in getting an accountant and a bank account set up...

We're also looking at other countries now to move forward. The other country we initially had thought about setting up a organization a branch in the Netherlands, and we are. We've got all our documentation about halfway finished with that. "

3 people working on this right now. So I think. I think Miriam reached out to like something crazy like 17 accounting. And financial firms in France, mostly in Paris. And literally, yeah. Haven't gotten a response back. So we're working on it. And if this doesn't work, we're we're not gonna yeah, we need to pivot and come up with a different plan

This is where things stand 9 years later.

Actually, turns out there's even been a bit more progress. A current board member, Sylvain Corlay, who joined the board in 2018, provided some more context in an email to me on March 1st:

When I joined the board, I was really keen on pushing forward the subject of building the organization in Europe - however there was a lot of resistance in the beginning about it on the NumFOCUS board...

Momentum was lost for personal, professional, and later pandemic reasons.

So we had lost the initial energy to get this done. We are back on pushing this subject because having EU-based operations would be very beneficial at many levels for NumFOCUS.

An unforeseen challenge arose when we discovered that non-EU individuals cannot have admin access to French bank accounts. To address this, we have partnered with a bank that grants both myself and Didrik[⁴] administrative access, while US-based individuals have "accountant access" to view all transactions.

In our search for an accounting firm, I initially hoped to engage the same firm that QuantStack uses. However they declined because they did not work with nonprofits. (Typically, voluntary associations do not employ accounting firms). NumFOCUS staff reached out to multiple accounting firms without success. However, in the meantime, we managed to get the accounting firm that we use for QuantStack to agree to take this on, and to give NumFOCUS a hefty discount on their rates. So we are now moving forward

So there's some progress here, and that's good news. It's too bad that there's no way for anyone to know this, because the board has not released meeting minutes for the past 11 months.

Elections

After I made my post following the Town Hall, Leah Silen reached out to me to talk about my concerns, but I had not yet synthesized my thoughts. In my reply to her I decided to focus on just the elections, expressed interest in the Election Committee, and asked about how many seats will be getting filled, and based on Leah's reply, the plan was that there would be four seats.

Term limits — what are they good for? Absolutely nothing. Say it again.

I followed up, wondering if the intent was to have the board shrinking to seven member, as I believe Logan Kilpatrick, Rosie Pongracz, and Larry Gray are the only board members who are not over their term limit, or are some board members who are over their term limit intending to stay on the board longer.

Leah replied

Because Katrina and James are officers, their terms are ending in July. We wanted to give the 4 new members who will be elected as soon as possible an opportunity to take advantage of their organizational knowledge and not have a majority of the board brand new at one time. We'll be holding another election in Q3 to fill these slots and fill any others required to move to the two-board structure - Technical and Administrative Boards.

How could Katrina Riehl and James Powell terms be ending in July if they are both over the term limit for serving on the board? They both began serving their terms in October 2018 [⁵] , each term lasts 2 years, and there's a possibility of renewal for an additional term, so 4 years later - back in 2022, should have been when their board tenure ended.

I understand the appeal of this approach, it's a "nice to have" for institutional knowledge to be passed on with overlapping board transition. Abiding by the bylaws of the organization, however, should be "non-negotiable". What does it say about the integrity or priorities of an organization that is willing to allow board members to continue to serve on the board, in violation of its own bylaws?

Each director shall hold office for a two year term with an option to renew for an additional two year term, for which he or she is elected and until his or her successor shall have been elected and qualified or until his or her earlier resignation, removal or death.
— Article 3: Directors; Section 5: Election and Term of Office

I am opposed to this plan, and find it problematic. A possible resolution would be to have current board members who are over the term limit, but are wishing to continue to serve NumFOCUS to do so by joining the Advisory Council.

Article 5 Section 2 of the NumFOCUS bylaws state that

The Board of Directors may, by resolution, establish a council of advisors (the “Advisory Council”) to be comprised of one or more individuals chosen by the Board of Directors at its sole discretion. The Board of Directors shall not be bound by any advice or decision of the Advisory Council. The members of the Advisory Council shall not have the rights or privileges of directors as set forth in these bylaws and shall have no power or authority over the operation of this corporation. A member of the Advisory Council may be removed at any time by the Board of Directors with or without cause.

Election committee

There's a small and seemingly shrinking set of volunteers who stepped up to help with the coming election. This github repository has a link to the meeting minutes, and the issues contain discussion of what's being worked on.

In some way, this whole post, and specifically this Elections portion of it, is among my contributions to Issue #5: communicate context to community.

One of the reasons the elections committee is shrinking [⁶] and having trouble making progress is because of a moving of goal posts. I guess I haven't spoken to other members of the elections committee about this, but that is one of the reasons I'm having a difficult time trying to engage on the elections front. As a concrete example, at the third meeting of the Election Committee (March 18th), Leah informed us that there would be seven seats in the upcoming election, instead of four, because the Board and Executive Director decided to move up the previous plan to split the board into administrative and technical boards to this upcoming election. I asked if the bylaws have been changed, and was told that they will be. But there's no way for anyone to know this, because the board has not released meeting minutes for the past 11 months.

Previous "Election" non-transparency

In May 2014, there was a post from Andy Terrel describing that

The current board was elected without terms kept in mind. While we love all our board members, we realize that a rotating board has many advantages. Going forward we would like to have yearly elections with each board member elected to a two year term. By alternating between half the board being elected each year, we will have a fair amount of overlap to keep the day to day affairs in order. Board members will not be limited to a specific number of terms.

At some point, the "half the board being elected each year" idea fell through, and was revised to allow the board member to renew for a second term without an election.

In 2018, there was an announcement of an election for the board.

I ran for a seat, along with 24 other people, and took a look at the bylaws and found that it wasn't really an election, because the new board members would be elected by the current board, per the bylaws. I suggested they change the word "election" to "endorsement" - because the input the board was seeking from the community was not actually binding in terms of who would get to be on the board. That change was accepted by Andy Terrel.

I made my suggestion on August 3rd. A blog post from July 20th announcing the election was edited, such that it appears now that this was always called an endorsement, but that change could not have been made until at the earliest the very last day of the Nomination period. Andy acknowledges after the board is selected:

In hindsight, we should not have used the word “election” anywhere, as many assume that the candidates with the highest vote counts will automatically be selected. We needed to apply a selection step in the process to ensure the resulting board had diversity, members willing to serve as officers, and the right mix of background and expertise to lead a rapidly growing non-profit organization.

The July 12, 2018 NF board meetings include the following plan for the election

Election process?
Nomination process; any member can nominate.

Voting by all members in leadership positions → to be defined in detail, e.g. 5 hours a month of community work.

Consensus ratification (can include possible adaptation for diversity, community representation, etc.)

Come back to this process by email. Nomination email end next week. Target voting date: end of August, one month before Summit.

But then the board, in announcing "election" results only said who was added to the board, and not the results of that endorsement (née election).

What they did announce was that "212 out of 763 eligible donating and volunteering members submitted endorsements", and that of the 25 candidates, twelve received more than an arbitrary 5% of the vote, which made the board deem them as "finalists", and five new board members were selected from among these finalists.

I tried to cajole board members (both then current, and then incoming) to release this data to the public, and though some were sympathetic, they as a body did not budge and kept it under wraps and out of scrutiny.

Here are some excerpts of me trying to convince board members to release the voting totals in 2018.

From an email thread with then board member Andy Terrel:

I think the trouble with the current outcome is it's completely opaque: it feels like a backroom deal.

I assert that any reason behind not releasing the numbers should be outweighed by the benefit of being transparent and honest about what happened. This applies equally in terms of reflecting community input, and in terms of being able to see if and how the board incorporated that information into their decision making process.

The board made some decisions, but there is no way to evaluate, scrutinize, or analyze that decision due to the absence of information. It may be uncomfortable that a prominent member of the community received too few, it may be uncomfortable that the someone with a wide level of community support was "passed over" for a board position for someone with less support. But whatever the realities are, they are just that: real. And we can have uncomfortable conversations if need be, or change the process for the future, but let's not sweep it under the rug. The current optics of the situation are that the board feels it knows what's best, made some difficult decisions, but does not wish to have any oversight or accountability for those decisions.

I'm very glad you wrote up the post reflecting the challenges with this process, this is a good thing and let's not stop there. Whatever challenges we have, we can only face them if we are honest about them. As Supreme Court Justice Louis Brandeis wrote over a hundred years ago: "Sunlight is said to be the best of disinfectants."

From email thread with then board member Lorena Barba:

I disagree that I am seeking answers where there are none. I am seeking transparency and clarity for what that broad-brush fuzzy indication of endorsement actually looked like. I do not think it is appropriate to keep the community in the dark about the actual totals.

I am not seeking justification for the board that was selected. Nor am I seeking to challenge it in any way. It already has been acknowledged that the process had its challenges and could be improved in the future, but those of us out of the loop cannot help improve it without understanding the facts about how it went.

From email explaining to Matplotlib lead developer Tom Caswell what likely transpired:

according to the minutes from August 27th - the advisory council provided 5 candidates recommended for positions on the board, and one to be added to the advisory council. The Board met and did just that on the 29th: https://numfocus.github.io/numfocus-board/2018/20180827/ But the advisory council also recommended that the vote totals be released, and were left with the impression that they would be. That's where things stood until the board met to do the selection. I'm inferring that since Peter Wang was the one proposed to be added to the board, it is most likely him that has a high number of votes that was passed over for a board position. I don't know why that is, but I do know it's not healthy to sweep that under the rug. I can speculate it may have something to do with the fact that Peter is still CTO of Continuum (Anaconda) while Travis is no longer there. It might also be the fact that Peter did not put any effort into his candidacy explanation paragraph. Who knows, but the point is that the Advisory board made the recommendation that I can only assume was followed but without following the numbers disclosure aspect. Uncomfortable though it may be, when there are so many doubts looming over an organization, not releasing those numbers and being tight lipped about it does further damage. Trust is being eroded.

So that was how the election went down in 2018.

Resignations

All organizations evolve, and board membership changes are expected. But you might be surprised to learn that two NumFOCUS board members resigned last year. It's true, Noa Tamir and Stéfan van der Walt resigned, I think in October. I'd point you to the board meeting minutes that documents this, to make sure I had it right, but the board has not released meeting minutes for the past 11 months,

Those two weren't the only ones who've resigned in the past. Here is what some of the former board members had to say about their decision to step down.

One former board member explained :

I resigned, and what it came down to was that I could not put the time or emotional energy into working to make NumFOCUS the organization I wanted it to be, instead of the organization that it was... The conflict in NumFOCUS between what I wanted (which is hard to narrow down, but I guess could come back to the idea of the student researcher being a key audience) and what would eventually make NumFOCUS more engaged with the communities with which it was already familiar (i.e., "data science," industry, etc) was getting harder to navigate. ... So when I resigned, it was because I was overwhelmed and not able to put in the effort that I wanted to -- but the reason I felt like I had to put in so much effort was because I felt like the organization was slipping further and further from what I wanted to see as its core mission.

Another former board member wrote that :

When I was on the board, I and others talked with many people who we thought had any kind of power to change things, and nothing changed. So this is not the first time, or probably even the second.

And when we video chatted added:

I left the board because I couldn't affect change.

A third former board member wrote me that part of the reason for their resignation was that :

I and others on the board had grave concerns about performance, but that no one seemed to be held accountable for that.

Paths forward and alternatives to NumFOCUS

My operating plan for making this post was to try to capture the various NumFOCUS transgressions as a way to illustrate why staying there is unlikely to work, given how many years and board members have churned through there, all while keeping the same executive director. Or that staying with NF will require a great deal of energy to keep the experience for individuals and projects from going sideways further. But a concern I have is that, aside from caring about the scientific-adjacent open source community, I don't actually have a need for an alternative to NumFOCUS to exist. I do not have any grants and no plans to apply for grants, so I worry about trying to put in a bunch of effort to stand something up while the people who actually have the grants figure out a better alternative for themselves (or just bite the bullet and stay with NF).

In responding to my email Rob Landley included this piece of wisdom:

Anyway, you don't "fix" stuff like this, the tree falls and a new tree grows in its place from a sprout

There are those who think that trying to reform NumFOCUS is a lost cause. Here's someone who's given up on trying to influence NumFOCUS, but also saying that it would be too much effort to create a new nonprofit.

NumFOCUS was founded to serve the needs of this community. I don't think many of us have the appetite of going through that process again, and there are options.

Jupyter has proposed departing NumFOCUS to transition to The Linux Foundation for its fiscal sponsorship. Though I focused on LF being a 501c6 in the Linux Foundation: Voltron of Bureaucracy section above, turns out they also own and operate two 501c3 foundations: The Linux Kernel Organization and LF Charities. The latter is the one Jupyter proposes transferring to, though Jupyter also proposes forming a Jupyter Foundation 501c6. I have not participated in developing this proposal, it was researched and proposed by the Jupyter Executive Council.
Some projects are looking into the Open Source Collective
There's also Code for Science and Society

Some of the people who asses a low probability of success for a NumFOCUS turnaround nevertheless also think that a more responsive NumFOCUS would be the best outcome.

You can join the discussion on LinkedIn, Mastodon, Twitter (most of the responses are on the LinkedIn post).

SDG refers to "Small Development Grants" ↩
The dozen on the "New Foundation Name?" thread in January 2012 were: Francesc Alted, Chuck Harris, Paul Ivanov, Jonathan March, Jarrod Millman, Travis Oliphant, Fernando Perez, Ilan Schnell, Anthony Scopatz, Andy Terrel, Peter Wang, Mark Wiebe ↩
I was deliberate here to apply the "treacherous" label to the action, not the people. Some are misinterpreting my words as "calling the board treacherous" — I am not. The distinction is akin to calling some act "irresponsible" without characterizing the people who err as "irresponsible". ↩
Didrik Pinte is a former NF board member (2012-2019). ↩
Along with Sylvain Corlay, Jane Herriman, Stefan van der Walt, October 11th, 2018 is the first time Katrina Riehl and James Powell are listed in the minutes as board members. ↩
I referred here to the fact that fewer people were attending each subsequent meeting and also fewer people were engaging with the committee's work asynchronously. But also it so happened that the next day after I originally made this post, the chair of our committee announced he is stepping away from the effort and provides a postmortem ↩

Money and California Propositions (2020)

2020-10-29T00:00:00-07:00

Ten years ago, I made some plots for how much money was contributed to and spent by the various proposition campaigns in California.

I decided to update these for this election, and here's the result:

Just in case you didn't get the full picture, here is the same data plotted on a common scale:

So, whereas 10 years ago, we had a total of ~$58 million on the election, the overwhelming amount of in support, this time, we had ~$662 million, an 11 fold increase!

The Cal-Access Campaign Finance Activity: Propositions & Ballot Measures source I used last time was still there, but there are way more propositions this time (12 vs 5), and the money details are broken out by committee, with some propositions have a dozen committees. Another wrinkle is that website has protected by some fancy scraping protection. I could browse it just fine in Firefox, even with Javascript turned off, but couldn't download it using wget, curl, or python, even after setting up all of the same headers, not just the User-Agent one. I would just get something like this:

<html>
<head>
<META NAME="robots" CONTENT="noindex,nofollow">
<script
src="/_Incapsula_Resource?SWJIYLWA=5074a744e2e3d891814e9a2dace20bd4,719d34d31c8e3a6e6fffd425f7e032f3">
</script>
<body>
</body></html>

or this

<html style="height:100%"><head><META NAME="ROBOTS" CONTENT="NOINDEX,
NOFOLLOW"><meta name="format-detection" content="telephone=no"><meta
name="viewport" content="initial-scale=1.0"><meta http-equiv="X-UA-Compatible"
content="IE=edge,chrome=1"></head><body style="margin:0px;height:100%"><iframe
id="main-iframe"
src="/_Incapsula_Resource?SWUDNSAI=9&xinfo=7-5676648-0%200NNN%20RT%281603990408902%200%29%20q%280%20-1%20-1%20-1%29%20r%280%20-1%29%20B12%284%2c316%2c0%29%20U19&incident_id=258000270014056206-28484364857902791&edet=12&cinfo=04000000&rpinfo=0&cts=P7QQVpGir0pkf9hcyxbNqBbXI4e5zF975eQFSdodvpp8NdgYKJeqrlW3TSO8Zzh5"
frameborder=0 width="100%" height="100%" marginheight="0px"
marginwidth="0px">Request unsuccessful. Incapsula incident ID:
258000270014056206-28484364857902791</iframe></body></html>'

Luckily, while I was working on the scraper, I took a break and found another source that already has the tabulations.

The only catch is that this source was last updated two weeks ago, so it's not the freshest data. Also, last time I had data for both contributions and for money spent, but this summed page only has contribution totals, not spending totals (the spending figures are still there)

But I figured it's good enough to get the big picture.

Also, here's the python code used to generate these plots (largely reused from last time, so don't expect it to be pretty).

# Create contributions  bar charts of committees supporting and opposing
# various propositions on the California Ballot for November 2020
# created by Paul Ivanov (https://pirsquared.org)

# figure(0) - Contributions by Proposition (as subplots)
# figure(2) - Contributions on a common scale

import numpy as np
from matplotlib import pyplot as plt
import locale

# fun times! without this next line, I got a
# ValueError: Currency formatting is not possible using the 'C' locale.
locale.setlocale(locale.LC_ALL, 'en_US.UTF-8')

elec = "Nov2020"
election =  "November 2020 Election"

# This part was done by hand by collecting data from CalAccess:
# https://www.sos.ca.gov/campaign-lobbying/cal-access-resources/measure-contributions/
# proposition: (yes, no) 
cont = {'14': (12810328, 250), 
        '15': (56320926, 60905901), 
        '16': (19926905, 1172614),
        '17': (1363887, 0), 
        '18': (835064, 0), 
        '19': (37794775, 45050), 
        '20': (4829677, 20471086), 
        '21': (40184953, 59379159), 
        '22': (188937777, 15896808), 
        '23': (6917438, 104405156), 
        '24': (5907002, 48368),
        '25': (13446871, 10181122)
}

def currency(x, pos):
    """The two args are the value and tick position"""
    if x==0:
        return "$0"
    if x < 1e3:
        return '$%f' % (x)
    elif x< 1e6:
        return '$%1.0fK' % (x*1e-3)
    return '$%1.0fM' % (x*1e-6)

from matplotlib.ticker import FuncFormatter
formatter = FuncFormatter(currency)

yes,no = range(2)
c = [(6.,.5,0),'blue']  # color for yes/no stance
c = ['red','blue']  # color for yes/no stance
c = [(1.,.5,0),'blue']  # color for yes/no stance
a = [.9,.9]             # alpha for yes/no stance
a = [.6,.5]             # alpha for yes/no stance
t = ['Yes','No ']       # text  for yes/no stance

raised,spent = range(2)
title = ["Contributed to", "Spent on" ] # reuse code by injecting title specifics
field = ['Contributions', 'Expenditures']

footer ="""
Total %s 1/1/2020-10/14/2020  (1/1/2020-10/16/2020 for Prop 15)
Data from http://www.sos.ca.gov/campaign-lobbying/cal-access-resources/
measure-contributions/2020-ballot-measure-contribution-totals 
cc-by Paul Ivanov (https://pirsquared.org)
""" # will inject field[col] in all plots

color = np.array((.9,.9,.54))*.9 # spine/ticklabel color
color = np.array((.52,.32,.12)) # spine/ticklabel color
color = np.array((.82,.42,.12)) # spine/ticklabel color
color = "gray"
color = np.array((85.,107,47)) / 255 # darkolivegreen

plt.rcParams['savefig.dpi'] = 200

def fixup_subplot(ax,color):
    """ Tufte-fy the axis labels - use different color than data"""
    spines = list(ax.spines.values())
    # liberate the data! hide right and top spines
    [ax.spines[s].set_visible(False) for s in ["top", "right"]]
    ax.yaxis.tick_left() # don't tick on the right

    # there's gotta be a better way to set all of these colors, but I don't
    # know that way, I only know the hard way
    [s.set_color(color) for s in spines]
    [s.set_color(color) for s in ax.yaxis.get_ticklines()]
    [s.set_visible(False) for s in ax.xaxis.get_ticklines()]
    [(s.set_color(color),s.set_size(8)) for s in ax.xaxis.get_ticklabels()]
    [(s.set_color(color),s.set_size(8)) for s in ax.yaxis.get_ticklabels()]
    ax.yaxis.grid(which='major',linestyle='-',color=color,alpha=.3)

adjust_dict = {'bottom': 0.052, 'hspace': 0.646815834767644,
 'left': 0.13732508948909858, 'right': 0.92971038073543777,
 'top': 0.94082616179001742, 'wspace': 0.084337349397590383}

# subplots for each proposition (Fig 0 and Fig 1)
col = 0 
f = plt.figure(col); f.clf(); f.dpi=100;
for i,p in enumerate(cont):
    ax = plt.subplot(len(cont),1, i+1)
    ax.clear()
    #p = i+14    #prop number
    for stance in [yes,no]:
        plt.bar(stance, cont[p][stance], color=c[stance], linewidth=0,
                align='center', width=.1, alpha=a[stance])
        lbl = locale.currency(round(cont[p][stance]), symbol=True, grouping=True)
        lbl = lbl[:-3] # drop the cents, since we've rounded
        ax.text(stance, cont[p][stance], lbl , ha='center', size=8)

    ax.set_xlim(-.3,1.3)
    ax.xaxis.set_ticks([0,1])
    ax.xaxis.set_ticklabels(["Yes on %s"%p, "No on %s"%p])

    # put a big (but faded) "Proposition X" in the center of this subplot
    common=dict(alpha=.1, color='k', ha='center', va='center', transform = ax.transAxes)
    ax.text(0.5, .9,"Proposition", size=8, weight=600, **common)
    ax.text(0.5, .50,"%s"%p, size=50, weight=300, **common)

    ax.yaxis.set_major_formatter(formatter) # plugin our currency labeler
    ax.yaxis.get_major_locator()._nbins=5 # put fewer tickmarks/labels

    fixup_subplot(ax,color)

adjust_dict.update(left=0.13732508948909858,right=0.92971038073543777)
f.subplots_adjust( **adjust_dict)

# Figure title, subtitle
extra_args = dict(family='serif', ha='center', va='top', transform=f.transFigure)
f.text(.5,.99,"Money %s CA Propositions"%title[col], size=12, **extra_args)
f.text(.5,.975,election, size=9, **extra_args)

#footer
extra_args.update(va='bottom', size=6,ma='center')
f.text(.5,0.0,footer%field[col], **extra_args)

f.set_figheight(20.); f.set_figwidth(5); f.canvas.draw()
f.savefig('CA-Props-%s-%s-Subplots.png'%(elec, field[col]))

# all props on one figure (Fig 2 and Fig 3)
f = plt.figure(col+2); f.clf()
adjust_dict.update(left= 0.045,right=.98, top=.91, bottom=.12)
f.subplots_adjust( **adjust_dict)
f.set_figheight(6.)
f.set_figwidth(16)

extra_args = dict(family='serif', ha='center', va='top', transform=f.transFigure)
f.text(.5,.99,"Money %s CA Propositions"%title[col], size=12, **extra_args)
f.text(.5,.96, election , size=9, **extra_args)

footer = footer.replace("/\n", "/") #.replace("\nc", "c")
#footer = footer.replace("\n", "") + "\n"
extra_args.update(ha='center', va='bottom', size=6,ma='center')
#f.text(adjust_dict['left'],0.0,footer%field[col], **extra_args)
f.text(.5,0.0,footer%field[col], **extra_args)

ax = plt.subplot(111)
for stance in [yes,no]:
    abscissa=np.arange(0+stance*.30,len(cont),1)
    total = sum([x[stance] for x in cont.values()])
    lbl = locale.currency(round(total),True,True)
    lbl = lbl[:-3] # drop the cents, since we've rounded
    lbl = t[stance]+" Total "+ lbl.rjust(12)
    plt.bar(abscissa,[cont[p][stance] for p in cont], width=.1, color=c[stance],
            alpha=a[stance],align='center',linewidth=0, label=lbl)
    for i,p in enumerate(cont):
        lbl = locale.currency(round(cont[p][stance]), symbol=True, grouping=True)
        lbl = lbl[:-3] # drop the cents, since we've rounded
        #ha = 'center' if i != 2 else "right" # tweek by hand to make numbers show up
        ax.text(abscissa[i], cont[p][stance], lbl , ha="center",
                size=8,rotation=00)

ax.set_xlim(xmin=-.3)
ax.xaxis.set_ticks(np.arange(.15,len(cont),1))
ax.xaxis.set_ticklabels(["Prop %s"%p for p in cont])
fixup_subplot(ax,color)

# plt.legend(prop=dict(family='monospace',size=9)) # this makes legend tied
# to the subplot, tie it to the figure, instead
handles, labels = ax.get_legend_handles_labels()
l = plt.figlegend(handles, labels,loc='upper right',prop=dict(family='monospace',size=9))
l.get_frame().set_visible(False)
ax.yaxis.set_major_formatter(formatter) # plugin our currency labeler
f.canvas.draw()
f.savefig('CA-Props-%s-%s.png'%(elec, field[col]))

plt.show()

aka: also known as

2020-10-08T00:00:00-07:00

I was chatting with Anthony Scopatz last week, and one of the things we covered was how it'd be cool to have a subcommand launcher, kind of like git, where the subcommands were swappable. If you're not familiar, git automatically calls out to git-something (note the dash) whenever you run

$ git something

and something is not one of the builtin git commands. For me, ~/bin is in my PATH, so

$ git lost
git: 'lost' is not a git command. See 'git --help'.
$ echo "echo how rude!" > ~/bin/git-lost; chmod +x ~/bin/git-lost
$ git lost
how rude!

And so what Anthony was talking about was having two commands that are supposed to do the same thing, and being able to switch between them. For example: maybe we have git-away and git-gone and both of them perform a similar function, and we wish call our preferred one when we run git lost.

One way to do this would be to copy or symlink our chosen version as git-lost, and replace that file whenever we wanted to switch between git-away and git-gone. Another would be to rename both as git-lost and store them in different folders, adjusting our PATH variable so the preferred version of git-lost would get called.

Now with git itself, you can do this by making a git alias. Here are some of my frequently used aliases from my ~/.gitconfig:

[alias]
    diffw = diff --word-diff=color --ignore-cr-at-eol
    root = rev-parse --show-toplevel
    lgo=log
    lazy=commit -a -m.

git diffw will highlight only parts of the line that were added and removed, instead of whole lines
git root prints the top-most directory of the current git repository
git lgo is a typo for git log (I also have several versions of checkout typos)
git lazy is for those frequent times when having to think of a commit message is too onerous.

So we would add lost=away to our alias block to have git lost call git-away and change that alias to lost=gone when we wanted to have git lost call git-gone, instead.

I wondered what a simple, generic version of such capability might look like and wrote a short shell script and then a README file that's ten times longer than the script itself to explain it. :)

  .---------------------.
 / aka - also known as /
/---------------------/

Simple, persistent, shell-based command aliasing

Usage: aka alias               show stored aliases
       aka alias NAME CMD...   store an alias for CMD
       aka NAME [...]          run CMD, optionally with more arguments

By default, aliases are stored in the .aka file of the current working directory. Optionally, the location of that file can be modified by setting the AKA_FILE environment variable. Tested on Debian, OpenBSD, and OpenWRT, aka uses portable shell syntax and should work everywhere that has a typical /bin/sh

Examples:

$ aka alias demo more

$ aka demo --version
more from util-linux 2.36

$ aka alias demo git

$ aka demo --version
git version 2.28.0

The command you alias can have parameters

$ aka alias l ls -f

$ aka l
.aka .. aka aka_tiny . README .git

And you can pass additional parameters

$ aka l -tR
.. aka_tiny aka .git .aka README .

You can see all of the currently stored aliases

$ aka alias
l='ls -f'
demo='git'

And edit them using your favorite text editor

$ cat .aka
alias demo='git'
alias l='ls -f'

Everything that is not aliased will get executed as a regular command.

$ aka python3 --version
Python 3.8.6

Why would anyone want this?

A practical use case might be to have

.
├── c_proj
│   └── .aka       | alias doit='make install'
├── go_proj
│   └── .aka       | alias doit='go run'
└── python_proj
    └── .aka       | alias doit='pip install .'

Now you can aka doit to your heart's content inside each of those folders, and have that execute the appropriate build commands for the type of project it is.

$ for x in *; do (cd $x && aka doit); done
make: *** No rule to make target 'install'.  Stop.
go run: no go files listed
ERROR: Directory '.' is not installable. Neither 'setup.py' nor 'pyproject.toml' found.

Installation

Grab aka, make it executable, and place it somewhere in your path.

The standard version of aka (~40 lines) that prints usage and has comments explaining the code is here:

https://git.sr.ht/~pi/aka/blob/main/aka

If you just want the business logic, functional equivalent (~10 lines) is here:

https://git.sr.ht/~pi/aka/blob/main/aka_tiny

How it works

aka is a tiny portable shell script that gets executed via /bin/sh, sources AKA_FILE and evals the positional parameters, thus applying aliases and whatever other shell script shenanigans stored in AKA_FILE.

For example, this means that you can put something like

alias my='AKA_FILE=~/.my_aliases aka'

in your shell's startup files, and thereafter use the my command as an aka invocation that always sources ~/.my_aliases file in your home directory, regardless of where you run it.

Notes

Initial prototype uses aliases, which may not work for programs that depend on establishing their behavior based on the argv[0] name they were called by (busybox, for example).

If you want to create aliases that include quotes, multiple commands, output redirection, or job control, you either have to escape the special characters like > and & with backslashes when creating the alias, or edit .aka file to add them directly in there:

$ aka alias redir_example echo \"hi\" \> log.txt
$ aka alias double_date date \;  date

In the example above, if we don't put a backslash in front of the semicolon, we will be terminating our aka alias for double_date with just one call do date, and calling the second date command immediately.

And finally, aka stores the aliases using single quotes, so if you have commands where you need to preserve single quotes, you should edit the .aka file by hand to change the alias definition to use double quotes.

https://direnv.net: "direnv is an extension for your shell. It augments existing shells with a new feature that can load and unload environment variables depending on the current directory." The direnv website has links to a half dozen similar projects.

Debian's update-alternatives Debian's alternatives system is similar: it lets you switch out raw XX command for all users, which is why you have to run update-alternatives as root. With aka, you're defining the aka XX commands and they only apply on a per-folder basis (or per AKA_FILE environment variable, if set).

License

aka is distributed under a 3-clause BSD license.

Author

aka was written by Paul Ivanov

https://sr.ht/~pi/aka

Changelog

2020-10-02: initial version

2020-10-03: added AKA_FILE environment variable

2020-10-08: publicly announced

2020-10-28: added notes about multiple commands, redirection, and quotes

Source code

Standard ~40 line aka with comments and usage printing:

#!/bin/sh
# aka, written by Paul Ivanov: https://git.sr.ht/~pi/aka
if [ $# -eq 0 ] ; then
    set -- -h # print usage on 0 arguments
fi

if [ $# -ge 1 ] ; then
    case $1 in
        -h|--help|-help|-H)
echo "Usage: aka alias               show stored aliases"
echo "       aka alias NAME CMD...   store an alias for CMD"
echo "       aka NAME [...]          run CMD, optionally with more arguments"
            exit 0
            ;;
    esac
fi

aliasFile=${AKA_FILE:-./.aka}
# Is this an alias creation? If so, we should have at least 3 arguments,
# such as:
#
# $ aka alias pager more
#    \     \     \    \
#     $0    $1    $2   $3
#
if [ $# -ge 3 ]  && [ $1 = alias ] ; then
    # cmd='pager'
    cmd=$2
    # make 'more' the new $1
    shift 2
    # remove previous aliases for 'pager'
    [ -e "$aliasFile" ] && grep -v "alias $cmd=" "$aliasFile"  > "$aliasFile"~
    echo "alias $cmd='$*'" >> "$aliasFile"~ && mv "$aliasFile"~ "$aliasFile"
    exit 0
fi

# Load up aliases...
[ -e "$aliasFile" ] && . "$aliasFile"

# ...and execute command
eval $*

Tiny ~10 line aka without comments:

#!/bin/sh
# aka, written by Paul Ivanov: https://git.sr.ht/~pi/aka
aliasFile=${AKA_FILE:-./.aka}
if [ $# -ge 3 ]  && [ $1 = alias ] ; then
    cmd=$2
    shift 2
    [ -e "$aliasFile" ] && grep -v "alias $cmd=" "$aliasFile"  > "$aliasFile"~
    echo "alias $cmd='$*'" >> "$aliasFile"~ && mv "$aliasFile"~ "$aliasFile"
    exit 0
fi
[ -e "$aliasFile" ] && . "$aliasFile"
eval $*

Lazy River of Curious Content 0

2020-05-17T00:00:00-07:00

This is the first post of what I'm calling a Lazy River of Curious Content. This is a way to review stuff that I've been doing, dealing with, or find interesting ~~during the week~~ recently (This was originally written two weeks ago, May 3rd, my shoddy internet connectivity kept me from posting it.). I'm loosely following the format that Justin Sherrill uses with great effect over at https://dragonflydigest.com

Learn NixOS by turning a Raspberry Pi into a Wireless Router Friend of the show, Anthony Scopatz, tried NixOS for the first time and provides a detailed report:

"While I had read the NixOS pamphlets, and listened politely when the faithful came knocking on my door at inconvenient times, I had never walked the path of functional Linux enlightenment myself"

Reading through that made me file away a todo of writing up how I use propellor (and why). But those todo sometimes just pile up for a while...

An interview of one of my long time nerd-crushes, Rob Pike. The questions focus on the Go programming language, but read carefully, specifically the tone and considerate nature of intent.

Owning up to still getting it wrong sometimes. Over on The Unix Historical Society list, there was a thread about the origin and meaning of Plan 9 from Bell Labs. Venerable computing deity Ken Thompson sends a private message to aforementioned computing demi-god Rob Pike, Rob forwards the admonishment back to the list.

I started playing with Krita a few weeks ago, using a tablet input device. I found out about Krita and felt empowered to try it thanks to this excellent explanatory video tutorial by David Revoy. Again, recurring theme of humility and gentle nature of creative high-output individuals. Thank you, David!

Fellow San Francisco Randonneurs cyclist Roy Ross passed away unexpectedly. A gentle soul. This thread on SFR gives a some sense of a quiet Deborah Ford put up some great photos of Roy, and the Metin Uz shared an album, with others pitching in

I set up a mumble server and have been mumbling with some old friends and family.

After Matthew and I chatted, I re-read his post about the XFree / X.Org fork and open source governance project death, then went hoping around some links and came across Rick Moen's fear of forking essay, and he followed up with this bit of Emacs / XEmacs history

I showed Matthias and Camille my first tracing using Krita, and Matthias made me aware of Grease Pencil - a way to do animations in Blender, and Camille sent me an awesome and sweet comic book she made about their cat.

Matthias showed me that there's a way in Git now to have git blame ignore stylistic commit changes - see ipython PR#12091 for how to set it up, and #12277 where we start using it. This was a pet-peeve of mine back when Nelle started to apply semi-automated PEP-8 formatting to parts of the matplotlib codebase - that it was making it more difficult to use git blame to track down when lines were changed functionally, and why.

And then, while we were chatting about a custom LaTeX completer PR that he has open, he arrived at "wouldn't it be cool to throw and exception with the table-flip emoji in your code?" ... and a short while later, he made that work:

No Video? download the mp4 from pirsquared.org/video/table_flip.mp4

You can also see it in it's original tweet form and send Matthias some love. And by the way, Matthias has a new gig at Quansight.

I've also tried out Mupen64Plus - a Nintendo 64 emulator (which I found out about via OpenBSD ports list). I found some ROMs that fell off the back of a truck, but without a proper controller, it's kind of difficult to play (I tried with an SNES-like USB controller, no dice). So now I've ordered a USB N64-like controller.

pheriday 3: infrastructure

2020-04-03T00:00:00-07:00

Looks like we can't inline audio for your browser. That's cool, just find the direct file links below.

paul's habitual errant ramblings (on Fr)idays

pheridays: 3

2020-04-10: A week ago, I recorded a 5 minute audio segment of some stuff I've been thinking about, but when I started to write it up I stumbled into and kept dropping down a deep technostalgic hole.

fall down along with me:

https://pirsquared.org/blog/pheriday-infrastructure.html

The recording is just shy of five minutes long, you can also download it in different formats, depending on your needs, if the audio tag above doesn't suit you:

https://pirsquared.org/pheridays/2020-04-03.ogg (2.9 Mb) https://pirsquared.org/pheridays/2020-04-03.mp3 (4.5 Mb) https://pirsquared.org/pheridays/2020-04-03.m4a (6.3 Mb)

Stuff I mentioned in the audio:

Propellor - "configuration management system using Haskell and Git" by Joey Hess

OpenWRT - specifically - reducing Bufferbloat

Mumble - "a free, open source, low latency, high quality voice chat application."

sourcehut.org - "the hacker's forge" also know as sr.ht by Drew DeVault

Jitsi - "Multi-platform open-source video conferencing"

OpenFire - "real time collaboration (RTC) server licensed under the Open Source Apache License." Extensible XMPP server, with plugins, like a Jitsi-based video meeeting one claled OpenFire Meetings.

Though this is the fourth installment, the last time I recorded and posted a rambling was back almost 8 years ago! In fact, it was 2012-08-03, so 7 years and 8 months, to the day.

Having control of your infrastructure is a longtime thread for me.

For starters - there's the bicycle. That's been my primary and preferred mode of transportation for 30 years. As a kid, I was empowered by the sense of freedom, independence, and self-sufficiency that came with a bike. All these years later, I'm still a fan. You can see just how happy I am on a bike at the top of this interview , thanks to a sweet photo that was taken by Robert Sexton right by the Golden Gate Bridge at the end of the Lucas Valley Populaire in 2015.

Those of you who knew me back in college might remember how at UC Davis I ran my own "pirate" internet radio station - KPVL - with the cheeky tagline of "More broadcasters than listeners". (I say "pirate" because it has not relation to the actual KPVL radio station). But there are earlier remnants and traces of my efforts to exercise control and build my own reality.

I think it was in 1999 that my brother Mike and I started using Redhat (6), then Mandrake Linux 6.5, dual booting on a computer at home and I separately around the same time I got myself an sdf.org account. Though I wasn't sophisticated enough to have a constant internet connection in high school, I was lucky enough to get an account on Robert Chin's laya.com server. The url was - p.laya.com - it's long gone, but luckily, Archive.org has a copy from 2001.

Wow. I just took a look and so much came flooding back.

Here's the thing: April is an anniversary of sorts for me. Back in 1999, it marks my first time breaking anonymity and pseudonimity and using my real name on the internet. I've written about this before under the title of Publisher's block ten years ago - just about half way between now and then. This time, though, let me inline the piece I linked to as proof of the deliberate nature of my lack of anonymity.

An account of my life at 15, as I live it.
traces of my awareness of the world, I can look back at later
My first attempt at a memoir
My goal is to capture my many thoughts emotions, behaviors, incidents, and
acquaintances
and to arrive only at an exponential number of those,
hoping yet being afraid that it might be zero
making everything about me: one

I'm glad I can now reflect on the kind of kid I was, thanks to the amazing folks who had the foresight to start archiving all of the web for The Way Back Machine. I used that p.laya.com page as a todo list and notes for myself using a hipster combination of the default file index listing with a FOOTER.html. It was captured in 2001, I was in 17, but some of this was written when I was 15 or 16 (I found contents from November 2000), I make mention to my then freeshell.org account (it's now been ivanov@ since 2012). I link to the source code of a MUD - ftp://ftp.game.org/pub/mud/diku/merc/rom/tartarus/tartarus.tgz - which is a broken link now, but I found a mirror over here: which is amazing, because just a week or two ago, I was hanging out with fellow SciPy 2020 program co-chairs Madicken Munk and Gil Forsyth over video chat after one of our meetings and I was happily reporting about how one of the positive things to come out of the shelter in place for me is that "I've fixed my mutt configuration and started using it again!" - but they both heard "mutt" as "MUD" and got very excited by that prospect. So much so that we all agreed that we'll have to follow up and actually follow through to build a MUD. And I brought up how at some point in high school I was mildly active in a pair of MUDs, and wanted to make my own, but never got around to it.

The last link I left for myself on there points to pinkmonkey.com - a homeschooling resource - which is probably handy for the parents with little ones these days.

Here's the most concrete infrastructure project I can find from then: I collected bookmarks from my friends to share them. The "service" lived at http://p.laya.com/bookmarks - and predates del.icio.us and pinboard. I bet I "advertised" it in my AIM profile.

If you're curious, there's a link to the archive.org copy near the end of this post, but I had this urge to show it to you much closer to its original glory.

Let me set the scene: It's Friday in April, the year is 2020, I'm running Windows 10 on my work laptop in poorly connected home in California, where a pandemic has most of the state's residents staying put at home for the several weeks already, and I decided to make a screenshot using the tool du jour of yesteryear

Netscape Navigator!

The timestamp on my bookmark website says I last updated it on: Thu Aug 30 20:14:14 PDT 2001

OldVersion.com tells me that the latest release for Windows that Netscape 4.79 was released in November of that year, and the closest antecedent version available is 4.72 (from February 2000).

I downloaded it and tried fiddling around with the compatibility settings, but without any luck.

Then I tried 4.79, and nope, that didn't work, either. So then I tried Netscape 6.01 - release February 2001.

I happened to have Chrome running at the time because in Firefox I have 1500 tabs open -- fifteen hundred and seven! ;) -- whereas in Chrome it's under 500, so I was trying to tread lightly. How do I know these numbers? For Chrome I found an extension that allows me to copy into the clipboard all open tabs' urls as plain text. It helpfully announces how many such tabs were copied. In Firefox one of the webextension examples gives you a counter.

Do you remember the web without tabs? Time was, you wanted to visit another webpage, you got two option: you navigate away from whatever you're looking at now, or you hit Ctrl-N to make an new window. I think most people used one or a few windows. But you were not gonna be crazy and open more than a dozen windows. I would have, and probably tried but I couldn't. And session saving across crashes or clean exits? Forget it! That what your history and bookmarks are for, grasshopper.

But let's get back to the task at hand: this was the lower right of my screen...

and I decide to start taking screenshots of this journey, click it, and let Windows 10 apply the compatibility settings, and then I'm faced with

the most improbable error message:

WAT?!

I didn't think Chrome had any ancestry shared with the Mosaic super-tree, but whatever - you can't exactly argue with software from 2001, and I have an important screenshot to take...

So now I've quit Chrome, just in case, and going to retry....

no dice....

Damn, what could it be...I've got the Bloomberg Terminal open, I know portions of it are built on Chromium browser technology (had to look it up if this was officially stated somewhere - it is). Ok, so maybe that's what causing the false positive? I close that, and...

...

I hardly have anything open anymore ...is it VLC?

...

nope... Ok, what's left still open... Snipping tool I'm using to capture this epic adventure, a few WSL Debian console windows... the voice recorder that started this post... Task manager and Sysinternals' Process Explorer - where I was checking if perhaps somehow the failed attempt at running what was probably a 16 bit version of Navigator 4.72 was still lingering somewhere... SumatraPDF, Windows Terminal (Preview), gVim, and ...

Zotero?!?

BINGO!!!!!!!!!

Oh right - I guess Zotero uses XUL technology. I didn't really think much about it, but Zotero did start off life as a Firefox extension, and the standalone version came out later, makes sense that it would have grabbed a browser when it struck out on its own.

At this point I had already sent Madicken, who works at NCSA where Mosaic, the progenitor of Netscape hails from, the first two images... So I wanted to play with fire a bit....

Now that I've closed Zotero - can I have Firefox 74 open while installing Netscape 6.01?

Rats! same error...

how about Chrome again?

oh yeah! Sweet. The world makes sense again.

Back to the setup.exe...

I scroll through the EULA - and randomly stop on this section:

12. HIGH RISK ACTIVITIES.  The Product is not
fault-tolerant and is not designed, manufactured or
intended for use or resale as on-line control
equipment in hazardous environments requiring
fail-safe performance, such as in the operation of
nuclear facilities, aircraft navigation or
communication systems, air traffic control, direct
life support machines, or weapons systems, in which
the failure of the Product could lead directly to
death, personal injury, or severe physical or
environmental damage ("High Risk Activities").

By the way - here's a good idea I came across a few months ago: throw EULAs (End User License Agreements) into some publicly indexed version control repo (I saw folks using gists for just that sort of thing: here's the Netscape 6.01 EULA.txt)

Let the folks at Redmond host it.

Fine. I click next...

Interesting - there's a "Read Me" button... I do as I'm told, so I click it.

Oh, a README.txt popped up in Notepad.exe - full contents of that are in that same gist as the EULA

but in them, there's a link to the full release notes over at

http://home.netscape.com/eng/mozilla/ns6/relnotes/6.0.html

Let's try to go there now:

Wat?!

Yeah, I'm sure Yahoo! engineers are jumping right on that.

How did a Netscape site ended up redirecting to Yahoo...

oh right, so AOL bought Netscape (1999), merged with Time Warner in 2001, was spun out again in 2009, after some rough times, and then purchased by Verizon in 2015. In the meantime, Yahoo acquired Geocities (1999) and shut it down in 2009 (yes, I'm still mad! All I remember was that it had some crispy banners, one of which was a scan of a sweet pencil lettering I made of my nick at the time - "ShadowKnight"). No one really cares what happened to Yahoo in the interim, aside from some massive data breaches, until finally, Verizon bought Yahoo in 2017 and merged AOL and Yahoo into one division.

Ok, so fine, maybe fine folks at The Internet Archive (archive.org) can help us with the WayBackMachine.

And indeed we can see what it looked like originally. This site has seen so many redirects over the years - it'd be a fun exercise to go through all of the indexed versions of this kind of site to see how people tried to preserve links. For example, I found out in 2010 it 301s ("Temporary redirect") to http://www.netscape.com/eng/mozilla/ns6/relnotes/6.0.html which then 302s ("Permanent redirect") to http://www.propeller.com/eng/mozilla/ns6/relnotes/6.0.html - which was indexed but happens to be a 404 ("Page not found") error page, at least in 2008. But this wasn't what we came here for, so this yak can roam free among the hills, the valleys, and the caverns of our minds.

... for now...

Where were we?

Oh right, we have to choose an install option. Back in the day I might have clicked "recommended" here, but we're not back in the day, and who wants to play life on easy mode? Let's go custom to see what the options are (I am a control freak, after all)

and immediately get another pop up:

Whoa, let's pause here for a moment.

I am digging such a consent model. The installer is establishing trust: it will not try to do anything behind my back and without my permission. I'm sure it will never abuse that trust. As a 2001 user, I sure am glad that folks involved with computing have such a well-develop sense of ethics. In 2001, the Future is bright. Computing will be filled with transparency. Consumer software and services will be built by folks with a strong moral compass. These are people with principles. With the dot-com bubble burst, we've swiftly inoculated tech from sleazy opportunists. It won't fall victim to the excesses and greed rivaling Wall Street in the 80s...

Which reminds me - how is it that this brilliant video only has 277 thousand views? Here's a excerpt:

Your users won't always understand just how much economic sense it makes to sell them out. And you don't want to alienate them, that would drive down their value.

"It won't always kk ( video )

By the way - the embed code I used above uses the youtube-nocookie.com domain - which is still from the folks at Goolag, but does what it says on the tin and doesn't issue cookies. Also, did you know there used to be a way to disable those annoying related video links from popping up at the end of the video? It's true. You used to be able to just append a rel=0 query parameter to not show related videos. But the corporate overlord bean counters didn't like that. They had to make sure that kids would get glued to the site by feeding them progressively conspiratorial garbage content. So that watching any video would nearly guarantee to pull them into the black hole cesspool of maximally "engaging" "content". What was that quote about users again? Ah yes:

Your users won't always understand just how much economic sense it makes to sell them out. And you don't want to alienate them, that would drive down their value.

Ok, so, in fairness, rel=0 query parameter still does something. It limits the suggested videos to the channel they are from. That's good. But what happens if we follow one of those links? First, we end up on the full youtube site, so that means the cookies are back. Hurray for surveillance capitalism! Also, the recommendations on the right are curated specifically for us, and not limited to the channel the previous video was one. Goodbye, rel=0!

So I have to install Navigator, but I can unselect Mail, Instant Messenger, and Spell Checker... I don't need mail, but whatever, let's just go with the defaults.

Oh, look at all this wonderful bundled crapware. Just in case you had any doubts.

That last one made me throw up a little in my mouth.

I opt for just the classic skin - and it tells me that the total download size will be 9959 K.

I thought about censoring this next screenshot, but 15 year old me wouldn't have like that... What's in the shot is in the shot..

Alright, and when the installation finished here's what we're greeted with:

Did software in 2001 try to phone home? "activation.netscape.com could not be found." sure seems so.

Yes, No, Cancel?

What if I cancel?

Alright, let's go for broke and get that retro look...

Remember all those redirects? Well the browser froze when I got overzealous, clicked on "Interact" at the bottom there and chose to open chat... And on the next load, it crashed... And again...

It just kept crashing...

In case anyone else gets stuck on the same issue ;) I got around this by using the Profile Manager, where I had the option to start the browser in Work Offline mode. Then I turn the "Work Online" option on after the browser loaded (which you can do by plugging together that cute outlet pair on the bottom left).

I do some ego surfing and go to my own site first.

http://pirsquared.org

I got too fancy with my unicode... But hey, this is totally functional.

I made this Loading gif via a screen capture tool and then it finally clicked that not only did I not use Netscape 6 - I remember most everyone's experience was to stick to the 4.x series, because it was so much more usable and not bloated with nonsense, etc, etc.

Alright, but at least I got 6 to run on Windows 10 and that works...When it doesn't crash, anyway... But I did get a error about youtube-nocookie.com... (some of the time, at least)...

And then I realized that I can't go to any site that has https... Because... you know, the protocol that provides that 's' has changed over the years, and our 2001 browser could do SSL 2 or 3 or TLS 1.0... But my website uses TLS 1.3...

I couldn't run to duckduckgo, either, since it redirects plain http to the https endpoint and that also runs TLS 1.3... I couldn't even go to archive.org to view my old site directly on the way back machine, because archive.org run TLS 1.2 at the moment.

It's difficult to find any place that still runs such outdated standards... I tried to search for just a TLS 1.0 test server - but didn't find anything suitable... But then I happen to flip through the recent changelog for Firefox:

Cool - so now we know if we want to find TLS 1.0 and 1.1 website, we should turn to the government of... damn a specific country wasn't specified...

But wait a minute... Firefox 74.0 came out on March

How did Mozilla release an update to a version of Firefox that was in the hands of a bunch of users without... umn...what's the word I'm looking for here...you know, that thing no one seems to think is a thing anymore... user consent?

https://wiki.mozilla.org/Firefox/Normandy/PreferenceRollout

This is the way consent ends
This is the way consent ends
This is the way consent ends
Not with a bang but a whimper

I'm late to the party - this has been going on for about three years - with what I now recall was caused a bit of a splash back in 2017 (Drew DeVault covers in "Firefox is on a slippery slope"). But I didn't know the extent of it. Who has the time to pay attention to the way in which all the software they use changes in anti-social ways.

Anyway, if you don't want the fine folks at Firefox to change your preferences out from under you, I think you go to about:config and switch the app.normandy.enabled setting to false. And if you're interested in specifics of how you've been a guinea pig: about:studies will tell you. And you can go to about:preferences#privacy to disable them.

But I digress...

Let's wrap this up...

At this point, I used my 2020 browser to grab historical snapshot of my old bookmarks site, stripping off the tastefully annotated Way Back Machine user interface insertions, and serve it locally over http via python -m http.server, making sure to change the URL bar to make a historically accurate re-enactment. And now that you know how I got here, you can fully appreciate the effort that went into this next screenshot:

So much so, that I couldn't resist making a video a scroll through :

No video? Oh well.

You can download it as webm, mp4, ogg, or gif.

If you're interested - you can go http://p.laya.com/bookmarks/ (archival copy) or see the rest of p.laya.com from 2001

That's all

The ongoing crisis has been a circuit breaker to our usual patterns. I am taking advantage of this affordance to experiment with and establish channels of communications that are not controlled by others.

Uvas Gold 200

2019-09-17T00:00:00-07:00

My poem about a rainy 200k was published in the Fall 2019 issue of American Randonneur (a quarterly magazine published by Randonneurs USA)

I've been doing samizdat poetry for as long as I've had a web presence (since 1999), but I am now officially a published poet! (I am deliberately not counting the embarrasing hackjob that was published in a youth anthology when I was in 8th grade.)

You can find "Uvas Gold 200" on page 26 - either directly on this skeuomorphic leafing viewer or the PDF, but I'm republishing both the exposition blurb and the poem below. If you prefer to listen, I recorded a reading of it that you can download in different flavors: a local audio only, a local video, or the embeded video version below.

Uvas Gold 200k starts and ends in Fremont, CA and was held on Saturday, December 1st, 2018. The ride frontloads the climbing by going nearly half-way up Mount Hamilton, the tallest peak in the San Francisco Bay Area, and then leads down south as far as Gilroy, "Garlic Capital of the World." Our RBA, Rob Hawks, sent this note the evening before the ride: "While we once had 75 riders signed up, we are now down to 59 even though a number of new names have been added since this was sent out on Monday." It had been a foul weather week, and by the time we were ready to start in rain at a balmy 48 degrees Fahrenheit, there were fewer than thirty people who would brave the cold wetness. The rain was at its worst the first half of the ride, so my memories are skewed toward that portion, with the poem content reflecting that. Call it Inverted Randonesia, the second half of the ride was just spent processing how I made it through the beginning of the day and how epic it was.

Weaving upstream the icy slush
Mount Ham – no cheese
Rough grind of gears and teeth
Steam locomotive breath
The paceline I abandon
Time stretched – warm pizza dough
Intent, attention rising
I take a pull
Thick spray across my rhythmic fogging glasses

Our feet make Revolutions, Randonneur!
With pavement gliding under top tube
It's all the same to you
Mile-marker Marxist
Unbound by thermal fear
"Fair-weather cyclist" you are not

A dreamy cold wet blanket smoky charcoal skyscape
High pitched frying sizzle
Tire fully splashing
The tug of war of winds where we're the rope
Off camber turn, soprano yodeling disc brakes yards ahead
Refreshing wafting garlic wake up – dreamy ride continues
Alert again in the slow slither through Coyote Creek

K-Hound chasing, headwind facing
Camaraderie, half-way in Gilroy supermarket shack
Then on the road again, occasional "Car back!"
December, Northern California, Reign of Rain

                   _
                  / \
                A*   \^   -
             ,./   _.`\\ / \
            / ,--.S    \/   \
           /  `"~,_     \    \
     __o           ?
   _ \<,_         /:\
--(_)/-(_)----.../ | \
--------------.......J

My first DNF (Ft Bragg 600k)

2019-05-13T00:00:00-07:00

It's been six years since my first ride with The San Francisco Randonneurs and four years since my first 200k. I've ridden 18 rides that are at least that distance since then (3x 300k, 2x 400k, 1x 600k), completing my first Super Randonneur Series (2-, 3-, 4-, and 600k in one year) last year after not riding much the year before that. And this weekend I had my first DNF result on the Fort Bragg 600k. I Did Not Finish.

The best response to my choice of abandoning the ride to enjoy the campground came from Peter Curley, who said "That was a very mature decision." A clear departure from typical randonneuring stubbornness and refusal to give up, I celebrated my decision to quit as a victory when I arrived at the campground and made my announcement to the volunteers. I think I was so energetic about it that they did not believe me. I was being kind to myself, to my body, and at peace with the decision by the time I rolled in.

I ride for myself. I ride because it feels good. I ride to challenge my body, to be alone: just my thoughts, my legs, and the road. This weekend, my legs felt strong. Stronger than they did a year ago when I finished the ride. But my objective was different. Paris-Brest-Paris is a 1200 kilometer ride across France which must be completed in under 90 hours that runs once every four years. I have been telling anyone who would listen once they found out about my cycling proclivity about how it is my intent to ride PBP this year for the last two years. That was still my intent this year, and it was my intent during much of the 600k.

But I was not willing to do so at any cost. Thoughts have been lingering about whether these distances are worth it for me since the 2018 Hopland 400k, where I threw up at the last control and struggled to the finish. Nausea again caught me just outside of this same campground in the outbound direction on the Fort Bragg 600k ride last year. This year on Hopland, though riding stronger overall and conserving my strength, I joined a fantastically energetic group and rode too hard before that last control and threw up there again. Though getting much more sleep and feeling fresher at the end, I finished within minutes of my time the previous year (24 hours).

I rode mostly alone on this ride. After letting go of the lead pack on the Camino Alto climb, only a pair of Santa Cruz Randonneurs' jerseys passed me between there and Point Reyes Station. I rolled out of Point Reyes on my own, gaining the only company I had for this ride by catching up to and following a pair of riders on the climbs on Point Reyes-Petaluma road. My legs felt good, I was not pushing too hard, just a sustained and seemingly sustainable effort. I made it to Healdsburg at half past noon. Six and a half hours to cover 88 miles. Perhaps I ate too much here, opting for both a small chicken noodle soup, and some sushi, whereas I usually eat just the soup. Still making progress to Cloverdale, I briefly chatted with a Texan who was visiting and exploring the local routes, crossed and then rode along the cycling leg of an Iron Man event that was going in the opposite direction.

I was starting to be affected by the heat. It wasn't that warm, but I needed to stop at a brand name coffee place in Cloverdale, and had a fizzy water with just a tad bit of juice to regain composure for the climb up 128. The climb rolled along nicely, I shifted down and just kept pedaling at a steady pace and easy effort. But I wasn't eating anymore. By the time I got to Yorkville, I needed another break, and luckily Michael had just stopped at the Yorkville Market, so I decided to stop, too. I got another carbonated beverage, a cherry cola this time, and in what might have been another mistake - honey-mustard flavored pretzel pieces. When I first started riding brevets, salted peanut-butter-filled pretzel pieces were a staple go-to in my basket - I'd just reach down to grab a few every couple of minutes. I thought about reaching for the simpler salted pretzel sticks this time, because I never have trouble consuming simple salted unflavored bread products, but the lizard brain sabotaged the rational sensibilities and swayed the scales in favor of the honey-mustard, which also had some sugar. As a little glimpse into my somewhat delirious state, I thought I had seen four to six cats in the shop. When I asked the proprietor, it turns out there were only two. She wished me a good ride, to be careful on highway 1, and asked if I thought I could make it to the camp before dark. I expressed serious doubts about that, but it turns out I was wrong.

I got to Boonville, another dozen miles out, and took another break, the nausea kicking in again. The next five miles I started to get cold, the sun not yet setting but the cold ocean headwind was starting to take its toll. I stopped to get dressed, and deliberately cycled past the campground without stopping there for another rest though I probably could have used it. And then something clicked in my head. A had been trying to fit a great deal many things in life but cycling had begun dwarfing all my other joys and pursuits this year. My reward for finishing the 600k would have been to train more, probably ride the NorCal Brevet week in June, to get ready for PBP and ride that in August. Instead of PBP, I'll be taking a break.

Randonneuring can be addictive. You keep putting goals out and keep accomplishing them, so you set higher, more challenging goals. Rinse and repeat. This year, cycling has started to become a kind of rat race for me. Yes, I still enjoy cycling, very much, but it has started to grind me down. I lost sight of why I enjoy this sport so much to begin with.

I decided to stop. I decided to exert and reaffirm control over what I value in life. The campground was by far the biggest highlight of my 2018 finish on this ride. I was the very last rider into the overnight control, rolling in half-past 3am or so. I sat myself down on one of the folding chairs in front of the fire pit, was fed without having to get up, and drifted off in that same spot around 4am. Two hours later, before I could open my eyes, my olfactory system was cannonballed into a perfect aroma pool of fresh coffee and bacon. "This is AMAZING!" That campground served as my magical site of rejuvenation after just two hours of sleep. My spirits lifted, though I did hang out longer than most at breakfast that year, in no rush to head out, taking in the gurgling creek, the majestic towering redwoods all around, inhaling and ingesting the sights and sounds and smells all around.

So less than 12 hours and 145 miles into the ride, and a few miles past the Indian Creek Campground that was the planned overnight, I decided to turn around and go directly there. It was about 40 miles to the Safeway in Fort Bragg, 42 miles back. I could have done it. I could have taken a break, even after I got to the campground. There was plenty of time left on the clock.

I could have finished the ride. The weather was great, and I had plenty of clothes to keep me warm, something I learned to appreciate on this ride the year before, confusing being cold with being exhausted, and quickly regaining my legs after finally throwing a jacket on. I could have finished the ride, but I also needed to get off the treadmill, the hamster wheel that was starting to grind away at a sense of balance in life. I've been spending more and more time on the bike recently, and not because I was so thoroughly enjoying it. It had become almost a job. Not a particularly unpleasant one, I like riding bikes, but the compulsory nature of it did not sit well with me. I rode in the rain to work. I rode in the hotel gym morning before and again at night after while traveling for a conference the week before. I was just biking too much. Too many other parts of my life were getting squeezed out to make room for the PBP effort. I still want to ride PBP, but letting go of this year's attempt is what I need to focus on other important aspects of life.

So after I had a beer and regained composure, I switched modes. Chatting with the volunteers during much of the night. Then as the riders started to come in, we'd offer them food, refilling plates when they were done and still hungry, insisting they can stay seated. Reed, a fellow campground DNFer, and I kept mentioning how we were earning our keep. Conversation flowed, lots of chuckling, story telling, sharing of experiences. I'll treasure these hours I spent in camp instead of on the bike. Chuck graciously offered me a tent and a sleeping bag and I was in bed a quarter to 11pm, fed and off the bike for 5 hours by then. It would probably have taken me again until 3am had I continued to Fort Bragg. I got 7 hours of sleep, instead of 2. It was a DNF, but it was also a DHF - I Did Have Fun. :)

The campground volunteers were rock solid! Eric and Arthur the big goofball dog, Grace and Rick, the two co-owners of Bo the smaller goofball dog, Aaron, Sourav, Chuck, Robert, Reed. Thank you all for such an memorable experience. It was a privilege to spend several hours with you. Thank you.

We cleaned up and packed up in the morning after the riders left. Chuck has an amazing van set up to carry up to 8 (!!) bikes, offered a ride for our bikes. Sourav offered to deliver our bodies. On the way back, Reed, Sourav and I discussed just how special and unique the San Francisco Randonneurs community is. The level of and attention to detail, the shared sense of purpose, the volunteer involvement. Thank you Rob Hawks. Thank you to everyone who volunteers to support SFR rides. Thank you to everyone who rides, you give us plausible deniability: we can't all be crazy :)

Carrying my bike as I climbed up the long sets of stairs from the BART station platform this morning, I had a liberating realization. It was the first time in a long while that I was not internally justifying the effort as "this will help me on PBP". I was once again doing it for the sole simple reason that this is just what I do.

                   _
                  / \
                A*   \^   -
             ,./   _.`\\ / \
            / ,--.S    \/   \
           /  `"~,_     \    \
     __o           ?
   _ \<,_         /:\
--(_)/-(_)----.../ | \
--------------.......J

PyCon2019 poem

2019-05-03T00:00:00-07:00

I'm back in Cleveland for another Pycon. Yesterday was my first full day here. Along with Matt Seale, I was a helper at Matthias Bussonnier tutorial ("IPython and Jupyter in Depth: High productivity, interactive Python). The sticky system is efficient at signaling when someone in a classroom needs help, and a lot of folks don't know that this practice was popularized by Software Carpentry workshops and continues to be used at The Carpentries.

I stepped out for a coffee refill and bumped into a large contingent of Bloomberg folks I'd never met (Princeton office). I guess we have something like 90 people at the conference this year, and I made the usual and true remark about how I go to conferences to meet the other people who work at our company. Then after his tutorial concluded, Matthias and I bumped into Tracy Teal, exchanged some stickers, and chatted about The Carpentries, Jupyter, organizing conferences, governance and sponsorship models, and a bunch of other stuff.

Matthias was a good influence in trying to participate in the swag bag stuffing, but it turned out we had almost an hour to ~~kill~~ maim (we decided injury's a better fate than death for that hour). Luckily, we bumped into John Lam and proceeded to chat away and entertain one another. The swag bag stuffing was really fun - highly recommend it to everyone! Smart people performing a task that allows for lots of banter and brief social interaction turn while there's music and a chance to move around turns out to be a good way to have time continue flying by. Too many people to name here, but Scopz was there and we made terrible puns, as usual.

After the swag bag stuffing party, reconnected with Tracy and we chatted some more while walking around and picking up swag (mostly socks), joined by Gil Forsyth who handed me Xonsh sticker, bumped into Saul Shanabrook. Then stopped by to say hi to Jess, a friend of Anna Vu's, whom I only ever see at PyCon), and then chatted with Scott Sanderson, met Hugo Bowne-Anderson, and got to briefly greet Jonathan Helmus.

I walked Matthias to his hotel, then went back to Butcher and the Brewer (again, Matthias joined me for dinner there the night before), and wrote out the poem you'll find below. After I wrote it, a bunch of my Bloomberg colleagues showed up, and I ended up chatting for long time with John Purviance. In keeping with tradition, though we work at the same company, I met John by accident at a coffee shop during last year's Pycon (he works in the New York office, whereas I'm in SF).

One day down, and it's already been a wonderful Pycon.

Pycon 2019

May 2nd, Pycon Cleveland Two, as well
The social wealth observed, created
not jaded, lots to say and listen to
Bring energy and silliness, too
My objective to subjectively inject 
a crack and tear decorum fabric
be vulnerable
a buzzing bee
break up the swag bag stuffing monotony
walk the path, switch sides, talk trash
Great folks, give stickers, smiles, rehash
A splash of warmth, a squak, a "BAM!"
A nudge - we jam enjoying this team work
not work at all - a play - a sandbox
roles all clear yet much room left 
to improvise
Devise a way to make us laugh
each finds their path
rocking forth and back
swaying fro and too
a rhythm of a train a-moving
Clapping celebration when we're done
"Hey-hey! We're done!" - a dinner earned
and well deserved - many socks collected
New people met and old friends reconnected
Tutorials and opening reception done
and now Pycon 2019's officially begun

Get in it

2018-06-12T00:00:00-07:00

Two weeks ago, Project Jupyter had our only planned team meeting for 2018. There was too much stuff going on for me to write a poem during the event as I had in previous years (2016, and 2017), so I ended up reading one of the pieces I wrote during my evening introvert breaks in Cleveland at PyCon a few weeks earlier.

Once again, Fernando and Matthias had their gadgets ready to record (thank you both!). The video below was taken by Fernando.

click to view the video

`Get in it`

Time suspended
Gellatinous reality - the haze
submerged in murky drops summed
in swamp pond of life

believe and strive, expand the mind
A state sublime, when in your prime you came to
me and we were free to flow and fling our
cares, our dreams, our in-betweens, our
rêves perdues, our residue -- the lime of light
the black of sight -- all these converge and
merge the forks of friction filled with fright
and more -- the float of logs that plunges deep
beyond the fray, beyond the keep -- a leap of faith
the lore of rite, with passage clear, let
fear subside, the wealth of confidence will
rise and iron out wrinkles of doubt

Commit to change and stash your pride
then push your luck, and make amends.
Branch out your thoughts, reset assumptions
then checkout.

The force of pulls t'wards master class
Remote of possibilities. Rehash the past
Patch up the present -- what's the diff?

There's nothing left -- except to glide -- and
soar beyond your frame of mind.  try not to pry
cry, freedom, cry.

SciPy 2018 dates and call for abstracts

2017-12-14T00:00:00-08:00

I'm helping with next year's SciPy conference, so here are the details:

July 9-15, 2018 | Austin, Texas

Tutorials: July 9-10, 2018
Conference (Talks and Posters): July 11-13, 2018
Sprints: July 14-15, 2018

SciPy 2018, the 17th annual Scientific Computing with Python conference, will be held July 9-15, 2018 in Austin, Texas. The annual SciPy Conference brings together over 700 participants from industry, academia, and government to showcase their latest projects, learn from skilled users and developers, and collaborate on code development. The call for abstracts for SciPy 2018 for talks, posters and tutorials is now open. The deadline for submissions is February 9, 2018.

Conference Website: https://scipy2018.scipy.org
Submission Website: https://easychair.org/conferences/?conf=scipy2018

Talks and Posters (July 11-13, 2018)

In addition to the general track, this year will have specialized tracks focused on:

Data Visualization
Reproducibilty and Software Sustainability

Mini Symposia

Astronomy
Biology and Bioinformatics
Data Science
Earth, Ocean and Geo Science
Image Processing
Language Interoperability
Library Science and Digital Humanities
Machine Learning
Materials Science
Political and Social Sciences

There will also be a SciPy Tools Plenary Session each day with 2 to 5 minute updates on tools and libraries.

Tutorials (July 9-10, 2018)

Tutorials should be focused on covering a well-defined topic in a hands-on manner. We are looking for awesome techniques or packages, helping new or advanced Python programmers develop better or faster scientific applications. We encourage submissions to be designed to allow at least 50% of the time for hands-on exercises even if this means the subject matter needs to be limited. Tutorials will be 4 hours in duration. In your tutorial application, you can indicate what prerequisite skills and knowledge will be needed for your tutorial, and the approximate expected level of knowledge of your students (i.e., beginner, intermediate, advanced). Instructors of accepted tutorials will receive a stipend.

Mark Your Calendar for SciPy 2018!

SciPy 2017 tips

2017-07-10T00:00:00-07:00

After missing it for a couple of years, I am happy to be back in Austin, TX for SciPy this week!

Always invigorating and exhilarating, Scientific Computing with Python (SciPy) has remained a top quality venue for getting together with fellow Pythonistas, especially the academically-bent variety.

As a graduate student eight years ago, I was fortunate enough to be one of receive sponsorship and attended my first SciPy - SciPy 2009. This was the last time it was held at CalTech in Pasadena, CA.

@ivanov/status/2648139681

The following year, in 2010, at the first SciPy held in its now usual spot in Austin, TX, each attendee got a bottle of delicious salsa!

Here are some oy my thoughts about attending this wonderful conference.

Conference Tips

bring a sweatshirt -- Yes, I know Austin's hot, but at the AT&T center, they don't mess around and crank the air conditioning all the way up to 11!

join the slack group -- This year, there's a Slack group for SciPy: the link to join is in a pair of emails with the titles "Getting the most out of SciPy2017" and "Getting the most out of SciPy2017-UPDATED", both from SciPy2017 Organizers. So far at the tutorials slack has served as a useful back channel for communicating repo URLs and specific commands to run, signaling questions without interrupting the speakers' flow.

engage with others during the breaks, lunch, etc -- There are lots of tool authors here and we love chatting with users (and helping you become contributors and authors yourselves). Not your first SciPy and feeling "in-your-element"? Make the effort to invite others into the conversations and lunch outings you're having with old friends - we're all here because we care about this stuff.

take introvert breaks (and be considerate of others who may be doing the same) - I'm an introvert. Though I enjoy interacting with others (one-on-one or in small groups is best for me), it takes a lot of energy and at some point, I run out of steam. That's when I go for a walk, stepping away from the commotion to just have some quiet time.

be kind to yourself (especially at the sprints) -- Between the tutorials, all of the talks, and the sprints that follow, there will be a flurry of activity. Conferences are already draining enough without trying to get any work done, just meeting a bunch of new people and taking in a lot of information. It took a lot of false starts for me to have productive work output at sprints, but the best thing I've learned about them is to just let go of trying to get a lot done. Instead, try to get something small and well defined done or just help others.

Stuff to do in Austin

The conference already has a great list of Things to do in Austin, as well as Restaurants, so I'll just mention a few of my personal favorites.

Barton Springs Pool. Take a nice dip in the cool waters, and grab a delicious bite from one of the food trucks at The Picnic food truck park.

Go see the bats. The Congress Ave bridge in Austin is home to the largest urban bat colony in the world. You can read more about this here, but the short version is that around sunset (8-9pm) - a large number of bats stream out from underneath the bridge to go feed on insects. Some days, they leave in waves (this Saturday there were two waves, the first was smaller, but many people left thinking that was the entire show).

I hope you enjoy SciPy2017!

June 1st, 2017

2017-06-03T00:00:00-07:00

We had another biannual Jupyter team meeting this week, this time it was right nearby in Berkeley. Since I had read a poem at the last meeting, I was encouraged to keep that going and decided to make this a tradition. Here's the result, as delivered this past Friday, recorded by Fernando Pérez (thanks, Fernando!).

`June 1st, 2017`

We struggle -- with ourselves and with each other
we plan -- we code and write
the pieces and ideas t'wards what we think is right
but we may disagree -- about
the means, about the goals, about the
shoulders we should stand on --
where we should stand, what we should stretch toward
shrink from, avoid, embrace --
a sense of urgency - but this is not a race

We can't erase the past
but we have built this future present
There's much to learn, to do...

Ours not the only path, no one coerced you here
You chose this -- so did I and here we are --
still at the barricades and gaining ground
against the old closed world:
compute communication comes unshackled

March 29th, 2017

2017-04-07T00:00:00-07:00

What's missing -- feels like there's something missing --
The capacity is there -- the job's not stressful but
I somehow fail at the ignition stage - all this
fuel just sitting around -- un-utilized potential
How do I light that fire? Set it ablaze
in a daze caught up in the haze of comfort
I need to challenge myself, raising tides lift
all boats, but they also drown 
livestock
cows, horses, and goats, seeking refuge in hills
that once covered in grass now fill up like
lifeboats. 
Doctors in white coats 
say "Keep your spirits up" -- hope floats.

November 9th, 2016

2016-11-21T00:00:00-08:00

Two weeks ago, I went down to San Luis Obispo, California for a five day Jupyter team meeting with about twenty five others. This was the first such meeting since my return after being away for two years, and I enjoyed meeting some of the "newer" faces, as well as catching up with old friends.

It was both a productive and an emotionally challenging week, as the project proceeds along at breakneck pace on some fronts yet continues to face growing pains which come from having to scale in the human dimension.

On Wednesday, November 9th, 2016, we spent a good chunk of the day at a nearby beach: chatting, decompressing, and luckily I brought my journal with me and was able to capture the poem you will find below. I intended to read it at a local open mic the same evening, but by the time I got there with a handful of fellow Jovyans for support, all of the slots were taken. On Friday, the last day of our meeting, I got the opportunity to read it to most of the larger group. Here's a recording of that reading, courtesy of Matthias Bussonnier (thanks, Matthias!).

`November 9th, 2016`

The lovely thing about the ocean is
that it
is
tireless 
It never stops
incessant pendulum of salty foamy slush
Periodic and chaotic
raw, serene 
Marine grandmother clock  
crashing against both pier
and rock

Statuesque encampment of abandonment
recoiling with force
and blasting forth again
No end in sight
a train forever riding forth
and back
along a line
refined yet undefined
the spirit with
which it keeps time 
in timeless unity of moon's alignment

I. walk. forth.

Forth forward by the force
of obsolete contrition
the vision of a life forgotten
Excuses not
made real with sand, wet and compressed
beneath my heel and toes, yet reeling from
the blinding glimmer of our Sol
reflected by the glaze of distant hazy surf
upon whose shoulders foam amoebas roam

It's gone.
Tone deaf and muted by

anticipation
each coming wave
breaks up the pregnant pause
And here I am, barefoot in slacks and tie
experiencing sensations
of loss, rebirth and seldom 
kelp bulbs popping in my soul.

Jupyter's Gravity

2016-08-31T00:00:00-07:00

I'm switching jobs.

For the past two years I've been working with the great team at Disqus as a member of the backend and data teams. Before that, I spent a half-dozen years mostly not working on my thesis at UC Berkeley but instead contributing to to the scientific Python ecosystem, especially matplotlib, IPython, and the IPython notebook, which is now called Jupyter. So when Bloomberg reached out to me with a compelling position to work on those open-source projects again from their SF office, such a tremendous opportunity was hard to pass up. You could say Jupyter has a large gravitational pull that's hard to escape, but you'd be huge nerd. ;)

I have a lot to catch up on, but I'm really excited and looking forward to contributing on these fronts again!

in transit

2016-05-26T00:00:00-07:00

Standing impatient, platform teeming, almost noon
Robo voices read off final destinations
But one commuter's already at his
He reached for life's third rail

There is no why in the abyss
There's only closing credit hiss
The soundtrack's gone, he didn't miss
Reaching for life's third rail

We ride on, now, relieved and moving forward
Each our own lives roll forth, for now
But now is gone, for one among us
Who reached for life's third rail

We rock, to-fro, and reach each station
Weight shifting onto forward foot
Flesh, bone ground up in violent elation
And bloody rags, hours ago a well worn suit

I ride the escalator up and pensive
About what did and not occur today
Commuter glut, flow restricted
A crooked kink in public transport hose resolved.

My first 200 K

2015-04-19T00:00:00-07:00

Yesterday, I rode my longest bike ride to date - the El Cerrito-Davis 200K - with the San Francisco Randonneurs. A big thank you to all the volunteers and randos who made my first 200k so much fun.

First, for the uninitiated, an aside about randonneuring:

I discovered the sport because I'm a cheapskate. I had gotten more and more into cycling over the past 2 years or so, and though I was riding through the East Bay hills mostly alone, I wanted to do a "Century" - a 100 mile ride. Looking up local rides I found out that, while most centuries cost a nontrivial amount of money for the poor grad student I was back then ($60-$200), the San Francisco Randonneur rides were all $10-$20. As I dug deeper and learned about the sport, I found out that the reason for low cost, is that rando rides are unsupported - randonneuring is all about self sufficiency. You are expected to bring gear to fix your own flats, as well as carry or procure your own snacks and beverages.

The Randonneurs USA (RUSA) website succinctly summarizes the sport.

Randonneuring is long-distance unsupported endurance cycling. This style of riding is non-competitive in nature, and self-sufficiency is paramount. When riders participate in randonneuring events, they are part of a long tradition that goes back to the beginning of the sport of cycling in France and Italy. Friendly camaraderie, not competition, is the hallmark of randonneuring.

This description is strikingly similar to my beliefs both about cycling and computing, so I knew I found a new activity I would greatly enjoy. This has been borne out on three previous occasions when I have participated in the ~100 km Populaire rides, which are intended as a way to introduce riders to the sport, yet still be within reach of wide variety of cycling abilities.

We moved in late December, and between unpacking, rainy weekends, and being sick - I haven't been able to get much riding in. However, I've been wanting to do a 200K for a long time - my century, coveted for years, seemed within reach more than ever. To seal my commitment to the ride, I went ahead and ordered a spiffy looking SFR cycling jersey:

Here's the 213km (132 miles) ride map and elevation profile, what follows is my ride report.

Start Control

The ride started at 8am at a Starbucks by El Cerrito BART station, so I just rode there from my house as I frequently do on my morning commute. I prepared my bike, packed my gear and food the night before, and the only thing I needed to do was to fill up my water bottle before the ride started. I showed up a dozen minutes before the start, with most riders already assembled, got my short drip and a heated up croissant, and totally failed to get any water. It's not a super terrible thing, I frequently don't end up drinking much at the beginning of my rides, but I had now set myself up to ride to the second control (44 km / 27 miles) without any water.

Randonneuring isn't about racing - as you traverse from control to control, there's just a window of time that you have for each control, and so long as you make it through each control withing that time window, you complete the event! There is no ordinal placement, the first rider to finish has just as much bragging rights as the last. The only time people talk about time is when they're trying to make a new personal best. The second control was open roughly between 9 and 11am, so I could have stopped off somewhere to pickup water, but that didn't fit into my ride plan.

You see, the cue sheet is two pages - with the first page dedicated to getting you to the second control, as there are a lot of turns to make in the East bay until you get to Benicia. Accordingly, my plan was to stick with the "fast" group who know the route well, so that I wouldn't have to look at the cue sheet at all. This was a success - and I got to chat with Jesse, whom I rode a fair chunk of the 111 km Lucas Valley Populaire back in October (here's a good video of the first chunk of that ride).

Control #2

When 7 of us got to the Benicia control at 9:45, I volunteered to guard the bikes as folks filed into the gas station mart to buy something (getting a timestamped receipt is how you prove that you did the ride from one control to the next within the allotted time). Lesson learned: I should have used that time to shed my warm second jersey and long pants, as it had warmed up by then. It just so happens that by the time I went inside, there was a line for the bathroom, and by the time I got out, the fast group was just heading out, yet I still needed to change.

Because keeping up with the "fast" group wasn't that big of a deal and actually rather fun, I really wanted to try to catch up to them, so I stepped it up, and told a couple of other riders that I'd do my best to pull us to them (in case you didn't know - the aerodynamics of cycling make it much easier for those behind the leader to keep up the pace, even if that pace isn't something they could comfortably do on their own). We didn't succeed in catching them during the first 10 mile stretch of road, but I kept pushing with a high cadence in my highest gear, and about 5 miles later, we caught up with them! The only problem was, by this point, I had wasted so much energy, that I couldn't make the last 20 feet to them, though I was happy to see the two guys who'd been letting my push the whole time, use their fresher legs and join the group. So there I was, a few dozen feet from my intended riding partners, but as more and more pavement went past our wheels, the distance between us slowly widened.

Somewhat disheartened by this (though, again, happy that I all of my pulling wasn't all for naught, since I bridged two others to the group), and now overheating, I decide to stop by the side of the road, removed my long sleeved shirt, put on sunscreen (Lesson learned: don't forget your ears, too - they're the only part of me that burned). A fellow rando rider Eric went past, to my smiling cheer of "Go get 'em!". Then when pair of riders asked if I'm alright, I nodded, and decided to hop back on the bike and ride with them for a bit. It was nice to let someone else pull for a bit, but shortly thereafter, we started the first serious ascent, and my heart was pounding too hard from exhaustion and the heat - and I had to stop to again to catch my breath and get some more food in me.

Luckily, when I resumed riding up that hill, I had a mental shift, gave myself a break, and given how tired my legs were already, even though I wasn't even half-way through the ride, I reminded myself that I'll just spin in a low gear if I have to, there's no rush, I'm not racing anyone, and though I know this won't end up being as good of a 200k as it could have been had I trained more in recent months, it was still up to me to enjoy the ride. One of the highlights of the ride were all of the different butterflies I got to see along the way that I started noticing after this change in mental attitude.

After the long climb, followed by a very nice descent, I got onto the Silverado Trail for 14 miles of a straight road with minimal elevation changes. Though my legs were again cooperating more, it was starting to get kind of old, and then out of nowhere, one of the riders I had pulled earlier rides past me, but then proceeds to slow down and ride along side me for a chat. He hadn't been able to keep up with the fast group for very long, and ended up stopping somewhere along the way to eat, which is why I didn't notice that I had passed him. We took turns pulling for each other, which took away from the monotony of Silverado, but he had a lot more in the tank, and I again wasn't able to keep up, losing him with 4 miles to go to St Helena.

Needing another break, with 4 miles to the next control, I decided I would need to spend a while there, recuperating, if I am to make it through the rest of the ride.

Control #3

I got to Model Bakery at 1:10pm, with many familiar faces from earlier in the day already enjoying their food, but ended up staying there until 2:30 - eating my food, drinking water, just letting my legs rest.

I ended up riding out solo, and really enjoyed the early parts of 128 (after missing the turnoff by a hundred feet) - luckily this was just the spot I had my last stop at, so I quickly turned around and got back on the road.

The problem was it kept getting hotter and hotter - it seemed that I couldn't go a mile without taking a good gulp of water. I still stuck to my strategy of just spinning fast without really pushing hard, since recovering from being out of breath is way faster than waiting for exhausted legs to obey your commands. I made it a good chunk of the way to Winters, but still ended up having to stop half way up the climb near Lake Berryessa. Another SFR rider, Julie, climbed past me, checking if I was OK as she went by. It's great to have that kind of camaraderie along the ride, a couple of people even gently expressed their concern that my rear wheel was out of true - which I knew but kept putting off getting it fixed. These nudges gave me more resolve to get that taken care of.

Finishing off the last of the Haribo Gummi Candy Gold-Bears that I brought with me (and you know you're tired and dehydrated when it takes effort to just chew), I got back on the bike and headed further up the hill. Then, finally, I didn't think I could ever be so cheered up by road sign (hint: they only put "TRUCKS USE LOWER GEAR" signs at the top of big hills).

Control #4

I finally got to Winters at 5:37pm, got myself a Chai Smoothie, and Julie, who unfortunately was going to miss her train home, proposed that we ride together the rest of the way to Davis. Again - though I really enjoy my alone time while cycling, it's also quite fun to have strong riders to ride with, so as the sun started descending behind us, and no longer scorchingly hot, we set out for the final 17 miles to Davis at a good clip, given how much riding we had already done.

Finish Control

Dodging drunk college kids (it was Picnic Day at UC Davis) was the last challenge of the ride. As a UCD alum, this was a homecoming of sorts, so I lead the way through town as we made our way to the Amtrak station. We finished just before 7, and I caught the 7:25pm train back to the Bay Area - enjoying the company of a handful of other randonneurs.

Thanks for reading my ride report!

                   _
                  / \
                A*   \^   -
             ,./   _.`\\ / \
            / ,--.S    \/   \
           /  `"~,_     \    \
     __o           ?
   _ \<,_         /:\
--(_)/-(_)----.../ | \
--------------.......J

pedestrian musings

2014-08-25T00:00:00-07:00

I walk in monologue 
    through Berkeley's Hills
Feet pressing into sidewalk firmly
I eat the pensive mood 
    solitude brings
And bite into the juiciness of
    self-reflection
I write, first time in years,
    free verse impromptu
Taking few dozen steps
    between each pair of lines
I yearn, on tip-toes
    stretching high, to be expressive
A mode of being longtime
    self-denied
I'm walking home - from job
    I'll soon be leaving
To find myself believing once 
    again
That which I do defines 
    me not and feeling
That which I am is
    good. enough. a lot.

starting my job search

2014-06-14T00:00:00-07:00

I am starting to look for a job in the San Francisco Bay Area.

Since many recruiters ask for and presumably look at GitHub profiles, I decided to give mine a little facelift:

In case you aren't familiar, that banner was motivated by Joel Spolsky's Smart and Gets Things Done, which is a book about hiring good developers . So I decided to tweet it out, mentioning @spolsky and he favorited it!

@ivanov/status/476932602587123712

Yesterday, I decided to tweet out an image that's at the top of my resume as a standalone tweet- mentioning Joel Spolsky again, and he liked it well enough to retweet it to his 90 thousand followers, so it's been getting plenty of love.

@ivanov/status/477477547957944321

@ivanov/status/477520571907842048

Perhaps unsurprisingly, the only person to contact me as a result of this so far is a reporter from Business Insider :

My editor would like to post it on our site as an example of a creative way to format a resume... I'm wondering if we can get your permission to do this?

So that's what prompted this post: I simply added my name and a Creative Commons Attribution Licence (CC-BY) to the two images, and then sent my permission along.

Outside of that, no prospective employers have gotten in touch. But like I always say: you can't win the lottery if you don't buy a ticket. And since I also enjoy mixing metaphors, I'll just keep on fishing!

indenting with tabs

2014-04-03T00:00:00-07:00

** This post was written as an IPython Notebook. You can view it on nbviewer, or download it. **

Greg Wilson asked on the IPython mailing list:

Subject: easiest way to insert a literal tab character in a code
cell? 
Greg Wilson, on 2014-04-03 18:37,  wrote:
> Hi,
> I'd like to put literal tab characters in cells, but of course tab means 
> "indent" to the editor.  What's the easiest way to do this?
> Thanks,
> Greg
> p.s. because I'm going to write Makefiles in the notebook...

The easiest way to do this is to just get a tab character somewhere that you can copy, and then paste it in.

In [1]:

print("\t")

In [2]:

    # I copy pasted the output of the cell above here

An alternative solution is to make a string with tabs and insert it into another cell, using IPython machinery.

In [3]:

ip = get_ipython()

In [4]:

ip.set_next_input("\tMakefiles are awesome")

In []:

    Makefiles are awesome

If you have a file on disk or on the web, you can also just use the %load magic to do this.

In [5]:

%load /home/pi/file_with_tabs

In []:

default:
    cat /etc/issue
    whoami

Such files can be written with the %%writefile cell magic... but of course you need to have inserted tabs there in some manner.

In [6]:

%%writefile ~/file_with_tabs
default:
    cat /etc/issue
    whoami

Overwriting /home/pi/file_with_tabs

In [7]:

!make -f /home/pi/file_with_tabs

cat /etc/issue
Debian GNU/Linux jessie/sid \n \l

whoami
pi

The more involved, but more usable way¶

We can set up CodeMirror to insert tabs instead of spaces.

In [8]:

%%javascript

IPython.tab_as_tab_everywhere = function(use_tabs) {
    if (use_tabs === undefined) {
        use_tabs = true; 
    }

    // apply setting to all current CodeMirror instances
    IPython.notebook.get_cells().map(
        function(c) {  return c.code_mirror.options.indentWithTabs=use_tabs;  }
    );
    // make sure new CodeMirror instances created in the future also use this setting
    CodeMirror.defaults.indentWithTabs=use_tabs;

    };

The reason we attach tab_as_tab_everywhere to IPython is because when we use the %%javascript magic, any variables we define there must be called in the same cell that defined it - they get their own closure. The reason we do this is to allow the notebook javascript to not get screwed up when there are javascript errors. We could have attached it to window or CodeMirror or anything else that's already in javascript-land.

I covered how to add functions like this to the custom.js file in your profile in my post about disabling blinking in the notebook. That way these little functions are available in every notebook, without you having to insert a cell defining them.

Now we've got code that allows us to apply the change to all current and future cells. We leave it as an exercise for the interested reader to modify that code and make a little button in the toolbar, to toggle it on a per-cell basis.

Hints:

You can get to the code mirror instance via IPython.notebook.get_selected_cell().code_mirror

** This post was written as an IPython Notebook. You can view it on nbviewer, or download it. **

bipython 0.1.0

2014-04-01T00:00:00-07:00

the boldly indiscriminate python interpreter

"...because you shouldn't have to choose."

PROLOGUE

Two interpreters, both alike in dignity,
In fair Pythona, where we lay our scene,
From ancient grudge break to new mutiny,
Where civil code makes git commits unclean.
From forth the fatal loins of these two foes
A newer kind of stranger's given life;
Whose misadventured piteous overthrows
Doth with its birth bury its parents' strife.

ACT I

Enter bpython and ipython

bpython

I'm a fancy terminal-based interface to the Python interpreter. I give you
inline syntax highlighting and auto-completion prompts as you type, and I'll
even automatically show you a little tooltip with a docstring and parameter
list as soon as you hit ( to make the function call, so you always know
what you're doing! I'm svelte and proud of it - I don't try to do all of the
shenanigans that ipython does with the shell and the web, but the cool kids
love my rewind feature for demos. I strive to make interactive python coding
a joy!

ipython

I'm an awesome suite of interactive computing ideas that work together.
For millennia, I've given you tab-completion and object introspection via
obj? instead of help(obj) in Python. I also have sweet shell features,
special magic commands (%run, %timeit, %matplotlib, etc.) and a
history mechanism for both input (command history) and output (results
caching).

More recently, I've decoupled the REPL into clients and kernels, allowing
them to run on independent of each other. One popular client is the
IPython Notebook which allows you to write code and prose using a web
browser, sending code to the kernel for execution and getting rich media
results back inline. The decoupling of clients and kernels also allows
multiple clients to interact with the same kernel, so you can hook-up to
that same running kernel from the terminal. The terminal workflow makes
more sense for some things, but my user interface there isn't as polished
as bpython's.

Enter bipython

bipython

By your powers combined... I am bipython!

Exeunt

The Power is Yours!

pip install  bipython
easy_install bipython

bipython requires ipython, pyzmq, bpython, and urwid.

For now, you'll need to have a running ipython kernel before running bipython. You can do this by either opening a notebook or running ipython console. It won't always be like this, I'll fix it as soon as I can, but it'll be sooner with your help over ivanov/bipython.

After that, just run bipython and enjoy the ride.

Here's a walkthrough of ipython, bpython, and bipython:

The screencast is 20 minutes long, but here I'll play it back double speed. There's no sound, and you can pause at any time and select / copy portion of the text as you like. Changing the browser font size in the usual way works, too. (click here if the embed didn't work)

stem-and-leaf plots in a tweet

2014-03-12T00:00:00-07:00

Summary: I describe stem plots, how to read them, and how to make them in Python, using 140 characters.

My friend @JarrodMillman, whose office is across the hall, is teaching a computational statistics course that involves a fair amount programming. He's been grading these homeworks semi-automatically - with python scripts that pull the students' latest changes from GitHub, run some tests, spit out the grade to a JSON file for the student, checks it in and updates a master JSON file that's only accessible to Jarrod. It's been fun periodically tagging along and watching his suite of little programs develop. He came in the other day and said "Do you know of any stem plot implementation in python? I found a few, and I'm using one that's ok, but it looks too complicated."

For those unfamiliar - a stem plot, or stem-and-leaf plot is a more detailed kind of histogram. On the left you have the stem, which is a prefix to all entries on the right. To the right of the stem, each entry takes up one space just like a bar chart, but still retains information about its actual value.

So a stem plot of the numbers 31, 41, 59, 26, 53, 58 looks like this:

That last line is hard to parse for the un-initiated. There are three entries to the right of the 50 stem, and these three entries 3 8 and 9 is how the numbers 53, 58, and 59 are concisely represented in a stem plot

As an instructor, you can quickly get a sense of the distribution of grades, without fearing the binning artifact caused by standard histograms. A stem-plot can reveal subtle patterns in the data that are easy to missed with usual grading histograms that have a binwidth of 10. Take this distribution, for example:

70:XXXXXXX
80:XXXXXXXXXXX
90:XXXXXXX

Below are two stem plots which have the same profile as the above, but tell a different story:

 7|7888999
 8|01123477899
 9|3467888

Above is a class that has a rather typical grade distribution that sort of clumps together. But a histogram of the same shape might come from data like this:

 7|0000223
 8|78888999999
 9|0255589

This is a class with 7 students clearly struggling compared to the rest.

So here's the code for making a stem plot in Python using NumPy. stem() expects an array or list of integers, and prints all stems that span the range of the data provided.

from __future__ import print_function
import numpy as np
def stem(d):
    "A stem-and-leaf plot that fits in a tweet by @ivanov"
    l,t=np.sort(d),10
    O=range(l[0]-l[0]%t,l[-1]+11,t)
    I=np.searchsorted(l,O)
    for e,a,f in zip(I,I[1:],O): print('%3d|'%(f/t),*(l[e:a]-f),sep='')

Yes, it isn't pretty, a fair amount of code golfing went into making this work. It is a good example for the kind of code you should not write, especially since I had a little bit of fun with the variable names using characters that look similar to others, especially in sans-serif typefaces (lI10O). Nevertheless, it's kind of fun to fit much functionality into 140 characters.

Here's my original tweet: @ivanov/status/443980372192137216

You can test it by running it on some generated data:

>>> data = np.random.poisson(355, 113)
>>> data
array([367, 334, 317, 351, 375, 372, 350, 352, 350, 344, 359, 355, 358,
   389, 335, 361, 363, 343, 340, 337, 378, 336, 382, 344, 359, 366,
   368, 327, 364, 365, 347, 328, 331, 358, 370, 346, 325, 332, 387,
   355, 359, 342, 353, 367, 389, 390, 337, 364, 346, 346, 346, 365,
   330, 363, 370, 388, 380, 332, 369, 347, 370, 366, 372, 310, 348,
   355, 408, 349, 326, 334, 355, 329, 363, 337, 330, 355, 367, 333,
   298, 387, 342, 337, 362, 337, 378, 326, 349, 357, 338, 349, 366,
   339, 362, 371, 357, 358, 316, 336, 374, 336, 354, 374, 366, 352,
   374, 339, 336, 354, 338, 348, 366, 370, 333])
>>> stem(data)
 29|8
 30|
 31|067
 32|566789
 33|00122334456666777778899
 34|02234466667788999
 35|001223445555577888999
 36|12233344556666677789
 37|0000122444588
 38|0277899
 39|0
 40|8

If you prefer to have spaces between entries, take out the sep='' from the last line.

>>> stem(data)
 29| 8
 30|
 31| 0 6 7
 32| 5 6 6 7 8 9
 33| 0 0 1 2 2 3 3 4 4 5 6 6 6 6 7 7 7 7 7 8 8 9 9
 34| 0 2 2 3 4 4 6 6 6 6 7 7 8 8 9 9 9
 35| 0 0 1 2 2 3 4 4 5 5 5 5 5 7 7 8 8 8 9 9 9
 36| 1 2 2 3 3 3 4 4 5 5 6 6 6 6 6 7 7 7 8 9
 37| 0 0 0 0 1 2 2 4 4 4 5 8 8
 38| 0 2 7 7 8 9 9
 39| 0
 40| 8

To skip over empty stems, add e!=a and in front of print. This will remove the 300 stem from the output (useful for data with lots of gaps).

>>> stem(data)
 29| 8
 31| 0 6 7
 32| 5 6 6 7 8 9
 33| 0 0 1 2 2 3 3 4 4 5 6 6 6 6 7 7 7 7 7 8 8 9 9
 34| 0 2 2 3 4 4 6 6 6 6 7 7 8 8 9 9 9
 35| 0 0 1 2 2 3 4 4 5 5 5 5 5 7 7 8 8 8 9 9 9
 36| 1 2 2 3 3 3 4 4 5 5 6 6 6 6 6 7 7 7 8 9
 37| 0 0 0 0 1 2 2 4 4 4 5 8 8
 38| 0 2 7 7 8 9 9
 39| 0
 40| 8

Thanks for reading.

@ivanov/status/443981782635921408

disabling blinking

2014-01-29T00:00:00-08:00

Background: Text editing in the IPython Notebook is provided by an excellent JavaScript-based CodeMirror text editor. This might be more detail than you want, but I'm a vision scientist so I hope you'll indulge me.

The cursor is meant to tell the user the current location.

The human visual system has a pre-cortical lag of roughly 50-90 ms (read more about P1).

That's how long it takes from something changing on the screen to cause an avalanche of photons to barrel towards your eyeball, be phototransduced and processed by several stages of cells in the retina, finally causing retinal ganglion cells to fire an action potential down their axons through the optic nerve, make its way to a processing relay station called the LGN, with those cells firing action potential down their axons, with those spikes finally ending up in the primary visual cortex.

By ~150 ms, our brains have processed the visual input enough to perform a non- trivial ammount of object recognition.

The default blink rate for CodeMirror is 530ms.

That's as slow as molasses in January!

Say that I've been distracted and looked away from the screen. When I look back at the scree, half of the time it will take 3 times longer for me to get the information that I want ("where's my cursor") than if that cursor was clearly visible at all times. Now it's not always that bad, because sometimes my gaze will land on the screen and even though the cursor isn't visible, it appears in a few milliseconds, and so it takes just as long as if the cursor was there the whole time. But if I happen to be particularly unlucky (there's a reason I don't gamble), it can take 6 times longer.

Try it out

Here's the bit of JavaScript code you need to disable blinking in CodeMirror.

CodeMirror.defaults.cursorBlinkRate=0

If you type that into the JavaScript console of your webbrowser, that setting will apply to all cells created in the current IPython Notebook. If you don't know how to open your browser's Javascript console, don't frett, just make a new cell with just the following lines in there, execute it, and make a new cell to see how you like it.

%%javascript
CodeMirror.defaults.cursorBlinkRate=0

Make the change stick

IPython has a notion of profiles to allow for different kinds of configurations. If this is news to you, you've probably just been using the default profile and not known it. In the shell, run the ipython profile create command to be sure (don't worry, if you alreay have a profile, this won't overwrite it). Now ipython locate profile will tell you the directory which contains all of the configuration for the default profile.

In [1]:

!ipython profile create

In [2]:

!ipython locate profile

/home/pi/.ipython/profile_default

In [3]:

x = !ipython locate profile

In [4]:

cd $x.s

/home/pi/.ipython/profile_default

In [5]:

ls

db/  history.sqlite  history.sqlite-journal  ipython_config.py  ipython_nbconvert_config.py  ipython_notebook_config.py  log/  pid/  security/  startup/  static/

There's a lot of stuff there, but we just need to add our one line to the end of the file in ``static/custom/custom.js``

In [6]:

cd static/custom/

/home/pi/.ipython/profile_default/static/custom

In [7]:

ls

custom.css  custom.js

In [8]:

!echo "codemirror.defaults.cursorblinkrate=0" >> custom.js

## "I want it all and I want it now!" You say you don't want to save your current notebook and reload it to get the updated CodeMirror settings? You just want all cells in the current notebook to change their behavior? Well, OK, Freddie:

In [9]:

%%javascript
var rate = 0;
// apply setting to  all current CodeMirror instances
IPython.notebook.get_cells().map(
    function(c) {  return c.code_mirror.options.cursorBlinkRate=rate;  }
);

// make sure new CodeMirror instance also use this setting
CodeMirror.defaults.cursorBlinkRate=rate;

I hope you enjoyed this little IPython customization detour. If you want more information about how to get rid of blinking in other programs you use every day, [here is an invaluable resource on that matter](http://www.jurta.org/en/prog/noblink). Remember, blinking in user interfaces is bad, [but blinking in vision is very important](http://www.brower.co.uk/opticians/blinking.html). ** This post was written as an IPython Notebook. You can view it on [nbviewer](http://nbviewer.ipython.org/url/pirsquared.org/blog/notebooks/notebook-blink.ipynb), or [download it](/blog/notebooks/notebook-blinking.ipynb) **

What happens to a talk deferred...

2013-06-17T00:00:00-07:00

I'll be in Austin, TX for SciPy next week, and I'll be giving an updated talk about tools for reproducible research.

What follows is the abstract of a talk that I really wanted to give and submitted to the general track which did not get in. I care deeply about this topic, and I'm hoping to have conversations with others about it. The talk was deferred to a poster, so if you don't bump into me at other times, seek me out during the poster session (mine's poster #15, 10:35 AM - 11:35 AM on June 27th). This post is intended as a pre-conference warm up to that.

Navigating the Scientific Python Communities - the missing guide.

The awful truth: our newcomers have a deluge of options to wade through as they begin their journey. What tools are available, how do I install them, how do I make them work together -- all hard questions facing a budding scipythonista.

On developer-friendly platforms, the popular approach is to just install numpy, scipy, matplotlib, and ipython using package management facilities provided by the operating system. On Mac OS X and Windows, the least painful, bootstrapping approach is to use a Python distribution like Python(X,Y), EPD, Anaconda, or Sage.

Both of these paths obscure a reality which must be stated explicitly: the development of packages is fundamentally decentralized. The scientific python ecosystem consists of a loose but thriving confederation of projects and communities.

The chaos of installation options and lack of centralization around a canonical solution, which on the surface appears to be a point of weakness that is detrimental to community growth, is a symptom of one of its greatest strengths. Namely, what we have is a direct democracy for user-developers. They vote with their feet: filing bug reports, testing pre-release versions, participating in mailing lists, writing and reviewing pull requests, and advertising the tools they use in talks, papers, and conversations at conferences.

I will discuss why this is the case, why this is a good thing, and how we can embrace it. The talk will present the ethos and expectations of an effective newcomer, as well as resources and strategies for incremental progress toward becoming a master scipythonista.

tags-vs-categories in Pelican

2013-05-24T14:55:00-07:00

So, following up on my transition to pelican, here's a little help for navigating a pelican site. If you've ever wondered why sometimes there's a highlighted menu item at the top, and other times there isn't, read on.

Pages

The menu items need not be limited to categories, however. With the popular notmyidea template (which is what I've customized for my journal here), if you create any pages (by putting the file in a content/pages/ directory), then the title of that page will appear in front of all of the categories in the menu. These pages don't show up in RSS feeds, though.

Menuitems

Additionally, you can specify other MENUITEMS in pelicanconf.py, which will go in front of all pages and categories. MENUITEMS is a list of tuples, each tuple should be composed of a link name, and a url. This is how I link to the listing of all of my blog posts:

MENUITEMS = [('all', '/blog/archives.html')]

Summary

So there you have it - now you know the differences between categories, tags, pages, and menu items, as far as Pelican is concerned. For me, the bottom line is that I'll probably stick to just using categories, and will try to use tags sparingly. In my transition from WordPress, I cleaned up and removed a whole bunch of tags (since they were either redundant (the same tags kept appearing together), or were too specific (most tags were only used for one post).

embracing my hypertextuality

2013-05-24T14:41:00-07:00

Well, it's happened again, I've jumped (back) on the static blog engine bandwagon. Early versions of my site were generated literally using #define ,#include, gcc, and a Makefile...). Back then I was transitioning away from using livejournal, and decided to use WordPress so I wouldn't have to roll my own RSS feed generator.

I tolerated WP for a while - and the frequency of my posts was so low, that it wasn't much of an issue.

Except for the security upgrades. I logged into the wp-admin console way more times than I cared to just to press the little "upgrade" box. The reason is that wordpress keeps everything in a database that gets queried every time someone hits the site. I was never comfortable with the fact, because content can be lost in case of database corruption during either an upgrade or a security breech. Also, my content just isn't that dynamic. The WP-cache stuff just seemed overkill, since I don't get that many visitors.

But lately, I've found myself wanting to write more, to post more, but also shying away from it because I hate dealing with the WordPress editor. And I also hate being uncertain about whether any of it will survive the next upgrade, or the next security hole, whichever I happen to stub my toe on first.

And the thing is, I really like to use version control for everything I do. I liked my blog posts to be just text files I can check into version control. I also like typing "make" to generate the blog, and now I get to!

For added fun, I'm hoping that writing my posts in markdown will make it easier to coordinate my gopher presence, since it's pretty close.

For posterity, I'm capturing what the first version of my Pelican-based blog looked like. I did the same thing when I moved to WordPress.

I ran into some confusing things about transitioning to Pelican, so I thought I'd note them here, for the benefit of others.

Unadulturated code blocks
nikola?
Redirects
- also known as: not breaking the web
Enabling table of contents for posts
Categories and tags
That's it for now

Unadulturated code blocks

I like to use indentation as a proxy for the venerable <tt> tag - which uses a monospace font.

If you just want to use an indentation, but do not want the indented text parsed as a programming language, put a :::text at the top of that block. Here's what I mean. Take this Oscar Wilde quote, where I've inserted a line break for drammatic effect, for example:

A gentleman is one who never hurts anyone's feelings
unintentionally.

Now, you see that ugly red box around the apostrophe? Well, That's because all I did was indent the two lines. If I just put a :::text above the quote, indented to the same level,

    :::text
    A gentleman is one who never hurts anyone's feelings
    unintentionally.

the result will render like this.

A gentleman is one who never hurts anyone's feelings
unintentionally.

As the pelican documentation specifies, this is also the way you can also specify the specific programming language you want, so :::python would be one way to not make pygments guess. You can get a list of all supported languages here, just use one of the short names for your language of choice.

And you should really do this, even if you aren't bothered by the red marks, because the code highlighting plugin goes in and tokenizes all of those words and surrounds them in <span> tags. Here's the HTML generated for the first version:

<div class="codehilite"><pre><span class="n">A</span> <span class="n">gentleman</span> <span class="n">is</span> <span class="n">one</span> <span class="n">who</span> <span class="n">never</span> <span class="n">hurts</span> <span class="n">anyone</span><span class="err">&#39;</span><span class="n">s</span> <span class="n">feelings</span>
<span class="n">unintentionally</span><span class="p">.</span>
</pre></div>

and here's the version generated when you add the :::text line at the top:

<div class="codehilite"><pre>A gentleman is one who never hurts anyone&#39;s feelings
unintentionally.
</pre></div>

nikola?

At some point, having a dialogue with myself, I wrote in here "or should I not use pelican and use nicola instead?"

Ok, tried it - nikola takes too long to startup -

nikola --help
real    0m1.202s
user    0m0.876s
sys     0m0.300s

pelican --help
real    0m0.639s
user    0m0.496s
sys 0m0.132s

I'm sure it's a fine static blogging engine - and Damian Avila's already written IPython Notebook converters for it, but it just feels like it tries to do too many things. Constraints are good. I'll stick with Pelican for now. (Though I did use the nikola wordpress import tool to grab the wp-upload images from my WordPress blog)

another set of instructions I consulted: Kevin Deldycke's WordPress to Pelican, which is how I did get my articles out using exitwp which I patched slightly, so files got saved as .md, and preserve other format properties.

Redirects

also known as: not breaking the web

I wanted to preserve rss feeds, and also not break old WordPress style /YYYY/MM/DD urls - the Nikola wp-import script had created a url remapping scheme in a file called url_map.csv.

I don't have that many posts, so I just added them in by hand:

Options +FollowSymLinks
RewriteEngine on
RedirectMatch 301 /blog/feed/ /blog/feeds/all.atom.xml
RedirectMatch 301 /blog/2006/10/18/todd-chritien-greens-choice-voting/ /blog/todd-chritien-greens-choice-voting.html
RedirectMatch 301 /blog/2007/01/04/changelogs-with-dates-gui-goodness/ /blog/changelogs-with-dates-gui-goodness.html
...

Enabling table of contents for posts

If you want to include a table of contents within a post using [TOC], you must enabled the markdown toc processor with a line like this is your pelicanconf.py:

MD_EXTENSIONS =  [ 'toc', 'codehilite','extra']

Categories and tags

Ok, so this was never clear to me in wordpress, either - but what's the difference between a tag and a category? is it the case that a post can only belong to one category, whereas it can have any number of tags?

I think I used categories as tags on wordpress. Looks like all posts on Pelican can have at most one category. Turns out this little aside was long enough to turn into its own post, so if you're interested, pelican tags-vs-categories has got you covered.

That's it for now

Thanks to Preston Holmes (@ptone) for encouraging me to transition away from WordPress, and pointing me to this post by Gabe Weatherhead (@MacDrifter) for how to do that. It should be said that the pelican documentation itself is very good for getting you going. Additionally, I consulted this post by Steve George which has a good description to get you started, and also covers a bunch of little gotchas, and lots of pointers. Also, thanks to Jake Vanderplas (@jakevdp) for his writeup on transitioning to Pelican, which I will consult later for incorporating IPython notebooks into my markdown posts, in the future. This is good enough for now. LTS.

Cycling

2013-05-23T00:00:00-07:00

A page about cycling in the East Bay (and the San Francisco Bay Area in general).

This started off as the sort of resource that I wish I had when visiting a new town: so hi there, out-of-towner, welcome to our corner of the world. Hope you enjoy your stay and find these resources helpful.

Routes
Local Bike Shops
Resources

You need a bike, a water bottle, a jersey or a saddle bag (to store your snack, phone, patch kit). Everything else is optional.

Routes

Though I consider myself an avid cyclist, the reality is that I am only mildly so. I have not been on a ton of rides in the area, and don't know them well at all outside of the East Bay - I have yet to cycle across the Golden Gate Bridge - and the Marin Headlands (an omission I plan to rectify soon). I did mountain bike a bit in Marin when I was a kid, and rode a fair bit around the Peninsula while I was in high school. I got my first roadie the summer before heading off to UC Davis. It was a wonderfully light steel blue Peugeot from the 70s (you can see a picture of it here). I convinced (my then future roommate) Philip to get a matching white one. But it was taken from me my first year in grad school (2006). I hadn't been cycling much since, until I started up again recently. Below, you will find a concise guide to some of my favorite rides in the area. As far as my actual experiences, you can get a flavor of the rides I go on by reading my cycling log.

20-40 km

Here is the overview: If you want a shorter ride with a great view - I recommend this 24 km (15 mi) loop from UC Berkeley to Grizzly Peak. The ascent to Grizzly Peak Rd is steady and manageable. The road does get pretty rough when you turn off of College until you get to Broadway, but you avoid the dangerous Ashby route to Tunnel Road. Recently, I've been doing a similar ride that avoids that rough patch by going down College Ave all the way to Broadway. As an alternative option, instead of going up to Grizzly Peak, you can turn off for a slightly shorter (but really fun) descent that ends at the historic Claremont Hotel.

What's nice about these loops is that you can do them back-to-back for a total of ~50 km. But If you want to do a ride that long with more variety, read on for the longer routes.

50 km and beyond

If you have some time (about twice the time it takes to do the shorter routes above), the Redwood-Pinehurst route will take you through an incredible canyon that's filled with redwoods! You can take the loop in the other direction - Pinehurst-Redwood.

Another local favorite is the "Three Bears" ride, which has some truly scenic views all throughout (62 km, ~1 km gain).

My longest ride was the El Cerrito to Davis 200K

My previous longest ride took me half-way up Mt Diablo - the views from there are breathtaking. Can't wait to go all the way up! (115.5 km, 1825 m gain)

Centuries

~~I haven't yet completed a century.~~ April 2014 - El Cerrito-Davis 200K was my first century! (132 miles)

Here are some local ones that I have my eye on.

Grizzly Peak Century is one that goes through some of the local East Bay routes I described above.

Best of The Bay Century likewise starts off in the East Bay routes I'm most familiar with, and heads south all the way to San Jose, before returning a bit to Fremont BART station.

Foxy's Fall Century is an organized ride based out of Davis, CA, where I was an undergrad.

Davis Double Century is in my sights as well. Maybe next year I'll get on the board of the California Triple Crown.

There are a lot of long rides listed on the Different Spokes San Francisco website. San Francisco Randonneurs run a very regular brevet series.

Routes to avoid

Ohlone Greenway: it's not fun - there are too many stops and unprotected crossings. This is a nice route to take a stroll through, on a bike it's just frustrating.

Local Bike Shops

I want to keep track of reasonable places that rent bicycles in the area (in particularly road bikes), as well as routes you should be sure to checkout (with variations, caveats, and general overview).

Missing Link Bicycle Cooperative - One can't say enough good things about this place, super friendly atmosphere. They're located right downtown Berkeley, they both sell gear and fix bikes, and even have two complete sets of tools and stands you can use to fix up your bike for free!

Blue Heron Bikes - a relatively new shop (opened in 2012), in "Westbrae" right of the Ohlone Greenway, I cycle past it every day on my commute. Soon after they first opened, I chatted with the owner, Rob, and he's a great guy.

Wheels of Justice Cyclery - The wheels of justice grind slowly, but exceedingly fine.

Resources

Bike East Bay - a very active local group (previously went by the name East Bay Bike Coalition).

Albany Strollers and Rollers - Albany, California, is a little 1.5 square mile city just north of Berkeley which I call home.

Berkeley Critical Mass - 2nd Friday of each month - departs at 6pm from Downtown Berkeley BART station.

East Bay Bike Party - a monthly evening ride.

Just Ride - my review of Grant Petersen's book.

Here's some ASCII art I made that I now use as my email signature:

                   _
                  / \
                A*   \^   -
             ,./   _.`\\ / \
            / ,--.S    \/   \
           /  `"~,_     \    \
     __o           ?
   _ \<,_         /:\
--(_)/-(_)----.../ | \
--------------.......J

Just Ride

2013-05-17T00:00:00-07:00

This is a pretty good book - though it's more of a set of short little blog posts combined into a book. It's nice to read the very opinionated thoughts of someone who's been dealing with bicycles for a very long time.

The whole time I was reading this, I felt like this is the type of stuff that my pal Jon G would have said, and in a similar style to the way he would have said it. I look forward to reading Jon's treatises on cycling in the future.

Reading this book made me proud to and convinced me to continue to never wear clipless pedals and cycling shoes (I have toe clips), and having a kickstand on my roadie. I'm an unracer - and to quote the title of one of the little chapters: "Racing ruins the breed".

I also learned how to corner and turn properly - which has been invaluable in my windy descents lately. Finally, I was reminded to not just count miles, because every ride counts ("No ride too short"). Also it makes as much sense to count time spent on the bike, and the amount of elevation gain, and the number of days biked, period. Thanks to my commute, this means my days biked per week stays above 5 during the most of the year in California.

I also learned a new word: Beausage (byoo-sidj). My current bike doesn't have any, but my last laptop certainly does.

It also felt kind of cool that the author is local. Grant Petersen founded and still runs Rivendell Bicycle Works in Walnut Creek, CA.

Cycling log

2013-04-17T00:00:00-07:00

Why do I keep a log? Read here. This log is terse, if you want longer descriptions of the rides, read my advice for cycling in the East Bay.

2018-06-17 Bay Trail to Mountain View I <3 the Bay Trail - especially the gravel portions which Jon had introduced me to almost two years ago (2016-08-07). Had a coffee and waffle at Blue Bottle in Jack London Square near the beginning, and took an eggs benedict breakfast in Newark before crossing via Dumbarton Bridge.

88.4 km +127 m

2018-06-09 Treasure Bica loop - I've sort of lost my cadance of logging these rides. I guess the ones I do alone all end up blending together, unless they're particularly long.

60.6 km +771 m

2018-05-19 Fort Bragg 600k - Wow. What an adventure. We started at 6am and I made it to Fort Bragg (mile ~180) at 10pm after riding along the coast to beautiful fog filtered sunset. Stayed there for an hour, then finally headed back to a campground at mile 225, because Kaley was leaving (otherwise I don't know how long I would have stayed there). Again, couldn't keep up with her the whole time, but didn't fall too far behind. At the camp, I slept in a chair in front of a fire pit from 3:15am to about 5:15, when I woke up to fresh eggs, bacon, and coffee being made for riders faster than me who were getting ready to head out. Left the campground at 6:15am, and needed to be at Pt Reyes Station by 6pm to make time cut off. Rode with Juliayn and Tom for a while - from Cloverdale to Healdsburg. Then took a longer break as they continued on and hooked up with Pierre - having a pleasant chat and good company almost all the way back to Freestone, where Pierre got a flat and insisted I keep going on my own. After figuring out that I wasn't as tired as much as I was cold - especially through the headwinds at Keys Creek - I proceeded to layer back up. I got to Point Reyes Station at 4:23pm (plenty ahead of the 6 something cutoff time), and then proceeded to ride more leisurely, taking a nap in a bus stop booth on Sir Francis Drake, getting tea and a morning bun at a cafe in Fairfax, and finally finishing at 8:20pm at Chrissy Field back in SF. Pierre ended up catching up right at the end of the last climb out of Sausalito and we rode in together. So that's 38 hours and 19 minutes elapsed time for this 600k. You can read more about this annual ride from Rob Hawks.

600 km +5900 m 38 hours 19 minutes

Hopland 400k - the beer and hard cider in Hopland seemed like a good idea at the time... Rode a big chunk of this with Phillip again - but had to let him go shortly after Petaluma safeway - I was too slow and nauseous after throwing up.

400 km +3921 m 24 hours 10 minutes

2018-05-07 Treasure Bica loop - The building that was in the median at the stop sign as you enter treasure island is gone. Saw a coyote near the top of South Park Dr.

60.6 km +771 m

2018-04-14 Treasure Bica + Wildcat 72.5 km +979 m 4 hours

2018-04-11 - Hawk Hill and Sausalito. Mario and Shawn joined me to brave the rain.

2018-03-30 A local 100k Treasure island, Bica, Papa Bear, Orinda. I should make something like this into a RUSA perm and ride it for a P-12

97.2 km, 1846m gain

2018-03-10 Healdsburg 300k - Yet another wonderful day on the bike. I stayed at my brother's in the city to make the 6am start, but as a result my breakfast was a banana just before the store. So after I got to the first control quickly, 9:10 - I decided to have a nice breakfast and stayed till about 9:45. Rode with Eri[ck]a, Matt and Mike for a while to the second control. Caught up with Jack in Samuel P Taylor Park and paired up for a bit. He let me go on the last climb on Sir Francis Drake.

My legs felt good - I was able to power through Marin .

Here's Brian Chun's video of the ride.

300.2 km +2549 m 15 hours 34 min

2018-03-04 Treasure Bica loop - The building that was in the median at the stop sign as you enter treasure island is gone. Saw a coyote near the top of South Park Dr.

59.2 km +758 m

2018-02-26 - Mt Hamilton 200k - amazing ride - we rode through the snow in the Bay Area (the peak had a good bunch of it, including some on the road) - lucked out into a paceline through most of Mines Rd to beat the wind. Rode a chunk of this with Phillip. Saw a fox riding through the dark on Pinehurst.

see Eric Walstad's photos, or Greg Merrit's video

201.9 km +2869 m

2018-02-17 - Treasure Bica, with more Bay Trail - got rained out so decided to skip the tunnel road climb. 55.4 km +400 m

2018-02-10 - Treasure Bica loop - 59.2 km +758 m

2018-02-03 - Treasure Bica loop - fell off the wagon there in keeping these updated.. missed a few rides - Sara and Kayvon, and Greg, too. 59.2 km +758 m

2017-08-12 Another coffee run - took a more creative descent. [48km, 1000m gain, 3 hours]

2017-08-05 Coffee run - the descent on Tunnel Road is the worst it's been in many years - way too many potholes and broken asphalt 53km, 1000m gain, 3 hours)

I rode several times between May and July, but somehow neglected to document any of thos rides

2017-05-07 Grizzly Peak Century - Rode to the start to add an extra 20k and 300m of climbing. (140km, 2100m gain)

2017-04-30 Treasure Island - rode with Sara and Kayvon - so good to see them both again.

2017-04-01 Tunnel Road + Orinda dash - A reasonably short ride with Greg

2017-02-25 SF Tri Club in the East Bay - Kilian invited me and it was great ride. I had to take easy since I haven't ridden any significant amount lately. (113km, 1710m gain, 7 hours)

-- 2017-03-22 Direct to Orinda and back - left my water bottle there the weekend prior

-- 2017-03-15 Happy Valley loop

-- 2017-02-25 Pt Richmond

2017-02-18 Pt Richmond

2017-02-11 Pt Richmond

2017-01-29 Ciba and back - a lovely weather ride with Greg. I stuck around for a Hawk Hill loop before heading home. First time at Ciba - great coffee!

2017-01-16 Happy valley loop - saw Gabi hiking in Tilden.

2017-01-13 Friday morning ride - it's been a long time since I did one of these. This time I went alone, fueling up at Bica before heading up the last bit of Broadway.

2017-01-02 Tour the Flats - Greg and I decided to start the year off right by going all the way across the Bay Bridge. It was cold. The flats gods tried to deter our plans, and halted our progress only 1.3 miles into the ride from our rendez-vous point. But Greg fixed it with grace, and we were on our way. Only to get another flat as soon as we made it to Yerba Buena Island. Once he patched that up, we decided to reward ourselves with cinnamon rolls from Ikea for our heroism.

2016-10-23 Happy valley loop - didn't take any snacks with me, was OK.

2016-10-18 Social ride - With Sara and Kayvon.

2016-10-09 Russian River 200k (modified) - Stork and I had a great day on our bikes. On the way back, we decided to get off Highway 1 and instead backtracked our morning rollout (for calmer road conditions, and at the cost of some extra climbing. This was my first ride in 2 months and it showed - I had to go painfully slow from Bodega Bay to Valley Ford, spinning in low gear without being able to provide any power.

2016-08-13 El Cerrito - Davis 200k - a my 5 consecutive monthly 200k. Left El Cerrito at 7am and got to St Helena at noon, which is 45 faster than usual for me - but it was so hot that, despite the extra time cushion, I ended up rolling into Davis at 6pm - which is an hour longer than last time. (213km, 2472m climbing, 11 hours total)

2016-08-12 Friday morning social ride - Danny and Jon, over Grizzly Peak (42 km, 759m gain, 1.5 hours)

2016-08-07 Bay Trail - Oakland to Fremont with Jon - beautiful day, the mixed terrain was very welcome. (90km, 137m gain, 5 hours)

2016-08-05 Friday morning social ride - just Danny and I, took Claremont. (40km, 678m gain, 1.5 hours)

2016-07-23 Happy valley loop - good to be back on the bike, it was a hot one. (57km, 1062m gain, 2.5 hours of saddle time)

2016-07-10 Cloverdale to SF 200k - Stayed up front at the beggining and got a chance to ride with some fast riders through the first two controls, and then from just before Sir Francis Drake climb. (202km, 1694 gain, 8 hours 40 minutes)

2016-07-09 Boonville Lollipop 300k - My first 300k! (310km, 3188m gain, 14 hours 56 minutes)

2016-07-07 Hawk Hill with Janet! We saw whales when we stopped part way down the descent.

2016-07-03 Return to Reliez - haven't ridden on Reliez Valley Rd for over a year, maybe two... Decided to correct that today. (73km, 1463m gain, 3 hours 40 minutes)

2016-07-01 Friday with Jon - had a good conversation about 3D modeling tools, and a possible stealth project... (52km, 860m gain)

2016-06-25 Russian River 200k - what a lovely ride! (205km, 2026m gain, 9 hours 25 minutes)

2016-06-24 Friday with Danny and Jon - good thing I did this ride, because the only water I had at the start of the 200k the next day was a single mostly full bottle from this outing.

2016-06-11 Happy valley loop - Saw a gorgeous Stellar's Jay in the woods, and two young deer on the Wildcat on the way back. (57km, 1062m gain, 2.5 hours of saddle time)

2016-05-22 El Cerrito - Davis 200k - a solo ride, the second in my quest for my first R-12. Took a bunch of photos. (213km, 2472m climbing, 10 hours total)

2016-05-12 Bike to Work day! A sweet leisurely ride from the office to Hawk Hill (my first time there) with Danny and Janet. Danny and I rode the ferry from Oakland, so I had another 13km before the SF portion of the ride. Danny has a few photos on his Strava (56km, 650m gain, 3 hours)

2016-05-08 Bear-y Happy Nice and foggy morning ride, fueled by Doughnut Dolly! (75km, 1174 m gain, 3 hours)

2016-05-01 Grizzly Peak Century - Spontaneously ended up riding with Mark within the first few miles, and after meeting at the first rest stop, we rode most of the 110 mile version of the ride together at a brisk pace. The conversation sure made the miles go by a lot faster. Also bumped into randonneurs Rob Hawks and Barb who volunteered separate rest stops - nice to see familiar faces. (190km, 3101 m gain, ~ 10.5 hours total)

2016-04-30 The ride of two Pauls - I took Paul The Elder on his first ride in the East Bay. It was a beautiful morning, which we started at Semifreddi's. I had to ditch Paul at the top of Happy Valley to head home, but he was a good sport about it. (56km, 1062m gain, 3.5 hours of saddle time)

2016-04-29 Another Friday with Danny - Danny and I took on Grizzly Peak - I returned home the more direct way, opting out of the extra climbing. (33km, 579m gain, 2 hours)

2016-04-16 El Cerrito - Davis 200k - a wonderful day for a ride (213km, 2472m climbing, 9 hours of saddle time, 10 hours total)

2016-04-10 Happy LaMo return via Arlington again - the weather kept me from leaving earlier in the morning in trying for Mt Diablo. Climbed around Albany Hill when I got back just to add a little extra. (57km, 1062m gain, 3 hours of saddle time)

2016-04-03 Longer training ride, the 200k is only two weeks away, so I extended my usual loop by going through Moraga. Pleasant sounds of bubbling creeks on Redwood Road: delightful. (88 km, 1720m gain, 4.5 hours)

2016-04-01 (Anti) Social ride - Danny slept in, so I took on Grizzly Peak by myself. (42 km, 759m gain, 2.5 hours)

2016-03-26 Happy LaMo return via Arlington - trying to get ready for the El Cerrito-Davis 200k (57km, 1062m gain, 3 hours of saddle time)

2016-03-25 Friday with Dmatt - Danny and I decided to take on Grizzly Peak. (42 km, 759m gain, 2.5 hours)

Wildcat Only - wow, not

-- Friday with Dmatt

-- 2016-01-23 A run - ran to El Cerrito BART station, then the North Berkeley one, once around UC Berkeley campus and back. 16.7km, 222m gain

-- 2016-01-15 Tunnel Road to Claremont - Danny was freezing, Jon's back tire had a slow leak, but we still got out there! I wore a jacket and was much warmer. (40km, 678m gain, 1.5 hours)

2016-01-08 Tunnel Road to Claremont - with Jon, I was very cold, was shivering at Fourne for quite a while after the Claremont descent. Jon and I came up with and idea for an electronics project. (40km, 678m gain, 1.5 hours) 2016-01-01 Wildcat Only - wow, not cycling for two months really made me weak... The cold temperature (5-7°C / 41-45° F) didn't help. (31km, 560m gain, 2 hours)

2015-10-25 Happy LaMo return via Arlington - saw one turkey doing a pretty dance for another turkey... in the middle of the road... Turkey's aren't very smart, it turns out. (57km, 1062m gain, 3 hours of saddle time)

2015-10-30 Tunnel Road to Claremont - Jon, Danny, and I. Saw some turkeys, one of whom had beef with us and took it out on the car behind us.

2015-10-25 Happy LaMo return via Arlington (2.5 hours of saddle time)

2015-10-16 Tunnel Road to Claremont - Just Danny and I today.

2015-10-10 Winters 200k - continuing my tradition of forgetting to put water in my bottles before rolling out - rode 'dry' and ended up filling them at the first control (38km into the ride). For the first half of the ride, I somehow ended up in a group with 2 anciens and 2 anciennes (Robert, Barry, Kris, and Renee). Having done only two 200k rides before this, I was actually surprised to keep up with them for as long as I did. We got to the Winters control (93km) at 10:10 - I was aiming for 10 hours which would have only put me there a bit before noon. The cardiac hill climb is where I couldn't keep up anymore, though our paths still crossed at the next control (124km) - but I decided to sit and cool off for a bit as they rolled out from there (and finished exactly an hour before me). I was pretty much alone for the second part of the ride, which is too bad, because the headwind was quite strong and took a lot out of me, but I'm happy with my performance. (Photos from an SFR member) (200km, 2266m gain, 9 hours, 28 minutes)

2015-10-04 Willow Ave Park and Ride Scoped out the Winters 200K start control for next week. (51 km, 460 m gain, 2 hours)

2015-10-02 Tunnel Road to Claremont - All of us again (Sara, Kayvon, Danny and Jon) - met up at Blue Bottle this time. Fournée post-ride stop again! (40km, 678m gain, 1.5 hours)

2015-09-27 Pt Reyes Station loop with Danny and Stork! Danny's longest ride ever - but he paid for it with two flats. I played with staying in the big gear for several of the climbs, and the new-to-me segments of the Cross Marin Trail were delightful. (150km, 8 hours)

2015-09-25 Tunnel Road to Claremont - Our biggest turnout! Sara, Kayvon, Danny and Jon all showed up. Jon got a flat halfway up Tunnel Road. Our first time at Fournee bakery afterwards, yummy! (40km, 678m gain, 1.5 hours)

2015-09-18 Tunnel Road to Centennial - A ride with Jon and "Hawkeye" Danny! (40km, 678m gain, 1.5 hours)

2013-09-14 Lucas Valley Populaire: Danny's first randonneuring event, it was a really nice, overcast day. (113km, 1417m gain, 6 hours)

2015-09-06 Happy Valley First weekend ride with Danny! (56km, 1057m gain, 3 hours)

2015-09-04 Tunnel Road to Claremont - With Danny and Jonathan! Rode up Euclid for good measure afterwards. (40km, 678m gain, 1.5 hours)

2015-09-01 Tunnel Road to Centennial - Sara, Kayvon, and Danny! (37km, 600m gain, 1.5 hours)

2015-08-27 Wildcat and South Park - went for a run with Stéfan afterwards (1.5 hours)

2015-08-21 Tunnel Road to Centennial - Just DMatt and I this time, we split off with Danny taking Centennial down, whereas I stayed on Grizzly and came down Spruce. (37km, 600m gain, 1.5 hours)

2015-08-09 Wildcat Moraga Redwood - a good weekend ride. (80km, 1.5km gain, 3.5 hours saddle time)

2015-08-07 Tunnel Road to Centennial - another Friday morning with Sara and Danny. Danny's first time up to Grizzly Peak. (37km, 600m gain, 2 hours)

2015-08-03 Freestone Bread Run 200K - Awesome ride with SF Randonneurs, - here's an album where I can be seen holding a coffee at the beginning of the ride. (205km, 3km of climbing, 11 hours 45 minutes of saddle time, 3 wild blackberry pickings on the side of the road)

2015-07-31 Tunnel Road to Claremont with Sara and Danny (DMatt's first time!). Sadly, found out that Nobolom is closing after 40 years this Sunday, glad to have gotten a chance to go there one last time.

2015-07-13 Biked to Oakland for coffee and bacon at Sara and Kayvon's place, followed by a ride to Concord for a long lunch with Kenji. Took BART from Pleasant Hill to Rockridge, and then biked home from Rockridge. (3 hour of saddle time)

2015-07-10 Tunnel Road up and down with Sara and Kayvon. We persevered, despite the rain, and it mostly cleared up and dried out by mid-ride. (1 hour of saddle time)

2015-07-04 Happy LaMo return via Arlington (2.5 hours of saddle time)

2015-06-26 Tunnel Road to Claremont - another Friday morning ride with Sara and Kayvon. Ascended on Kavon's single speeder. (1.5 hours of saddle time)

2015-06-14 Happy LaMo return via Arlington (2.5 hours saddle time)

2015-06-12 Tunnel Road to Claremont - joined by Sara and Kayvon again (1.5 hours of saddle time)

2015-06-07 Morgan Territory and Mount Diablo (junction only). A ride with Jeff and his team (Team Roaring Mouse) - it was waaayy too hot - but the company was good. (66.6 miles, 5 hours altogether with waits / regroups)

2015-06-05 Tunnel Road to Claremont - joined by Sara and Kayvon (1.5 hours of saddle time)

2015-05-30 Happy LaMo return via Redwood Wow! First ride on my new Surly Long Haul Trucker - and it rolls like a dream. I was able to push myself the entire ride, thanks to the higher gears, as well as the wide handle bars that allowed for easier breathing during recovery. (4 hours of saddle time)

2015-05-25 Went up Centennial after checking out a Surly Long Haul Trucker at The Missing Link. Was cold at Grizzly peak, only had enough time to go down to Wagner Ranch and come back (1.5 hours of saddle time)

2015-05-23 Happy LaMo return via Redwood. Inaugural ride in my brand new SF Randonneurs jersey! (4 hours of saddle time)

2015-05-10 Happy LaMo return via Kensington (2.5 hours of saddle time)

2015-05-02 Happy LaMo - yay! (2.5 hours of saddle time)

2015-04-18 El Cerrito - Davis 200k with SF Randonneurs. (213 km, 9.5 hours of saddle time, 11 total)

2015-04-05 Happy LaMo return via Redwood. Downhill portion of tunnel road is seriously messed up now. (5 hours of saddle time)

2015-03-29 Wildcat only. Was sick last week, still not fully recovered, but figured I'd get at least a short ride in. (35.5 km, 665 m gain, 1.5 hours of saddle time)

2015-03-15 Happy LaMo return via Pinehurst, then Euclid and return via Kensington. Glad I had the time to push myself a bit, to get back into shape for the El Cerrito 200 km. (4.25 hours of saddle time)

2015-03-08 Happy LaMo return via Kensington (2.5 hours of saddle time)

2015-03-01 Wildcat only, return through Kensington. First ride of the year? Yikes. I did go on a few runs though. Need to make this a regular morning ride before work a few times per week. (1.5 hours of saddle time)

2014-10-18 SF Randoneurs' Lucas Valley Populaire - awesome ride again this year - my time was 5h 35 minutes - and I did take a few nice breaks, including the extravagance of a double espresso and a morning in Fairfax on the tail end of the ride. Seriously considering my first 200K in a few weeks. Been trying to get in shape the last couple of weeks by compensating for a lack of saddle time with running for an hour twice per week.

A few out and back to Wildcat Canyon

2014-09-07 Happy LaMo - Wasn't as hot, and I took some snacks with me, so didn't even need to stop in Orinda as I made my way home. (56km, 1057 m gain, 3 hours of saddle time)

2014-08-31 Happy LaMo - finally working my way back up to a longer ride. It was pretty hot today. (56km, 1057 m gain, 3 hours of saddle time)

2014-08-24 Wildcat and back A good pace,

2014-08-10 Wildcat and back via Arlington First ride in a long time due to back injury that's been slowing me down. The top of Wildcat Canyon is so incredibly smooth now.

2014-06-14 Wildcat and back via Arlington short ride again today.

2014-06-01 Never Again a fun "shortcut" from Inspiration Point ended up costing me an extra hour, and had me off-roading in my poor roadie. I did see 5 bunnies up there while riding through the park, though. (61 km, 1254 m gain, 3 hours of saddle time)

2014-04-27 Reliez LaMo - Got a flat in as soon as I got on Pleasent Hill Rd, had to walk to Hank and Frank's who fixed me up. Again, did this after a 5k run up to campus, but could actually walk down stair the next day (unlike last time) (km, m gain, 3 hours of saddle time)

2014-04-20 Happy LaMo - met up with Lena in Orinda and at Inspiration Point. Did this after a 5k run up to campus. (56km, 1057 m gain, 2 hours 20 minutes of saddle time)

2014-04-12 - Happy LaMoRedwood - longer ride

2014-04-06 Happy LaMo - good to be back on my bike. Beautiful weather today. (56km, 1057 m gain, 2 hours 40 minutes of saddle time)

2014-03-?? - A Bay Trail ride with Jon.

2014-03-09 South Park - got a flat at the beginning of Wildcat, so had to turn back and walk down for an hour and a half all the way to Wheels of Justice where Matt fixed it in 7 minutes, and I was on my way back up. Found out I can't shift into my lowest gear when I got to South Park, so that was a painful ascent.

2014-02-16 to Happy Valley Rd - actually felt like I really pushed hard the whole ride - stayed in my highest gear all along the top of Wildcat Canyon Rd on the way out. Brought just enough snacks to stay filled with energy. (56.7 km, 1291 m gain)

2014-02-12 Brazilian Room and back only had an hour, but that's an hour longer than I rode in the last 3 weeks. (18 km, 339 m gain)

2014-01-19 Wildcat Only same ride again, faster pace this time. (35.5 km, 665 m gain)

2014-01-18 Wildcat Only short ride to get back on that horse. (35.5 km, 665 m gain)

2014-01-10 Grizzly Peak My first ride of the year (and in well over a month), wearing jeans and with my messenger bag on, right after therapy. The best thing for depression is exercise.

2013-12 Bay Trail short ride with Jon and Sofia ()

2013-11 Grizzly Peak with Sara via South Park ()

2013-11 a ride with with Minh ()

2013-09-23 Bay Trail Stroll with an extra 30 kilo in tow. (15 km, 55 m gain)

2013-09-14 Lucas Valley Populaire: Took BART over to Embarcadero, and after an espresso and waffle at Blue Bottle at the Ferry Building with my friend JB, we headed over to Chrissy Field, where JB left me in the good company of Guy, James, Sara, and Kayvon. It was a great day for a nice long ride, we were mostly bringing up the rear the whole time, but I got to take a bunch of pictures. Without a doubt, I'm ready for my first 200k (112.6 km, 1505 m gain)

2013-09-08 Orinda via Happy Valley alone today. It was a hot one, so I took my shirt off and got some of the late-summer sunshine. (55.4 km, 1254 m gain)

2013-09-08 Bay Trail Stroll we skipped the Solano variety, and just took a new trailer to the bay trail (19.8 km, 78 m gain)

2013-09-07 The new bridge and Port of Oakland Rode across the new section of the Bay Bridge with my buddy Jon, then he showed me a couple of parks around the Port of Oakland. (45 km, 75 m gain)

2013-09-06 Moraga-Wildcat Started off as the Mutton-Chop ride with Cal Cycling. Met tri-athlete and optometry student Christie, who said she's thinking of going to Moraga when we got to the intersection of Skyline and Grizzly Peak. She didn't object to me tagging along, and we proceeded at a pretty damn fast pace. By the time we were climbing out back into Tilden, I could barely keep up, but it was a great ride. (55.3 km, 1062 m gain)

2013-09-02 Orinda via Happy Valley same ride as the day before, more people were out because of labor day. Stayed in the highest gear the whole time on Wildcat on my way out. (55.4 km, 1254 m gain)

2013-09-01 Orinda via Happy Valley A cyclist with a Portland jersey let me lead but kept me honest up Wildcat, then he got ahead of me on the downhill - it sucks to not have a reasonable high gear. Didn't visit Granny's house on the way up the Happy Valley hill. (55.4 km, 1254 m gain)

2013-08-31 Orinda via Upper Happy Valley rode with Dan, his first time up Papa Bear. (52.5 km, 1239 m gain)

2013-08-24 Orinda and back - had an iced green atomic sludge thing from Peet's in Orinda. (47.4 km, 966 m gain)

2013-08-20 Grizzly Peak I was running late this morning, Dan waited patiently for me at Bica again, and then we went up to Grizzly Peak and split off our separate ways. (37.2 km, 702 m gain)

2013-08-19 To Bica and Back This hipster cafe was giving away free coffee on account of their third anniversary, so I met up there with Dan, had a good conversation catching up, and then headed home. (17.4 km, 86 m gain)

2013-08-17 Pinehurst - MacArthur Got to do some exploring around upper Rockridge while waiting for Jon, and then we did the Pinehurst - Redwood ride together. (64.4 km, 1171 m gain)

2013-08-14 Grizzly Peak morning ride with Sara. Bumped into Matt Davis during our pre-ride coffee at Actual Cafe (he just moved to the neighborhood) (32.5 km, 722 m gain)

2013-08-11 A ride to Concord where I met up and chatted with my old friend Janet. We chatted for a few hours, and I did not have enough time left to ride all the way home, so I took BART from Concord to Rockridge, and took a ride home from Rockridge BART. (56 km, 1182 m gain)

2013-08-01 Pinehurst - Reliez - Spruce Finally went on a good long ride. Bonked and took a 10 minute nap in the sun at Wagner Ranch elementary before heading up Wildcat Canyon. (96.6 km, 2048 m gain)

2013-07-22 Tunnel - Claremont loop Another morning ride with Sara. (27 km, 583 m gain)

2013-07-21 A ride to Jon and Sarah's new place! I dropped off some freshly made preserves from strawberries we had picked from our farm the day before. Redwood is pretty steep over there. (50 km, 809 m gain)

2013-07-13 Tunnel - Claremont loop First ride with Sara Emery! Neither of us had ridden in a few weeks due to travel and being sick, so we took it easy. Actual Cafe apparently only opens at 8am on the weekend, so we doped at Cole Coffee in Rockridge. (35 km, 591 m gain)

2013-07-03 Marin - Tunnel Rd loop - first ride in over a week due to extensive conferencing and travel. Met David, who has a cool rig with an aerodynamic bubble out front, vertical steering, and an electric regenerative assist. (33.7 km, 765 m gain)

2013-06-22 The Populaire (SF Randoneurs) my first randoneuring event! (116.3 km, 1637 m gain)

2013-06-18 Mt Diablo (half-way) - My first metric century. Ran out of time and decided to not try for the summit this time. Good thing I didn't, it was dark by the time I made it back to Tilden. Took a new basket I got at a big box for $15 in preparation for the Populaire on Saturday - but made the mistake of carrying my U-lock and empty metal sandwich container - tons of rattling the whole time, except for the slow ascents. Mt Diablo's south side road is freshly paved and the views are breathtaking. (115.5 km, 1825 m gain)

2013-06-14 Broadway-Centennial Another morning ride with Dan, another double espresso cubano doping dosage at Actual (pre-ride). Dan found some rope for his boat as we approached campus. (32 km, 671 m gain)

2013-06-13 Spin class with Randy. Climbed some hills, had several streaks around 300 watts and higher, which seemed impossibly high a couple of months ago. Randy told me that the North side of Mt Diablo is steeper, but he prefers it - South side is nicely paved and better for descents. (22 km, 35 minutes)

2013-06-13 Broadway Terrace - Grizzly Peak - First time going up Broadway Ter - it's steeper than Tunnel Rd, and my left calf was very unhappy about that. Also, I Wish I had a higher gear for downhills - another cyclist in a red jersey that I passed during the climb caught up with me on the descent from Grizzly Peak, but pedaling at those speeds and my gearing is useless. We chatted briefly after Centennial, and he was impressed that I wasn't wearing cleats - strapless toe-clips for the win! (24.5 km, 638 m gain)

2013-06-11 Tunnel Rd Only Road again with Dan Coates this morning. We had pre-ride espressos at Actual, and post-ride pastries with coffee at Nabolom, with thick fog sandwiched in between. (36 km, 637 m gain)

2013-06-09 Happy-Reliez Overcast and a bit chilly. New slightly thicker jersey and long sleeved shirt kept me adequately warm. In good spirits, was told I'm "going way too fast" as I passed a cyclist going up Happy Valley road. :) (77.5 km, 2062 m gain)

2013-06-04 Centennial-Happy-Lamorinda Succeeded in doing a standing climb all the way up Happy Valley Rd. (64.5 km, 1604 m gain)

2013-06-01 Mt Diablo Prelude I flirted with Mt Diablo, but got a flat and was stranded in Danville until the cavalry came and picked me up. (50.5 km, 1216 m gain)

2013-05-31 Wildcat Only it was really hot on the other side of the hill. (35.5 km, 665 m gain)

2013-05-30 Morning Claremont Loop - Dan Coates joined me for a ride. I was doping (Espresso Cubano at Actual Cafe), but carried my bag the whole way, as a handicap. (27 km, 583 m gain)

2013-05-28 Spin Class at RSF: speed (30-30 sprints and standing endurance). We got a bonus 15 minutes at the end. (29 km, 60 minutes)

2013-05-28 Redwood-Pinehurst-Tunnel - glad to see dozens of cyclists going up Tunnel Road in pairs as I was coming down. (50 km, 1099 m gain)

2013-05-26 South Park-Papa-Happy-Reliez-Bear Creek-Happy-Orinda-Wildcat Had an extra hour today, made good use of it with some extra climbing. Went through the blocked off section of El Toyonal - lots of horses, and also a terrible road, wouldn't recommend it. (90 km, 2276 m gain)

2013-05-25 Wildcat-Papa Bear-Happy Valley-Lamorinda - First time taking Happy Valley Rd - rough on the ascent, but buttery smooth on the way down. Stopped for sugary coffee in Orinda. I like ridewithgps's interface a lot better, and the elevation gain it recorded was 1.5 km, a full 500m later than MapMyRide - because MMR smooths things over more. I like the interface better (you can zoom in on portion of a ride quicker), so I think I'm going to stick with RWGPS from now on. (72 km, 1550 m gain)

2013-05-23 Tunnel-Claremont Right leg kind of bumming - but surprised at how much strength my legs had at the top of Grizzly Peak - stayed in high gear all along even that last hill. (23 km, 405 m gain)

2013-05-21 Inspiration Point-Wildcat Roundtrip-Grizzly Peak - South Park Drive still kicks my butt, so I'll have to do it more often. (37 km, 690 m gain)

2013-05-17 - Spin class at RSF, got there 10 minutes late, but hey, every ride counts! (16 km, 35 minutes)

2013-05-17 - Tunnel Rd-Claremont-Tunnel Rd-Grizzly Peak Tunnel Rd uphill is so nice, today I rode it twice! On the second go around, on the rolling part of Grizzly Peak Blvd, I hit the Bohemian Rhapsody solo - and left a cycling team in the dust! (48 km, 893 m gain)

2013-05-16 - Redwood-St Mary's-Reliez Valley-Three Bears-Grizzly Peak (93.5km, 1451 m gain)

2013-05-14 - Tunnel-Pinehurst-Redwood-Tunnel A good ride, solid two hours. Watched a raptor taking off right in front of me while cycling up Redwood, beautiful bird. (49.5 km, 822 m gain)

2013-05-11 - Three bears - Papa first Helped one guy find his cycling computer. Saw a wild turkey with a bunch of chicks. Was way to windy on San Pablo Dam Rd as I approached 80, and all along 80. I wouldn't like cycling in the pacific northwest - apparently it's always windy there. (53.5 km, 589 m gain)

2013-05-10 - Claremont-Tunnel Rd. Morning ride - thick fog along Grizzly Peak Blvd and Skyline. Bumped into and chatted with Kirk McKusick while enjoying my cup of coffee after the ride.
(26.5 km, 443 m gain)

2013-05-08 - Centennial-Grizzly Peak-Tunnel Rd Was told "hey bro, call it out!" when I passed another cyclist on Grizzly Peak, which I'm ok with, but he kept yelling something long after I was out of hearing range. My first time going down Tunnel Rd. (22.5 km, 441 m gain)

2013-05-06 - Page Mill-Altamont Loop, from Stanford. After the first day of Software Carpentry at Stanford, I spent an hour in front of the Thai restaurant where Bernard, Ariel, and I had dinner trying to get my bike lock unlocked. Bob suggested I put WD-40 in the lock, and it worked to revive it! (22.5 km, 214 m gain)

2013-05-05 - Page Mill-Altamont Loop Saw three deer on Page Mill. Remarkable how easy this was compared to what a Page Mill ride was like a decade ago for me. (14.4 miles, 702 ft gain)

2013-05-03 - Centennial-Grizzly Peak-Claremont Short ride. Didn't have much time, just enough for a small climb. But every ride counts. (9.4 miles, 1398 ft gain)

2013-05-01 - Redwood-Pinehurst, again. Concentrated on cadence. A fellow cyclist I passed on the Tunnel Road climb stayed not too far behind the whole way up, and caught up near the top. I told him "Thanks for keeping me honest", to which he replied "It was the other way around". He took a left onto Grizzly, as I continued down Skyline. Briefly stopped at the Canyon post office at 4:30pm, as they were closing up. Will have to send postcards from there, sometime. (31 miles, 2772 ft gain)

2013-04-30 - Redwood-Pinehurst Pushed through in higher gears as much as possible, strength training. Took a slightly different route to get to the hills, taking College all the way to Broadway. (31 miles, 2772 ft gain)

2013-04-28 - Wildcat-Three Bears-Wildcat My first time riding Three Bears! I was not super fond of the beginning of the Three Bears route (Alhambra Valley Rd) - it's quite choppy and not as nicely paved as most other places I'm used to riding, but having the road talk to you is quite encouraging and novel, to me. Examples of signage in the asphalt: "Momma" at the beginning of the first big hill, or "feeding zone", or "almost" within the last 100 yards of the "Papa Bear" hill. (38.5 miles, 3051 ft gain)

2013-04-27 - Wildcat-Orinda-Moraga-Pinehurst-Grizzly Peak, with a Rosie finish

Fueled by two cups of coffee (one from Nabolom, one from Local-123), and pastries from Nabolom, Crixa Cakes, and Starter Bakery - I decided to binge on the pain, as well. I particularly loved slicing through Camino Pablo and Moraga Way in high gear. Made it almost all the way up Pinehurst in second lowest gear - only had to switch to the lowest for the last 100 yards, or so. The finish was anything but rosy - a very strong wind from the bay made me wish I had a sail with me. (44 miles, 2782 ft gain)

2013-04-24 - Grizzly Peak loop, same as a week ago, had to wade through the thick low-lying clouds all along the ridge - near zero visibility up there this time. Bumped into Brandon at the gym afterwards. Turns out the VS ride I missed yesterday was just him and Zack, and they went up Spruce and came down Centennial. (15 miles, 1575 ft gain)

2013-04-23 - Claremont Only. Missed the Vision Science ride, and had to get back quickly, so only had time for a very short (but fairly intense climb) ride. Claremont has too much automotive activity this time of day (4-5pm). (6.25 miles, 974 ft gain)

2013-04-21 - Rosie the Riveter to Lake Anza. Amazing how a bar I ate along the way made me not even tired at the end of the ride. The last hill (Canon Drive) was easier than it had been all winter long. Snacks FTW!

Wore boardshorts today, and amazing how many fewer nods I got from fellow cyclists - it's like they don't recognize their own if I don't look the part. It's the cadence, not the clothes that make a cyclist. (20 miles, 1119 ft gain)

2013-04-19 - Longer ride: Redwood-Pinehurst, ran out of steam (completely bonked) at the bottom of the last hill on Pinehurst to get back to the ridge, and had to just slowly waddle my Resolved to never go on rides longer than 15 miles without a snack. (30 miles, 2680 ft gain)

2013-04-17 - Rode the Grizzly Peak loop, from campus to Broadway and then up. Dan Coates would have joined, but had a flat, and turns out the glue in his patch kit was all dry. Brandon Lujan also stopped by, on his bike, wondering why no one responded to his Bay-to-Breakers thread on vs-social. I watched the sun set behind Mount Tam as I rode along the Grizzly Peak ridge, and got back to the gym just at twilight.
(15 miles, 1575 ft gain)

Why keep a log?

This is a log that's inspired by the Davis Wednesday Night Skate Log. I'm in there somewhere at least a couple of times (01/21/04, 09/29/04) and my brother Mike even more frequently. Another point of inspiration is Ned "Bike Nomad" Konz's trip logs and Steven Roberts' "Miles with Maggie" portion of his bicycle adventures.

About a month after I started keeping this log, I came across yet another example of why I like keeping logs (and reading them) - this one is from Rob, the owner of Blue Heron Bikes in Berkeley. Read Rob's October Bike Vacation

I don't keep track of every single time I get on a bicycle - that would be too tedious, since I have a 6 mile round-trip home-campus commute every day. These are just rides I do for fun and exercise. I get no sense of accomplishment keeping track of rides where I'm riding in a manner as to minimize my sweatiness during my trips to and from campus.

Here's some ASCII art I made that I now use as my email signature:

                   _
                  / \
                A*   \^   -
             ,./   _.`\\ / \
            / ,--.S    \/   \
           /  `"~,_     \    \
     __o           ?
   _ \<,_         /:\
--(_)/-(_)----.../ | \
--------------.......J

remembering John Hunter (1968-2012)

2012-09-03T14:51:53-07:00

John Hunter, the author of matplotlib, passed away on August 28th, 2012. He will be dearly missed.

Please donate to the John Hunter Memorial Fund. A giant in our community, John lead by example and gave us all so much. This is one small way we can give back to his family.

what follows are excerpts from my paper journal over the past week:

John Hunter passed away this morning,
Oh my god. John Hunter was incredibly
kind and warm -- I can't believe he bought
me this laptop. I can't believe that such
a wonderful man could be dead -- so suddenly.

    What a tremendous loss.

Dear Merlin,
    John Hunter died yesterday --
getting unstuck. It is entirely appropriate
to be stuck -- to feel it, smell it, taste it
feel it drilling though your head

I keep pacing around the house.
I just need to leave.
    calmly.
My thoughts are with John's
family. & with the folks in Louisiana.

    A rat in a maze --
panicked -- & the water
level rises still. escalates.

    Our loss of John makes me
want to code furiously.

I feel the urge to code furiously,
but only have the capacity to tweet about
it, and lack thereof to censor myself.

I don't believe I'm losing
my mind. I believe I never had
one to begin with.
  I'm not losing my mind. I never
  had one to begin with.
  I'm not losing my mind.
  That would imply I had one
  to begin with.
Left the house without shoes --
waiting for my laptop to charge.

  I want to share this with the
  group -- so I can come to
terms with it myself. forgive myself.

  Connection with a stranger -- can be a form of escape.
but it can help you gain perspective on
your own life. I think it has for
me.
  I am a severely broken
person. Ok. Time to get shoes,
shower, & then go to try &
talk w/ Greg Wilson.

I must be mistaken. Maybe I was.
Who's to say that I wasn't.

The scientific python community lost one
of it's giants this week, and
I lost an important mentor.

Remembering John Hunter (JDH)

    People keep dying. I don't know
how to deal with that -- I feel like
I never really processed dedushka's death --
nor Ken Green's, nor Jessi Debaca's,
nor babushka's

... I lost another mentor. Most of
these people (all?) don't know that they
were mentors to me -- but they were.
I looked up to John -- I secretly
wanted to please him -- but did not
even dare to do so directly (I
have Fernando Perez to thank for getting
my first contribution into Matplotlib.

I remember feeling really bad after my first
sprints at SciPy 2009 -- John actually
knew who I was -- and wanted me
to work on some matplotlib stuff --
  I ended up doing some rote work with
David Warde Farley (and felt kind
of like a third wheel - since dwf
(pronounced "dwoof" - did I get
that right, David?) is more than
capable as a command-line
cowboy (David doesn't know this --
but I learned a great deal from
just sitting next to him and watching
him string together standard unix
tools, pipe-after-pipe -- to clean
up some scipy wiki content, to
try and export it to a new site.
David's also a role model in other ways --
he's very calm and collected (unless
he senses you've pushed the bozo button from
some punditry vending machine - instead of
understanding and engaging with the full complexity of some
social or political issue)

    John was extremely kind and understanding -- 
he wanted to invite & welcome me to
code alongside other matplotlib developers,
but there was no expectation.

  I got to hang out with John the 
most at the PyData conference in
Mountain View in late February. He was
giving a matplotlib talk, and was seeking
feedback on what to talk about --
& how to do it. He's warm -- and always
had this kind smile about him.

    After hanging out with John at PyData --
this email arrived in my inbox

Fernnado knew about the email -- & I was at
a barbecue at his house -- to sort of
celebrate an awesome week of lots of
Python Giants in from out of town.
  Fernando was giddy -- "Have
you checked your email?" -- with a
twinkle in his eye "Let's go have you
check your email right now" -- & he
walked over to turn on his desktop machine.

    On the mailing lists -- I always tried to
emulate John's approach -- helping everyone
even if in the slightest manner.

I was offered commit rights sort
of out of the blue -- after sending
a couple of pull requests.

    We are a community.
We need to remember John, & keep
remembering John -- for many, many,
many years to come.

I feel the urge to reach
out to everyone I know & don't know.
I'm desperate -- i think I've been that
way since I was a little kid -- I
remember having the same feelings when
we were leaving Moscow -- a ten-year old,
talking to friends and my teachers,
many for the very last time.

I am posting this, but it isn't finished.

This is broken, half-finished, confused,
necessarily so -- because there is no way
to mend what we have lost. This
is an expression of my current state --
and if I don't let this out, it will just
keep ricocheting around in my head for years
to come.

    This is a first pass. This is me
grieving. This will never be enough. This
is just a start.

videos of John Hunter's talks:

matplotlib: Lessons from middle age. (Scipy 2012)

Advanced Matplotlib Tutorial (PyData)

SciPy 2009 - Advanced tutorial 3: Advanced topics in matplotlib

(beginning) SampleDoc

Scipy '09 Panel on Visualization tools

Scipy '02 Core Projects update

NIPS Workshop on Machine Learning Open Source Software (MLOSS)

Fernando Perez and John Hunter

Sept 2009: "some fun stories like 'Jeez, you guys have some crazy examples. I am surprised there isn't dolphins swimming around inside a sphere.' So now there is."

pheriday 2: termcasting overview

2012-08-05T01:12:33-07:00

pheriday 2: termcasting overview (2012-08-03) from Paul Ivanov on Vimeo.

paul's habitual errant ramblings (on Fr)idays (2012-08-03)

show notes:
http://pirsquared.org/blog/2012/08/04/termcasting/
gopher://sdf.org/1/users/ivanov/pheridays/2012-08-03 (yes, gopher!)

try to not say "uuuuhhhhmmnn"

BAM/PFA Summer Cinema on Center Street http://bampfa.berkeley.edu/filmseries/summercinema

SciPy 2012 videos up, go check them out! (I have!) http://www.youtube.com/nextdayvideo (removed nextdayvideo internal box url, by request)

Software Carpentry: Record and Playback post http://software-carpentry.org/2012/07/record-and-playback/

termcasting: a review of what's out there.

http://termcast.org and http://alt.org/nethack/ mostly nethack stuff, both just use telnet protocol, only live sessions (though there are "TV" scripts to re-run sets of ttyrec files).

(playterm vs ascii.io vs shelr.tv)

tldr: playterm.org supports ttyrec files, but has the most primitive player. Players on ascii.io and shelr.tv can both seek. shelr.tv can also speed up playback! Downside is both of those have their own recorder programs (though at least shelr leverages script or ttyrec)

http://playterm.org/ - supports ttyrec files, outgoing links for author and related article, comments. - most primitive player (https://github.com/encryptio/jsttyplay/) - Pause only - only terminal sized of 80x24 or 120x35 - supports tags and comments - service only (code for playterm.org does not seem to be available, though jsttyplay is doing the hardest part of actual playback)

http://ascii.io/ - supports non-standard terminal size - player can seek. - aesthetic thumbnail previews - login via github or twitter credentials (for uploads) - code for website available (ruby and javascript) https://github.com/sickill/ascii.io - code for recorder available (python) https://github.com/sickill/ascii.io-cli

http://shelr.tv/ - supports non-standard terminal size - player can seek. - player playback speed can be increased (currently up to 10x of real time) - supports tags, comments and voting up/down on a video - shelr can playback from command line ("shelr play http://shelr.tv/records/4f8f30389660802671000012.json") - code for website available (ruby and javascript) [AGPLv3] https://github.com/shelr/shelr.tv - code for recorder available (ruby) [GPLv3] https://github.com/shelr/shelr

my wanted list for termcasting
should support ttyrec files (upload and download)
live-streaming (like ustream - but for coding)
termcast.org has ttrtail which does just this
quick "encrypt" switch - to keep streaming, but start GPG encrypting the stream as it goes out - so you can still look at it later. This would make it easy to leave the streaming on all the time
a .tty editor that's like a video editor cut out portions [i.e. dead time]

This is a low-bandwidth way of capturing what I'm working on and thinking about. Now, I'm going to try to record everything I do! "ttyrec -e screen -x". I've only done it a couple of times so far while coding, but I find being able to go back and re-view (and review) what I worked on at the end of the day to be really helpful.

I was inspired by Joey Hess' "git-annex coding in haskell" where he reviews and narrates some of the code he wrote, after he wrote it. http://joeyh.name/screencasts/git-annex_coding_in_haskell/

P.S. It's Saturday now. I tried to save some local diskspace by running recordmydesktop using the --on-the-fly-encoding option, and that was a mistake. The audio and video were (un)hilariously desynchronized - the audio ran for 9:48, but the video wanted to be just 7:30. Audacity came to the rescue by allowing me to change the tempo to be 30% faster, which made the syncing better. And then I used avconv to stitch in the faster audio.

tools used: Debian GNU/Linux sid, recordmydesktop, xmonad, fbpanel, screen, chromium, cheese, xcompmgr, audacity, avconv

pheriday 1: software carpentry, digital artifacts, visiting other OSes

2012-07-27T23:26:43-07:00

Here's pheriday 1, another edition of paul's habitual errant ramblings (on Fr)idays

pheriday 1: software carpentry, digital artifacts, visiting other OSes (2012-07-27) from Paul Ivanov on Vimeo.

2012-07-27.mp4 (28 MB) 2012-07-27.avi (71 MB) 2012-07-27.ogv (321 MB)

show notes

I had three topics I wanted to cover today, and ended up spending about an hour thinking about what I was going to say and which resources I was going to include. This was too long, and the end result was still very rambling, but I think I'll get better at this with more practice.

SDF Public Access UNIX System http://sdf.org gopher://sdf.org/1

try to not say "uuuuhhhhmmnn"
be lazy (software carpentry)
flipside of "publisher's block" (git-annex)
visiting another country (windows 8 release preview)

As usual, I didn't know what I was really trying to say in 0, and here's a really good overview of what I meant: Software Carpentry

If you have 90 seconds, watch the pitch

play it in 60 seconds, instead: mplayer -af scaletempo -speed 1.5 Software_Carpentry_in_90_Seconds-AHt3mgViyCs.flv

publisher's block

Jaron Lanier's You Are Not A Gadget What I mention in the video is not at all the main point of Lanier's book (which is quite good!), and in fact, his book is a critique of (over) digitization. Nevertheless, I'm only pointing out that there are redeemable aspects of an increasingly digital artifact producing life, such as preservation.

David Weinberger’s Everything is Miscellaneous

My review of David Weinberger’s Everything is Miscellaneous, where I go into more depth about "information overload".

git-annex Excellent project. The technical details is that when you "annex" files, they are renamed to long hash of their contents (bit rot resistant!) and stored in a .git/annex/objects directory, whereas in place of where the file was, you get a symlink to the original file, which gets added to git. So git only keeps track of symlinks, and additionally has a git-annex branch that keeps track of all known annexes, so that you can copy, move, and drop files from the ones that are accessible. Very handy!

Haiku OS

tools used: Debian GNU/Linux sid, recordmydesktop, xmonad, fbpanel, screen, iceweasel, cheese, xcompmgr, youtube-dl, mplayer, screen

gopher version of this post (proxy)

pheriday 0: scientist-hacker howto (video post)

2012-07-20T23:55:41-07:00

Hey everyone, here's pheriday 0, the first of paul's habitual errant ramblings (on Fr)idays

pheriday 0: scientist-hacker howto (2012-07-20) from Paul Ivanov on Vimeo.

Berkeley Kite Festival (510 Families)

Merlin Mann's Most Days (specifically the travel day one on 2009-01-11)

Sad that I missed SciPy Conference this year. One of the things I like doing at scipy is nerding it up with my friends, seeing each others workflows, showing off vim tricks, etc. This video was my attempt at scratching that itch, a little bit. As I mention in the video, this is take 2. Take 1 ended when I ran out disk space, but needless to say, it was more awesome than this. It seems I am cursed with losing first takes, see also a summary of last year's SciPy conference, where this exact same thing happened.

NumFOCUS: NumPy Foundation for Open Code for Usable Science

NumFOCUS Google Group see thread titled: "[Funding] Notes from Funding BOF at SciPy2012"

TLDP: The Linux Documentation Project (page I was scrolling through)

Transition to Gopher was rough this time, it was better during the first take.

Lorance Stinson's w3m (better) gopher support Use this if, for example, going to w3m gopher://sdf.org you get errors like:

[unsupported] '/1' doesn't exist! [unsupported] This resource cannot be located.

It still took some tweaking, shoot me an email for details

Robert Bigelow's About | Gopher & GopherSpace

Here's the HTTP Proxied version of the above: Gopher proxy provided by Floodgap

SDF Public Access UNIX System http://sdf.org gopher://sdf.org/1

Eric S. Raymond's How To Become a Hacker Howto

Fernando Perez' Py4Science Starter Kit

Q: Why are you using "Chromium --incognito"? I have chronic tabitis, and this is one way of mitigating that problem. If the browser crashes or I shutdown my computer, I won't have those tabs around anymore.

programs used: Debian GNU/Linux sid, recordmydesktop, xmonad, fbpanel, screen, chromium, cheese, xcompmgr, mutt, wyrd, tail, w3m

gopher version of this post (proxy)

Ada Lovelace Day: remembering Shirley Theis and Evelyn Silvia

2011-10-07T17:01:29-07:00

In case you didn't know it - today is Ada Lovelace Day!

Now, as any self-respecting Computer Science degree-wielding person should, I, too, think it's important to celebrate the day named after the world's very first programmer.

For me, the first math teacher I remember making a big difference was Shirley Theis - who taught me Algebra in 8th grade at McKinley Middle School in Redwood City, CA. Mrs Theis, an energetic dynamo in her mid fifties, was a deeply motivated and caring teacher, who expected a lot out of her students, but never in a disciplinary manner. She was full of enthusiasm, which projected out and infected even the most timid or disaffected student: in her class, you couldn't be just a sack of potatoes planted in your seat.

She often lead class in a nearly theatrical manner - pacing back and forth, egging students on by eagerly repeating their partial responses, getting exponentially more excited if the student was on the right track, barely containing herself from jumping up and down in anticipation of that lightbulb going off -- and yet just as quickly waning in her enthusiasm,becoming a personified caricature of hopelessness and despair to let you know the instant a response was starting to go astray.

It may have been the only math class I've ever taken where there were group assignments - we would work with a partner or a few classmates in trying to figure out an assignment, first trying it solo, and then putting our heads together to figure out why our answers disagree and which is the right one. I believe it was Mrs. Theis who succinctly captured a value I hold in high regard: "it's not about how far you go - it's about how many people you bring with you."

There was one other mathematics teacher I had in my life who clearly stands out: it was Professor Evelyn Silvia who had a comparable level of enthusiasm and energy, and from whom I had the pleasure of taking the first upper-division math course (Math 108 - Intro to Abstract Math) during my second quarter at UC Davis. Dr. Silvia was the real deal - she cared, gesticulated, encouraged us to question why something was true, and had an approach which demanded we each take ownership of our education. The book for the course, Introduction to Abstract Mathematics: A Working Excursion by D.O. Cutler and E.M. Silvia was a blue workbook - each of us had our own copy, and there were blanks left out for us to write our own answers to the exercises. The fact that the book had blanks for me to fill in was so inviting, there was a kind of "working mathematician" approach that came with it with that it made me really enjoy and look forward to working through the material. I still have mine.

Dr. Silvia was incredibly sharp, not just intellectually but also interpersonally. Not only could she gauge when the class was lost, but she also had a knack for spotting if something was affecting you outside of class. She was really committed to helping you not just as a student, but as a person. I remember spending hours at Mishka's, or Cafe Roma, or the CoHo, reading and writing, wanting to do well and not let Silvia down, because she invested so much energy and placed a great deal of trust in us.

So thank you both, Shirley Theis and Evelyn Silvia - you both encouraged me to grow a lot as a person, challenged my concept of what it means to be a student, and by your example provided a template of what it means to be an effective teacher, which I've imitated and embraced with pleasure in my own teaching.

(tagged scipy to spread word of Ada Lovelace day to Planet SciPy)

vim-ipython two-way integration! (updated: 2011-08-02)

2011-07-28T23:28:10-07:00

I'm very pleased to share with you a demo the forthcoming vim-ipython integration which will work with IPython 0.11(trunk).

You can either use the Flash player below, or download the OggVorbis file (14MB) update: vim-ipython 'shell' demo (9.6MB). The blog-free form of this post is here.

If you like what you see and want to try it, you can get the details from the vim-ipython github page ~~and it currently requires 4 line changes to IPython, which are currently in this pull request.~~ (Fixed to work on IPython trunk with no changes).

Big thanks to Min for walking me through the new IPython kernel manager during the SciPy2011 sprints.

Get the Flash Player to see this video, or download it in OggVorbis format

UPDATE: 2011-08-02

vim-ipython ‘shell’ mode.

Get the Flash Player to see this video, or download it in OggVorbis format

Just in case, here are the same videos as above, but hosted on Youtube:

If you're have any issues, try searching for your error on the vim-ipython github issues page, and if you don't find it, please file a new one, and I'll help you out there.

Money and CA Propositions

2010-06-07T17:28:14-07:00

Since tomorrow we'll be having another one of those practice democracy drills here in California, I thought I'd put together a few bar charts.

There are five propositions on tomorrow's ballot. In researching them, Lena came across the Cal-Access Campaign Finance Activity: Propositions & Ballot Measures.

Unfortunately, for each proposition, you have to click through each committee to get the details for the amount of money they've raised and spent. Here's a run-down in visual form, the only data manipulation I did was round to the nearest dollar. Note: no committees formed to support or oppose Proposition 13.

Here's how much money was raised, by proposition:

Just in case you didn't get the full picture, here is the same data plotted on a common scale:

And the same two plots for money spent ((I don't fully understand what these numbers mean, as some groups' "Total Expenditures" exceed their "Total Contributions" and still had positive "Ending Cash")):

It could just be my perception of things, but I get pretty suspicious when there's a ton of money involved in politics, especially when it's this lopsided.

The only thing I have to add is you should Vote "YES" on Prop 15, because Larry Lessig says so, and so do the Alameda County Greens!

Update #1: Let me write it out in text, so that the search engines have an easier time finding this. According to the official record from Cal-Access (Secretary of State), as of May 22nd, 2010, there were $54.4 million spent in support of various propositions, most notably $40.5 million on Prop 16, $8.9 million on Prop 17, and $4.6 million on Prop 14. Compare that with a "grand" total of less than $1.2 million spent to oppose them, with a trivial $78 thousand (!!) to oppose Prop 16's $40.5 million deep pockets.

Update #2: The California Voter Foundation included more recent totals (they don't seem to be that different), as well as a listing of the top 5 donors for each side of a proposition in their Online Voter Guide.

Also, here's the python code used to generate these plots (enable javascript to get syntax highlighting):

# Create contributions and expenditures bar charts of committees supporting and
# opposing various propositions on the California Ballot for June 8th, 2010
# created by Paul Ivanov (http://pirsquared.org)

# figure(0) - Contributions by Proposition (as subplots)
# figure(1) - Expenditures by Proposition (as subplots)
# figure(2) - Contributions on a common scale
# figure(3) - Expenditures on a common scale

import numpy as np
from matplotlib import pyplot as plt
import locale

# This part was done by hand by collecting data from CalAccess:
# http://cal-access.sos.ca.gov/Campaign/Measures/
prop = np.array([
     4650694.66, 4623830.07    # Yes on 14 Contributions, Expenditures
    , 216050, 52796.71         # No  on 14 Contributions, Expenditures
    , 118807.45, 264136.30     # Yes on 15 Contributions, Expenditures
    , 200750.01, 86822.79      # No  on 15 Contributions, Expenditures
    , 40706258.17, 40582036.58 # Yes on 16 Contributions, Expenditures
    , 83187.29, 78063.91       # No  on 16 Contributions, Expenditures
    , 10328675.12, 8932786.06  # Yes on 17 Contributions, Expenditures
    , 1229783.79, 965218.48    # No  on 17 Contributions, Expenditures
    ])
prop.shape = -1,2,2

def currency(x, pos):
    """The two args are the value and tick position"""
    if x==0:
        return "$0"
    if x < 1e3:
        return '$%f' % (x)
    elif x< 1e6:
        return '$%1.0fK' % (x*1e-3)
    return '$%1.0fM' % (x*1e-6)

from matplotlib.ticker import FuncFormatter
formatter = FuncFormatter(currency)

yes,no = range(2)
c = [(1.,.5,0),'blue']  # color for yes/no stance
a = [.6,.5]             # alpha for yes/no stance
t = ['Yes','No ']       # text  for yes/no stance

raised,spent = range(2)
title = ["Raised for", "Spent on" ] # reuse code by injecting title specifics
field = ['Contributions', 'Expenditures']

footer ="""
Data from CalAccess: http://cal-access.sos.ca.gov/Campaign/Measures/
'Total %s 1/1/2010-05/22/2010' field extracted for every committee
and summed by position ('Support' or 'Oppose').  No committees formed to
support or oppose Proposition 13. cc-by Paul Ivanov (http://pirsquared.org).
""" # will inject field[col] in all plots

color = np.array((.9,.9,.34))*.9 # spine/ticklabel color
plt.rcParams['savefig.dpi'] = 100

def fixup_subplot(ax,color):
    """ Tufte-fy the axis labels - use different color than data"""
    spines = ax.spines.values()
    # liberate the data! hide right and top spines
    [s.set_visible(False) for s in spines[:2]]
    ax.yaxis.tick_left() # don't tick on the right

    # there's gotta be a better way to set all of these colors, but I don't
    # know that way, I only know the hard way
    [s.set_color(color) for s in spines]
    [s.set_color(color) for s in ax.yaxis.get_ticklines()]
    [s.set_visible(False) for s in ax.xaxis.get_ticklines()]
    [(s.set_color(color),s.set_size(8)) for s in ax.xaxis.get_ticklabels()]
    [(s.set_color(color),s.set_size(8)) for s in ax.yaxis.get_ticklabels()]
    ax.yaxis.grid(which='major',linestyle='-',color=color,alpha=.3)

# for subplot spacing, I fiddle around using the f.subplot_tool(), then get
# this dict by doing something like:
#    f = plt.gcf()
#    adjust_dict= f.subplotpars.__dict__.copy()
#    del(adjust_dict['validate'])
#    f.subplots_adjust(**adjust_dict)

adjust_dict = {'bottom': 0.12129189716889031, 'hspace': 0.646815834767644,
 'left': 0.13732508948909858, 'right': 0.92971038073543777,
 'top': 0.91082616179001742, 'wspace': 0.084337349397590383}

for col in [raised, spent]: #column to plot - money spent or money raised
    # subplots for each proposition (Fig 0 and Fig 1)
    f = plt.figure(col); f.clf(); f.dpi=100;
    for i in range(len(prop)):
        ax = plt.subplot(len(prop),1, i+1)
        ax.clear()
        p = i+14    #prop number
        for stance in [yes,no]:
            plt.bar(stance, prop[i,stance,col], color=c[stance], linewidth=0,
                    align='center', width=.1, alpha=a[stance])
            lbl = locale.currency(round(prop[i,stance,col]), symbol=True, grouping=True)
            lbl = lbl[:-3] # drop the cents, since we've rounded
            ax.text(stance, prop[i,stance,col], lbl , ha='center', size=8)

        ax.set_xlim(-.3,1.3)
        ax.xaxis.set_ticks([0,1])
        ax.xaxis.set_ticklabels(["Yes on %d"%p, "No on %d"%p])

        # put a big (but faded) "Proposition X" in the center of this subplot
        common=dict(alpha=.1, color='k', ha='center', va='center', transform = ax.transAxes)
        ax.text(0.5, .9,"Proposition", size=8, weight=600, **common)
        ax.text(0.5, .50,"%d"%p, size=50, weight=300, **common)

        ax.yaxis.set_major_formatter(formatter) # plugin our currency labeler
        ax.yaxis.get_major_locator()._nbins=5 # put fewer tickmarks/labels

        fixup_subplot(ax,color)

    adjust_dict.update(left=0.13732508948909858,right=0.92971038073543777)
    f.subplots_adjust( **adjust_dict)

    # Figure title, subtitle
    extra_args = dict(family='serif', ha='center', va='top', transform=f.transFigure)
    f.text(.5,.99,"Money %s CA Propositions"%title[col], size=12, **extra_args)
    f.text(.5,.96,"June 8th, 2010 Primary", size=9, **extra_args)

    #footer
    extra_args.update(va='bottom', size=6,ma='left')
    f.text(.5,0.0,footer%field[col], **extra_args)

    f.set_figheight(6.); f.set_figwidth(3.6); f.canvas.draw()
    f.savefig('CA-Props-June8th2010-%s-Subplots.png'%field[col])

    # all props on one figure (Fig 2 and Fig 3)
    f = plt.figure(col+2); f.clf()
    adjust_dict.update(left= 0.06,right=.96)
    f.subplots_adjust( **adjust_dict)
    f.set_figheight(6.)
    f.set_figwidth(7.6)

    extra_args = dict(family='serif', ha='center', va='top', transform=f.transFigure)
    f.text(.5,.99,"Money %s CA Propositions"%title[col], size=12, **extra_args)
    f.text(.5,.96,"June 8th, 2010 Primary", size=9, **extra_args)

    extra_args.update(ha='left', va='bottom', size=6,ma='left')
    f.text(adjust_dict['left'],0.0,footer%field[col], **extra_args)

    ax = plt.subplot(111)
    for stance in [yes,no]:
        abscissa=np.arange(0+stance*.30,4,1)
        lbl = locale.currency(round(prop[:,stance,col].sum()),True,True)
        lbl = lbl[:-3] # drop the cents, since we've rounded
        lbl = t[stance]+" Total"+ lbl.rjust(12)
        plt.bar(abscissa,prop[:,stance,col], width=.1, color=c[stance],
                alpha=a[stance],align='center',linewidth=0, label=lbl)
        for i in range(len(prop)):
            lbl = locale.currency(round(prop[i,stance,col]), symbol=True, grouping=True)
            lbl = lbl[:-3] # drop the cents, since we've rounded
            ax.text(abscissa[i], prop[i,stance,col], lbl , ha='center',
                    size=8,rotation=00)

    ax.set_xlim(xmin=-.3)
    ax.xaxis.set_ticks(np.arange(.15,4,1))
    ax.xaxis.set_ticklabels(["Proposition %d"%(i+14) for i in range(4)])
    fixup_subplot(ax,color)

    # plt.legend(prop=dict(family='monospace',size=9)) # this makes legend tied
    # to the subplot, tie it to the figure, instead
    handles, labels = ax.get_legend_handles_labels()
    l = plt.figlegend(handles, labels,loc='lower right',prop=dict(family='monospace',size=9))
    l.get_frame().set_visible(False)
    ax.yaxis.set_major_formatter(formatter) # plugin our currency labeler
    f.canvas.draw()
    f.savefig('CA-Props-June8th2010-%s.png'%field[col])

plt.show()

Immigration in the US, contextualized (with pictures)

2010-05-29T03:07:19-07:00

So I probably don't need to tell you this since you already know, but

Arizona sucks!

It turns out that even documented immigrants agree, and I have the graphs to prove it!

You see, it all started when I took a great Visualization course this past term which was taught by Maneesh Agrawala. Maneesh gave enough structure for the assignments, but also left some aspect of each open ended. For example, our first assignment had a fixed dataset which everyone had to make a static visualization of, but the means by which we did that was entirely up to us. A lot of people used Excel (in graduate level CS class? gross!), some people wrote little programs (I wrote mine in python using matplotlib and numpy, and did some cool stuff that I will have to post about another time and contribute back to matplotlib), there was even a poor sap who did it all in Photoshop, as I recall, but anything was fair game. Turns out we could even just draw or make something by hand and turn it in!

The second assignment, the source of my graphs which quantitatively demonstrate the suckiness of Arizona, required us to use interactive visualization software to iteratively develop a visualization by first asking a question, then making a visualization to address this question, and going back several times refine the question and make successive visualizations.

On thing to keep in mind is that, overall, naturalized citizens are both an exclusive and a discerning lot. In most cases, you have to be a permanent resident (have a Green card) for 5 years before you can apply. And there are quotas for how many people can get a Green card every year, so there are lots of hoops to jump through. Given the amount of effort involved, wouldn't it be nice to look at a breakdown of naturalized citizens by state? Because that would give us an idea about which states immigrants percieve as, for lack of a better word, "awesome", or if you're not so generous, "least sucky". I bet you'll feel that this second description is more appropriate once you take a look at the data, but keep my "least sucky" premise in mind as you read my original write-up which focused on a different angle (but from which we can still draw some reasonable conclusions). I'll return to make a few more comments about the title of this post after the copy-pasted portion.

here's my original write-up:

begin cut --->

There are three kinds of lies: lies, damned lies, and statistics.

As an immigrant, I've always had the subjective feeling that about half of the people I'm acquainted with are either themselves immigrants, or the children of immigrants. The US prides itself in being a melting pot, a country built by immigrants, so I wanted to dive into the data that would help me understand just how large of a role immigration plays in terms of the entire country. The question I started with, for the purpose of this assignment is this:

What's the relationship between naturalizations and births in the US?

But what I really wanted was to find out was what kind of question do I need to ask to get the answer that would be consistent with my world view. :)

To do this, I started with the DHS 2008 Yearbook of Immigration Statistics, which was linked from the class website.

The file I started with was natzsuptable1d.xls, which required cleanup before I could read it into Tableau. Turns out that even though "importing" to tableau format is supposed to speed things up, it seems very fragile and would regularly fail when I tried converting type to Number (there were some non-numeric codes, like 'D' for 'Data withheld to limit disclosure). *NOT* importing to Tableua's desired format also had the added benefit of allowing me to change the .xls files externally, and having all the adjustments made in Tableau, without having to re-import the data source.

Frustratingly, the last column and last row kept not getting loaded in Tableau! I also ran into an issue which I think had to do with the 'Unknown' country of origin and 'Unknown' state of naturalization which made the totals funky. It took a while to figure out, but there was a problem with Korea, because there was a superscript 1 by it, indicating that data from North and South Korea were combined.

I was trying to use the freshest data possible, so I used the CDC's National Vital Statistics System report titled Births: Preliminary Data for 2007. I just had to copy paste the desired data, and massage it to fit the proper order columns in the excel table I already had handy. I put zeros for U.S. Armed Services Posts and similar territories which is probably not accurate, but this data was not available in the reports that I found. Interesting factoid: according to NVSS (CDC), in 2007 there were more people born in NYC than the rest of the state combined. (about 129K vs 126.5K). The only caveat with this data is that it contains only 98.7% of the data. The states with some missing portion of their data tabulated are Michigan (at 80.2% completeness), Georgia (86.4%), Louisiana (91.4%), Texas (99.4%), Alaska (99.7%), Nevada (99.7%), Delaware (99.9%). Thus, state-level analysis for MI, GA, and LA may be distorted.

The data I had from DHS is for Fiscal Year 2008, which, as it turns out, goes from October 1st, 2007 - Sept 30th, 2008. Thus, no matter which combination of NVSS and DHS datasets I used, there would necessarily be a mismatch in the date range covered by each, so I settled with describing my visualization as "using the latest available data", noting the actual dates for each dataset in the captions. Also, the NVSS report contained a graph of births over time, which fluctuates very modestly from year-to-year, thus the visualization would not change qualitatively if I had 2008 birth data on hand.

I was having a really hard time trying to get a look at the data I wanted to see in one sheet, and ended up trying to make a dashboard that combined several sheets. I couldn't figure out a good way to link the different states across datasets. I struggled for quite a while to pull out the data that I wanted to look at, and ended up having to copy past everything from DHS and NVSS (transposed) onto a new sheet in Gnumeric.

Here's the result:

[caption id="" align="alignnone" width="744" caption="Initial visualization"][/caption]

So, in all of the US, about 1 in 5 new american citizens is an immigrant, or for every four births, we have one naturalization. That was kind of unsatisfying. I've lived in California the entire time I've been in the US, and I feel that at least California is more diverse than that. There's all those states in the middle of the country that few people from the rest of the world would want to immigrate to, yet the people living in them are still having babies, throwing off the numbers which would otherwise support my subjective world view...

So I decided to look at the breakdown by state.

Broken down by state, what's the relationship between naturalizations and births in the US?

[caption id="" align="alignnone" width="1226" caption="my second iteration"][/caption]

I added the reference lines so that you could both read off the approximate total easier, and be able to do proportion calculations visually, instead of mentally. This started looking promising, as I've only lived in California, and it looks like it's got quite a lot of immigrants as a portion of total new citizens.

It was still kind of hard to see the totals, so I decide to create my very first calculated field - which would had the very simple formula [Births in 2007]+[Total Naturalized]. Using this new field, I could now make a map, to see the growth broken down geographically. This was just a way of reaffirming my earlier bias against the middle states having babies without attracting a sufficient number of immigrants to conform to my world view.

[caption id="" align="alignnone" width="1072" caption="gratuitous map (was too easy to do using the software)"][/caption]

In the breakdown by state bar graph, it was also difficult to visually compare the total births by state, because they all started at a different place, depending on the number of naturalizations for that state. So I decided to split the single bar and make small multiples for each state.

[caption id="" align="alignnone" width="1278" caption="back to something more interpretable"][/caption]

It's interesting that the contribution of naturalizations slightly changes the ordering of the growth of states. For example, Florida has fewer births than New York, yet it's total growth is larger, because it naturalized 30,000 more people than New York. With this small multiples arrangement, it was now possible to do positional comparisons across categories, not just between naturalizations and totals. Turns out that more people get naturalized in California than are born in the entire state of New York. And since New York has the third highest number of births annually, more people got naturalized in California than are born in any state other than CA and TX.

This was too large of a graph, and the story I'm interested in is really the ratio between the birth and naturalizations (the closer to 1:1, the better), so I made another calculated field, which is exactly such a ratio, multiplied by a factor of a thousand, so I could give it a sensible description (Naturalizations per 1000 births). This refines my question

For every 1000 people born in the US, how many many immigrants become naturalized?

I then ordered on these ratios, and decided to filter the top states. Guam would have made the cut, but it is not a state, and (though I didn't mention it earlier) it's NVSS birth data was only 77% complete, so I excluded it. Fifteen is a nice odd number, but it actually marked a nice transition, as after Texas, everything else is less than 200 naturalizations per 1,000 births.

The small multiples bar graphs still looked too busy, and there was redundancy in the data, which didn't tell a succinct story. So I switched to just look at the ratios alone. This revealed, that, indeed, the fact that I've been living in California makes my perspective quite unique, as it is one of three states, along with Florida and New Jersey, to have an outstandingly large number of naturalizations compared to births. It is so high, indeed, that it puts the naturalization per births rate in these three states at more than twice the national average!

Looking at ratio alone tells us about the diversity in each states growth, but carries more meaning in the context of total growth . Thus, added the combined totals (naturalizations and births) as a size variable, for context. The alternating bands to both make it easier to read off the rows, and to aid the comparison of sizes by framing every data point in a common reference window. It obviates that California is the state with 864,261 new citizens because fills the frame completely.

Final question: What are the Top 15 "Melting Pot" States?

[caption id="" align="alignnone" width="1095" caption="almost done, would be nice to include context from the visualization I started with"][/caption]

Ordering the data in this way also shed light on the small but still very diverse states that would not have otherwise made the cut (and did not pop out in any manner on my previous bar graphs). Rhode Island and Hawaii got it going on, in terms of attracting immigrants.

Certainly the fact that I'm an immigrant myself also greatly influences whom I associate with, further skewing my world view towards a 1:1 ratio, but I'm actually quite impressed with just how close to that ratio is in California - 1:1.9. Of course, the data I've analyzed does not include the American-born 1st generation of children, nor does it take into account the number of immigrants living in the US that do not have citizenship. All of these factors would surely push the ratio even closer toward 1:1.

I decided to combine the US total growth information, since it's gives further perspective on the entire data set, such as the fact that California accounts for about 16% of total US growth. It also sheds light on how the US average was calculated. A new "twice the nat'l avg" line makes explicit the three most diverse outlier states mentioned before. I also changed the colors to match the convention used in the bar charts made earlier. The US combined total line semantically links the data plotted with the national growth bar chart - i.e. the green dots are formed by the sum of born and naturalized citizens.

[caption id="" align="aligncenter" width="1259" caption="What are the Top 15 "Melting Pot" States?"][/caption]

<---- end of cut

Ok, so, to be honest, it turns out that I wrote a large chunk of this post (Arizona suckage included) before I actually looked back at my visualizations, only going off my memory that it wasn't in the top 10. So Arizona is just below the national average in this "Melting Pot" ratio (a measure I made up, the number of naturalization per 1000 births). Since it is #12, some might say, "Paul, Arizona's on your top 15 list", to which I'll reply: "So's Texas."

I guess I just wanted to share these purdy graphs I made a few months back, and it seemed like there was a somewhat topical angle on them a few weeks back, when I remembered that I hadn't posted them on here yet. Anyway, I'd love to hear back your thoughts.

on facebook, you die a quiet death...

2010-02-27T06:56:13-08:00

So I finally got around to quitting facebook. I came to the HackerDojo tonight, and Waleed was working on sending out notifications for users of his app, because Facebook is disabling notifications as of March 1st.

Notifications was the way I was planning to announce my departure, because the "app" I created for that purpose would still be around after I deleted my account. But I was only able to send out a few of these notifications before facebook would not let me anymore.

Here's the message (with the cool icon I made) as it appeared on Facebook:.

But that wasn't going to stop me: so I made a status update - which contained essentially the same message:

Hey guys, as I alluded to in a status update a few weeks back:

I'm leaving Facebook.

This walled garden isn't the way the web was meant to be, and I *refuse* to
continue participating in it.

You can always find me by either searching for "Paul Ivanov", or through the
various means listed [on my website](//pirsquared.org/personal.html).

best, 
-pi

And then I was off to "Deactivate" myself. Beware - Facebook will play on your heartstrings by telling you that a couple of your friends will miss you - putting their photos right there and beckoning you to message them - but I was prepared to deflect this Faustian bargain. Though after I said "yes, really", it asked me for my password, and following that also sent a CAPTCHA! ("the escapade" indeed!)

and then, just like that: POOF! I vanished. Without any trace. The fake temporary account I created had no friends - my last (and all) status updates lost in the ether (at least as far as facebook users are concerned). But rest assured that the data lingers - permenantly - as facebook was immediately suggesting that my new fake account, who now had no friends to speak of - connect with the people I had been friends with, even though my account was gone.

For most of my friends, I got your birthdays, schools and jobs, as well as the most recent photo, and all of the friend-to-friend connection among you. For most - I actually had a stale copy (of everything but the photos) from 2007, which when I originally wrote the export scripts I used, and Facebook was just starting to open up its API and quickly devolving to a MySpace clone. It's not like I even frequently used the site - but reading about this reminded me to quit for good.

So I finally did.

Publisher's Block

2009-12-26T01:55:55-08:00

One of the reasons I find it so difficult to get more than a couple of entries in per year, is that I know they aren't going anywhere after I post them. They're sticking around for a while, and if they're full of trivial crap then that doesn't reflect very well on me. Posting about trivial stuff was ok when I was still trying to establish a sense of identity. These days, when I write something public, say on a mailinglist, I agonize over every detail because I know that this digital breadcrumb with my name attached will be around forever. So I keep raising the stakes to myself, neurotically checking over every possible extra whitespace in a patch I send in, sinking hours into something that should have taken 15 minutes.

I'm finally getting to the point where I realize it's a problem that, for example, even when I'm texting someone, I try to get all of the spelling and punctuation correct.

It's slowing me down.

I've had a lot of half-written blog posts that, after stepping away from them for a short while just don't seem significant enough. I try to only publish pieces that either I think about for a while, or that I'm not hearing/reading others write about. But I'm always mindful about adding noise. The way I see it, when it became super easy for anyone to publish online, a lot of content flooded in that I simply don't care for. Same idea with web 2.0 - because of Ruby on Rails, Django, and other web frameworks, writing a fancy (but useless) website became super easy - and now we're oversaturated with them ((Though this problem will probably sort itself out with time. I didn't intent to write about this now, so I'll just keep that remark without developing it further)). So there's this internal tension: I think there's too much crap-content out there but at the same time my internal filter keeps me from publishing anything. I rarely express my thoughts about what I find important in writing anymore. Others don't seem to make such a big deal about self-filtering, and are much more prolific writers/bloggers/coders, etc.

LTS

So here's a new acronym-sized motto to help correct this behavior, which is starting to get sprinkled in comments in the software I'm writing for my research: LTS. Life's too short.

I use it as a reminder of what in the past was one of my frequently used maxims: most things in life are pass or fail. This doesn't mean that it's ok to do a half-assed job on everything, but given that there's a limited amount of time, I should focus my efforts only on that which is truly important. Typos in a text message or extra trailing whitespace do not qualify as such.

I wasn't always this careful about what I publish. I've had some form of internet presence (as embarrassing as it may seem now) since I was in middle school. It started in one of those geocities neighborhoods, I don't even remember any details right now, probably because my brothers helped me to set it up. I didn't use my real name until I started a poetry website freshman year in high school.

I used my full name, because I wanted to express my thoughts and have them be connect back to my persona, not a pseudonym that I might grow tired of. I was quite explicit about this at the time. And I didn't filter myself, I just counted a total of 20 poems on there which were written in the course of a year. None of them really make me cringe, and some I'm still quite proud of.

I had nothing to gain by hiding behind an alias. I think that attaching my real name somehow made my thoughts sincere. I started blogging socially my senior year in high school (livejournal), and looking back on the first entry there, I was just trying to capture day-to-day events and thoughts. Vim, THE editor, is mentioned five times in the first two entries :) . But there are some very candid and thoughtful remarks in there, too.

It's kind funny to have your more than 10 year old website cited in a Yahoo! Answer to the question: "What is the best way to live life to the fullest?". I mean, it is yahoo answers, we're really scraping the bottom of the barrel when it comes to content (( in fact, Elaine absolutely refuses to read anything on that site anymore, despite the fact that frequently, her google search string is verbatim the same as the question which comes up as one of the top results)) , but it's still cool. Yeah, ok, so it's doubly embarrassing because the citation is just for the lyrics to "The Sunscreen Song". I'm ok with that.

And I'm very grateful for my many friends and colleagues who, by their example, continue to give me the courage to release my thoughts and code out in the open. Thank you.

As I was putting my finishing touches on this post, I found a recent entry on Scot Hacker's blog titled "(I Don’t Care About) Facebook and Privacy" that covers similar ground: "For me, it’s simple: If what you have to say shouldn’t be said to the whole world, then don’t say it online." I agree, and it's a more sensible standard than my "everything you say will forever be connected to you, so don't screw it up!" But just to be clear, this should only apply to things you intend to write up and release: I absolutely oppose Eric Schmidt's dismissal of privacy. Eric says, "If you have something that you don't want anyone to know, maybe you shouldn't be doing it in the first place." Due to its construction, it bears striking similarity to Scot's quote above with which I mostly agree. But to me, Eric's statement is a 1984-sized world apart.

Anyway, hopefully I've adequately explained my "publisher's block", and there are many related topics left to explore, but this is where I'll have to end this post for now. LTS.

Standing up to the Madness is an excellent read

2009-05-02T22:38:33-07:00

My labmate Tim sent me an email on Wednesday (April 15th) saying that Amy Goodman "Democracy Now! fame, and my heroin" [sic] was speaking on campus at noon. The place was packed, and it's the best way I could have imagined to snap back out of the Qualifying Exam bubble I've spent the last several months in, and re-engage with the world at large.

One of the excuses for the tour is the paperback release of Standing up to the Madness: _ Ordinary Heroes in Extraordinary Times_ by Amy and David Goodman.

Now that I'm a tenured grad student, I can actually allow myself to read for pleasure - guilt free! So I went to the library that Thursday, and picked up the hardcover, which came out last year.

What I liked about this book is what sets it apart from other political books of today. Amy and David don't just provide us with a laundry list of wrongdoing by the Bush administration, congress, various governmental agencies, as well as highlighting some of the ongoing local struggles. Though the book is chock-full of such details, they are all provided in the context of a particular vignette. What's more - instead of simply stating the problems, or providing an outline of the authors' opinions regarding what course of action should be taken, the book highlights the work average citizens have already done to oppose injustice, censorship, racism, etc. One example is T-shirt "terrorist" Raed Jarrar, who wore a shirt with the words "We will not be silent" - written in both English and Arabic - a reference to the White Rose - and was forced to put another shirt over it because JetBlue customers were threatened or offended. With the help of the ACLU, Jarrar sued the TSA and JetBlue, who ended up paying $240,000 to settle the discrimination charges.

Like Hochschild's King Leopold's Ghost ((which, after I first read it in 2001 became my measuring stick for gauging the quality of non-fiction)), this book is non-fiction that reads like fiction. Not because it is well-written, though it is, but because of the shocking realities of the content. Leadership cannot be taught, it can only be revealed. Standing up to the Madness gives us dozens of snapshots of the ongoing work of ordinary heroes.

Duopoly (or why I'm not voting for Obama)

2008-07-04T13:12:02-07:00

Let me ask you a question: Do you think that the two-party system is good for the United States?

I find it very difficult to engage in debates about national politics because the average citizen has so little influence over these matters. I think that it's much more worthwhile to get informed about and involved in local politics, because that's where someone like me can actually have influence.

Nevertheless my own answer to the question is that it's probably not a good thing. There's this high-dimensional landscape of issues that people care and have different ideas about - reproductive rights, gun control, immigration, education, social programs, the size of government, taxation, the list goes on and on. Yet that gets projected down to this one dimensional line with just "Left" and "Right" with optional "far" and "center" prefixes.

And, sadly, the common consensus is that on election day you have only two possible boxes to check. A single decision. One bit. 0 or 1.

The Democrats and Republicans are playing a small concessions type of game. They sort of shuffle around slightly to appeal to enough of those voters who aren't already automatically voting for them. If you only vote for one or the other, they have no reason to change - they already have your vote.

Voters in safe rarely contested states, have the unique opportunity to vote their conscience without fear ((Electoral College: bug or feature?)). When I twittered about Obama's support for the FISA Compromise, Philip, a disappointed California voter replied: "our voting system forces us to vote strategically and i'll be voting obama ." This doesn't make any sense to me! Obama will carry California. Democrats almost automatically get California ((The only way the Democrats might not get California is if Arnold runs as VP for a moderate Republican, and that just is not happening this year.)) .

So why give in? You're not happy with the Democratic candidate ((There are more reasons to not be happy)), the candidate who will carry California regardless of how you vote, yet you still feel unable to voice your disapproval in the electoral arena. David wrote: "I'm not going to throw away my vote on the green party," but aren't you just throwing away your vote to the democrats, instead?

The role of third parties is to emphasize new and different ideas, to bring folks who've given up hope back to the table, and to make the major parties shift in MEANINGFUL ways. Here are some great YouTube clips on the role of third parties in the US: Part One, Part Two, Part Three, Part Four, Part Five.

If you still have doubts about voting for a third party candidate and/or you live in a swing state - consider the votepact.org proposal: find a fellow kindred heart on the other side of the political spectrum who's also unhappy with the candidate on their side, and together vote for a third party (fill out your absentees together over coffee).

My first foray into the production of motion pictures

2007-11-28T05:11:10-08:00

I want to thank Curt Siffert for granting me permission to use his song "All Aboard (v2)" which you can download (for free) here. This is the first video ((yes, okay, it's more of a slideshow with an intro, but my brother Mike told me that Ken Burns would be proud)) I've ever made, but I've wanted to make films for as long as I've been writing ((creatively, which would be 1999 - warning: link contains some extremely cheesy content, including an early version of what evolved into this journal)) (even before Sally said "Hey guys, I'm going to make movies!" and then did), so I'm glad I've finally started.

I put this together for a video contest here at I-house. You can see all of the videos for the contest here, the winner was Life At I-House, A Glimpse by KirstyandEliana.

Napoleon@Home - Distributed World Domination

2007-10-13T21:07:29-07:00

I'm about to give this Lightning Talk at SHDH20.

Weinberger's talk and OLPC

2007-08-08T05:45:17-07:00

Here's David Weinberger talking about Everything is Miscellaneous (the book I reviewed here and many others did here). The hour-long talk stands on its own and covers much of the book, though I don't recommend watching it if you're planning to read the book.

I went to Linux World Expo today and played with one of these upcoming One-Laptop-Per-Child project's XO-1 laptops at the Creative Commons booth. _ Photo by Scott Beale / Laughing Squid (cc) While OLPC is a noble effort, I think it still feels like another example of trying to solve a problem with technology where technology is not the bottleneck. For example, I was bummed that they took away the hand-crank power-supply a while back because I think this severely limits who'll be able to eventually use these. The UI and networking stuff is pretty novel, but my overall impression is that it's too gadgety. I felt pretty lost in all just the buttons on the keyboard, but then again I only used it for 20 minutes and this wasn't made for me. With that said, I'm not holding my breath, but it _could be a great thing if this takes off. I say "could" because technology by itself just isn't enough ((A point the OLPC project acknowledges in their vision.)). This is a point I keep coming back to again and again.

uncomfortably sincere

2007-08-07T06:34:08-07:00

From my paper journal:

23:37 May 10th, 2007 Thursday

So what good is all of this if we don't engage one another - on a very real level? "Only a Sith Lord deals in absolutes," man - so let's get off the condescending trips and the polarizing anti-discourse. Let's use our full range - zero, one, two, five, seven, eight, ten, etc and not just on or off - there's an infinitude of wonder in between and out in every dimension.

I resolve to hesitantly dip my toes in, from time to time, instead of being all in or all out. Talk to a beat stranger - but not every beat stranger. Give up some left over food to the guys on Bancroft and Telegraph (the Shattuck hobos are too hip for me - but I hope they aren't for someone else -- I know they aren't). I can just eavesdrop on the world from time to time - I can't always be wide-eyed gulping from the fire hydrant of information flowing at 100 terabits per second eyes glazed over passed out exhausted gasping for a sense of self disoriented head-spun hours or days later. Just a drinking fountain or a tap and a few cups or liters a day would be fine. No need to parch yourself and dry up like a raisin all the time. It's ok to wrinkle and shrivel - and you don't need to burst, either - just be uncomfortably sincere.

I think I'm going to try that.

Related brief thought:

17:36 June 22nd, 2007 Friday

Bumper Sticker Activists (Telegraph in Berkeley)

The last thing we need is more Bumper Sticker Activism. Wearing a clever T-shirt does not constitute civic participation.

thoughts about the sea of information

2007-07-31T04:23:27-07:00

I just finished reading ((In three evening sittings at Moe's Books)) David Weinberger's Everything is Miscellaneous and I find it to be a pretty engaging description of how the state of knowledge evolved with time, and now it has given me a chance to write down some thoughts.

The basic gist of the book is that knowledge is no longer tied to the physical (e.g. books), which used to limit how one went about organizing and finding it (e.g. Dewey decimal system). Now we can attach as much metadata as our hearts desire, which technology helps us sift through to help us find what we want. Instead of each book having a particular place, as in a warehouse, or a relative position (alphabetical within a subject), an individual leaf of information lives on a multitude of trees simultaneously, and the trees themselves are dynamically created and rearranged for each user on the fly.

The first few chapters focused on how knowledge has been historically organized over the centuries. I did skim through a few of the middle chapters, it seemed to be pretty straightforward commentary on the digital lives most of us now lead - user created content, social tags and lists, auto-recommendation, etc. Some over-simplified, in that sometimes unavoidable awkwardness that comes out of describing something neat and complex yet obvious to those leading digital lives. It was refreshing to read about the downsides of scientific publications like Nature and Science (e.g. good science isn't enough ((some might even argue "isn't required")) to publish because of how few articles get in, the research has to be "sexy") and how the new comer PLoS One aims to correct these shortcomings. Because this was just the topic that was discussed at the Neuroscience retreat last year (in a lecture about the then-upcoming PLoS One), scientists care about this stuff and it comes back every so often.

Although I never considered it myself, I totally got it when Danae started her Master of Library Science. I would argue that more than anything else, what we're producing most of in the world today is information. Perhaps capture and disseminate is a more appropriate description. Information, by itself, is agnostic to how it gets used (or abused). But the Cliff Stoll-ian side of me says that we should be weary of the exponentially growing amount of information, and not just for the obvious Big Brother / privacy reasons (e.g. "Plate reader draws objections of ACLU").

The non-obvious threat of information is that we're drowning in it (my claim). Here I'm glad Weinberger mentions Cass Sunstein's book Republic.com ((Republic.com starts with a succinct vignette: "the daily me")), the basic thesis of which ((on my quick skimming at the UCD bookstore this past Picnic Day.)) is that with more and more information out there, we can all end up listening, watching, and reading only that which reinforces our world view - drowning out everything else without even having to plug up our ears and going "LALALALALA", but by finding podcasts, channels, and blogs where others are doing the "LALALALALA" for us.

In many ways, this leads to huge portions of the population nonsensically parroting something like "Evolution is just a theory" to one another. Scientific theories both explain observed phenomena (why living organisms share so much of their DNA) and make predictions about future observations (my niece's hair color based on that of her parents, or maybe one you don't hear about so often: regular use of antibacterial soap might be a bad idea, placing evolutionary pressure on the bacteria to evolve immunity to the soap). Moreover simpler or more elegant, straightforward theories are preferred (aka Occam's Razor). Which is why Intelligent Design is on par with Flying Spaghetti Monsterism, not science. But this has been better described in other places and elsewhere (suggestions welcome). The point is that I'm worried that there's no way anyone get through to the people that end up isolating themselves in their own feedback loops. I worry that not enough people engage enough to think on their own. Technology can't fix this problem. No amount of metadata will ever be enough (( a point I think the book misses)).

In this entry, I've linked to Wikipedia a few times, and while I agree it should not be regularly used for primary research, I also welcome the explicit uncertainty inherent in a publicly editable wiki, as it reflects the tentative nature of information, and I think we should be somewhat skeptical about a great deal. I have also been recommended, though I have not yet read Manuel Castells' The Internet Galaxy, though perhaps it is more topical for a future post I've been brewing for a while. Has anyone read it? ...Anyway, this is my first pass at processing this stuff, hope it's not too scatterbrained (( Cory Doctrow does a better job reviewing the book.)).

And it begins again...

2007-07-20T01:04:03-07:00

So I finally bit the bullet and put up my own blog. It was just one of those wait and see things for a while, but now I find myself reading most things via rss feeds, so I really had no excuse not to move on from my livejournal. I was afraid of abandoning my lj-friends - but Yuan found a happy medium with cross-posting back to her lj (though now that she has an rss feed I read her entries first in Thunderbird, sometimes days ahead of visiting my friends page)

I'm still getting settled in, so this isn't quite live yet.

Anyway, I've had a couple of entries on the back burner that I've been working on, and they feel serious enough to warrant having their own place, instead of being a part of a corpus I started almost six years ago (in high school, no less). More and more people I know host their own blogs and it's always nice to have a fresh start (though I've reposted a hand full of my most recent LJ entries to get a running start).

visualizing world statistics (Gapminder - Hans Rosling)

2007-07-03T11:04:00-07:00

Graph: **CO2 emissions per capita versus Time ** Above: a plot I made using Gapminder. When I first tried this tool a few months ago, I was left confused and unimpressed. Luckily, since then, I've stumbled upon the following two explanatory videos (~20 min each).

last year and this year.

After watching the videos, you can play with Gapminder yourself as it is a web-based tool.

More info and tool links at gapminder.org.

View of the Golden Gate Bridge from I-house Cafe (google maps streetview)

2007-05-30T21:34:00-07:00

goddamn google just doesn't know what to do with itself, anymore.

View of the Golden Gate Bridge from I-house Cafe

Oh yeah, feel free to drag the little man around the blue highlighted streets, and then rotate / zoom in the overlaid photo. Here's the official demo (cheesy video). In San Fracisco, almost every street is completely covered. They also did much of Mountain View, Palo Alto, San Jose, and all of Manhattan, that I've checked.

Rediculous.

The practical and the ideological

2007-03-15T03:46:00-07:00

To start off with the latter: on Friday, after dinner with Robert and Julia at Zachary's, we went to a screening of An Unreasonable Man - which filled the gap in my knowledge of Ralph Nader between Unsafe at Any Speed / Nader's Raiders and the 2000 election. Fascinating balanced documentary. You can still see it this week, but it'll only be around the theatres a short while.

The practical: After getting lunch with Robert and Jon on Saturday, I got the chance to hear recent UCSB alum Logan Green talk about Zimride, this new cool webapp he's just put together. Carpooling made easy and safe. Here's what it looks like:

Zimride integrates with facebook, so you actually get to know something about your potential drivers/hitchers, and they might even end up being someone you know! Moreover, you can advertise your ride via those facebook stalker feeds.

SLC punk'd!

2007-02-22T07:29:00-08:00

I'm going to a conference for a week in Utah. It'll be my first time in Park City, and my second time in Salt Lake City. Here's a map of downtown SLC, color-coded to emphasize the insanity:

**worst. planning. ever. ** (...and I don't buy their propagnda - next to the green arrow on the map above, you could be on N W Temple, between W N Temple and W S Temple [or is it E S Temple?])

coming to you live from my desk...

2007-01-22T07:43:00-08:00

So I've finally gotten a new phone and my camera that had been flakey for the last year decided to start working properly again - and I am happy with technology! In celebration I decided to take some pictures of my room (messy edition)

I imported these pictures using F-Spot, which has a decent tagging interface that I should make use of to catalogue a bunch of old photos. F-Spot also happily resized and exported them to Flickr, tags and all (other stuff also supported). Hopefully this also means I'll start taking pictures again.

A picture of me and some of my housemates...

2007-01-13T08:24:00-08:00

I-house is a pretty fun place to live (I'm off on the right side).

P.S. Ñ Ð½Ð°ÑÑ‚ÑƒÐ¿Ð°ÑŽÑ‰Ð¸Ð¼ ÑÑ‚Ð°Ñ€Ñ‹Ð¼ Ð½Ð¾Ð²Ñ‹Ð¼ Ð³Ð¾Ð´Ð¾Ð¼! happy old new year!

Changelogs with dates (!) + gui goodness

2007-01-04T03:31:00-08:00

... please date your software releases ... I've been trying out a lot of new software lately, and it's the most frustrating thing in the world to not be able to figure out WHEN a particular version came out.

Sure, there's a changelog that tells me all the neat bug fixes from the last version, but what good does that do me if I can't tell whether the software was last updated 8 years ago or 8 days ago? It's such a simple thing, but I can't believe at the number of projects out there that have no mention of release dates on their website.

... changing topics... Anyone have a gui svn client they particularly like? I've started playing with eSvn (0.7.0 testing version), it's clean and simple (someone else came to the same conclusion). It looks like this:

Subcommander has a neat looking log graph (bottom), and kdesvn has a history browser, anyone use stuff like that?

I've also tried subclipse. Yep, I finally bit the bullet and started trying Eclipse and other GUI stuff, like Inisght (gdb gui), although I continue to live in Vim (which has an CVS/SVN integration plugin I've found useful)

damn you, amazon.

2006-12-22T04:29:00-08:00

So I used to go to the Super Crown by my house all the time when I was in high school. It became sort of a ritual for me, whenever I was feeling in the dumps, not getting any work done, or just needed a break and a walk to refresh my mind, I'd head out, usually around 9 o'clock at night, and spend a few hours sitting on their comfortable couch seats (or on the floor, when those were occupied) reading the first few chapters of some book, usually technologically related. The only one I remember finishing entirely at Crown, in several visits, was ENIAC: The Triumphs and Tragedies of the World's First Computer, though I know I peeked into a bunch of others on telecommunications, AI, Unix, CS, etc. My thinking was that it's good to expose myself to just a little bit of a something that I didn't know anything about, and I'd walk away refreshed by the new knowledge. I treated the bookstore as a library (incidentally, the French word for bookstore is librairie, so you can't blame me there), and even took little notes of the things I had learned along the way, in part so that I could return to the page I read up to the previous time. I bought books, there, too, when I had the cash - Cliff Stoll, Steven Levy, Robert Pirsig, Daniel Quinn, Tim Berners-Lee, many others, too; I usually keep the receipt in the book (and would write the same sorts of notes on the receipts).

It became a really familiar place, the same classical music, the same new book smells. I never really had to talk to anyone, or say anything, so it very much became a place where I could go and clear my head, just sort of process my thoughts. Then it went out of business and closed, which really made me sad. Luckily, Tower Books, nearby, had just started operating, and though it didn't have as large of a selection, I migrated over there, and got used to the music, atmosphere, and the staff there. So Tower, too, became familiar with time, and they were open till midnight, which suited my fancy more. Going down there at night became a ritual, whenever I was feeling uninspired, I'd just head over to there and immerse myself in some new book, if only for a few hours. I think I'm kind of different that way. If you've ever been in a bookstore with me, you'll know that I always want to stick around for a while, even though I usually have no specific book in mind, I just like to go and sit and read something new for a while. For example, I always like to drop by Borders whenever I'm on University in Palo Alto, but also usually overlook that whoever it is that I'm with, whether it's Elaine, or Philip, or Jon, doesn't have the same approach to visiting bookstores.

Cody's Books on Telegraph closed earlier this year, though Moe's is still around. Now Tower's closing up shop, I just walked around the all of the empty shelves and saw very few books that'd be of any interest to me. Ended up picking up two DVDs: Before Stonewall, and Ð‘Ñ€Ð°Ñ‚ (Brother) for $6 each. It's really makes me uncomfortable and sad knowing that I won't have that little place to escape to, anymore. I'm not a big fan of changes like these.

Todd Chretien, Greens, Choice Voting

2006-10-18T04:29:00-07:00

Sentence long update on life: I'm at Berkeley studying Vision Science now.

I've started getting involved with the (currently small) Campus Greens organization (which meets Mondays at 7:10 in 200 Wheeler).

So today I heard Todd Chretien, Green senatorial candidate speak to a group of about 30 as part of the ASUC Speaker Series. Todd titled his talk "Why Students Should Never, Ever Vote for the Democrats," which I think is somewhat unfortunate. Todd has an eloquent platform and I share a lot of the same views, but I also think that the title incites the type of reaction that eliminates any possibility for reasonable discussion or discourse.

I think that people don't want to listen to you if you insult them, or just say something shocking - the novelty (if any) quickly wears off (it's taken me a while to figure this out, but I think I learned the difficulty in trying to actively engage those who support the Democrats when talking (ranting?) to Janet on the streets of Brussels over the summer).

I think that we need more boring nitty-gritty politics, because no one will hand over the helm to people with big ideas (even if they are the right ideas). The big picture is important, but it has to be negotiated with real, tangible, local progress.

Todd gave a short run through of his top three issues ( war in Iraq, education, the two party system), and then opened it up for Q & A. In answering the questions, he covered a lot of ground in both domestic and foreign policy, but I felt like it was a discussion of issues larger than those someone who admitted he had no chance of winning could hope to influence....

So as the last question for the night, after expressing these sentiments I asked what we could do locally, that's within our power, mentioning current choice voting efforts in Davis and Oakland. Unfortunately, Todd stuck to his anti-war protest-in-the-streets approach (even taking an outlandish pot shot at proportional representation by mentioning something about Hitler getting elected).

Most of my life I, too, have been a big ideas person, but I can't say I've accomplished much with them, which is why I'm trying something new...

By the way, Kenji and Philip, you continued work on important matters has been really inspiring.Here's my letter to the editor regarding choice voting that never got printed in the Davis Enterprise:

Until I came to UC Davis, I had never realized that there could be different voting systems. Choice voting is a way of reaching a majority (greater than 50%) consensus.

Choice voting allows everyone to vote their conscience without the fear of having your vote "wasted." After the polls close, if your top-ranked candidate, Alice, has the least amount of votes, she is eliminated and your vote transfers to your next choice, Bob, in your order of preference. This process ("instant run-off") continues until candidates reach enough votes to be elected (the threshold). This consensus building mechanism ensures that the elected officials will represent the greatest possible proportion of the voters.

Contrast this with the current system: candidate Mallory and Minnie, representing a minority of the population could get elected when multiple similar candidates (Alice, Bob, Chris, and Debra) representing the viewpoints of the majority of the population split the vote between one other.

This would not happen under choice voting, because when Alice is eliminated, those votes would go to the next choices of her supporters. This would provide more votes for the remaining majority candidates, ensuring that one of them gets elected.

I encourage Davis voters to vote yes on Measure L this November so that the City can continue looking into this effective system.

Paul Ivanov UC Davis Class of 2005

(cute choice voting promotional video)

Paul Ivanov’s Journal

Code generation tools: a degenerative disease for free and open source software

diff-jfk

*_fresh vs *_highlighted

Unredacted diffs

Some samples

Limitations of Minimal Magic

How diff-jfk came to be

Duplicate ambiguity

Previously unreleased files

Issues and Discussion

Parting thoughts

Current Challenges in Free Software and Open Source Development

We're sitting here, I'm supposed to be an ethical professional, and we're talking about productivity.

I mean, listen, we're talking about... productivity?

Not ethics.

Not ethics.

Productivity.

We're talking... about... productivity.

Not ethics... that we go out there and project into the world, we're talking about productivity.

oh! Oh! We do have this mechanism — it's called a license.

Oil and water

self <---> society

journey <---> destination

Athenian <---> Spartan

dogmatic <---> pragmatic

silly <---> serious

courteous <---> abrasive

pushing <---> steering

emotional <---> intellectual

(1/3)Bh = V

NumFOCUS concerns

NumFOCUS origin story

How a 501c6 differs from a 501c3

Money and California Propositions (2020)

aka: also known as

Examples:

Why would anyone want this?

Installation

How it works

Notes

Related Projects

License

Author

Changelog

Source code

Lazy River of Curious Content 0

pheriday 3: infrastructure

Netscape Navigator!

the most improbable error message:

WAT?!

BINGO!!!!!!!!!

Wat?!

How did a Netscape site ended up redirecting to Yahoo...

Where were we?

Yes, No, Cancel?

How Mozilla released an update without user consent

Let's wrap this up...

That's all

Uvas Gold 200

My first DNF (Ft Bragg 600k)

PyCon2019 poem

Pycon 2019

Get in it

Get in it

SciPy 2018 dates and call for abstracts

July 9-15, 2018 | Austin, Texas

Talks and Posters (July 11-13, 2018)

Mini Symposia

Tutorials (July 9-10, 2018)

SciPy 2017 tips

Conference Tips

Stuff to do in Austin

June 1st, 2017

June 1st, 2017

March 29th, 2017

November 9th, 2016

November 9th, 2016

Jupyter's Gravity

in transit

`_fresh` vs `_highlighted`

How `diff-jfk` came to be

`Get in it`

`June 1st, 2017`

`November 9th, 2016`