1. embracing my hypertextuality

    Well, it's happened again, I've jumped (back) on the static blog engine bandwagon. Early versions of my site were generated literally using #define ,#include, gcc, and a Makefile...). Back then I was transitioning away from using livejournal, and decided to use WordPress so I wouldn't have to roll my own RSS feed generator.

    I tolerated WP for a while - and the frequency of my posts was so low, that it wasn't much of an issue.

    Except for the security upgrades. I logged into the wp-admin console way more times than I cared to just to press the little "upgrade" box. The reason is that wordpress keeps everything in a database that gets queried every time someone hits the site. I was never comfortable with the fact, because content can be lost in case of database corruption during either an upgrade or a security breech. Also, my content just isn't that dynamic. The WP-cache stuff just seemed overkill, since I don't get that many visitors.

    But lately, I've found myself wanting to write more, to post more, but also shying away from it because I hate dealing with the WordPress editor. And I also hate being uncertain about whether any of it will survive the next upgrade, or the next security hole, whichever I happen to stub my toe on first.

    And the thing is, I really like to use version control for everything I do. I liked my blog posts to be just text files I can check into version control. I also like typing "make" to generate the blog, and now I get to!

    For added fun, I'm hoping that writing my posts in markdown will make it easier to coordinate my gopher presence, since it's pretty close.

    For posterity, I'm capturing what the first version of my Pelican-based blog looked like. I did the same thing when I moved to WordPress.

    Embracing my

    I ran into some confusing things about transitioning to Pelican, so I thought I'd note them here, for the benefit of others.

    Unadulturated code blocks

    I like to use indentation as a proxy for the venerable <tt> tag - which uses a monospace font.

    If you just want to use an indentation, but do not want the indented text parsed as a programming language, put a :::text at the top of that block. Here's what I mean. Take this Oscar Wilde quote, where I've inserted a line break for drammatic effect, for example:

    A gentleman is one who never hurts anyone's feelings

    Now, you see that ugly red box around the apostrophe? Well, That's because all I did was indent the two lines. If I just put a :::text above the quote, indented to the same level,

        A gentleman is one who never hurts anyone's feelings

    the result will render like this.

    A gentleman is one who never hurts anyone's feelings

    As the pelican documentation specifies, this is also the way you can also specify the specific programming language you want, so :::python would be one way to not make pygments guess. You can get a list of all supported languages here, just use one of the short names for your language of choice.

    And you should really do this, even if you aren't bothered by the red marks, because the code highlighting plugin goes in and tokenizes all of those words and surrounds them in <span> tags. Here's the HTML generated for the first version:

    <div class="codehilite"><pre><span class="n">A</span> <span class="n">gentleman</span> <span class="n">is</span> <span class="n">one</span> <span class="n">who</span> <span class="n">never</span> <span class="n">hurts</span> <span class="n">anyone</span><span class="err">&#39;</span><span class="n">s</span> <span class="n">feelings</span>
    <span class="n">unintentionally</span><span class="p">.</span>

    and here's the version generated when you add the :::text line at the top:

    <div class="codehilite"><pre>A gentleman is one who never hurts anyone&#39;s feelings


    At some point, having a dialogue with myself, I wrote in here "or should I not use pelican and use nicola instead?"

    Ok, tried it - nikola takes too long to startup -

    nikola --help
    real    0m1.202s
    user    0m0.876s
    sys     0m0.300s
    pelican --help
    real    0m0.639s
    user    0m0.496s
    sys 0m0.132s

    I'm sure it's a fine static blogging engine - and Damian Avila's already written IPython Notebook converters for it, but it just feels like it tries to do too many things. Constraints are good. I'll stick with Pelican for now. (Though I did use the nikola wordpress import tool to grab the wp-upload images from my WordPress blog)

    another set of instructions I consulted: Kevin Deldycke's WordPress to Pelican, which is how I did get my articles out using exitwp which I patched slightly, so files got saved as .md, and preserve other format properties.


    also known as: not breaking the web

    I wanted to preserve rss feeds, and also not break old WordPress style /YYYY/MM/DD urls - the Nikola wp-import script had created a url remapping scheme in a file called url_map.csv.

    I don't have that many posts, so I just added them in by hand:

    Options +FollowSymLinks
    RewriteEngine on
    RedirectMatch 301 /blog/feed/ /blog/feeds/all.atom.xml
    RedirectMatch 301 /blog/2006/10/18/todd-chritien-greens-choice-voting/ /blog/todd-chritien-greens-choice-voting.html
    RedirectMatch 301 /blog/2007/01/04/changelogs-with-dates-gui-goodness/ /blog/changelogs-with-dates-gui-goodness.html

    Enabling table of contents for posts

    If you want to include a table of contents within a post using [TOC], you must enabled the markdown toc processor with a line like this is your pelicanconf.py:

    MD_EXTENSIONS =  [ 'toc', 'codehilite','extra']

    Categories and tags

    Ok, so this was never clear to me in wordpress, either - but what's the difference between a tag and a category? is it the case that a post can only belong to one category, whereas it can have any number of tags?

    I think I used categories as tags on wordpress. Looks like all posts on Pelican can have at most one category. Turns out this little aside was long enough to turn into its own post, so if you're interested, pelican tags-vs-categories has got you covered.

    That's it for now

    Thanks to Preston Holmes (@ptone) for encouraging me to transition away from WordPress, and pointing me to this post by Gabe Weatherhead (@MacDrifter) for how to do that. It should be said that the pelican documentation itself is very good for getting you going. Additionally, I consulted this post by Steve George which has a good description to get you started, and also covers a bunch of little gotchas, and lots of pointers. Also, thanks to Jake Vanderplas (@jakevdp) for his writeup on transitioning to Pelican, which I will consult later for incorporating IPython notebooks into my markdown posts, in the future. This is good enough for now. LTS.

    read more
  2. remembering John Hunter (1968-2012)

    John Hunter, the author of matplotlib, passed away on August 28th, 2012. He will be dearly missed.

    Please donate to the John Hunter Memorial Fund. A giant in our community, John lead by example and gave us all so much. This is one small way we can give back to his family.

    what follows are excerpts from my paper journal over the past week:

    John Hunter passed away this morning,
    Oh my god. John Hunter was incredibly
    kind and warm -- I can't believe he bought
    me this laptop. I can't believe that such
    a wonderful man could be dead -- so suddenly.
        What a tremendous loss.

    Dear Merlin,
        John Hunter died yesterday --
    getting unstuck. It is entirely appropriate
    to be stuck -- to feel it, smell it, taste it
    feel it drilling though your head

    I keep pacing around the house.
    I just need to leave.
    My thoughts are with John's
    family. & with the folks in Louisiana.
        A rat in a maze --
    panicked -- & the water
    level rises still. escalates.

        Our loss of John makes me
    want to code furiously.

    I feel the urge to code furiously,
    but only have the capacity to tweet about
    it, and lack thereof to censor myself.

    I don't believe I'm losing
    my mind. I believe I never had
    one to begin with.
      I'm not losing my mind. I never
      had one to begin with.
      I'm not losing my mind.
      That would imply I had one
      to begin with.
    Left the house without shoes --
    waiting for my laptop to charge.
      I want to share this with the
      group -- so I can come to
    terms with it myself. forgive myself.
      Connection with a stranger -- can be a form of escape.
    but it can help you gain perspective on
    your own life. I think it has for
      I am a severely broken
    person. Ok. Time to get shoes,
    shower, & then go to try &
    talk w/ Greg Wilson.

    I must be mistaken. Maybe I was.
    Who's to say that I wasn't.

    The scientific python community lost one
    of it's giants this week, and
    I lost an important mentor.
    Remembering John Hunter (JDH)
        People keep dying. I don't know
    how to deal with that -- I feel like
    I never really processed dedushka's death --
    nor Ken Green's, nor Jessi Debaca's,
    nor babushka's
    ... I lost another mentor. Most of
    these people (all?) don't know that they
    were mentors to me -- but they were.
    I looked up to John -- I secretly
    wanted to please him -- but did not
    even dare to do so directly (I
    have Fernando Perez to thank for getting
    my first contribution into Matplotlib.
    I remember feeling really bad after my first
    sprints at SciPy 2009 -- John actually
    knew who I was -- and wanted me
    to work on some matplotlib stuff --
      I ended up doing some rote work with
    David Warde Farley (and felt kind
    of like a third wheel - since dwf
    (pronounced "dwoof" - did I get
    that right, David?) is more than
    capable as a command-line
    cowboy (David doesn't know this --
    but I learned a great deal from
    just sitting next to him and watching
    him string together standard unix
    tools, pipe-after-pipe -- to clean
    up some scipy wiki content, to
    try and export it to a new site.
    David's also a role model in other ways --
    he's very calm and collected (unless
    he senses you've pushed the bozo button from
    some punditry vending machine - instead of
    understanding and engaging with the full complexity of some
    social or political issue)
        John was extremely kind and understanding -- 
    he wanted to invite & welcome me to
    code alongside other matplotlib developers,
    but there was no expectation.
      I got to hang out with John the 
    most at the PyData conference in
    Mountain View in late February. He was
    giving a matplotlib talk, and was seeking
    feedback on what to talk about --
    & how to do it. He's warm -- and always
    had this kind smile about him.
        After hanging out with John at PyData --
    this email arrived in my inbox

    Hey Paul, It was great seeing you again at pydata, and thanks for your help during the talk. We’ve decided that you could be a lot more productive in all your efforts to help ipython, mpl and others if you had a shiny new laptop, so I want to buy you one out of the mpl donations fund. If you spec out the machine you want and send me a link and details, and your preferred shipping address, I’ll order it for you and have it shipped to you. Pick a machine good enough that you can rely on it for a few years, because it looks like you get good use from these things! In other words, don’t feel compelled to be frugal on our behalf.  JDH

    Fernnado knew about the email -- & I was at
    a barbecue at his house -- to sort of
    celebrate an awesome week of lots of
    Python Giants in from out of town.
      Fernando was giddy -- "Have
    you checked your email?" -- with a
    twinkle in his eye "Let's go have you
    check your email right now" -- & he
    walked over to turn on his desktop machine.

        On the mailing lists -- I always tried to
    emulate John's approach -- helping everyone
    even if in the slightest manner.
    I was offered commit rights sort
    of out of the blue -- after sending
    a couple of pull requests.
        We are a community.
    We need to remember John, & keep
    remembering John -- for many, many,
    many years to come.
    I feel the urge to reach
    out to everyone I know & don't know.
    I'm desperate -- i think I've been that
    way since I was a little kid -- I
    remember having the same feelings when
    we were leaving Moscow -- a ten-year old,
    talking to friends and my teachers,
    many for the very last time.

    I am posting this, but it isn't finished.
    This is broken, half-finished, confused,
    necessarily so -- because there is no way
    to mend what we have lost. This
    is an expression of my current state --
    and if I don't let this out, it will just
    keep ricocheting around in my head for years
    to come.
        This is a first pass. This is me
    grieving. This will never be enough. This
    is just a start.

    videos of John Hunter's talks:

    matplotlib: Lessons from middle age. (Scipy 2012)

    Advanced Matplotlib Tutorial (PyData)

    SciPy 2009 - Advanced tutorial 3: Advanced topics in matplotlib

    (beginning) SampleDoc

    Scipy '09 Panel on Visualization tools

    Scipy '02 Core Projects update

    NIPS Workshop on Machine Learning Open Source Software (MLOSS)

    Fernando Perez and John Hunter

    Sept 2009: "some fun stories like 'Jeez, you guys have some crazy examples. I am surprised there isn't dolphins swimming around inside a sphere.' So now there is."

    read more
  3. pheriday 2: termcasting overview

    pheriday 2: termcasting overview (2012-08-03) from Paul Ivanov on Vimeo.

    paul's habitual errant ramblings (on Fr)idays (2012-08-03)

    show notes:
    gopher://sdf.org/1/users/ivanov/pheridays/2012-08-03 (yes, gopher!)

    1. try to not say "uuuuhhhhmmnn"

    BAM/PFA Summer Cinema on Center Street http://bampfa.berkeley.edu/filmseries/summercinema

    1. SciPy 2012 videos up, go check them out! (I have!) http://www.youtube.com/nextdayvideo (removed nextdayvideo internal box url, by request)

    Software Carpentry: Record and Playback post http://software-carpentry.org/2012/07/record-and-playback/

    1. termcasting: a review of what's out there.

    http://termcast.org and http://alt.org/nethack/ mostly nethack stuff, both just use telnet protocol, only live sessions (though there are "TV" scripts to re-run sets of ttyrec files).

    (playterm vs ascii.io vs shelr.tv)

    tldr: playterm.org supports ttyrec files, but has the most primitive player. Players on ascii.io and shelr.tv can both seek. shelr.tv can also speed up playback! Downside is both of those have their own recorder programs (though at least shelr leverages script or ttyrec)

    http://playterm.org/ - supports ttyrec files, outgoing links for author and related article, comments. - most primitive player (https://github.com/encryptio/jsttyplay/) - Pause only - only terminal sized of 80x24 or 120x35 - supports tags and comments - service only (code for playterm.org does not seem to be available, though jsttyplay is doing the hardest part of actual playback)

    http://ascii.io/ - supports non-standard terminal size - player can seek. - aesthetic thumbnail previews - login via github or twitter credentials (for uploads) - code for website available (ruby and javascript) https://github.com/sickill/ascii.io - code for recorder available (python) https://github.com/sickill/ascii.io-cli

    http://shelr.tv/ - supports non-standard terminal size - player can seek. - player playback speed can be increased (currently up to 10x of real time) - supports tags, comments and voting up/down on a video - shelr can playback from command line ("shelr play http://shelr.tv/records/4f8f30389660802671000012.json") - code for website available (ruby and javascript) [AGPLv3] https://github.com/shelr/shelr.tv - code for recorder available (ruby) [GPLv3] https://github.com/shelr/shelr

    1. my wanted list for termcasting
    2. should support ttyrec files (upload and download)
    3. live-streaming (like ustream - but for coding)
    4. termcast.org has ttrtail which does just this
    5. quick "encrypt" switch - to keep streaming, but start GPG encrypting the stream as it goes out - so you can still look at it later. This would make it easy to leave the streaming on all the time
    6. a .tty editor that's like a video editor cut out portions [i.e. dead time]

    This is a low-bandwidth way of capturing what I'm working on and thinking about. Now, I'm going to try to record everything I do! "ttyrec -e screen -x". I've only done it a couple of times so far while coding, but I find being able to go back and re-view (and review) what I worked on at the end of the day to be really helpful.

    I was inspired by Joey Hess' "git-annex coding in haskell" where he reviews and narrates some of the code he wrote, after he wrote it. http://joeyh.name/screencasts/git-annex_coding_in_haskell/

    P.S. It's Saturday now. I tried to save some local diskspace by running recordmydesktop using the --on-the-fly-encoding option, and that was a mistake. The audio and video were (un)hilariously desynchronized - the audio ran for 9:48, but the video wanted to be just 7:30. Audacity came to the rescue by allowing me to change the tempo to be 30% faster, which made the syncing better. And then I used avconv to stitch in the faster audio.

    tools used: Debian GNU/Linux sid, recordmydesktop, xmonad, fbpanel, screen, chromium, cheese, xcompmgr, audacity, avconv

    read more
  4. pheriday 1: software carpentry, digital artifacts, visiting other OSes

    Here's pheriday 1, another edition of paul's habitual errant ramblings (on Fr)idays

    pheriday 1: software carpentry, digital artifacts, visiting other OSes (2012-07-27) from Paul Ivanov on Vimeo.

    2012-07-27.mp4 (28 MB) 2012-07-27.avi (71 MB) 2012-07-27.ogv (321 MB)

    show notes

    I had three topics I wanted to cover today, and ended up spending about an hour thinking about what I was going to say and which resources I was going to include. This was too long, and the end result was still very rambling, but I think I'll get better at this with more practice.

    SDF Public Access UNIX System http://sdf.org gopher://sdf.org/1

    1. try to not say "uuuuhhhhmmnn"
    2. be lazy (software carpentry)
    3. flipside of "publisher's block" (git-annex)
    4. visiting another country (windows 8 release preview)

    As usual, I didn't know what I was really trying to say in 0, and here's a really good overview of what I meant: Software Carpentry

    If you have 90 seconds, watch the pitch

    play it in 60 seconds, instead: mplayer -af scaletempo -speed 1.5 Software_Carpentry_in_90_Seconds-AHt3mgViyCs.flv

    1. publisher's block

    Jaron Lanier's You Are Not A Gadget What I mention in the video is not at all the main point of Lanier's book (which is quite good!), and in fact, his book is a critique of (over) digitization. Nevertheless, I'm only pointing out that there are redeemable aspects of an increasingly digital artifact producing life, such as preservation.

    David Weinberger’s Everything is Miscellaneous

    My review of David Weinberger’s Everything is Miscellaneous, where I go into more depth about "information overload".

    git-annex Excellent project. The technical details is that when you "annex" files, they are renamed to long hash of their contents (bit rot resistant!) and stored in a .git/annex/objects directory, whereas in place of where the file was, you get a symlink to the original file, which gets added to git. So git only keeps track of symlinks, and additionally has a git-annex branch that keeps track of all known annexes, so that you can copy, move, and drop files from the ones that are accessible. Very handy!

    Haiku OS

    tools used: Debian GNU/Linux sid, recordmydesktop, xmonad, fbpanel, screen, iceweasel, cheese, xcompmgr, youtube-dl, mplayer, screen

    gopher version of this post (proxy)

    read more
  5. pheriday 0: scientist-hacker howto (video post)

    Hey everyone, here's pheriday 0, the first of paul's habitual errant ramblings (on Fr)idays

    pheriday 0: scientist-hacker howto (2012-07-20) from Paul Ivanov on Vimeo.

    Berkeley Kite Festival (510 Families)

    Merlin Mann's Most Days (specifically the travel day one on 2009-01-11)

    Sad that I missed SciPy Conference this year. One of the things I like doing at scipy is nerding it up with my friends, seeing each others workflows, showing off vim tricks, etc. This video was my attempt at scratching that itch, a little bit. As I mention in the video, this is take 2. Take 1 ended when I ran out disk space, but needless to say, it was more awesome than this. It seems I am cursed with losing first takes, see also a summary of last year's SciPy conference, where this exact same thing happened.

    NumFOCUS: NumPy Foundation for Open Code for Usable Science

    NumFOCUS Google Group see thread titled: "[Funding] Notes from Funding BOF at SciPy2012"

    TLDP: The Linux Documentation Project (page I was scrolling through)

    Transition to Gopher was rough this time, it was better during the first take.

    Lorance Stinson's w3m (better) gopher support Use this if, for example, going to w3m gopher://sdf.org you get errors like:

    [unsupported] '/1' doesn't exist! [unsupported] This resource cannot be located.

    It still took some tweaking, shoot me an email for details

    Robert Bigelow's About | Gopher & GopherSpace

    Here's the HTTP Proxied version of the above: Gopher proxy provided by Floodgap

    SDF Public Access UNIX System http://sdf.org gopher://sdf.org/1

    Eric S. Raymond's How To Become a Hacker Howto

    Fernando Perez' Py4Science Starter Kit

    Q: Why are you using "Chromium --incognito"? I have chronic tabitis, and this is one way of mitigating that problem. If the browser crashes or I shutdown my computer, I won't have those tabs around anymore.

    programs used: Debian GNU/Linux sid, recordmydesktop, xmonad, fbpanel, screen, chromium, cheese, xcompmgr, mutt, wyrd, tail, w3m

    gopher version of this post (proxy)

    read more
  6. Ada Lovelace Day: remembering Shirley Theis and Evelyn Silvia

    In case you didn't know it - today is Ada Lovelace Day!

    Now, as any self-respecting Computer Science degree-wielding person should, I, too, think it's important to celebrate the day named after the world's very first programmer.

    For me, the first math teacher I remember making a big difference was Shirley Theis - who taught me Algebra in 8th grade at McKinley Middle School in Redwood City, CA. Mrs Theis, an energetic dynamo in her mid fifties, was a deeply motivated and caring teacher, who expected a lot out of her students, but never in a disciplinary manner. She was full of enthusiasm, which projected out and infected even the most timid or disaffected student: in her class, you couldn't be just a sack of potatoes planted in your seat.

    She often lead class in a nearly theatrical manner - pacing back and forth, egging students on by eagerly repeating their partial responses, getting exponentially more excited if the student was on the right track, barely containing herself from jumping up and down in anticipation of that lightbulb going off -- and yet just as quickly waning in her enthusiasm,becoming a personified caricature of hopelessness and despair to let you know the instant a response was starting to go astray.

    It may have been the only math class I've ever taken where there were group assignments - we would work with a partner or a few classmates in trying to figure out an assignment, first trying it solo, and then putting our heads together to figure out why our answers disagree and which is the right one. I believe it was Mrs. Theis who succinctly captured a value I hold in high regard: "it's not about how far you go - it's about how many people you bring with you."

    There was one other mathematics teacher I had in my life who clearly stands out: it was Professor Evelyn Silvia who had a comparable level of enthusiasm and energy, and from whom I had the pleasure of taking the first upper-division math course (Math 108 - Intro to Abstract Math) during my second quarter at UC Davis. Dr. Silvia was the real deal - she cared, gesticulated, encouraged us to question why something was true, and had an approach which demanded we each take ownership of our education. The book for the course, Introduction to Abstract Mathematics: A Working Excursion by D.O. Cutler and E.M. Silvia was a blue workbook - each of us had our own copy, and there were blanks left out for us to write our own answers to the exercises. The fact that the book had blanks for me to fill in was so inviting, there was a kind of "working mathematician" approach that came with it with that it made me really enjoy and look forward to working through the material. I still have mine.

    Dr. Silvia was incredibly sharp, not just intellectually but also interpersonally. Not only could she gauge when the class was lost, but she also had a knack for spotting if something was affecting you outside of class. She was really committed to helping you not just as a student, but as a person. I remember spending hours at Mishka's, or Cafe Roma, or the CoHo, reading and writing, wanting to do well and not let Silvia down, because she invested so much energy and placed a great deal of trust in us.

    So thank you both, Shirley Theis and Evelyn Silvia - you both encouraged me to grow a lot as a person, challenged my concept of what it means to be a student, and by your example provided a template of what it means to be an effective teacher, which I've imitated and embraced with pleasure in my own teaching.

    (tagged scipy to spread word of Ada Lovelace day to Planet SciPy)

    read more
  7. vim-ipython two-way integration! (updated: 2011-08-02)

    I'm very pleased to share with you a demo the forthcoming vim-ipython integration which will work with IPython 0.11(trunk).

    You can either use the Flash player below, or download the OggVorbis file (14MB) update: vim-ipython 'shell' demo (9.6MB). The blog-free form of this post is here.

    If you like what you see and want to try it, you can get the details from the vim-ipython github page and it currently requires 4 line changes to IPython, which are currently in this pull request. (Fixed to work on IPython trunk with no changes).

    Big thanks to Min for walking me through the new IPython kernel manager during the SciPy2011 sprints.

    UPDATE: 2011-08-02

    vim-ipython ‘shell’ mode.

    Just in case, here are the same videos as above, but hosted on Youtube:

    If you're have any issues, try searching for your error on the vim-ipython github issues page, and if you don't find it, please file a new one, and I'll help you out there.

    read more
  8. Money and CA Propositions

    Since tomorrow we'll be having another one of those practice democracy drills here in California, I thought I'd put together a few bar charts.

    There are five propositions on tomorrow's ballot. In researching them, Lena came across the Cal-Access Campaign Finance Activity: Propositions & Ballot Measures.

    Unfortunately, for each proposition, you have to click through each committee to get the details for the amount of money they've raised and spent. Here's a run-down in visual form, the only data manipulation I did was round to the nearest dollar. Note: no committees formed to support or oppose Proposition 13.

    Here's how much money was raised, by proposition:


    Just in case you didn't get the full picture, here is the same data plotted on a common scale:

    Money Raised (common

    And the same two plots for money spent ((I don't fully understand what these numbers mean, as some groups' "Total Expenditures" exceed their "Total Contributions" and still had positive "Ending Cash")):

    Money Spent

    Money Spent (common scale)

    It could just be my perception of things, but I get pretty suspicious when there's a ton of money involved in politics, especially when it's this lopsided.

    The only thing I have to add is you should Vote "YES" on Prop 15, because Larry Lessig says so, and so do the Alameda County Greens!

    Update #1: Let me write it out in text, so that the search engines have an easier time finding this. According to the official record from Cal-Access (Secretary of State), as of May 22nd, 2010, there were $54.4 million spent in support of various propositions, most notably $40.5 million on Prop 16, $8.9 million on Prop 17, and $4.6 million on Prop 14. Compare that with a "grand" total of less than $1.2 million spent to oppose them, with a trivial $78 thousand (!!) to oppose Prop 16's $40.5 million deep pockets.

    Update #2: The California Voter Foundation included more recent totals (they don't seem to be that different), as well as a listing of the top 5 donors for each side of a proposition in their Online Voter Guide.

    Also, here's the python code used to generate these plots (enable javascript to get syntax highlighting):

    # Create contributions and expenditures bar charts of committees supporting and
    # opposing various propositions on the California Ballot for June 8th, 2010
    # created by Paul Ivanov (http://pirsquared.org)
    # figure(0) - Contributions by Proposition (as subplots)
    # figure(1) - Expenditures by Proposition (as subplots)
    # figure(2) - Contributions on a common scale
    # figure(3) - Expenditures on a common scale
    import numpy as np
    from matplotlib import pyplot as plt
    import locale
    # This part was done by hand by collecting data from CalAccess:
    # http://cal-access.sos.ca.gov/Campaign/Measures/
    prop = np.array([
         4650694.66, 4623830.07    # Yes on 14 Contributions, Expenditures
        , 216050, 52796.71         # No  on 14 Contributions, Expenditures
        , 118807.45, 264136.30     # Yes on 15 Contributions, Expenditures
        , 200750.01, 86822.79      # No  on 15 Contributions, Expenditures
        , 40706258.17, 40582036.58 # Yes on 16 Contributions, Expenditures
        , 83187.29, 78063.91       # No  on 16 Contributions, Expenditures
        , 10328675.12, 8932786.06  # Yes on 17 Contributions, Expenditures
        , 1229783.79, 965218.48    # No  on 17 Contributions, Expenditures
    prop.shape = -1,2,2
    def currency(x, pos):
        """The two args are the value and tick position"""
        if x==0:
            return "$0"
        if x < 1e3:
            return '$%f' % (x)
        elif x< 1e6:
            return '$%1.0fK' % (x*1e-3)
        return '$%1.0fM' % (x*1e-6)
    from matplotlib.ticker import FuncFormatter
    formatter = FuncFormatter(currency)
    yes,no = range(2)
    c = [(1.,.5,0),'blue']  # color for yes/no stance
    a = [.6,.5]             # alpha for yes/no stance
    t = ['Yes','No ']       # text  for yes/no stance
    raised,spent = range(2)
    title = ["Raised for", "Spent on" ] # reuse code by injecting title specifics
    field = ['Contributions', 'Expenditures']
    footer ="""
    Data from CalAccess: http://cal-access.sos.ca.gov/Campaign/Measures/
    'Total %s 1/1/2010-05/22/2010' field extracted for every committee
    and summed by position ('Support' or 'Oppose').  No committees formed to
    support or oppose Proposition 13. cc-by Paul Ivanov (http://pirsquared.org).
    """ # will inject field[col] in all plots
    color = np.array((.9,.9,.34))*.9 # spine/ticklabel color
    plt.rcParams['savefig.dpi'] = 100
    def fixup_subplot(ax,color):
        """ Tufte-fy the axis labels - use different color than data"""
        spines = ax.spines.values()
        # liberate the data! hide right and top spines
        [s.set_visible(False) for s in spines[:2]]
        ax.yaxis.tick_left() # don't tick on the right
        # there's gotta be a better way to set all of these colors, but I don't
        # know that way, I only know the hard way
        [s.set_color(color) for s in spines]
        [s.set_color(color) for s in ax.yaxis.get_ticklines()]
        [s.set_visible(False) for s in ax.xaxis.get_ticklines()]
        [(s.set_color(color),s.set_size(8)) for s in ax.xaxis.get_ticklabels()]
        [(s.set_color(color),s.set_size(8)) for s in ax.yaxis.get_ticklabels()]
    # for subplot spacing, I fiddle around using the f.subplot_tool(), then get
    # this dict by doing something like:
    #    f = plt.gcf()
    #    adjust_dict= f.subplotpars.__dict__.copy()
    #    del(adjust_dict['validate'])
    #    f.subplots_adjust(**adjust_dict)
    adjust_dict = {'bottom': 0.12129189716889031, 'hspace': 0.646815834767644,
     'left': 0.13732508948909858, 'right': 0.92971038073543777,
     'top': 0.91082616179001742, 'wspace': 0.084337349397590383}
    for col in [raised, spent]: #column to plot - money spent or money raised
        # subplots for each proposition (Fig 0 and Fig 1)
        f = plt.figure(col); f.clf(); f.dpi=100;
        for i in range(len(prop)):
            ax = plt.subplot(len(prop),1, i+1)
            p = i+14    #prop number
            for stance in [yes,no]:
                plt.bar(stance, prop[i,stance,col], color=c[stance], linewidth=0,
                        align='center', width=.1, alpha=a[stance])
                lbl = locale.currency(round(prop[i,stance,col]), symbol=True, grouping=True)
                lbl = lbl[:-3] # drop the cents, since we've rounded
                ax.text(stance, prop[i,stance,col], lbl , ha='center', size=8)
            ax.xaxis.set_ticklabels(["Yes on %d"%p, "No on %d"%p])
            # put a big (but faded) "Proposition X" in the center of this subplot
            common=dict(alpha=.1, color='k', ha='center', va='center', transform = ax.transAxes)
            ax.text(0.5, .9,"Proposition", size=8, weight=600, **common)
            ax.text(0.5, .50,"%d"%p, size=50, weight=300, **common)
            ax.yaxis.set_major_formatter(formatter) # plugin our currency labeler
            ax.yaxis.get_major_locator()._nbins=5 # put fewer tickmarks/labels
        f.subplots_adjust( **adjust_dict)
        # Figure title, subtitle
        extra_args = dict(family='serif', ha='center', va='top', transform=f.transFigure)
        f.text(.5,.99,"Money %s CA Propositions"%title[col], size=12, **extra_args)
        f.text(.5,.96,"June 8th, 2010 Primary", size=9, **extra_args)
        extra_args.update(va='bottom', size=6,ma='left')
        f.text(.5,0.0,footer%field[col], **extra_args)
        f.set_figheight(6.); f.set_figwidth(3.6); f.canvas.draw()
        # all props on one figure (Fig 2 and Fig 3)
        f = plt.figure(col+2); f.clf()
        adjust_dict.update(left= 0.06,right=.96)
        f.subplots_adjust( **adjust_dict)
        extra_args = dict(family='serif', ha='center', va='top', transform=f.transFigure)
        f.text(.5,.99,"Money %s CA Propositions"%title[col], size=12, **extra_args)
        f.text(.5,.96,"June 8th, 2010 Primary", size=9, **extra_args)
        extra_args.update(ha='left', va='bottom', size=6,ma='left')
        f.text(adjust_dict['left'],0.0,footer%field[col], **extra_args)
        ax = plt.subplot(111)
        for stance in [yes,no]:
            lbl = locale.currency(round(prop[:,stance,col].sum()),True,True)
            lbl = lbl[:-3] # drop the cents, since we've rounded
            lbl = t[stance]+" Total"+ lbl.rjust(12)
            plt.bar(abscissa,prop[:,stance,col], width=.1, color=c[stance],
                    alpha=a[stance],align='center',linewidth=0, label=lbl)
            for i in range(len(prop)):
                lbl = locale.currency(round(prop[i,stance,col]), symbol=True, grouping=True)
                lbl = lbl[:-3] # drop the cents, since we've rounded
                ax.text(abscissa[i], prop[i,stance,col], lbl , ha='center',
        ax.xaxis.set_ticklabels(["Proposition %d"%(i+14) for i in range(4)])
        # plt.legend(prop=dict(family='monospace',size=9)) # this makes legend tied
        # to the subplot, tie it to the figure, instead
        handles, labels = ax.get_legend_handles_labels()
        l = plt.figlegend(handles, labels,loc='lower right',prop=dict(family='monospace',size=9))
        ax.yaxis.set_major_formatter(formatter) # plugin our currency labeler
    read more

« Page 2 / 2