Other articles


  1. pheriday 3: infrastructure

    paul's habitual errant ramblings (on Fr)idays

    pheridays: 3

    2020-04-10: A week ago, I recorded a 5 minute audio segment of some stuff I've been thinking about, but when I started to write it up I stumbled into and kept dropping down a deep technostalgic hole.

    fall down along with me:

    https://pirsquared.org/blog/pheriday-infrastructure.html

    The recording is just shy of five minutes long, you can also download it in different formats, depending on your needs, if the audio tag above doesn't suit you:

    https://pirsquared.org/pheridays/2020-04-03.ogg (2.9 Mb)
    https://pirsquared.org/pheridays/2020-04-03.mp3 (4.5 Mb)
    https://pirsquared.org/pheridays/2020-04-03.m4a (6.3 Mb)

    --

    Stuff I mentioned in the audio:

    Propellor - "configuration management system using Haskell and Git" by Joey Hess

    OpenWRT - specifically - reducing Bufferbloat

    Mumble - "a free, open source, low latency, high quality voice chat application."

    sourcehut.org - "the hacker's forge" also know as sr.ht by Drew DeVault

    Jitsi - "Multi-platform open-source video conferencing"

    OpenFire - "real time collaboration (RTC) server licensed under the Open Source Apache License." Extensible XMPP server, with plugins, like a Jitsi-based video meeeting one claled OpenFire Meetings.


    Though this is the fourth installment, the last time I recorded and posted a rambling was back almost 8 years ago! In fact, it was 2012-08-03, so 7 years and 8 months, to the day.


    Having control of your infrastructure is a longtime thread for me.

    For starters - there's the bicycle. That's been my primary and preferred mode of transportation for 30 years. As a kid, I was empowered by the sense of freedom, independence, and self-sufficiency that came with a bike. All these years later, I'm still a fan. You can see just how happy I am on a bike at the top of this interview , thanks to a sweet photo that was taken by Robert Sexton right by the Golden Gate Bridge at the end of the Lucas Valley Populaire in 2015.

    Those of you who knew me back in college might remember how at UC Davis I ran my own "pirate" internet radio station - KPVL - with the cheeky tagline of "More broadcasters than listeners". (I say "pirate" because it has not relation to the actual KPVL radio station). But there are earlier remnants and traces of my efforts to exercise control and build my own reality.

    I think it was in 1999 that my brother Mike and I started using Redhat (6), then Mandrake Linux 6.5, dual booting on a computer at home and I separately around the same time I got myself an sdf.org account. Though I wasn't sophisticated enough to have a constant internet connection in high school, I was lucky enough to get an account on Robert Chin's laya.com server. The url was - p.laya.com - it's long gone, but luckily, Archive.org has a copy from 2001.

    Wow. I just took a look and so much came flooding back.

    Here's the thing: April is an anniversary of sorts for me. Back in 1999, it marks my first time breaking anonymity and pseudonimity and using my real name on the internet. I've written about this before under the title of Publisher's block ten years ago - just about half way between now and then. This time, though, let me inline the piece I linked to as proof of the deliberate nature of my lack of anonymity.

    An account of my life at 15, as I live it.
    traces of my awareness of the world, I can look back at later
    My first attempt at a memoir
    My goal is to capture my many thoughts emotions, behaviors, incidents, and
    acquaintances
    and to arrive only at an exponential number of those,
    hoping yet being afraid that it might be zero
    making everything about me: one
    

    I'm glad I can now reflect on the kind of kid I was, thanks to the amazing folks who had the foresight to start archiving all of the web for The Way Back Machine. I used that p.laya.com page as a todo list and notes for myself using a hipster combination of the default file index listing with a FOOTER.html. It was captured in 2001, I was in 17, but some of this was written when I was 15 or 16 (I found contents from November 2000), I make mention to my then freeshell.org account (it's now been ivanov@ since 2012). I link to the source code of a MUD - ftp://ftp.game.org/pub/mud/diku/merc/rom/tartarus/tartarus.tgz - which is a broken link now, but I found a mirror over here: which is amazing, because just a week or two ago, I was hanging out with fellow SciPy 2020 program co-chairs Madicken Munk and Gil Forsyth over video chat after one of our meetings and I was happily reporting about how one of the positive things to come out of the shelter in place for me is that "I've fixed my mutt configuration and started using it again!" - but they both heard "mutt" as "MUD" and got very excited by that prospect. So much so that we all agreed that we'll have to follow up and actually follow through to build a MUD. And I brought up how at some point in high school I was mildly active in a pair of MUDs, and wanted to make my own, but never got around to it.

    The last link I left for myself on there points to pinkmonkey.com - a homeschooling resource - which is probably handy for the parents with little ones these days.

    Here's the most concrete infrastructure project I can find from then: I collected bookmarks from my friends to share them. The "service" lived at http://p.laya.com/bookmarks - and predates del.icio.us and pinboard. I bet I "advertised" it in my AIM profile.

    If you're curious, there's a link to the archive.org copy near the end of this post, but I had this urge to show it to you much closer to its original glory.

    Let me set the scene: It's Friday in April, the year is 2020, I'm running Windows 10 on my work laptop in poorly connected home in California, where a pandemic has most of the state's residents staying put at home for the several weeks already, and I decided to make a screenshot using the tool du jour of yesteryear

    Netscape Navigator!

    The timestamp on my bookmark website says I last updated it on: Thu Aug 30 20:14:14 PDT 2001

    OldVersion.com tells me that the latest release for Windows that Netscape 4.79 was released in November of that year, and the closest antecedent version available is 4.72 (from February 2000).

    I downloaded it and tried fiddling around with the compatibility settings, but without any luck.

    Then I tried 4.79, and nope, that didn't work, either. So then I tried Netscape 6.01 - release February 2001.

    I happened to have Chrome running at the time because in Firefox I have 1500 tabs open -- fifteen hundred and seven! ;) -- whereas in Chrome it's under 500, so I was trying to tread lightly. How do I know these numbers? For Chrome I found an extension that allows me to copy into the clipboard all open tabs' urls as plain text. It helpfully announces how many such tabs were copied. In Firefox one of the webextension examples gives you a counter.

    Do you remember the web without tabs? Time was, you wanted to visit another webpage, you got two option: you navigate away from whatever you're looking at now, or you hit Ctrl-N to make an new window. I think most people used one or a few windows. But you were not gonna be crazy and open more than a dozen windows. I would have, and probably tried but I couldn't. And session saving across crashes or clean exits? Forget it! That what your history and bookmarks are for, grasshopper.

    But let's get back to the task at hand: this was the lower right of my screen...

    Downloaded Netscape601.exe

    and I decide to start taking screenshots of this journey, click it, and let Windows 10 apply the compatibility settings, and then I'm faced with

    the most improbable error message:

    Setup detected another instance of Netscape 6 is currently running

    :)

    WAT?!

    I didn't think Chrome had any ancestry shared with the Mosaic super-tree, but whatever - you can't exactly argue with software from 2001, and I have an important screenshot to take...

    So now I've quit Chrome, just in case, and going to retry....

    Setup detected another instance of Netscape 6 is currently running

    no dice....

    Damn, what could it be...I've got the Bloomberg Terminal open, I know portions of it are built on Chromium browser technology (had to look it up if this was officially stated somewhere - it is). Ok, so maybe that's what causing the false positive? I close that, and...

    Setup detected another instance of Netscape 6 is currently running

    ...

    I hardly have anything open anymore ...is it VLC?

    ...

    Setup detected another instance of Netscape 6 is currently running

    nope... Ok, what's left still open... Snipping tool I'm using to capture this epic adventure, a few WSL Debian console windows... the voice recorder that started this post... Task manager and Sysinternals' Process Explorer - where I was checking if perhaps somehow the failed attempt at running what was probably a 16 bit version of Navigator 4.72 was still lingering somewhere... SumatraPDF, Windows Terminal (Preview), gVim, and ...

    Zotero?!?

    BINGO!!!!!!!!!

    Initial Netscape 6 Setup screen

    Initial Netscape 6 Setup screen

    Oh right - I guess Zotero uses XUL technology. I didn't really think much about it, but Zotero did start off life as a Firefox extension, and the standalone version came out later, makes sense that it would have grabbed a browser when it struck out on its own.

    At this point I had already sent Madicken, who works at NCSA where Mosaic, the progenitor of Netscape hails from, the first two images... So I wanted to play with fire a bit....

    Now that I've closed Zotero - can I have Firefox 74 open while installing Netscape 6.01?

    Setup detected another instance of Netscape 6 is currently running

    Rats! same error...

    how about Chrome again?

    Initial Netscape 6 Setup screen

    Initial Netscape 6 Setup screen

    oh yeah! Sweet. The world makes sense again.

    Back to the setup.exe...

    I scroll through the EULA - and randomly stop on this section:

    12. HIGH RISK ACTIVITIES.  The Product is not
    fault-tolerant and is not designed, manufactured or
    intended for use or resale as on-line control
    equipment in hazardous environments requiring
    fail-safe performance, such as in the operation of
    nuclear facilities, aircraft navigation or
    communication systems, air traffic control, direct
    life support machines, or weapons systems, in which
    the failure of the Product could lead directly to
    death, personal injury, or severe physical or
    environmental damage ("High Risk Activities").
    

    By the way - here's a good idea I came across a few months ago: throw EULAs (End User License Agreements) into some publicly indexed version control repo (I saw folks using gists for just that sort of thing: here's the Netscape 6.01 EULA.txt)

    Let the folks at Redmond host it.

    Fine. I click next...

    Interesting - there's a "Read Me" button... I do as I'm told, so I click it.

    Oh, a README.txt popped up in Notepad.exe - full contents of that are in that same gist as the EULA

    but in them, there's a link to the full release notes over at

    http://home.netscape.com/eng/mozilla/ns6/relnotes/6.0.html

    Let's try to go there now:

    "Yahoo! will be right back... Thank yo for your patience."

    Wat?!

    Yeah, I'm sure Yahoo! engineers are jumping right on that.

    How did a Netscape site ended up redirecting to Yahoo...

    oh right, so AOL bought Netscape (1999), merged with Time Warner in 2001, was spun out again in 2009, after some rough times, and then purchased by Verizon in 2015. In the meantime, Yahoo acquired Geocities (1999) and shut it down in 2009 (yes, I'm still mad! All I remember was that it had some crispy banners, one of which was a scan of a sweet pencil lettering I made of my nick at the time - "ShadowKnight"). No one really cares what happened to Yahoo in the interim, aside from some massive data breaches, until finally, Verizon bought Yahoo in 2017 and merged AOL and Yahoo into one division.

    Ok, so fine, maybe fine folks at The Internet Archive (archive.org) can help us with the WayBackMachine.

    And indeed we can see what it looked like originally. This site has seen so many redirects over the years - it'd be a fun exercise to go through all of the indexed versions of this kind of site to see how people tried to preserve links. For example, I found out in 2010 it 301s ("Temporary redirect") to http://www.netscape.com/eng/mozilla/ns6/relnotes/6.0.html which then 302s ("Permanent redirect") to http://www.propeller.com/eng/mozilla/ns6/relnotes/6.0.html - which was indexed but happens to be a 404 ("Page not found") error page, at least in 2008. But this wasn't what we came here for, so this yak can roam free among the hills, the valleys, and the caverns of our minds.

    ... for now...

    Where were we?

    Oh right, we have to choose an install option. Back in the day I might have clicked "recommended" here, but we're not back in the day, and who wants to play life on easy mode? Let's go custom to see what the options are (I am a control freak, after all)

    Custom Install Option selected

    and immediately get another pop up:

    "Create install directory?" prompt

    Whoa, let's pause here for a moment.

    I am digging such a consent model. The installer is establishing trust: it will not try to do anything behind my back and without my permission. I'm sure it will never abuse that trust. As a 2001 user, I sure am glad that folks involved with computing have such a well-develop sense of ethics. In 2001, the Future is bright. Computing will be filled with transparency. Consumer software and services will be built by folks with a strong moral compass. These are people with principles. With the dot-com bubble burst, we've swiftly inoculated tech from sleazy opportunists. It won't fall victim to the excesses and greed rivaling Wall Street in the 80s...

    Which reminds me - how is it that this brilliant video only has 277 thousand views? Here's a excerpt:

    Your users won't always understand just how much economic sense it makes to sell them out. And you don't want to alienate them, that would drive down their value.

    By the way - the embed code I used above uses the youtube-nocookie.com domain - which is still from the folks at Goolag, but does what it says on the tin and doesn't issue cookies. Also, did you know there used to be a way to disable those annoying related video links from popping up at the end of the video? It's true. You used to be able to just append a rel=0 query parameter to not show related videos. But the corporate overlord bean counters didn't like that. They had to make sure that kids would get glued to the site by feeding them progressively conspiratorial garbage content. So that watching any video would nearly guarantee to pull them into the black hole cesspool of maximally "engaging" "content". What was that quote about users again? Ah yes:

    Your users won't always understand just how much economic sense it makes to sell them out. And you don't want to alienate them, that would drive down their value.

    Ok, so, in fairness, rel=0 query parameter still does something. It limits the suggested videos to the channel they are from. That's good. But what happens if we follow one of those links? First, we end up on the full youtube site, so that means the cookies are back. Hurray for surveillance capitalism! Also, the recommendations on the right are curated specifically for us, and not limited to the channel the previous video was one. Goodbye, rel=0!

    So I have to install Navigator, but I can unselect Mail, Instant Messenger, and Spell Checker... I don't need mail, but whatever, let's just go with the defaults.

    Oh, look at all this wonderful bundled crapware. Just in case you had any doubts.

    That last one made me throw up a little in my mouth.

    I opt for just the classic skin - and it tells me that the total download size will be 9959 K.

    I thought about censoring this next screenshot, but 15 year old me wouldn't have like that... What's in the shot is in the shot..

    Alright, and when the installation finished here's what we're greeted with:

    Did software in 2001 try to phone home? "activation.netscape.com could not be found." sure seems so.

    Yes, No, Cancel?

    What if I cancel?

    Alright, let's go for broke and get that retro look...

    Remember all those redirects? Well the browser froze when I got overzealous, clicked on "Interact" at the bottom there and chose to open chat... And on the next load, it crashed... And again...

    Netscape 6 Frozen on Yahool Will be be right back page

    Netscape 6 Frozen on Yahool Will be be right back page It just kept crashing...

    In case anyone else gets stuck on the same issue ;) I got around this by using the Profile Manager, where I had the option to start the browser in Work Offline mode. Then I turn the "Work Online" option on after the browser loaded (which you can do by plugging together that cute outlet pair on the bottom left).

    I do some ego surfing and go to my own site first.

    http://pirsquared.org

    pirsquared.org using Netscape 6

    pirsquared.org using Netscape 6

    I got too fancy with my unicode... But hey, this is totally functional.

    I made this Loading gif via a screen capture tool and then it finally clicked that not only did I not use Netscape 6 - I remember most everyone's experience was to stick to the 4.x series, because it was so much more usable and not bloated with nonsense, etc, etc.

    Alright, but at least I got 6 to run on Windows 10 and that works...When it doesn't crash, anyway... But I did get a error about youtube-nocookie.com... (some of the time, at least)...

    And then I realized that I can't go to any site that has https... Because... you know, the protocol that provides that 's' has changed over the years, and our 2001 browser could do SSL 2 or 3 or TLS 1.0... But my website uses TLS 1.3...

    I couldn't run to duckduckgo, either, since it redirects plain http to the https endpoint and that also runs TLS 1.3... I couldn't even go to archive.org to view my old site directly on the way back machine, because archive.org run TLS 1.2 at the moment.

    It's difficult to find any place that still runs such outdated standards... I tried to search for just a TLS 1.0 test server - but didn't find anything suitable... But then I happen to flip through the recent changelog for Firefox:

    Firefox 74 Reverting deprecation of TLS 1.0 and 1.1

    Cool - so now we know if we want to find TLS 1.0 and 1.1 website, we should turn to the government of... damn a specific country wasn't specified...

    But wait a minute... Firefox 74.0 came out on March

    How did Mozilla release an update to a version of Firefox that was in the hands of a bunch of users without... umn...what's the word I'm looking for here...you know, that thing no one seems to think is a thing anymore... user consent?

    How Mozilla released an update without user consent

    https://wiki.mozilla.org/Firefox/Normandy/PreferenceRollout

    This is the way consent ends
    This is the way consent ends
    This is the way consent ends
    Not with a bang but a whimper
    

    I'm late to the party - this has been going on for about three years - with what I now recall was caused a bit of a splash back in 2017 (Drew DeVault covers in "Firefox is on a slippery slope"). But I didn't know the extent of it. Who has the time to pay attention to the way in which all the software they use changes in anti-social ways.

    Anyway, if you don't want the fine folks at Firefox to change your preferences out from under you, I think you go to about:config and switch the app.normandy.enabled setting to false. And if you're interested in specifics of how you've been a guinea pig: about:studies will tell you. And you can go to about:preferences#privacy to disable them.

    But I digress...

    Let's wrap this up...

    At this point, I used my 2020 browser to grab historical snapshot of my old bookmarks site, stripping off the tastefully annotated Way Back Machine user interface insertions, and serve it locally over http via python -m http.server, making sure to change the URL bar to make a historically accurate re-enactment. And now that you know how I got here, you can fully appreciate the effort that went into this next screenshot:

    My bookmarks Gallore site in Netscape 6

    My bookmarks Gallore site in Netscape 6

    So much so, that I couldn't resist making a video a scroll through :

    You can download it as webm, mp4, ogg, or gif.

    If you're interested - you can go http://p.laya.com/bookmarks/ (archival copy) or see the rest of p.laya.com from 2001

    That's all


    The ongoing crisis has been a circuit breaker to our usual patterns. I am taking advantage of this affordance to experiment with and establish channels of communications that are not controlled by others.


    permalink
  2. SciPy 2018 dates and call for abstracts

    I'm helping with next year's SciPy conference, so here are the details:

    July 9-15, 2018 | Austin, Texas

    Tutorials: July 9-10, 2018
    Conference (Talks and Posters): July 11-13, 2018
    Sprints: July 14-15, 2018
    

    SciPy 2018, the 17th annual Scientific Computing with Python conference, will be held July 9-15, 2018 in Austin, Texas. The annual SciPy Conference brings together over 700 participants from industry, academia, and government to showcase their latest projects, learn from skilled users and developers, and collaborate on code development. The call for abstracts for SciPy 2018 for talks, posters and tutorials is now open. The deadline for submissions is February 9, 2018.

    Talks and Posters (July 11-13, 2018)

    In addition to the general track, this year will have specialized tracks focused on:

    • Data Visualization
    • Reproducibilty and Software Sustainability

    Mini Symposia

    • Astronomy
    • Biology and Bioinformatics
    • Data Science
    • Earth, Ocean and Geo Science
    • Image Processing
    • Language Interoperability
    • Library Science and Digital Humanities
    • Machine Learning
    • Materials Science
    • Political and Social Sciences

    There will also be a SciPy Tools Plenary Session each day with 2 to 5 minute updates on tools and libraries.

    Tutorials (July 9-10, 2018)

    Tutorials should be focused on covering a well-defined topic in a hands-on manner. We are looking for awesome techniques or packages, helping new or advanced Python programmers develop better or faster scientific applications. We encourage submissions to be designed to allow at least 50% of the time for hands-on exercises even if this means the subject matter needs to be limited. Tutorials will be 4 hours in duration. In your tutorial application, you can indicate what prerequisite skills and knowledge will be needed for your tutorial, and the approximate expected level of knowledge of your students (i.e., beginner, intermediate, advanced). Instructors of accepted tutorials will receive a stipend.

    Mark Your Calendar for SciPy 2018!

    permalink
  3. SciPy 2017 tips

    After missing it for a couple of years, I am happy to be back in Austin, TX for SciPy this week!

    Always invigorating and exhilarating, Scientific Computing with Python (SciPy) has remained a top quality venue for getting together with fellow Pythonistas, especially the academically-bent variety.

    As a graduate student eight years ago, I was fortunate enough to be one of receive sponsorship and attended my first SciPy - SciPy 2009. This was the last time it was held at CalTech in Pasadena, CA.

    The following year, in 2010, at the first SciPy held in its now usual spot in Austin, TX, each attendee got a bottle of delicious salsa!

    SciPy2010 Salsa Stack

    Here are some oy my thoughts about attending this wonderful conference.

    Conference Tips

    bring a sweatshirt -- Yes, I know Austin's hot, but at the AT&T center, they don't mess around and crank the air conditioning all the way up to 11!

    join the slack group -- This year, there's a Slack group for SciPy: the link to join is in a pair of emails with the titles "Getting the most out of SciPy2017" and "Getting the most out of SciPy2017-UPDATED", both from SciPy2017 Organizers. So far at the tutorials slack has served as a useful back channel for communicating repo URLs and specific commands to run, signaling questions without interrupting the speakers' flow.

    engage with others during the breaks, lunch, etc -- There are lots of tool authors here and we love chatting with users (and helping you become contributors and authors yourselves). Not your first SciPy and feeling "in-your-element"? Make the effort to invite others into the conversations and lunch outings you're having with old friends - we're all here because we care about this stuff.

    take introvert breaks (and be considerate of others who may be doing the same) - I'm an introvert. Though I enjoy interacting with others (one-on-one or in small groups is best for me), it takes a lot of energy and at some point, I run out of steam. That's when I go for a walk, stepping away from the commotion to just have some quiet time.

    be kind to yourself (especially at the sprints) -- Between the tutorials, all of the talks, and the sprints that follow, there will be a flurry of activity. Conferences are already draining enough without trying to get any work done, just meeting a bunch of new people and taking in a lot of information. It took a lot of false starts for me to have productive work output at sprints, but the best thing I've learned about them is to just let go of trying to get a lot done. Instead, try to get something small and well defined done or just help others.

    Stuff to do in Austin

    The conference already has a great list of Things to do in Austin, as well as Restaurants, so I'll just mention a few of my personal favorites.

    Barton Springs Pool. Take a nice dip in the cool waters, and grab a delicious bite from one of the food trucks at The Picnic food truck park.

    Go see the bats. The Congress Ave bridge in Austin is home to the largest urban bat colony in the world. You can read more about this here, but the short version is that around sunset (8-9pm) - a large number of bats stream out from underneath the bridge to go feed on insects. Some days, they leave in waves (this Saturday there were two waves, the first was smaller, but many people left thinking that was the entire show).

    I hope you enjoy SciPy2017!

    permalink
  4. starting my job search

    I am starting to look for a job in the San Francisco Bay Area.

    Since many recruiters ask for and presumably look at GitHub profiles, I decided to give mine a little facelift:

    Smart and Gets Things Done Github Contribution
Graph:

    In case you aren't familiar, that banner was motivated by Joel Spolsky's Smart and Gets Things Done, which is a book about hiring good developers . So I decided to tweet it out, mentioning @spolsky and he favorited it!

    Yesterday, I decided to tweet out an image that's at the top of my resume as a standalone tweet- mentioning Joel Spolsky again, and he liked it well enough to retweet it to his 90 thousand followers, so it's been getting plenty of love.

    Paul Ivanov's Visual Resume

    Perhaps unsurprisingly, the only person to contact me as a result of this so far is a reporter from Business Insider :

    My editor would like to post it on our site as an example of a creative way to format a resume... I'm wondering if we can get your permission to do this?

    So that's what prompted this post: I simply added my name and a Creative Commons Attribution Licence (CC-BY) to the two images, and then sent my permission along.

    Outside of that, no prospective employers have gotten in touch. But like I always say: you can't win the lottery if you don't buy a ticket. And since I also enjoy mixing metaphors, I'll just keep on fishing!

    permalink
  5. indenting with tabs

    2014 04 03 technology

    python

    This post was written as an IPython Notebook. You can view it on nbviewer, or download it.

    Greg Wilson asked on the IPython mailing list:

    Subject: easiest way to insert a literal tab character in a code
    cell? 
    Greg Wilson, on 2014-04-03 18:37,  wrote:
    > Hi,
    > I'd like to put literal tab characters in cells, but of course tab means 
    > "indent" to the editor.  What's the easiest way to do this?
    > Thanks,
    > Greg
    > p.s. because I'm going to write Makefiles in the notebook...
    

    The easiest way to do this is to just get a tab character somewhere that you can copy, and then paste it in.

    In [1]:
    print("\t")
    
    
    
    
    In [2]:
        # I copy pasted the output of the cell above here
    

    An alternative solution is to make a string with tabs and insert it into another cell, using IPython machinery.

    In [3]:
    ip = get_ipython()
    
    In [4]:
    ip.set_next_input("\tMakefiles are awesome")
    
    In []:
        Makefiles are awesome
    

    If you have a file on disk or on the web, you can also just use the %load magic to do this.

    In [5]:
    %load /home/pi/file_with_tabs
    
    In []:
    default:
        cat /etc/issue
        whoami
    

    Such files can be written with the %%writefile cell magic... but of course you need to have inserted tabs there in some manner.

    In [6]:
    %%writefile ~/file_with_tabs
    default:
        cat /etc/issue
        whoami
    
    Overwriting /home/pi/file_with_tabs
    
    
    In [7]:
    !make -f /home/pi/file_with_tabs
    
    cat /etc/issue
    Debian GNU/Linux jessie/sid \n \l
    
    whoami
    pi
    
    

    The more involved, but more usable way

    We can set up CodeMirror to insert tabs instead of spaces.

    In [8]:
    %%javascript
    
    IPython.tab_as_tab_everywhere = function(use_tabs) {
        if (use_tabs === undefined) {
            use_tabs = true; 
        }
    
        // apply setting to all current CodeMirror instances
        IPython.notebook.get_cells().map(
            function(c) {  return c.code_mirror.options.indentWithTabs=use_tabs;  }
        );
        // make sure new CodeMirror instances created in the future also use this setting
        CodeMirror.defaults.indentWithTabs=use_tabs;
    
        };
    

    The reason we attach tab_as_tab_everywhere to IPython is because when we use the %%javascript magic, any variables we define there must be called in the same cell that defined it - they get their own closure. The reason we do this is to allow the notebook javascript to not get screwed up when there are javascript errors. We could have attached it to window or CodeMirror or anything else that's already in javascript-land.

    I covered how to add functions like this to the custom.js file in your profile in my post about disabling blinking in the notebook. That way these little functions are available in every notebook, without you having to insert a cell defining them.

    Now we've got code that allows us to apply the change to all current and future cells. We leave it as an exercise for the interested reader to modify that code and make a little button in the toolbar, to toggle it on a per-cell basis.

    Hints:

    You can get to the code mirror instance via IPython.notebook.get_selected_cell().code_mirror

    This post was written as an IPython Notebook. You can view it on nbviewer, or download it.

    permalink
  6. bipython 0.1.0

    bipython
logo

    the boldly indiscriminate python interpreter

    "...because you shouldn't have to choose."

    PROLOGUE

    Two interpreters, both alike in dignity,
    In fair Pythona, where we lay our scene,
    From ancient grudge break to new mutiny,
    Where civil code makes git commits unclean.
    From forth the fatal loins of these two foes
    A newer kind of stranger's given life;
    Whose misadventured piteous overthrows
    Doth with its birth bury its parents' strife.

    ACT I

    Enter bpython and ipython

    bpython

    I'm a fancy terminal-based interface to the Python interpreter. I give you
    inline syntax highlighting and auto-completion prompts as you type, and I'll
    even automatically show you a little tooltip with a docstring and parameter
    list as soon as you hit ( to make the function call, so you always know
    what you're doing! I'm svelte and proud of it - I don't try to do all of the
    shenanigans that ipython does with the shell and the web, but the cool kids
    love my rewind feature for demos. I strive to make interactive python coding
    a joy!

    ipython

    I'm an awesome suite of interactive computing ideas that work together.
    For millennia, I've given you tab-completion and object introspection via
    obj? instead of help(obj) in Python. I also have sweet shell features,
    special magic commands (%run, %timeit, %matplotlib, etc.) and a
    history mechanism for both input (command history) and output (results
    caching).

    More recently, I've decoupled the REPL into clients and kernels, allowing
    them to run on independent of each other. One popular client is the
    IPython Notebook which allows you to write code and prose using a web
    browser, sending code to the kernel for execution and getting rich media
    results back inline. The decoupling of clients and kernels also allows
    multiple clients to interact with the same kernel, so you can hook-up to
    that same running kernel from the terminal. The terminal workflow makes
    more sense for some things, but my user interface there isn't as polished
    as bpython's.

    Enter bipython

    bipython

    By your powers combined... I am bipython!

    Exeunt

    The Power is Yours!

    pip install  bipython
    easy_install bipython
    

    bipython requires ipython, pyzmq, bpython, and urwid.

    For now, you'll need to have a running ipython kernel before running bipython. You can do this by either opening a notebook or running ipython console. It won't always be like this, I'll fix it as soon as I can, but it'll be sooner with your help over ivanov/bipython.

    After that, just run bipython and enjoy the ride.

    Here's a walkthrough of ipython, bpython, and bipython:

    The screencast is 20 minutes long, but here I'll play it back double speed. There's no sound, and you can pause at any time and select / copy portion of the text as you like. Changing the browser font size in the usual way works, too. (click here if the embed didn't work)

    permalink
  7. stem-and-leaf plots in a tweet


    Summary: I describe stem plots, how to read them, and how to make them in Python, using 140 characters.

    My friend @JarrodMillman, whose office is across the hall, is teaching a computational statistics course that involves a fair amount programming. He's been grading these homeworks semi-automatically - with python scripts that pull the students' latest changes from GitHub, run some tests, spit out the grade to a JSON file for the student, checks it in and updates a master JSON file that's only accessible to Jarrod. It's been fun periodically tagging along and watching his suite of little programs develop. He came in the other day and said "Do you know of any stem plot implementation in python? I found a few, and I'm using one that's ok, but it looks too complicated."

    For those unfamiliar - a stem plot, or stem-and-leaf plot is a more detailed kind of histogram. On the left you have the stem, which is a prefix to all entries on the right. To the right of the stem, each entry takes up one space just like a bar chart, but still retains information about its actual value.

    So a stem plot of the numbers 31, 41, 59, 26, 53, 58 looks like this:

     2|6
     3|1
     4|1
     5|389
    

    That last line is hard to parse for the un-initiated. There are three entries to the right of the 50 stem, and these three entries 3 8 and 9 is how the numbers 53, 58, and 59 are concisely represented in a stem plot

    As an instructor, you can quickly get a sense of the distribution of grades, without fearing the binning artifact caused by standard histograms. A stem-plot can reveal subtle patterns in the data that are easy to missed with usual grading histograms that have a binwidth of 10. Take this distribution, for example:

    70:XXXXXXX
    80:XXXXXXXXXXX
    90:XXXXXXX
    

    Below are two stem plots which have the same profile as the above, but tell a different story:

     7|7888999
     8|01123477899
     9|3467888
    

    Above is a class that has a rather typical grade distribution that sort of clumps together. But a histogram of the same shape might come from data like this:

     7|0000223
     8|78888999999
     9|0255589
    

    This is a class with 7 students clearly struggling compared to the rest.

    So here's the code for making a stem plot in Python using NumPy. stem() expects an array or list of integers, and prints all stems that span the range of the data provided.

    from __future__ import print_function
    import numpy as np
    def stem(d):
        "A stem-and-leaf plot that fits in a tweet by @ivanov"
        l,t=np.sort(d),10
        O=range(l[0]-l[0]%t,l[-1]+11,t)
        I=np.searchsorted(l,O)
        for e,a,f in zip(I,I[1:],O): print('%3d|'%(f/t),*(l[e:a]-f),sep='')
    

    Yes, it isn't pretty, a fair amount of code golfing went into making this work. It is a good example for the kind of code you should not write, especially since I had a little bit of fun with the variable names using characters that look similar to others, especially in sans-serif typefaces (lI10O). Nevertheless, it's kind of fun to fit much functionality into 140 characters.

    Here's my original tweet:

    You can test it by running it on some generated data:

    >>> data = np.random.poisson(355, 113)
    >>> data
    array([367, 334, 317, 351, 375, 372, 350, 352, 350, 344, 359, 355, 358,
       389, 335, 361, 363, 343, 340, 337, 378, 336, 382, 344, 359, 366,
       368, 327, 364, 365, 347, 328, 331, 358, 370, 346, 325, 332, 387,
       355, 359, 342, 353, 367, 389, 390, 337, 364, 346, 346, 346, 365,
       330, 363, 370, 388, 380, 332, 369, 347, 370, 366, 372, 310, 348,
       355, 408, 349, 326, 334, 355, 329, 363, 337, 330, 355, 367, 333,
       298, 387, 342, 337, 362, 337, 378, 326, 349, 357, 338, 349, 366,
       339, 362, 371, 357, 358, 316, 336, 374, 336, 354, 374, 366, 352,
       374, 339, 336, 354, 338, 348, 366, 370, 333])
    >>> stem(data)
     29|8
     30|
     31|067
     32|566789
     33|00122334456666777778899
     34|02234466667788999
     35|001223445555577888999
     36|12233344556666677789
     37|0000122444588
     38|0277899
     39|0
     40|8
    

    If you prefer to have spaces between entries, take out the sep='' from the last line.

    >>> stem(data)
     29| 8
     30|
     31| 0 6 7
     32| 5 6 6 7 8 9
     33| 0 0 1 2 2 3 3 4 4 5 6 6 6 6 7 7 7 7 7 8 8 9 9
     34| 0 2 2 3 4 4 6 6 6 6 7 7 8 8 9 9 9
     35| 0 0 1 2 2 3 4 4 5 5 5 5 5 7 7 8 8 8 9 9 9
     36| 1 2 2 3 3 3 4 4 5 5 6 6 6 6 6 7 7 7 8 9
     37| 0 0 0 0 1 2 2 4 4 4 5 8 8
     38| 0 2 7 7 8 9 9
     39| 0
     40| 8
    

    To skip over empty stems, add e!=a and in front of print. This will remove the 300 stem from the output (useful for data with lots of gaps).

    >>> stem(data)
     29| 8
     31| 0 6 7
     32| 5 6 6 7 8 9
     33| 0 0 1 2 2 3 3 4 4 5 6 6 6 6 7 7 7 7 7 8 8 9 9
     34| 0 2 2 3 4 4 6 6 6 6 7 7 8 8 9 9 9
     35| 0 0 1 2 2 3 4 4 5 5 5 5 5 7 7 8 8 8 9 9 9
     36| 1 2 2 3 3 3 4 4 5 5 6 6 6 6 6 7 7 7 8 9
     37| 0 0 0 0 1 2 2 4 4 4 5 8 8
     38| 0 2 7 7 8 9 9
     39| 0
     40| 8
    

    Thanks for reading.

    permalink
  8. disabling blinking

    2014 01 29 technology

    python vision

    Background: Text editing in the IPython Notebook is provided by an excellent JavaScript-based CodeMirror text editor. This might be more detail than you want, but I'm a vision scientist so I hope you'll indulge me.

    The cursor is meant to tell the user the current location.

    The human visual system has a pre-cortical lag of roughly 50-90 ms (read more about P1).

    That's how long it takes from something changing on the screen to cause an avalanche of photons to barrel towards your eyeball, be phototransduced and processed by several stages of cells in the retina, finally causing retinal ganglion cells to fire an action potential down their axons through the optic nerve, make its way to a processing relay station called the LGN, with those cells firing action potential down their axons, with those spikes finally ending up in the primary visual cortex.

    By ~150 ms, our brains have processed the visual input enough to perform a non- trivial ammount of object recognition.

    The default blink rate for CodeMirror is 530ms.

    That's as slow as molasses in January!

    Say that I've been distracted and looked away from the screen. When I look back at the scree, half of the time it will take 3 times longer for me to get the information that I want ("where's my cursor") than if that cursor was clearly visible at all times. Now it's not always that bad, because sometimes my gaze will land on the screen and even though the cursor isn't visible, it appears in a few milliseconds, and so it takes just as long as if the cursor was there the whole time. But if I happen to be particularly unlucky (there's a reason I don't gamble), it can take 6 times longer.

    Try it out

    Here's the bit of JavaScript code you need to disable blinking in CodeMirror.

    CodeMirror.defaults.cursorBlinkRate=0
    

    If you type that into the JavaScript console of your webbrowser, that setting will apply to all cells created in the current IPython Notebook. If you don't know how to open your browser's Javascript console, don't frett, just make a new cell with just the following lines in there, execute it, and make a new cell to see how you like it.

    %%javascript
    CodeMirror.defaults.cursorBlinkRate=0
    

    Make the change stick

    IPython has a notion of profiles to allow for different kinds of configurations. If this is news to you, you've probably just been using the default profile and not known it. In the shell, run the ipython profile create command to be sure (don't worry, if you alreay have a profile, this won't overwrite it). Now ipython locate profile will tell you the directory which contains all of the configuration for the default profile.

    In [1]:
    !ipython profile create
    
    In [2]:
    !ipython locate profile
    
    /home/pi/.ipython/profile_default
    
    In [3]:
    x = !ipython locate profile
    
    In [4]:
    cd $x.s
    
    /home/pi/.ipython/profile_default
    
    In [5]:
    ls
    
    db/  history.sqlite  history.sqlite-journal  ipython_config.py  ipython_nbconvert_config.py  ipython_notebook_config.py  log/  pid/  security/  startup/  static/
    
    There's a lot of stuff there, but we just need to add our one line to the end of the file in ``static/custom/custom.js``
    In [6]:
    cd static/custom/
    
    /home/pi/.ipython/profile_default/static/custom
    
    In [7]:
    ls
    
    custom.css  custom.js
    
    In [8]:
    !echo "codemirror.defaults.cursorblinkrate=0" >> custom.js
    
    ## "I want it all and I want it now!" You say you don't want to save your current notebook and reload it to get the updated CodeMirror settings? You just want all cells in the current notebook to change their behavior? Well, OK, Freddie:
    In [9]:
    %%javascript
    var rate = 0;
    // apply setting to  all current CodeMirror instances
    IPython.notebook.get_cells().map(
        function(c) {  return c.code_mirror.options.cursorBlinkRate=rate;  }
    );
    
    // make sure new CodeMirror instance also use this setting
    CodeMirror.defaults.cursorBlinkRate=rate;
    
    I hope you enjoyed this little IPython customization detour. If you want more information about how to get rid of blinking in other programs you use every day, [here is an invaluable resource on that matter](http://www.jurta.org/en/prog/noblink). Remember, blinking in user interfaces is bad, [but blinking in vision is very important](http://www.brower.co.uk/opticians/blinking.html). ** This post was written as an IPython Notebook. You can view it on [nbviewer](http://nbviewer.ipython.org/url/pirsquared.org/blog/notebooks/notebook-blink.ipynb), or [download it](/blog/notebooks/notebook-blinking.ipynb) ** permalink
  9. What happens to a talk deferred...

    2013 06 17 technology

    python talk

    I'll be in Austin, TX for SciPy next week, and I'll be giving an updated talk about tools for reproducible research.

    What follows is the abstract of a talk that I really wanted to give and submitted to the general track which did not get in. I care deeply about this topic, and I'm hoping to have conversations with others about it. The talk was deferred to a poster, so if you don't bump into me at other times, seek me out during the poster session (mine's poster #15, 10:35 AM - 11:35 AM on June 27th). This post is intended as a pre-conference warm up to that.

    Navigating the Scientific Python Communities - the missing guide.

    The awful truth: our newcomers have a deluge of options to wade through as they begin their journey. What tools are available, how do I install them, how do I make them work together -- all hard questions facing a budding scipythonista.

    On developer-friendly platforms, the popular approach is to just install numpy, scipy, matplotlib, and ipython using package management facilities provided by the operating system. On Mac OS X and Windows, the least painful, bootstrapping approach is to use a Python distribution like Python(X,Y), EPD, Anaconda, or Sage.

    Both of these paths obscure a reality which must be stated explicitly: the development of packages is fundamentally decentralized. The scientific python ecosystem consists of a loose but thriving confederation of projects and communities.

    The chaos of installation options and lack of centralization around a canonical solution, which on the surface appears to be a point of weakness that is detrimental to community growth, is a symptom of one of its greatest strengths. Namely, what we have is a direct democracy for user-developers. They vote with their feet: filing bug reports, testing pre-release versions, participating in mailing lists, writing and reviewing pull requests, and advertising the tools they use in talks, papers, and conversations at conferences.

    I will discuss why this is the case, why this is a good thing, and how we can embrace it. The talk will present the ethos and expectations of an effective newcomer, as well as resources and strategies for incremental progress toward becoming a master scipythonista.

    permalink

Page 1 / 3 »