Other articles

1. aka: also known as

   2020 10 08 technology

   shell hello-world git

   I was chatting with Anthony Scopatz last week, and one of the things we covered was how it'd be cool to have a subcommand launcher, kind of like git, where the subcommands were swappable. If you're not familiar, git automatically calls out to git-something (note the dash) whenever you run

   $ git something
   

   and something is not one of the builtin git commands. For me, ~/bin is in my PATH, so

   $ git lost
   git: 'lost' is not a git command. See 'git --help'.
   $ echo "echo how rude!" > ~/bin/git-lost; chmod +x ~/bin/git-lost
   $ git lost
   how rude!
   

   And so what Anthony was talking about was having two commands that are supposed to do the same thing, and being able to switch between them. For example: maybe we have git-away and git-gone and both of them perform a similar function, and we wish call our preferred one when we run git lost.

   One way to do this would be to copy or symlink our chosen version as git-lost, and replace that file whenever we wanted to switch between git-away and git-gone. Another would be to rename both as git-lost and store them in different folders, adjusting our PATH variable so the preferred version of git-lost would get called.

   Now with git itself, you can do this by making a git alias. Here are some of my frequently used aliases from my ~/.gitconfig:

   [alias]
       diffw = diff --word-diff=color --ignore-cr-at-eol
       root = rev-parse --show-toplevel
       lgo=log
       lazy=commit -a -m.
   

   git diffw will highlight only parts of the line that were added and removed, instead of whole lines
   git root prints the top-most directory of the current git repository
   git lgo is a typo for git log (I also have several versions of checkout typos)
   git lazy is for those frequent times when having to think of a commit message is too onerous.

   So we would add lost=away to our alias block to have git lost call git-away and change that alias to lost=gone when we wanted to have git lost call git-gone, instead.

   I wondered what a simple, generic version of such capability might look like and wrote a short shell script and then a README file that's ten times longer than the script itself to explain it. :)

     .---------------------.
    / aka - also known as /
   /---------------------/
   
   Simple, persistent, shell-based command aliasing
   
   Usage: aka alias               show stored aliases
          aka alias NAME CMD...   store an alias for CMD
          aka NAME [...]          run CMD, optionally with more arguments
   

   By default, aliases are stored in the .aka file of the current working directory. Optionally, the location of that file can be modified by setting the AKA_FILE environment variable. Tested on Debian, OpenBSD, and OpenWRT, aka uses portable shell syntax and should work everywhere that has a typical /bin/sh

   Examples:

   $ aka alias demo more
   
   $ aka demo --version
   more from util-linux 2.36
   
   $ aka alias demo git
   
   $ aka demo --version
   git version 2.28.0
   

   The command you alias can have parameters

   $ aka alias l ls -f
   
   $ aka l
   .aka .. aka aka_tiny . README .git
   

   And you can pass additional parameters

   $ aka l -tR
   .. aka_tiny aka .git .aka README .
   

   You can see all of the currently stored aliases

   $ aka alias
   l='ls -f'
   demo='git'
   

   And edit them using your favorite text editor

   $ cat .aka
   alias demo='git'
   alias l='ls -f'
   

   Everything that is not aliased will get executed as a regular command.

   $ aka python3 --version
   Python 3.8.6
   

   Why would anyone want this?

   A practical use case might be to have

   .
   ├── c_proj
   │   └── .aka       | alias doit='make install'
   ├── go_proj
   │   └── .aka       | alias doit='go run'
   └── python_proj
       └── .aka       | alias doit='pip install .'
   

   Now you can aka doit to your heart's content inside each of those folders, and have that execute the appropriate build commands for the type of project it is.

   $ for x in *; do (cd $x && aka doit); done
   make: *** No rule to make target 'install'.  Stop.
   go run: no go files listed
   ERROR: Directory '.' is not installable. Neither 'setup.py' nor 'pyproject.toml' found.
   

   Installation

   Grab aka, make it executable, and place it somewhere in your path.

   The standard version of aka (~40 lines) that prints usage and has comments explaining the code is here:

   https://git.sr.ht/~pi/aka/blob/main/aka

   If you just want the business logic, functional equivalent (~10 lines) is here:

   https://git.sr.ht/~pi/aka/blob/main/aka_tiny

   How it works

   aka is a tiny portable shell script that gets executed via /bin/sh, sources AKA_FILE and evals the positional parameters, thus applying aliases and whatever other shell script shenanigans stored in AKA_FILE.

   For example, this means that you can put something like

   alias my='AKA_FILE=~/.my_aliases aka'
   

   in your shell's startup files, and thereafter use the my command as an aka invocation that always sources ~/.my_aliases file in your home directory, regardless of where you run it.

   Notes

   Initial prototype uses aliases, which may not work for programs that depend on establishing their behavior based on the argv[0] name they were called by (busybox, for example).

   If you want to create aliases that include quotes, multiple commands, output redirection, or job control, you either have to escape the special characters like > and & with backslashes when creating the alias, or edit .aka file to add them directly in there:

   $ aka alias redir_example echo \"hi\" \> log.txt
   $ aka alias double_date date \;  date
   

   In the example above, if we don't put a backslash in front of the semicolon, we will be terminating our aka alias for double_date with just one call do date, and calling the second date command immediately.

   And finally, aka stores the aliases using single quotes, so if you have commands where you need to preserve single quotes, you should edit the .aka file by hand to change the alias definition to use double quotes.

   Related Projects

   https://direnv.net: "direnv is an extension for your shell. It augments existing shells with a new feature that can load and unload environment variables depending on the current directory." The direnv website has links to a half dozen similar projects.

   Debian's update-alternatives Debian's alternatives system is similar: it lets you switch out raw XX command for all users, which is why you have to run update-alternatives as root. With aka, you're defining the aka XX commands and they only apply on a per-folder basis (or per AKA_FILE environment variable, if set).

   License

   aka is distributed under a 3-clause BSD license.

   Author

   aka was written by Paul Ivanov

   https://sr.ht/~pi/aka

   Changelog

   2020-10-02: initial version

   2020-10-03: added AKA_FILE environment variable

   2020-10-08: publicly announced

   2020-10-28: added notes about multiple commands, redirection, and quotes

   Source code

   Standard ~40 line aka with comments and usage printing:

   #!/bin/sh
   # aka, written by Paul Ivanov: https://git.sr.ht/~pi/aka
   if [ $# -eq 0 ] ; then
       set -- -h # print usage on 0 arguments
   fi
   
   if [ $# -ge 1 ] ; then
       case $1 in
           -h|--help|-help|-H)
   echo "Usage: aka alias               show stored aliases"
   echo "       aka alias NAME CMD...   store an alias for CMD"
   echo "       aka NAME [...]          run CMD, optionally with more arguments"
               exit 0
               ;;
       esac
   fi
   
   aliasFile=${AKA_FILE:-./.aka}
   # Is this an alias creation? If so, we should have at least 3 arguments,
   # such as:
   #
   # $ aka alias pager more
   #    \     \     \    \
   #     $0    $1    $2   $3
   #
   if [ $# -ge 3 ]  && [ $1 = alias ] ; then
       # cmd='pager'
       cmd=$2
       # make 'more' the new $1
       shift 2
       # remove previous aliases for 'pager'
       [ -e "$aliasFile" ] && grep -v "alias $cmd=" "$aliasFile"  > "$aliasFile"~
       echo "alias $cmd='$*'" >> "$aliasFile"~ && mv "$aliasFile"~ "$aliasFile"
       exit 0
   fi
   
   # Load up aliases...
   [ -e "$aliasFile" ] && . "$aliasFile"
   
   # ...and execute command
   eval $*
   

   Tiny ~10 line aka without comments:

   #!/bin/sh
   # aka, written by Paul Ivanov: https://git.sr.ht/~pi/aka
   aliasFile=${AKA_FILE:-./.aka}
   if [ $# -ge 3 ]  && [ $1 = alias ] ; then
       cmd=$2
       shift 2
       [ -e "$aliasFile" ] && grep -v "alias $cmd=" "$aliasFile"  > "$aliasFile"~
       echo "alias $cmd='$*'" >> "$aliasFile"~ && mv "$aliasFile"~ "$aliasFile"
       exit 0
   fi
   [ -e "$aliasFile" ] && . "$aliasFile"
   eval $*
   

   permalink

2. pheriday 3: infrastructure

   2020 04 03 technology

   python pheriday infrastructure hello-world

   Looks like we can't inline audio for your browser. That's cool, just find the direct file links below.

   paul's habitual errant ramblings (on Fr)idays

   pheridays: 3

   2020-04-10: A week ago, I recorded a 5 minute audio segment of some stuff I've been thinking about, but when I started to write it up I stumbled into and kept dropping down a deep technostalgic hole.

   fall down along with me:

   https://pirsquared.org/blog/pheriday-infrastructure.html

   The recording is just shy of five minutes long, you can also download it in different formats, depending on your needs, if the audio tag above doesn't suit you:

   https://pirsquared.org/pheridays/2020-04-03.ogg (2.9 Mb)
   https://pirsquared.org/pheridays/2020-04-03.mp3 (4.5 Mb)
   https://pirsquared.org/pheridays/2020-04-03.m4a (6.3 Mb)
   

   --

   Stuff I mentioned in the audio:

   Propellor - "configuration management system using Haskell and Git" by Joey Hess

   OpenWRT - specifically - reducing Bufferbloat

   Mumble - "a free, open source, low latency, high quality voice chat application."

   sourcehut.org - "the hacker's forge" also know as sr.ht by Drew DeVault

   Jitsi - "Multi-platform open-source video conferencing"

   OpenFire - "real time collaboration (RTC) server licensed under the Open Source Apache License." Extensible XMPP server, with plugins, like a Jitsi-based video meeeting one claled OpenFire Meetings.

   ---

   Though this is the fourth installment, the last time I recorded and posted a rambling was back almost 8 years ago! In fact, it was 2012-08-03, so 7 years and 8 months, to the day.

   ---

   Having control of your infrastructure is a longtime thread for me.

   For starters - there's the bicycle. That's been my primary and preferred mode of transportation for 30 years. As a kid, I was empowered by the sense of freedom, independence, and self-sufficiency that came with a bike. All these years later, I'm still a fan. You can see just how happy I am on a bike at the top of this interview , thanks to a sweet photo that was taken by Robert Sexton right by the Golden Gate Bridge at the end of the Lucas Valley Populaire in 2015.

   Those of you who knew me back in college might remember how at UC Davis I ran my own "pirate" internet radio station - KPVL - with the cheeky tagline of "More broadcasters than listeners". (I say "pirate" because it has not relation to the actual KPVL radio station). But there are earlier remnants and traces of my efforts to exercise control and build my own reality.

   I think it was in 1999 that my brother Mike and I started using Redhat (6), then Mandrake Linux 6.5, dual booting on a computer at home and I separately around the same time I got myself an sdf.org account. Though I wasn't sophisticated enough to have a constant internet connection in high school, I was lucky enough to get an account on Robert Chin's laya.com server. The url was - p.laya.com - it's long gone, but luckily, Archive.org has a copy from 2001.

   Wow. I just took a look and so much came flooding back.

   Here's the thing: April is an anniversary of sorts for me. Back in 1999, it marks my first time breaking anonymity and pseudonimity and using my real name on the internet. I've written about this before under the title of Publisher's block ten years ago - just about half way between now and then. This time, though, let me inline the piece I linked to as proof of the deliberate nature of my lack of anonymity.

   An account of my life at 15, as I live it.
   traces of my awareness of the world, I can look back at later
   My first attempt at a memoir
   My goal is to capture my many thoughts emotions, behaviors, incidents, and
   acquaintances
   and to arrive only at an exponential number of those,
   hoping yet being afraid that it might be zero
   making everything about me: one
   

   I'm glad I can now reflect on the kind of kid I was, thanks to the amazing folks who had the foresight to start archiving all of the web for The Way Back Machine. I used that p.laya.com page as a todo list and notes for myself using a hipster combination of the default file index listing with a FOOTER.html. It was captured in 2001, I was in 17, but some of this was written when I was 15 or 16 (I found contents from November 2000), I make mention to my then freeshell.org account (it's now been ivanov@ since 2012). I link to the source code of a MUD - ftp://ftp.game.org/pub/mud/diku/merc/rom/tartarus/tartarus.tgz - which is a broken link now, but I found a mirror over here: which is amazing, because just a week or two ago, I was hanging out with fellow SciPy 2020 program co-chairs Madicken Munk and Gil Forsyth over video chat after one of our meetings and I was happily reporting about how one of the positive things to come out of the shelter in place for me is that "I've fixed my mutt configuration and started using it again!" - but they both heard "mutt" as "MUD" and got very excited by that prospect. So much so that we all agreed that we'll have to follow up and actually follow through to build a MUD. And I brought up how at some point in high school I was mildly active in a pair of MUDs, and wanted to make my own, but never got around to it.

   The last link I left for myself on there points to pinkmonkey.com - a homeschooling resource - which is probably handy for the parents with little ones these days.

   Here's the most concrete infrastructure project I can find from then: I collected bookmarks from my friends to share them. The "service" lived at http://p.laya.com/bookmarks - and predates del.icio.us and pinboard. I bet I "advertised" it in my AIM profile.

   If you're curious, there's a link to the archive.org copy near the end of this post, but I had this urge to show it to you much closer to its original glory.

   Let me set the scene: It's Friday in April, the year is 2020, I'm running Windows 10 on my work laptop in poorly connected home in California, where a pandemic has most of the state's residents staying put at home for the several weeks already, and I decided to make a screenshot using the tool du jour of yesteryear

   Netscape Navigator!

   The timestamp on my bookmark website says I last updated it on: Thu Aug 30 20:14:14 PDT 2001

   OldVersion.com tells me that the latest release for Windows that Netscape 4.79 was released in November of that year, and the closest antecedent version available is 4.72 (from February 2000).

   I downloaded it and tried fiddling around with the compatibility settings, but without any luck.

   Then I tried 4.79, and nope, that didn't work, either. So then I tried Netscape 6.01 - release February 2001.

   I happened to have Chrome running at the time because in Firefox I have 1500 tabs open -- fifteen hundred and seven! ;) -- whereas in Chrome it's under 500, so I was trying to tread lightly. How do I know these numbers? For Chrome I found an extension that allows me to copy into the clipboard all open tabs' urls as plain text. It helpfully announces how many such tabs were copied. In Firefox one of the webextension examples gives you a counter.

   Do you remember the web without tabs? Time was, you wanted to visit another webpage, you got two option: you navigate away from whatever you're looking at now, or you hit Ctrl-N to make an new window. I think most people used one or a few windows. But you were not gonna be crazy and open more than a dozen windows. I would have, and probably tried but I couldn't. And session saving across crashes or clean exits? Forget it! That what your history and bookmarks are for, grasshopper.

   But let's get back to the task at hand: this was the lower right of my screen...

   

   and I decide to start taking screenshots of this journey, click it, and let Windows 10 apply the compatibility settings, and then I'm faced with

   the most improbable error message:

   

   :)

   WAT?!

   I didn't think Chrome had any ancestry shared with the Mosaic super-tree, but whatever - you can't exactly argue with software from 2001, and I have an important screenshot to take...

   So now I've quit Chrome, just in case, and going to retry....

   

   no dice....

   Damn, what could it be...I've got the Bloomberg Terminal open, I know portions of it are built on Chromium browser technology (had to look it up if this was officially stated somewhere - it is). Ok, so maybe that's what causing the false positive? I close that, and...

   

   ...

   I hardly have anything open anymore ...is it VLC?

   ...

   

   nope... Ok, what's left still open... Snipping tool I'm using to capture this epic adventure, a few WSL Debian console windows... the voice recorder that started this post... Task manager and Sysinternals' Process Explorer - where I was checking if perhaps somehow the failed attempt at running what was probably a 16 bit version of Navigator 4.72 was still lingering somewhere... SumatraPDF, Windows Terminal (Preview), gVim, and ...

   Zotero?!?

   BINGO!!!!!!!!!

   

   

   Oh right - I guess Zotero uses XUL technology. I didn't really think much about it, but Zotero did start off life as a Firefox extension, and the standalone version came out later, makes sense that it would have grabbed a browser when it struck out on its own.

   At this point I had already sent Madicken, who works at NCSA where Mosaic, the progenitor of Netscape hails from, the first two images... So I wanted to play with fire a bit....

   Now that I've closed Zotero - can I have Firefox 74 open while installing Netscape 6.01?

   

   Rats! same error...

   how about Chrome again?

   

   

   oh yeah! Sweet. The world makes sense again.

   Back to the setup.exe...

   

   

   I scroll through the EULA - and randomly stop on this section:

   12. HIGH RISK ACTIVITIES.  The Product is not
   fault-tolerant and is not designed, manufactured or
   intended for use or resale as on-line control
   equipment in hazardous environments requiring
   fail-safe performance, such as in the operation of
   nuclear facilities, aircraft navigation or
   communication systems, air traffic control, direct
   life support machines, or weapons systems, in which
   the failure of the Product could lead directly to
   death, personal injury, or severe physical or
   environmental damage ("High Risk Activities").
   

   By the way - here's a good idea I came across a few months ago: throw EULAs (End User License Agreements) into some publicly indexed version control repo (I saw folks using gists for just that sort of thing: here's the Netscape 6.01 EULA.txt)

   Let the folks at Redmond host it.

   Fine. I click next...

   

   Interesting - there's a "Read Me" button... I do as I'm told, so I click it.

   Oh, a README.txt popped up in Notepad.exe - full contents of that are in that same gist as the EULA

   but in them, there's a link to the full release notes over at

   http://home.netscape.com/eng/mozilla/ns6/relnotes/6.0.html

   Let's try to go there now:

   

   Wat?!

   Yeah, I'm sure Yahoo! engineers are jumping right on that.

   How did a Netscape site ended up redirecting to Yahoo...

   oh right, so AOL bought Netscape (1999), merged with Time Warner in 2001, was spun out again in 2009, after some rough times, and then purchased by Verizon in 2015. In the meantime, Yahoo acquired Geocities (1999) and shut it down in 2009 (yes, I'm still mad! All I remember was that it had some crispy banners, one of which was a scan of a sweet pencil lettering I made of my nick at the time - "ShadowKnight"). No one really cares what happened to Yahoo in the interim, aside from some massive data breaches, until finally, Verizon bought Yahoo in 2017 and merged AOL and Yahoo into one division.

   Ok, so fine, maybe fine folks at The Internet Archive (archive.org) can help us with the WayBackMachine.

   And indeed we can see what it looked like originally. This site has seen so many redirects over the years - it'd be a fun exercise to go through all of the indexed versions of this kind of site to see how people tried to preserve links. For example, I found out in 2010 it 301s ("Temporary redirect") to http://www.netscape.com/eng/mozilla/ns6/relnotes/6.0.html which then 302s ("Permanent redirect") to http://www.propeller.com/eng/mozilla/ns6/relnotes/6.0.html - which was indexed but happens to be a 404 ("Page not found") error page, at least in 2008. But this wasn't what we came here for, so this yak can roam free among the hills, the valleys, and the caverns of our minds.

   ... for now...

   Where were we?

   Oh right, we have to choose an install option. Back in the day I might have clicked "recommended" here, but we're not back in the day, and who wants to play life on easy mode? Let's go custom to see what the options are (I am a control freak, after all)

   

   and immediately get another pop up:

   

   Whoa, let's pause here for a moment.

   I am digging such a consent model. The installer is establishing trust: it will not try to do anything behind my back and without my permission. I'm sure it will never abuse that trust. As a 2001 user, I sure am glad that folks involved with computing have such a well-develop sense of ethics. In 2001, the Future is bright. Computing will be filled with transparency. Consumer software and services will be built by folks with a strong moral compass. These are people with principles. With the dot-com bubble burst, we've swiftly inoculated tech from sleazy opportunists. It won't fall victim to the excesses and greed rivaling Wall Street in the 80s...

   Which reminds me - how is it that this brilliant video only has 277 thousand views? Here's a excerpt:

   Your users won't always understand just how much economic sense it makes to sell them out. And you don't want to alienate them, that would drive down their value.

   By the way - the embed code I used above uses the youtube-nocookie.com domain - which is still from the folks at Goolag, but does what it says on the tin and doesn't issue cookies. Also, did you know there used to be a way to disable those annoying related video links from popping up at the end of the video? It's true. You used to be able to just append a rel=0 query parameter to not show related videos. But the corporate overlord bean counters didn't like that. They had to make sure that kids would get glued to the site by feeding them progressively conspiratorial garbage content. So that watching any video would nearly guarantee to pull them into the black hole cesspool of maximally "engaging" "content". What was that quote about users again? Ah yes:

   Your users won't always understand just how much economic sense it makes to sell them out. And you don't want to alienate them, that would drive down their value.

   Ok, so, in fairness, rel=0 query parameter still does something. It limits the suggested videos to the channel they are from. That's good. But what happens if we follow one of those links? First, we end up on the full youtube site, so that means the cookies are back. Hurray for surveillance capitalism! Also, the recommendations on the right are curated specifically for us, and not limited to the channel the previous video was one. Goodbye, rel=0!

   

   So I have to install Navigator, but I can unselect Mail, Instant Messenger, and Spell Checker... I don't need mail, but whatever, let's just go with the defaults.

   

   Oh, look at all this wonderful bundled crapware. Just in case you had any doubts.

   

   That last one made me throw up a little in my mouth.

   I opt for just the classic skin - and it tells me that the total download size will be 9959 K.

   

   I thought about censoring this next screenshot, but 15 year old me wouldn't have like that... What's in the shot is in the shot..

   

   Alright, and when the installation finished here's what we're greeted with:

   

   Did software in 2001 try to phone home? "activation.netscape.com could not be found." sure seems so.

   Yes, No, Cancel?

   

   What if I cancel?

   

   Alright, let's go for broke and get that retro look...

   

   

   Remember all those redirects? Well the browser froze when I got overzealous, clicked on "Interact" at the bottom there and chose to open chat... And on the next load, it crashed... And again...

   

   It just kept crashing...

   In case anyone else gets stuck on the same issue ;) I got around this by using the Profile Manager, where I had the option to start the browser in Work Offline mode. Then I turn the "Work Online" option on after the browser loaded (which you can do by plugging together that cute outlet pair on the bottom left).

   I do some ego surfing and go to my own site first.

   http://pirsquared.org

   

   

   I got too fancy with my unicode... But hey, this is totally functional.

   I made this Loading gif via a screen capture tool and then it finally clicked that not only did I not use Netscape 6 - I remember most everyone's experience was to stick to the 4.x series, because it was so much more usable and not bloated with nonsense, etc, etc.

   Alright, but at least I got 6 to run on Windows 10 and that works...When it doesn't crash, anyway... But I did get a error about youtube-nocookie.com... (some of the time, at least)...

   

   And then I realized that I can't go to any site that has https... Because... you know, the protocol that provides that 's' has changed over the years, and our 2001 browser could do SSL 2 or 3 or TLS 1.0... But my website uses TLS 1.3...

   

   I couldn't run to duckduckgo, either, since it redirects plain http to the https endpoint and that also runs TLS 1.3... I couldn't even go to archive.org to view my old site directly on the way back machine, because archive.org run TLS 1.2 at the moment.

   It's difficult to find any place that still runs such outdated standards... I tried to search for just a TLS 1.0 test server - but didn't find anything suitable... But then I happen to flip through the recent changelog for Firefox:

   

   Cool - so now we know if we want to find TLS 1.0 and 1.1 website, we should turn to the government of... damn a specific country wasn't specified...

   But wait a minute... Firefox 74.0 came out on March

   How did Mozilla release an update to a version of Firefox that was in the hands of a bunch of users without... umn...what's the word I'm looking for here...you know, that thing no one seems to think is a thing anymore... user consent?

   How Mozilla released an update without user consent

   https://wiki.mozilla.org/Firefox/Normandy/PreferenceRollout

   This is the way consent ends
   This is the way consent ends
   This is the way consent ends
   Not with a bang but a whimper
   

   I'm late to the party - this has been going on for about three years - with what I now recall was caused a bit of a splash back in 2017 (Drew DeVault covers in "Firefox is on a slippery slope"). But I didn't know the extent of it. Who has the time to pay attention to the way in which all the software they use changes in anti-social ways.

   Anyway, if you don't want the fine folks at Firefox to change your preferences out from under you, I think you go to about:config and switch the app.normandy.enabled setting to false. And if you're interested in specifics of how you've been a guinea pig: about:studies will tell you. And you can go to about:preferences#privacy to disable them.

   But I digress...

   Let's wrap this up...

   At this point, I used my 2020 browser to grab historical snapshot of my old bookmarks site, stripping off the tastefully annotated Way Back Machine user interface insertions, and serve it locally over http via python -m http.server, making sure to change the URL bar to make a historically accurate re-enactment. And now that you know how I got here, you can fully appreciate the effort that went into this next screenshot:

   

   

   So much so, that I couldn't resist making a video a scroll through :

   You can download it as webm, mp4, ogg, or gif.

   If you're interested - you can go http://p.laya.com/bookmarks/ (archival copy) or see the rest of p.laya.com from 2001

   That's all

   ---

   The ongoing crisis has been a circuit breaker to our usual patterns. I am taking advantage of this affordance to experiment with and establish channels of communications that are not controlled by others.

   ---

   permalink

3. SciPy 2018 dates and call for abstracts

   2017 12 14 technology

   python scipy2018 conference

   I'm helping with next year's SciPy conference, so here are the details:

   July 9-15, 2018 | Austin, Texas

   Tutorials: July 9-10, 2018
   Conference (Talks and Posters): July 11-13, 2018
   Sprints: July 14-15, 2018
   

   SciPy 2018, the 17th annual Scientific Computing with Python conference, will be held July 9-15, 2018 in Austin, Texas. The annual SciPy Conference brings together over 700 participants from industry, academia, and government to showcase their latest projects, learn from skilled users and developers, and collaborate on code development. The call for abstracts for SciPy 2018 for talks, posters and tutorials is now open. The deadline for submissions is February 9, 2018.

   • Conference Website: https://scipy2018.scipy.org
   • Submission Website: https://easychair.org/conferences/?conf=scipy2018

   Talks and Posters (July 11-13, 2018)

   In addition to the general track, this year will have specialized tracks focused on:

   • Data Visualization
   • Reproducibilty and Software Sustainability

   Mini Symposia

   • Astronomy
   • Biology and Bioinformatics
   • Data Science
   • Earth, Ocean and Geo Science
   • Image Processing
   • Language Interoperability
   • Library Science and Digital Humanities
   • Machine Learning
   • Materials Science
   • Political and Social Sciences

   There will also be a SciPy Tools Plenary Session each day with 2 to 5 minute updates on tools and libraries.

   Tutorials (July 9-10, 2018)

   Tutorials should be focused on covering a well-defined topic in a hands-on manner. We are looking for awesome techniques or packages, helping new or advanced Python programmers develop better or faster scientific applications. We encourage submissions to be designed to allow at least 50% of the time for hands-on exercises even if this means the subject matter needs to be limited. Tutorials will be 4 hours in duration. In your tutorial application, you can indicate what prerequisite skills and knowledge will be needed for your tutorial, and the approximate expected level of knowledge of your students (i.e., beginner, intermediate, advanced). Instructors of accepted tutorials will receive a stipend.

   Mark Your Calendar for SciPy 2018!

   permalink

4. SciPy 2017 tips

   2017 07 10 technology

   python scipy2017 conference

   After missing it for a couple of years, I am happy to be back in Austin, TX for SciPy this week!

   Always invigorating and exhilarating, Scientific Computing with Python (SciPy) has remained a top quality venue for getting together with fellow Pythonistas, especially the academically-bent variety.

   As a graduate student eight years ago, I was fortunate enough to be one of receive sponsorship and attended my first SciPy - SciPy 2009. This was the last time it was held at CalTech in Pasadena, CA.

   @ivanov/status/2648139681

   The following year, in 2010, at the first SciPy held in its now usual spot in Austin, TX, each attendee got a bottle of delicious salsa!

   

   Here are some oy my thoughts about attending this wonderful conference.

   Conference Tips

   bring a sweatshirt -- Yes, I know Austin's hot, but at the AT&T center, they don't mess around and crank the air conditioning all the way up to 11!

   join the slack group -- This year, there's a Slack group for SciPy: the link to join is in a pair of emails with the titles "Getting the most out of SciPy2017" and "Getting the most out of SciPy2017-UPDATED", both from SciPy2017 Organizers. So far at the tutorials slack has served as a useful back channel for communicating repo URLs and specific commands to run, signaling questions without interrupting the speakers' flow.

   engage with others during the breaks, lunch, etc -- There are lots of tool authors here and we love chatting with users (and helping you become contributors and authors yourselves). Not your first SciPy and feeling "in-your-element"? Make the effort to invite others into the conversations and lunch outings you're having with old friends - we're all here because we care about this stuff.

   take introvert breaks (and be considerate of others who may be doing the same) - I'm an introvert. Though I enjoy interacting with others (one-on-one or in small groups is best for me), it takes a lot of energy and at some point, I run out of steam. That's when I go for a walk, stepping away from the commotion to just have some quiet time.

   be kind to yourself (especially at the sprints) -- Between the tutorials, all of the talks, and the sprints that follow, there will be a flurry of activity. Conferences are already draining enough without trying to get any work done, just meeting a bunch of new people and taking in a lot of information. It took a lot of false starts for me to have productive work output at sprints, but the best thing I've learned about them is to just let go of trying to get a lot done. Instead, try to get something small and well defined done or just help others.

   Stuff to do in Austin

   The conference already has a great list of Things to do in Austin, as well as Restaurants, so I'll just mention a few of my personal favorites.

   Barton Springs Pool. Take a nice dip in the cool waters, and grab a delicious bite from one of the food trucks at The Picnic food truck park.

   Go see the bats. The Congress Ave bridge in Austin is home to the largest urban bat colony in the world. You can read more about this here, but the short version is that around sunset (8-9pm) - a large number of bats stream out from underneath the bridge to go feed on insects. Some days, they leave in waves (this Saturday there were two waves, the first was smaller, but many people left thinking that was the entire show).

   I hope you enjoy SciPy2017!

   permalink

5. starting my job search

   2014 06 14 technology

   visualization hello-world

   I am starting to look for a job in the San Francisco Bay Area.

   Since many recruiters ask for and presumably look at GitHub profiles, I decided to give mine a little facelift:

   

   In case you aren't familiar, that banner was motivated by Joel Spolsky's Smart and Gets Things Done, which is a book about hiring good developers . So I decided to tweet it out, mentioning @spolsky and he favorited it!

   @ivanov/status/476932602587123712

   Yesterday, I decided to tweet out an image that's at the top of my resume as a standalone tweet- mentioning Joel Spolsky again, and he liked it well enough to retweet it to his 90 thousand followers, so it's been getting plenty of love.

   

   @ivanov/status/477477547957944321

   @ivanov/status/477520571907842048

   Perhaps unsurprisingly, the only person to contact me as a result of this so far is a reporter from Business Insider :

   My editor would like to post it on our site as an example of a creative way to format a resume... I'm wondering if we can get your permission to do this?

   So that's what prompted this post: I simply added my name and a Creative Commons Attribution Licence (CC-BY) to the two images, and then sent my permission along.

   Outside of that, no prospective employers have gotten in touch. But like I always say: you can't win the lottery if you don't buy a ticket. And since I also enjoy mixing metaphors, I'll just keep on fishing!

   permalink

6. indenting with tabs

   2014 04 03 technology

   python

   ** This post was written as an IPython Notebook. You can view it on nbviewer, or download it. **

   Greg Wilson asked on the IPython mailing list:

   Subject: easiest way to insert a literal tab character in a code
   cell? 
   Greg Wilson, on 2014-04-03 18:37,  wrote:
   > Hi,
   > I'd like to put literal tab characters in cells, but of course tab means 
   > "indent" to the editor.  What's the easiest way to do this?
   > Thanks,
   > Greg
   > p.s. because I'm going to write Makefiles in the notebook...
   

   The easiest way to do this is to just get a tab character somewhere that you can copy, and then paste it in.

   In [1]:

   print("\t")
   

   In [2]:

       # I copy pasted the output of the cell above here
   

   An alternative solution is to make a string with tabs and insert it into another cell, using IPython machinery.

   In [3]:

   ip = get_ipython()
   

   In [4]:

   ip.set_next_input("\tMakefiles are awesome")
   

   In []:

       Makefiles are awesome
   

   If you have a file on disk or on the web, you can also just use the %load magic to do this.

   In [5]:

   %load /home/pi/file_with_tabs
   

   In []:

   default:
       cat /etc/issue
       whoami
   

   Such files can be written with the %%writefile cell magic... but of course you need to have inserted tabs there in some manner.

   In [6]:

   %%writefile ~/file_with_tabs
   default:
       cat /etc/issue
       whoami
   

   Overwriting /home/pi/file_with_tabs
   
   

   In [7]:

   !make -f /home/pi/file_with_tabs
   

   cat /etc/issue
   Debian GNU/Linux jessie/sid \n \l
   
   whoami
   pi
   
   

   The more involved, but more usable way¶

   We can set up CodeMirror to insert tabs instead of spaces.

   In [8]:

   %%javascript
   
   IPython.tab_as_tab_everywhere = function(use_tabs) {
       if (use_tabs === undefined) {
           use_tabs = true; 
       }
   
       // apply setting to all current CodeMirror instances
       IPython.notebook.get_cells().map(
           function(c) {  return c.code_mirror.options.indentWithTabs=use_tabs;  }
       );
       // make sure new CodeMirror instances created in the future also use this setting
       CodeMirror.defaults.indentWithTabs=use_tabs;
   
       };
   

   The reason we attach tab_as_tab_everywhere to IPython is because when we use the %%javascript magic, any variables we define there must be called in the same cell that defined it - they get their own closure. The reason we do this is to allow the notebook javascript to not get screwed up when there are javascript errors. We could have attached it to window or CodeMirror or anything else that's already in javascript-land.

   I covered how to add functions like this to the custom.js file in your profile in my post about disabling blinking in the notebook. That way these little functions are available in every notebook, without you having to insert a cell defining them.

   Now we've got code that allows us to apply the change to all current and future cells. We leave it as an exercise for the interested reader to modify that code and make a little button in the toolbar, to toggle it on a per-cell basis.

   Hints:

   You can get to the code mirror instance via IPython.notebook.get_selected_cell().code_mirror

   ** This post was written as an IPython Notebook. You can view it on nbviewer, or download it. **

   permalink

7. bipython 0.1.0

   2014 04 01 technology

   python hello-world

   

   the boldly indiscriminate python interpreter

   "...because you shouldn't have to choose."

   PROLOGUE

   Two interpreters, both alike in dignity,
   In fair Pythona, where we lay our scene,
   From ancient grudge break to new mutiny,
   Where civil code makes git commits unclean.
   From forth the fatal loins of these two foes
   A newer kind of stranger's given life;
   Whose misadventured piteous overthrows
   Doth with its birth bury its parents' strife.

   ACT I

   Enter bpython and ipython

   bpython

   I'm a fancy terminal-based interface to the Python interpreter. I give you
   inline syntax highlighting and auto-completion prompts as you type, and I'll
   even automatically show you a little tooltip with a docstring and parameter
   list as soon as you hit ( to make the function call, so you always know
   what you're doing! I'm svelte and proud of it - I don't try to do all of the
   shenanigans that ipython does with the shell and the web, but the cool kids
   love my rewind feature for demos. I strive to make interactive python coding
   a joy!

   ipython

   I'm an awesome suite of interactive computing ideas that work together.
   For millennia, I've given you tab-completion and object introspection via
   obj? instead of help(obj) in Python. I also have sweet shell features,
   special magic commands (%run, %timeit, %matplotlib, etc.) and a
   history mechanism for both input (command history) and output (results
   caching).

   More recently, I've decoupled the REPL into clients and kernels, allowing
   them to run on independent of each other. One popular client is the
   IPython Notebook which allows you to write code and prose using a web
   browser, sending code to the kernel for execution and getting rich media
   results back inline. The decoupling of clients and kernels also allows
   multiple clients to interact with the same kernel, so you can hook-up to
   that same running kernel from the terminal. The terminal workflow makes
   more sense for some things, but my user interface there isn't as polished
   as bpython's.

   Enter bipython

   bipython

   By your powers combined... I am bipython!

   Exeunt

   The Power is Yours!

   pip install  bipython
   easy_install bipython
   

   bipython requires ipython, pyzmq, bpython, and urwid.

   For now, you'll need to have a running ipython kernel before running bipython. You can do this by either opening a notebook or running ipython console. It won't always be like this, I'll fix it as soon as I can, but it'll be sooner with your help over ivanov/bipython.

   After that, just run bipython and enjoy the ride.

   Here's a walkthrough of ipython, bpython, and bipython:

   The screencast is 20 minutes long, but here I'll play it back double speed. There's no sound, and you can pause at any time and select / copy portion of the text as you like. Changing the browser font size in the usual way works, too. (click here if the embed didn't work)

   permalink

8. stem-and-leaf plots in a tweet

   2014 03 12 technology

   python mathematics teaching visualization

   ---

   Summary: I describe stem plots, how to read them, and how to make them in Python, using 140 characters.

   My friend @JarrodMillman, whose office is across the hall, is teaching a computational statistics course that involves a fair amount programming. He's been grading these homeworks semi-automatically - with python scripts that pull the students' latest changes from GitHub, run some tests, spit out the grade to a JSON file for the student, checks it in and updates a master JSON file that's only accessible to Jarrod. It's been fun periodically tagging along and watching his suite of little programs develop. He came in the other day and said "Do you know of any stem plot implementation in python? I found a few, and I'm using one that's ok, but it looks too complicated."

   For those unfamiliar - a stem plot, or stem-and-leaf plot is a more detailed kind of histogram. On the left you have the stem, which is a prefix to all entries on the right. To the right of the stem, each entry takes up one space just like a bar chart, but still retains information about its actual value.

   So a stem plot of the numbers 31, 41, 59, 26, 53, 58 looks like this:

    2|6
    3|1
    4|1
    5|389
   

   That last line is hard to parse for the un-initiated. There are three entries to the right of the 50 stem, and these three entries 3 8 and 9 is how the numbers 53, 58, and 59 are concisely represented in a stem plot

   As an instructor, you can quickly get a sense of the distribution of grades, without fearing the binning artifact caused by standard histograms. A stem-plot can reveal subtle patterns in the data that are easy to missed with usual grading histograms that have a binwidth of 10. Take this distribution, for example:

   70:XXXXXXX
   80:XXXXXXXXXXX
   90:XXXXXXX
   

   Below are two stem plots which have the same profile as the above, but tell a different story:

    7|7888999
    8|01123477899
    9|3467888
   

   Above is a class that has a rather typical grade distribution that sort of clumps together. But a histogram of the same shape might come from data like this:

    7|0000223
    8|78888999999
    9|0255589
   

   This is a class with 7 students clearly struggling compared to the rest.

   So here's the code for making a stem plot in Python using NumPy. stem() expects an array or list of integers, and prints all stems that span the range of the data provided.

   from __future__ import print_function
   import numpy as np
   def stem(d):
       "A stem-and-leaf plot that fits in a tweet by @ivanov"
       l,t=np.sort(d),10
       O=range(l[0]-l[0]%t,l[-1]+11,t)
       I=np.searchsorted(l,O)
       for e,a,f in zip(I,I[1:],O): print('%3d|'%(f/t),*(l[e:a]-f),sep='')
   

   Yes, it isn't pretty, a fair amount of code golfing went into making this work. It is a good example for the kind of code you should not write, especially since I had a little bit of fun with the variable names using characters that look similar to others, especially in sans-serif typefaces (lI10O). Nevertheless, it's kind of fun to fit much functionality into 140 characters.

   Here's my original tweet: @ivanov/status/443980372192137216

   You can test it by running it on some generated data:

   >>> data = np.random.poisson(355, 113)
   >>> data
   array([367, 334, 317, 351, 375, 372, 350, 352, 350, 344, 359, 355, 358,
      389, 335, 361, 363, 343, 340, 337, 378, 336, 382, 344, 359, 366,
      368, 327, 364, 365, 347, 328, 331, 358, 370, 346, 325, 332, 387,
      355, 359, 342, 353, 367, 389, 390, 337, 364, 346, 346, 346, 365,
      330, 363, 370, 388, 380, 332, 369, 347, 370, 366, 372, 310, 348,
      355, 408, 349, 326, 334, 355, 329, 363, 337, 330, 355, 367, 333,
      298, 387, 342, 337, 362, 337, 378, 326, 349, 357, 338, 349, 366,
      339, 362, 371, 357, 358, 316, 336, 374, 336, 354, 374, 366, 352,
      374, 339, 336, 354, 338, 348, 366, 370, 333])
   >>> stem(data)
    29|8
    30|
    31|067
    32|566789
    33|00122334456666777778899
    34|02234466667788999
    35|001223445555577888999
    36|12233344556666677789
    37|0000122444588
    38|0277899
    39|0
    40|8
   

   If you prefer to have spaces between entries, take out the sep='' from the last line.

   >>> stem(data)
    29| 8
    30|
    31| 0 6 7
    32| 5 6 6 7 8 9
    33| 0 0 1 2 2 3 3 4 4 5 6 6 6 6 7 7 7 7 7 8 8 9 9
    34| 0 2 2 3 4 4 6 6 6 6 7 7 8 8 9 9 9
    35| 0 0 1 2 2 3 4 4 5 5 5 5 5 7 7 8 8 8 9 9 9
    36| 1 2 2 3 3 3 4 4 5 5 6 6 6 6 6 7 7 7 8 9
    37| 0 0 0 0 1 2 2 4 4 4 5 8 8
    38| 0 2 7 7 8 9 9
    39| 0
    40| 8
   

   To skip over empty stems, add e!=a and in front of print. This will remove the 300 stem from the output (useful for data with lots of gaps).

   >>> stem(data)
    29| 8
    31| 0 6 7
    32| 5 6 6 7 8 9
    33| 0 0 1 2 2 3 3 4 4 5 6 6 6 6 7 7 7 7 7 8 8 9 9
    34| 0 2 2 3 4 4 6 6 6 6 7 7 8 8 9 9 9
    35| 0 0 1 2 2 3 4 4 5 5 5 5 5 7 7 8 8 8 9 9 9
    36| 1 2 2 3 3 3 4 4 5 5 6 6 6 6 6 7 7 7 8 9
    37| 0 0 0 0 1 2 2 4 4 4 5 8 8
    38| 0 2 7 7 8 9 9
    39| 0
    40| 8
   

   Thanks for reading.

   @ivanov/status/443981782635921408

   permalink

9. disabling blinking

   2014 01 29 technology

   python vision

   Background: Text editing in the IPython Notebook is provided by an excellent JavaScript-based CodeMirror text editor. This might be more detail than you want, but I'm a vision scientist so I hope you'll indulge me.

   The cursor is meant to tell the user the current location.

   The human visual system has a pre-cortical lag of roughly 50-90 ms (read more about P1).

   That's how long it takes from something changing on the screen to cause an avalanche of photons to barrel towards your eyeball, be phototransduced and processed by several stages of cells in the retina, finally causing retinal ganglion cells to fire an action potential down their axons through the optic nerve, make its way to a processing relay station called the LGN, with those cells firing action potential down their axons, with those spikes finally ending up in the primary visual cortex.

   By ~150 ms, our brains have processed the visual input enough to perform a non- trivial ammount of object recognition.

   The default blink rate for CodeMirror is 530ms.

   That's as slow as molasses in January!

   Say that I've been distracted and looked away from the screen. When I look back at the scree, half of the time it will take 3 times longer for me to get the information that I want ("where's my cursor") than if that cursor was clearly visible at all times. Now it's not always that bad, because sometimes my gaze will land on the screen and even though the cursor isn't visible, it appears in a few milliseconds, and so it takes just as long as if the cursor was there the whole time. But if I happen to be particularly unlucky (there's a reason I don't gamble), it can take 6 times longer.

   Try it out

   Here's the bit of JavaScript code you need to disable blinking in CodeMirror.

   CodeMirror.defaults.cursorBlinkRate=0
   

   If you type that into the JavaScript console of your webbrowser, that setting will apply to all cells created in the current IPython Notebook. If you don't know how to open your browser's Javascript console, don't frett, just make a new cell with just the following lines in there, execute it, and make a new cell to see how you like it.

   %%javascript
   CodeMirror.defaults.cursorBlinkRate=0
   

   Make the change stick

   IPython has a notion of profiles to allow for different kinds of configurations. If this is news to you, you've probably just been using the default profile and not known it. In the shell, run the ipython profile create command to be sure (don't worry, if you alreay have a profile, this won't overwrite it). Now ipython locate profile will tell you the directory which contains all of the configuration for the default profile.

   In [1]:

   !ipython profile create
   

   In [2]:

   !ipython locate profile
   

   /home/pi/.ipython/profile_default
   

   In [3]:

   x = !ipython locate profile
   

   In [4]:

   cd $x.s
   

   /home/pi/.ipython/profile_default
   

   In [5]:

   ls
   

   db/  history.sqlite  history.sqlite-journal  ipython_config.py  ipython_nbconvert_config.py  ipython_notebook_config.py  log/  pid/  security/  startup/  static/
   

   There's a lot of stuff there, but we just need to add our one line to the end of the file in ``static/custom/custom.js``

   In [6]:

   cd static/custom/
   

   /home/pi/.ipython/profile_default/static/custom
   

   In [7]:

   ls
   

   custom.css  custom.js
   

   In [8]:

   !echo "codemirror.defaults.cursorblinkrate=0" >> custom.js
   

   ## "I want it all and I want it now!" You say you don't want to save your current notebook and reload it to get the updated CodeMirror settings? You just want all cells in the current notebook to change their behavior? Well, OK, Freddie:

   In [9]:

   %%javascript
   var rate = 0;
   // apply setting to  all current CodeMirror instances
   IPython.notebook.get_cells().map(
       function(c) {  return c.code_mirror.options.cursorBlinkRate=rate;  }
   );
   
   // make sure new CodeMirror instance also use this setting
   CodeMirror.defaults.cursorBlinkRate=rate;
   

   I hope you enjoyed this little IPython customization detour. If you want more information about how to get rid of blinking in other programs you use every day, [here is an invaluable resource on that matter](http://www.jurta.org/en/prog/noblink). Remember, blinking in user interfaces is bad, [but blinking in vision is very important](http://www.brower.co.uk/opticians/blinking.html). ** This post was written as an IPython Notebook. You can view it on [nbviewer](http://nbviewer.ipython.org/url/pirsquared.org/blog/notebooks/notebook-blink.ipynb), or [download it](/blog/notebooks/notebook-blinking.ipynb) ** permalink

Page 1 / 3 »