vscode you should know

search files
ctrl+p
type ? into search bar for help for other types of search

search editor commands
ctrl+shift+p or f1 or ctrl+p >

move line or selection up/down
alt+up/down

duplicate line or selection above/below
alt+shift+up/down

peek definition
alt+f12
you can scroll and edit in the pop-up overlay

goto definition
f12

add cursor at end of line or lines of selection
ctrl+shift+i

add cursor above below
ctrl+alt+shift+up/down

ctrl+`
open and toggle terminal window

ctrl+shit+`
open a new terminal

more here

pycharm you should know

search everywhere - file names, commands names, and symbol names across project
press shift key twice

search for a pycharm command by name or abbreviation or parts
ctrl+shift+a
use a space to separate parts of a search(f s) file save (fs) should work too

search for a file in the project space
ctrl+shift+n

open a recent file - searchable
ctrl+e

switch an open file or a open a panel
ctrl+tab start from the top of the open files list
ctrl+shift+tab start from the bottom of the panels list

return focus to editor window
esc

return focus to editor window and close last active panel
shif+esc

quick documentation popup for current cursor or selection
ctrl+q

quick popup of source snippet for current cursor or selection
ctrl+shit+i

return focus to last
navigate to previous or next edit location across files
ctrl+alt+left ctrl+alt+right

add a cursor AKA multi-cursor
alt+left mouse click
alt+shift+insert toggle multi-cursor mode for shifted selection

add multiple cursors and  rectangle selection
alt+shift+left mouse click and drag
use same selection method for rectangle pasting

add an import automatically to the top of the file by selecting from a popupalt+enter
use after typing the usage (datetime.datetime or just datetime)

duplicate the current line or selection
ctrl+d

move the current line or selection up or down
alt-shift-up or alt-shift-down or ctrl-shift-up or ctrl-shift-down

navigate to class method or function in current file by selecting from a popup
ctrl+f12

debug the current file
shift+f9

step into
f7

step over
f8

run the current file
shift+f10

run the current line or selection in the console
alt+shift+e

Python - exception avoidance

m1 = []

m2 = None

m1_total = m1 and max(m1) or None

m2_total = m2 and min(m2) or 0

J and Python Tips

J

$: self reference for recursion etc.

Have to experiment with it

Python

Use seq[:1] and seq[-1:] first and last instead of seq[0] and seq[-1] to avoid index out of range exception for empty sequences

Generally, for all index n use seq[n:n+1] instead of seq[n] to avoid exceptions without having to add extra logic checks

>>> [][3:4]

[]

I like ipython notebooks

I have been stretching my programming chops by working on the problems at http://projecteuler.net

I like mathematics. At a glance the exercises at http://projecteuler.net seemed very interesting. There are network walks and problems involving primes and other number types.

I used the typical development cycle at first. Write a script kill it if it runs too long. Optimize and repeat until I have a solution that runs in a reasonable amount of time. Then submit the solution. If it doesn't check out then write an alternate version by altering and trying to correct any perceived deficiencies. Check the solution against the generated solution and repeat.

In my idle time I worked these problems using this process mostly in a terminal on my personal server either in Emacs or in my alternate environment on my phone.

I had recently reacquainted myself with several programming languages I had encountered in the past: assembly, forth, aplus, j-console, lisp/scheme, prolog, java, ruby, JavaScript, coffeescript. And had also learned a new language or two that I wanted to get comfortable with: Haskell, f-sharp, Julia.

I figured I'd write a solution for each of the problems in each of the languages. I setup the environments for each and started with Haskell. What started out as a simple learning experience quickly proved to be more of a brain teaser than I expected. Learning Haskell while simultaneously stretching my programming chops was more work than I had expected.

After solving the first couple of problems in Haskell I switched to solving the first couple of problems in prolog. On the first prolog problem I ended up looking up a solution on Wikipedia that I reworked after reading and searching and understanding more and trying again. In prolog I am a neophyte at best and to call me a programmer would be generous.

When I got my prolog code to work and looked up the next problem it dawned on me that by solving these problems first in a language I am more comfortable in and am considered an expert in I might have a better chance of finishing a significant portion of the over 400 and growing list of problems presented at http://projecteuler.net and then I could attack the solutions in other languages from the point of knowing the solution in another language.

Python is one of my stronger languages. I know how to program in the functional style and in the OOP style. Which would be useful for thinking of solutions in some of the other languages. I started down that road with SL4A on my phone and python in a shell on my server with a pinch of git.

When I hit Wikipedia to lookup triangle numbers it was a real eye opener and led me to back to prime number theory to optimize a solution to a problem I had previously successfully solved involving primes. In that process and to my benefit I had created a virtualenv with a more recent version of ipython. Which includes notebook.

The command 'ipython notebook' in the the directory and virtualenv of your choosing is all it takes to get started. And I was hooked after I solved my my first http://projecteuler.net problem using a notebook. I teared through 5 or six more problems in less time than it took to solve one using traditional editing run cycles. And the benefit of a running shell environment that can be used for  documentation and has individual cells that only have to be evaluated if you want to add them to the solution makes for a very iterative development cycle.

I feel like I can finally keep a train of thought and I can feel my approach to coding at work improving despite the difference in environments.

I sidetracked myself last weekend and to get my head around the clusters tab of a notebook. The ease of launching clusters has me crashing running processes on my tiny micro AWS instance server with only 4 clusters. I resolved to drive farther off track and write some code in my project Euler notebook for spinning up and down temporary clusters using AWS big compute instances for brief period of time to divvy up some of the tougher problems. By writing solutions that are easy to process in parallel I could properly figure out solutions in short order.

Being able to start a solution and continue work on it in parts anywhere from any device is much more accessible than using ssh and a terminal. You have to try it to get it. Working on code in a browser seems like a toy at first. But then you throw some real problems at it and you begin to realize there is something to this style of programming. And then you shutdown your notebook kernel. And then you come back to it and realize you haven't lost anything. Everything you've typed is still there waiting to be evaluated or edited and reevaluated.

And should you get curious and ssh to your server and type 'ipython console --existing' you too will start to understand the power of ipython notebooks.

I know now that I will be using ipython notebooks for all the languages I plan on tackling. After I finish up to problem 15 in order I will start with ruby and Julia. Both languages are the natural next choice because they already have working iruby and ijulia solutions that have ipython notebooks built in.

I'll have to sidetrack again after getting up to problem 15 in each as I'll have to implement an ipython notebook extension to each of the other languages before I start to work on the problems in that language. The tight iterative train of thought development cycle of ipython notebooks is too useful to not use when I am a programmer and can plugin the solution. Which is the point of the entire exercise.

Crushing Python

I've begun thinking in Python a lot lately. ~/.mypyrc.py:

# standard library
from collections import defaultdict as DD,namedtuple as NT
import csv
from datetime import date as D,datetime as DT,time as T,timedelta as TD
import operator as op
import os
import re
from cStringIO import StringIO as SIO
import sys
import time
from urlllib2 import build_opener as BO,install_opener as IO,Request as R,urlopen as UO
from urlparse import urljoin as UJ,urlparse as UP
from zipfile import ZipFile as ZF
# third party libraries
from BeautifulSoup import BeautifulSoup as BS,BeautifulStoneSoup as BSS
import pytz
# initialization
IO(BO())

~/.profile or ~/.bash_profile:

PYTHONSTARTUP=~/.mypyrc.py
export PYTHONSTARTUP

Now starting python interactively loads the PYTHONSTARTUP file automagically:

python
>>> for f in ZF('a.zip').namelist:
...  print f
... 
>>> print BS(UO(R('http://google.com'))).find('title').text
>>> cols=zip(*csv.reader(open('a.csv')))
>>> rows=zip(*cols)

pypad - the python notepad web server

The plan is to take this simple python script and expand on it's note taking features with a GUI, console, and command line front end and additional storage options including a tiddlywiki style of appending the data store to the script.

The web server will eventually be an option rather than integral.

http://launchpad.net/pypad

I will fork this into a micro framework and also fork it into a python eval type remote or local shell. Maybe even a REPL embedded in a web page.

The Expectation of Lack of Privacy in Python

One feature of Python that I thankfully rarely have to think of is the whole private, protected, public meme that seems to always popup in programming languages. Programmers coming from other programming languages always want to know how they can make an object's properties private.

Well, besides that fact that black boxes are only good for figuring out what went wrong after the crash. I'd have to say that the whole privacy meme in programming stems from the same sort of administrator == all powerful == programmer way of thinking. Responsibilities are not entitlements. Responsibilities are work.

Here is an example I wrote to clear up what I felt was a miscommunication about using the underscore to create private variables in Python:

Here is a short example I wrote that demonstrates my point about property getters and setters. I have tested cutting and pasting the following into an interactive python 2.5.2 session launched by typing python on the command line without any parameters. The # commented lines should match the output line above it. I hope this clarifies any miscommunication we may have had.

class One(object):
  def __init__(self, value):
    self._x = value
  def getx(self):
    return self._x ** 2
  def setx(self, value):
    self._x = value + 1
  x = property(getx, setx)

class Two(object):
  def __init__(self, value):
    self._my_x = value
  def getx(self):
    """ directly accessing a "private" member of another class """
    return self._my_x._x
  def setx(self, value):
    """ directly setting a "private" member of another class """
    self._my_x._x = value + 2
  # Not defining a setter for the polite_x property makes it read only
  @property
  def polite_x(self):
    """ using the exposed property of another class """
    return self._my_x.x
  x = property(getx, setx)

one = One(10)
two = Two(one)
one._x
#10

one.x
#100

one.x = 10
one._x
#11

one.x
#121

two._my_x._x
#11

two._my_x.x
#121

two.x
#11

two.polite_x
#121

two.x = 10
two.polite_x
#144

two.x
#12

one._x
#12

two.polite_x = 1
#Traceback (most recent call last):
# File "<stdin>", line 1, in <module>
#AttributeError: can't set attribute

I realized 30 seconds after I sent the above example. By looking up the meaning of the underscore and double underscore. That the communication problem was about the underscore as opposed to the double underscore. I was talking about the single underscore because I was asked about the underscore and was only told underscore as feedback when I tried to clarify by asking whether they were referring to the double or single underscore. Which should have been my first clue that I may have my answer judged by someone that didn't fully understand the question they were asking at the time. Which is a big no no. Unless you want to be seen as incompetent because you are being judged by someone who doesn't understand the questions they are asking. And may not be able to understand the answers to those questions. I am loath to admit it but I was just as culpable in this misunderstanding as my understanding of the double underscore was incomplete at that moment. As my mistaken derision of the double underscore came from the accessibility of members like __repr__ and __str__ and __dict__ and __len__. Class members with a double underscore on both sides of the name are system methods and are not meant or implied to being private at all. My lack of interest in making any of my code private was probably the real source of my having no real interest in reading past the disclaimer in the Python documentation which starts with:

9.6 Private Variables
“Private” instance variables that cannot be accessed except from inside an object, don’t exist in Python.

Thus my second email missive of the morning. Making my point all over again using the double underscore. Luckily :). Thank you benevolent dictator for thumbing your nose at private variables!

The private members rewritten with double underscores make those members seem enforceably private. This is because of the exception raised when trying to access those members outside of object and class methods. But this is a misnomer as leading double underscores are actually a name mangling scheme used to hide those members from name collisions with child classes. They can still be accessed directly.

Here is the same example written with "private" members that generate the same results and allows the same access. Despite the miscommunication of apparent privacy. This is because the double underscore's real purpose is for predictable name mangling not privacy.

class One(object):
  def __init__(self, value):
    self.__x = value
  def getx(self):
    return self.__x ** 2
  def setx(self, value):
    self.__x = value + 1
  x = property(getx, setx)

class Two(object):
  def __init__(self, value):
     self.__my_x = value
  def getx(self):
    """ directly accessing a "private" member of another class """
    return self.__my_x._One__x
  def setx(self, value):
    """ directly setting a "private" member of another class """
    self.__my_x._One__x = value + 2
  # Not defining a setter for the polite_x property makes it read only
  @property
  def polite_x(self):
    """ using the exposed property of another class """
    return self.__my_x.x
  x = property(getx, setx)

one = One(10)
two = Two(one)
one._One__x
#10

one.x
#100

one.x = 10
one._One__x
#11

one.x
#121

two._Two__my_x._One__x
#11

two._Two__my_x.x
#121

two.x
#11

two.polite_x
#121

two.x = 10
two.polite_x
#144

two.x
#12

one._One__x
#12

two.polite_x = 1
#Traceback (most recent call last):
# File "<stdin>", line 1, in <module>
#AttributeError: can't set attribute


Bit Flipping in Python

In Python you can access bits directly with the &, |, ^, ~, <<, >> operators on integer objects.

An example bit transformation might be one that moves all of the set bits in a 12 bit number to the left and all of the unset bits in the same 12 bit number to the right.

For example the 12 bit number 101101000001 would be transformed into the 12 bit number 111110000000 using that transformation.

This example bit manipulation can be concisely written in Python as:

n1 = int("101101000001", 2)
bc = sum(1 for i in range(12) if n1 & (1 << i))
n2 = int(("1" * bc) + ("0" * (12 - bc)))

In Python 2.6 and later this example can more concisely be written as:

n1 = 0b101101000001
bc = sum(1 for i in range(12) if n1 & (1 << i))
n2 = sum(1 << i for i in range(11, (11 - bc), -1))

The generic 12 bit manipulation can be more re-usably written as:

# The 12 bits
b12 = tuple(1 << i for i in range(11, -1, -1))

# There are only 13 solutions
result_numbers = tuple(sum(b12[i] for i in range(c)) for c in range(13))
result_strings = tuple(("1" * i) + ("0" * (12 - i)) for i in range(13))

original_number = int("101101000001", 2)
bit_count = sum(1 for i in range(12) if original_number & b12[i])
transformed_number = result_numbers[bit_count]
print result_strings[bit_count]

The Python 2.6 and later version is:

# The 12 bits
b12 = tuple(1 << i for i in range(11, -1, -1))

# There are only 13 solutions
result_numbers = tuple(sum(b12[i] for i in range(c)) for c in range(13))
result_strings = tuple(bin(i) for i in result_numbers)

original_number = 0b101101000001
bit_count = sum(1 for i in range(12) if original_number & b12[i])
# Updated 2011-01-08
# in python 2.7 and later the above can be further optimized
# using range(original_number.bit_length) in place of range(12)
transformed_number = result_numbers[bit_count]
print result_strings[bit_count]

There is no need to define a function call or set of function calls. Bit manipulation easily lends itself to mapping list indexes to result sets that can be prebuilt to aid in readability and execution speed.

python no overhead debugger on demand

One of the best ad-hock troubleshooting ways to figure out what caused an exception post mortem.

Run:

python -i your_script.py

After the exception you will be left at the python interpreter prompt staring at your exception.

Type:

import pdb
pdb.pm()

And you will be put into the context of the stack of the last exception. This means you can print/examine the local variables at the point of failure after the failure has occurred without having to change a line of code or import pdb in advance.

Another method that requires a little bit of preparation is to put the import pdb and pdb.set_trace() in a signal handler trap. *Moved the import of pdb out of the signal handler as imports in functions are deprecated. That way you can do a kill -SIGNAL PID or if the signal you trap is INT you can just Ctrl-C to get dropped into the debugger at any point. The signal handler technique is good for testing release candidates and released code because the signal handler doesn't create any runtime overhead.

Signal handler example. Debugger starts with Ctrl-C:

import pdb

import signal


def int_handler(signal, frame):
  pdb.set_trace()


if __name__ == "__main__":
  signal.signal(signal.SIGINT, int_handler)

Put that at the top of your script and you can start debugging your script at any point by typing Ctrl-C. Resume programe execution by typing exit at the Pdb prompt.

Python File Type Plugin for Vim - Part 1

* Made a correction below. Using the ~/.vim/ftplugin directory for a file type plugin causes the plugin to be reloaded repeatedly causing any autocommands to be appended for BufWrite actions. This causes multiple executions of every autocommand every time you save the file. A better location is the ~/.vim/plugin directory. Where the plugin is loaded once at startup. I have changed the location below. *

I wanted to add some IDE style conveniences to my preferred editor, console Vim. Rather than reinventing the wheel I decided to search the Internet and start with what was already out there. I found a bunch of very useful scripts and snippets. In part 1 I am documenting some of what I found here by publishing a snippet that I have only made minor changes to to make it useful for my needs.

This script needs to be put in a file named python.vim and saved in ~/.vim/plugin. If the directory does not exist it needs to be created. The scripts create the pyflakes, pylinks, and pychecker commands that can be used at the vim command prompt, ':'. The pylint, pyflakes, pychecker packages will have to be installed as prerequisites for using these commands.

All three commands act on the current buffer's file name. Running one of the commands at the vim command prompt will open a Vim quickfix window which will seem familiar to anyone who has used a graphical IDE for development.

You can press c to close the quickfix window when it has the focus. You can double click with a mouse under X or press enter to go to the error or warning associated with the current quickfix line. This will take you to the line referred to by that message. You will have to restart vim if you have already loaded a python file during the currently loaded Vim session.

Here is the code:

function! PythonGrep(tool)
  set lazyredraw
  " Close any existing cwindows.
  cclose
  let l:grepformat_save = &grepformat
  let l:grepprogram_save = &grepprg
  set grepformat&vim
  set grepformat&vim
  let &grepformat = '%f:%l:%m'
  if a:tool == "pylint"
    let &grepprg = 'pylint --output-format=parseable --reports=n --indent-string="        "'
  elseif a:tool == "pychecker"

    let &grepprg = 'pychecker --quiet -q'

  elseif a:tool == "pyflakes"

     let &grepprg = 'pyflakes'

  else

    echohl WarningMsg

    echo "PythonGrep Error: Unknown Tool"

    echohl none

  endif

  if &readonly == 0 | update | endif

  silent! grep! %

  let &grepformat = l:grepformat_save

  let &grepprg = l:grepprogram_save

  let l:mod_total = 0

  let l:win_count = 1

  " Determine correct window height

  windo let l:win_count = l:win_count + 1

  if l:win_count <= 2 | let l:win_count = 4 | endif

  windo let l:mod_total = l:mod_total + winheight(0)/l:win_count |

  \ execute 'resize +'.l:mod_total

  " Open cwindow

  execute 'belowright cw '.l:mod_total

  nnoremap   c :cclose

  set nolazyredraw

  redraw!
endfunction

command! Pyflakes call PythonGrep('pyflakes')
command! PyFlakes call PythonGrep('pyflakes')
command! Pychecker call PythonGrep('pychecker')
command! PyChecker call PythonGrep('pychecker')
command! Pylint call PythonGrep('pylint')
command! PyLint call PythonGrep('pylint')

" These three are successively more informative and aggressive in their
" warnings with pyflakes as the least noisy. Only uncomment one.
"autocmd BufWrite *.{py} :Pyflakes
autocmd BufWrite *.{py} :Pychecker
"autocmd BufWrite *.{py} :Pylint

In Vim a single double quote, ", is considered the beginning of a comment for the rest of the line.

The benefit of having the autocmd run one of the Python lint programs every time you save the file is that you can never get too far from sane working code as any errors and warnings pop up. This, in your face, style of development can keep you from pulling your hair out over something as simple as a semicolon or comma typo and can also warn you about more complex problems and proper coding style.

More to follow. I will try to combine the output of the three commands as their output doesn't seem to overlap for the most part. And the noisier programs (pychecker and pylint) don't appear to give the same advice when it comes to warnings. pylinker is by far the noisiest of them all but all it's warnings are configurable.

Warning: If you are using tabs for indenting you will have to modify the pylint parameter '--indent-string='. Set it to the empty string for tabs, or the actual number of spaces used for indenting. In my code above I have it set for 8 spaces which is project specific.

Step One: Insert Foot in Mouth

This is a follow up to my Your Fancy Language is Burdensome and Dangerous post.

Python and Ruby don't have the APL array/vector/matrix syntax built in to the core language. But a little nudge from someone made me do the little bit of research I should have done before I wrote that post. And I rediscovered that I did not originate the idea that these languages need array/matrix math syntax thought.

There is a gem for Ruby called NArray and an egg for Python called Numpy that not only add array syntax but also coherent coercion rules and the ability to apply functions across arrays without calling map. I'm glad I found it because they are easy as pie to use and do exactly what I wanted and a heck of a lot more.

Check out this Python example:

from numpy import *
a = array([1,2,3])
b = array([4,5,6])
print a+b #equals array([5,7,9])
print (a+b)/4
print (a+b)%6

Outputs
[5,7,9]
[1,1,2]
[5,1,3]

As you can see Numpy also replicates the integer type bug/feature which is very C like but is being dropped in Python 3000 in favor of coercing integers to float on divide. You can typecast the divisor to float to get floating point division and float results.

Your Fancy Language is Burdensome and Dangerous

I found this quote about the PL/I language in Wikipedia to be very poignant today. The quote's sentiment and the PL/I language are from the 1960's.

PL/I is considered by some to be a turning point where it was found that large languages with a lot of complexity that allow programmers to do almost anything (such as coerce types with no semantic relationship to one another) are burdensome and dangerous rather than "more powerful."

It echoes my low opinion of most of the toy languages that seem to be escapees from a Compiler 101 college course masquerading as the cure for the inability of some people to program. I find it to be very relevant to the object oriented mass hysteria and popularity of the latest scripting languages of the moment.

My opinion of C++ with the STL and templates and generic algorithms and iterators and iostreams is lower than my original opinion of C++ before it became standardized. What problem does all that junk solve. More to the point, what problem did object oriented programming ever solve beyond speed to delivery?

Object oriented programming is a morass and a tar pit. It's the all promise and no delivery. It eats up 90% of your programming time forcing you to write witty little programming ditties when you should be writing elegant, concise solutions that aren't 90% object oriented overhead.

Object oriented code hides not only the method invocation but also confuses the issue as to which method is actually called. Multiple inheritance in C++ and other languages demonstrated that the only part of multiple inheritance worth saving in other new object oriented implementations was the interface definition which is basically a template which is basically a macro. It saves you the onerous task of cutting and pasting. In C it's called a macro and it's frowned upon because it obfuscates what the final code does and looks like.

In summary. First write code quickly using only 10% of your programming time. Second scratch head and guesstimate what your code actually does and what are it's unintended consequences. Third hope there aren't any bugs that make your invisible untraceable code do something you didn't intend and use more resources then are necessary to do it.

I'm not saying I don't appreciate Python, Ruby, Perl, C++, Java, and the like. I'm just saying it's not that hard to write the little bit of code it takes to implement most software projects. Especially since the main benefit of most new languages is their extensive built in libraries. Not that you couldn't do the same thing in C by including the proper headers and linking in the proper libraries.

To be brutally honest I think in assembler so even C seems a bit like unnecessary overhead and a bunch of macros hiding the actual code.

I've been reading up on a couple of APL like programming languages J and K but after reading some of the original code for the original J implementation all I see it a bunch of nested for loops creating everything on the stack and avoiding calls to the C standard library to allocate memory.

Now I like APL and I hope to write my own version of it one day because it solves a real programming problem through it's expressiveness. The problem it solves is how to communicate complex and simple vector and matrix operations programmatically to a computer without having to jam a for loop into your equation. The analogy would be the poor man's version tacked on to most of the recent programming languages with list comprehensions and regular expressions. Which are not the same thing.

There are some nice examples of K and J at Wikipedia. I'm not recommending checking out APL as anything other than an educational exercise. I could kick myself for not learning it sooner as it shows how a well thought out programming language was written way back in time that blows the doors off all the fancy flavor of the year languages which waste our time by forcing us programmers to learn an unending stream of new syntax with exceptions that can't even express programming logic as simply and concisely as the A language, APL(A Programming Language) could more than half a century before.