Monday, 8 January 2018

Rust: Lacking equivalence under simple refactorings

I've been writing a fair bit of Rust recently. I'm new to the language, and it's been alternating between making me very happy, and making me very frustrated. One of the things I've been finding frustrating is the number of places that I expect two pieces of code to be equivalent where they are not. Here's some sample code, which I'm going to use in a few examples:

#[derive(Debug)]
struct Foo {}

impl Foo {
    fn foo(&self) -> &Foo {
        self
    }

    fn bar(&mut self) -> &Foo {
        self
    }

    fn baz(&mut self) -> &mut Foo {
        self
    }
}

fn call_one() {
    let bar = Foo{}.bar();
    println!("{:?}", bar);
}

fn call_two() {
    let mut foo = Foo{};
    let bar = foo.bar();
    println!("{:?}", bar);
}

fn call_three() {
    let foo = Foo{}.foo();
    println!("{:?}", foo);
}

fn call_four() {
    let mut f = Foo{};
    let baz = f.baz();
    println!("{:?}", baz);
    let bar = f.bar();
    println!("{:?}", bar);
}

fn call_five() {
    let mut f = Foo{};
    println!("{:?}", f.baz());
    println!("{:?}", f.bar());
}

fn call_six() {
    let mut f = Foo{};
    {
        let baz = f.baz();
        println!("{:?}", baz);
    }
    let bar = f.bar();
    println!("{:?}", bar);
}

First, let's look at call_one and call_two. These look like they should be identical code; the simple application of an "inline" refactoring (or "extract variable", if going in the other direction). But in reality, call_two is fine, but the borrow checker is unhappy with call_one:

error[E0597]: borrowed value does not live long enough
  --> src/main.rs:19:26
   |
19 |     let bar = Foo{}.bar();
   |               -----      ^ temporary value dropped here while still borrowed
   |               |
   |               temporary value created here
20 |     println!("{:?}", bar);
21 | }
   | - temporary value needs to live until here
   |
   = note: consider using a `let` binding to increase its lifetime

So what's going on here? When you declare a variable with let, its lifetime begins, and it remains live until the variable it's bound to goes out of scope. But if you never bind an expression to a variable with let, its lifetime immediately ends, and it gets dropped. There seem to be some special-cases where this is transparently handled for you (look at call_three - it's exactly the same as call_one which the borrow checker finds problematic, but the borrow checker is fine with this one. The only difference is that bar borrows self mutably, whereas foo doesn't. I assume that internally, the compiler is somehow promoting foo to be an owned type, or creating a hidden temporary which owns the Foo. It would be nice if the same courtesy could be afforded to more mutable references until something actually tries to violate constraints, like actually modifying the reference, rather than just preventing anonymous mutable borrows entirely).  The big place I've seen this be a problem in real life is with the builder pattern; if I want to populate my builder with some arguments, and then build from it more than once, I need three statements: to assign the un-populated builder, to populate it, and to build it, where I'd really like two: to assign the populated builder, and to build it.

There is work underway in the language to solve part of this problem, with something called non-lexical lifetimes (NLL). Take a look at call_fourcall_five, and call_six. Again, these look like they should be equivalent, but call_four makes the borrow checker unhappy.

error[E0499]: cannot borrow `f` as mutable more than once at a time
  --> src/main.rs:39:15
   |
37 |     let baz = f.baz();
   |               - first mutable borrow occurs here
38 |     println!("{:?}", baz);
39 |     let bar = f.bar();
   |               ^ second mutable borrow occurs here
40 |     println!("{:?}", bar);
41 | }
   | - first borrow ends here

This time, the borrow checker is unhappy for the opposite reason: a variable's lifetime is bounded by its scope, and its scope continues until the end of the block in which it was declared, so baz, which mutably borrows f, precludes bar from being able to borrow it, even though baz could be analysed by the compiler to be dead. You can work around this by avoiding the assignment, as in call_five, or by introducing an inner scope, after which baz gets dropped, as in call_six. The NLL work will address this, and perform the liveness analysis to render these workarounds unnecessary.

Having to re-structure code which otherwise feels like it should be equivalent to satisfy the borrow checker is pretty frustrating, and occasionally means sacrificing clarity. It's great to see work being done to improve this situation, and I hope it continues to go further!


Pixel 2

For the last 6ish years, I’ve primarily been an iPhone user. Every few years, I’ve tried out a flagship Android phone as my main phone for a non-trivial period of time (most recently the Nexus 6), and I’ve always switched back to an iPhone. About six weeks ago, I got a Pixel 2, and I’ve been using it pretty much exclusively. Here are my thoughts on being both an Android user, and a Pixel 2 user.
tl;dr: Android is pretty good - definitely up to iOS quality. The hardware is pretty solid, but lacks a lot of finesse. If I switch back to an iPhone, it will be because of the physical build quality, but I’m not likely to do that at least until this phone dies.

Android

Android has gotten a lot better. The user experience is a lot more consistent and a lot less flaky than it used to be. I’ve only had Google Play Services crash once. The permissions model is a lot cleaner (you can deny specific permissions to specific apps, though more on that later). In the past, when I’ve switched from an iPhone to an Android phone, I’ve instantly felt painful regressions, and they wouldn’t go away no matter how much I got used to the operating system. This time, while there are things I prefer about each system, I can’t obviously point at one and say “I strongly prefer this one”. I don’t really have any major complaints about the core Android operating system any more. But there are some things I really like about it:
  • ChromeView integration. If I open a link in the Twitter app, I can select “Open in Chrome” from the drop-down menu, and my exact screen will be preserved, scroll-location and all, just in a standalone Chrome tab. This is amazing.
  • Chrome tabs are preserved offline a lot more than I saw on my iPhone, and the “download when next online” feature is pretty swanky too.
  • Gboard (Google’s swipe keyboard) is really nice (and because it’s better system-wide supported, works in most apps, unlike on iOS). Nonetheless, every now and then an app appears to use its own custom keyboard, or something, and I find that my attempts to swipe to type fail randomly.
  • Sharing intents work a lot nicer than iOS’s half-baked copy of them.
I still don’t understand why Phone and Contacts are separate apps, or when I’m meant to use each of them.

Region-locked app-store

My one big complaint, which I’m aware is a little niche, is around region locking in the Play store. For weird historical reasons, my Google account is US-based rather than UK-based. Accordingly, I cannot install the apps for one of my banks, or one of my credit cards, and the BBC iPlayer and iPlayer Radio apps don’t let me download content. On iOS, I could switch the country in which my account is registered through an official process, by entering a billing address and card in that country, which would let me download apps from the other country, and then switch back. On Android, I can only emulate this by signing in as a second user, which is a generally poorly supported flow. For one thing, it means I need to switch accounts to get to the apps, and for another, storage doesn’t really work properly (downloads from the BBC apps fail in obscure ways, for instance).

The workaround here would be to side-load the apps, but that’s kind of sketchy security-wise.

If you want to install apps from multiple regions, I recommend an iPhone.

Permissions

  • “All files and photos”. Oh god. If you want to give an app permission to save any files or photos or data, you need to give it permission to read, write, or delete all files and/or photos on your phone. This is very alarming! Please please please let apps have a private sandbox, and a separate permission for some kind of global storage!
  • Notifications are on by default. Accordingly, my notifications are super spammy, until I manually tweak them down. This is pretty annoying; I suspect that one of the reasons that iOS is less spammy notification-wise is that there’s a user prompt before notifications can ever be sent, and so app developers need to convince people to enable notifications with some good reason before they can spam.
  • Each bank / credit card app on my phone told me it needed permission to make and manage phone calls, and promised me it wouldn’t ever actually use it, but it needed to use it to verify security somehow. Weird.

Photos

The Photos app doesn’t appear to have a way to let me scroll through all of my taken photos, received photos from WhatsApp, received photos from each other app, screenshots, etc, in one view; I need to remember which one of the above the photo I’m looking for was, and open the correct folder. Not the end of the world, but a little annoying.

Android Pay

Android Pay appears to have less broad adoption from credit card companies and banks than Apple Pay does. But I don’t really use either, so it doesn’t matter to me that much.

WiFi Connections

My iPhone was pretty sluggish to connect to wifi, and often needed prompting, particularly when on the tube. My Pixel 2 appears to join wifi networks really quickly, and without prompting, which gives a much smoother experience on the tube.

Automatic re-enabling of Wifi

I found out about this feature yesterday! If you’ve disabled wifi, and Android detects that you’re near a wifi network which it thinks is good, it will turn your wifi back on! This is a terrible feature to enable by default! Firstly, if I’ve disabled my wifi, it’s for a reason, and you shouldn’t assume to know better. Secondly, I can only assume that it does this detection by probing, which means that 1) it’s using my battery to do so, and 2) it’s broadcasting my MAC address when I may have asked it not to.

The back button

I find it pretty hard to predict what the back button will do at any time. If I open a link from an email, and press back, it closes the Chrome tab and takes me back to my email. If I click a Twitter link from Signal, and press back, it takes me to the most recent Tweet before that which I opened from a link in Signal. What?

Camera

The camera is pretty good. A similar quality to what I’d expect on an iPhone (though slightly worse in low-light conditions). It takes “motion images” by default, which are kind of fun (they give a 1-3 second animation around the frame you’re actually taking), but a little annoying to share (if I try to email them to myself, they’re about 8MB each, and there doesn’t appear to be an easy way to just send one frame).

Pixel 2 software

This is something I’m actually really disappointed about. One of the joys of getting a Google experience phone is that vendors don’t mess with the UI, bundle in un-removable apps, etc. And yet, on my homescreen I cannot move or remove the search bar at the bottom of the screen (which I never use, but frequently accidentally tap on when I’m trying to open one of my pinned apps), and I cannot move or remove the “At a glance” display at the top of the screen. I really wish I could actually configure my device as I want to, without rooting it.

Pixel 2 hardware

Weirdly, this is my area where I have the most complaints. Which I wasn’t expecting, considering the price tag and marketing of the phone.

Screen

My screen is covered in scratches. I have no idea what from; I’ve not had any incidents which should have scratched it. I’ve used it exactly the same as my iPhone; keeping it in the same pocket with the same things, putting it on my same bed-side table. I can only assume the iPhone screen is made of a tougher material, or has a coating, or something. I’m actually pretty disappointed that my phone has several noticeable scratches six weeks in.

Fingerprint sensor

I really dislike the location of the fingerprint sensor. It’s on the back, just above where your index finger rests on the back of the phone. With my iPhone, if I had the phone on my desk, I could tap the sensor with my index finger, and interact with my phone. With my Pixel 2, I need to pick up the phone, and poke the back of it; this is a surprisingly much higher friction activity to do something like glance at a message I’ve received.

I also accidentally press the fingerprint sensor a lot when my phone is in my pocket when fumbling for my phone, or wallet, or whatever, which necessitates putting in my PIN to unlock (because it hits the fingerprint attempt limit). Minor, but annoying.

Speaker location

At night, I leave my phone face-down because it has a flashy light on it. The speaker appears to be on the front of the phone. So if I want to settle down with a podcast or music or something, I can’t really hear it. My iPhone didn’t suffer from this problem, as the speaker is at the bottom.

One USB-C port is awful

My biggest wish is that my phone had either a 3.5mm headphone jack, or two USB-C ports, so that I can plug in both a charger and headphones. I have some good bluetooth headphones, but sometimes I want to use cheap earphones, so I’ve bought a bluetooth receiver (which is excellent, for the record) that I can plug my earphones into, but I need to carry it around, and charge it, and just ugh.

I looked at getting a splitter which allows you to plug both headphones and power into one end, and plugs into the USB-C port on the phone. As far as I can tell, there are several available, but most of them probably don’t work with any particular phone. Ugh.

The bundled USB-C -> 3.5mm adapter is awful

You need to jiggle it around before it actually works. And there’s no indication on the phone anywhere that headphones are plugged in, so reliably I will start playing music out of the phone speaker, and then keep jiggling the adapter until it plays through the earphones. This is simply not acceptable.

USB-C is far from ubiquitous

I bought a set of USB-C - USB-A cables, so that I can plug my phone into existing chargers, or my laptop, or whatever, to charge. Bundling one of these would have been nice.

Size

The phone is big. Bigger than I’d like. I can just about use it with one hand (I have pretty big hands). But most annoying, big without it being for the screen. About 1.3cm are added at both the top and bottom around the screen for… Black plastic. Seems annoying. I’d love my phone to just be 2.5cm shorter.

Sharing by Bluetooth

I miss Airdrop. I’ve tried sharing images over bluetooth from my phone several times, both to other phones and computers. It’s failed 100% of the time. I just email stuff now, but I’d rather not have to.

Wednesday, 6 January 2016

Ethical Banking and Saving

I generally seek to have a positive effect on the world where I can, particularly where it’s easy to achieve with only small changes in behaviour. I have a bank account. I have a savings account. Right now, the profits of these largely go to banks who, while serving a useful function, aren’t necessarily my dream beneficiaries.

I spent a while trying to find some good places to put my money, and figured I might as well write up my research in case it’s useful to anyone else. I was definitely trying to solve my own problems, so I should probably describe my aims: I live in the United Kingdom (in London). I have a current account, which I largely use for direct debits and to pay off a credit card with which I buy most things. It is always in credit. I have a few tens of thousands of pounds of savings. My life is fairly unstable, so I want relatively easy access to those savings (instant, or at most three months notice), in case I decide to move country, or buy a house, or start a company, or who knows what. I don’t particularly care about making the most interest I could make (I’m not a huge fan of making money from having money), but some would be nice, and more would be nicer. I don’t want an ISA. I travel a lot, and have a lot of foreign transactions. So practically, what I’m looking for is:
  • A place to put tens of thousands of pounds of savings with at most 90 days of notice to withdraw (with probably some in instant access).
  • A current account which handles me traveling well, offers a debit card, direct debits, and good online banking.
With that established, the next question is, what are my ethical aims? What are the available families of ethical ideologies? I have found only a fairly small handful of “ethical” banks (they’re not all banks, but I’m going to use that term broadly), and many fewer who offer current accounts. The broad families seem to be:
  • Banks who only lend people/companies/groups who they judge to be making an actively positive influence in the world (Triodos Bank, Ecology Building Society, Charity Bank, Shared Interest). I call these “whitelisters” because they look for actively good things which they approve of, rather than invest anywhere that isn’t negative (“blacklisting”).
  • Banks who lend to local communities, who don’t necessarily vet their borrowers for ethics, but whose target is mostly people or local enterprises rather than large companies, and whose scale means they probably work out ok (most of the building societies, credit unions, arguably Metro Bank).
  • Religious banks (Al Rayan Bank, Reliance Bank).

What will I be doing?

For my current account, I really value convenience and customer service, so despite being far from the most ethical choice, I’ll be opening a current account with Metro Bank, with an aim to keep it pretty empty.

For instant access savings, I will be saving with Triodos and one of my local credit unions.

For term-notice savings, I will be saving with Ecology.

How did I get to that decision?

What follows is a brief narrative about each of the banks I investigated, starting with those who offer current accounts:

Reliance Bank

I did not investigate Reliance particularly heavily; they’re the bank of the Salvation Army, and I have problems with that particular organisation, so they were never really a contender for me. But some relevant information:
  • They offer current accounts (with everything you’d expect), and savings accounts which don’t pay any meaningful interest.
  • They blacklist certain trades (tobacco, alcohol, gambling, pornography, arms, the socially irresponsible, and those who don’t uphold the Universal Declaration of Human Rights).
  • They pay their employees the living wage.
  • They have only one branch, but have arrangements with other banks to allow you to use their branches for paying in.
  • Foreign transactions have a flat £1.25 fee.

Al Rayan Bank (formerly: Islamic Bank of Britain)

Loosely speaking, the Qur’an forbids charging interest on loans and generally making money from money (and accordingly, making interest on savings). Al Rayan, accordingly, don’t formally pay interest, but instead share the profits of the bank with their members. It’s certainly a subtle dance. One thing that this means is that they don’t quote interest rates, they quote “expected profit rates” which may be missed.
  • They offer current accounts and savings accounts. Their instant access savings don’t provide a meaningful expected profit, but if you’re willing to take a notice account or fixed-term account, they’re fairly competitive (ranging from 1% 60-day-notice to 2.88% 3-year-fixes).
  • Their investments are all in property or low-risk commodities; they blacklist the standard “evil” industries, and require actual assets underlying their investments.
  • They pay their employees the living wage.
  • They have five branches and three agencies spread across particularly Islamic cities of England.
  • There are limits on what countries accept their debit cards, what locations accept their debit cards (e.g. not petrol pumps), and they seem not to be part of the major ATM networks, so expect some fees. If you’re not near one of their branches, the accounts sound like hassle to operate.

Metro Bank

Metro Bank were founded in 2010 with an aim towards customer service. They do not brand themselves as particularly ethically focused (though they do talk about being community-focused, responsible, and charitable); their focus is definitely on profit via good customer service. In particular, they neither blacklist nor whitelist specific industries or groups. They don’t have an investment bank, they’re a small-ish straightforward deposit-taker-loan-lender retail group.
  • They offer current accounts (optimising for ease-of-use) and savings returning decent rates (0.75% instant access, 1% 90 day term).
  • European transactions charge no fee. Outside Europe, 1.9% is charged.
  • Their branches (“stores”) are spread mostly across London and the South East, and are open seven days a week.
  • They don’t have any particular ethical stance with their lending.

Building Societies

Building societies tend to offer decent rates of interest on savings, often times also offering current accounts. They tend to have a small number of branches in a local area, and to use their savings deposits exclusively to back mortgages (often only locally) though occasionally also for loans and/or insurance. If you’re looking for local community investment (where community here literally means “the people around you”), they’re not a bad choice. They’re formally owned and run by their members, and that gives some control over direction and aims. The largest few (Nationwide, Yorkshire, Coventry, and some others) have many branches across the nation (often times as the result of mergers of smaller societies), but quite quickly you get into the realms of them being very localised, and either only being accessible to those in the area, or possibly able to be operated by post or online.

Credit Unions

Credit unions operate fairly similarly to Building Societies; the notable differences are:
  • They require members to have a common bond (typically an area in which they live or work).
  • They are typically more strongly targeted to those of lesser means; they typically offer small loans rather than mortgages, and also many try to offer less obscene alternatives to increasingly prevalent payday loans.
  • Along those lines, they are typically less choosy with to whom they will lend. For instance, most credit union websites explicitly talk about the benefits their members are probably on. They try to provide banking to all.
  • They explicitly don’t seek to maximise profit, and they tend not to pay precise guaranteed interest; instead each year the members look at the profit made, and share it out as a dividend across the members.
  • They typically only have one or two branches, and if they offer current accounts, tend to charge a fee for operating them.
  • They frequently operate on much smaller scales than banks or building societies, and may not have full banking facilities. Some, for instance, manually process transactions (which may add a day or two to processing times), or require payments in to go to a shared account, with your account number as a reference, rather than actually allocating account numbers under a sort code.
They’re probably a more impactful place to put money than building societies, but they’re also likely to be a less convenient place to operate with; that said, they are frequently lauded for their customer service, and they definitely have very real impact in local communities.

Triodos Bank

Triodos Bank only lend money to people and organisations they view to be actively making a positive impact on society (whitelisting cultural, social, and environmental borrowers) rather than avoiding actively bad companies. Their definitions of positive are open to question (for instance, they happily fund homeopathy), but for the most part seems decent, and certainly better than most of the field.
  • They currently only operate savings accounts, though say current accounts should be coming in late 2016.
  • They pay their employees the living wage.
  • They offer 1% interest for online accounts, or more (but not much more) for term accounts of at least a year.
  • They are very transparent with their lending, actively listing all of their borrowers online.
  • They have only two branches in the UK, though as a savings-only provider this is probably less of an issue than for current account providers.
  • Their deposit protection is from the Netherlands, and so its limit is slightly subject to fluctuations on the GBP-EUR exchange rate.
One thing to note is that they also offer some ethical investment funds; note that they hold companies in these funds to a lower standard than those to whom they will lend, though still not a bad standard.

Ecology Building Society

The Ecology Building Society take deposits in savings accounts and lend only to support environmentally friendly, sustainable projects, particularly ecological new builds, energy-saving improvements, renovations, and preference to affordable housing. They also very consciously track and aim to improve their environmental footprint (e.g. publishing their “grams of CO2 per £ of new lending”).
  • They offer only savings accounts, but pay a good rate on them (1% instant access, up to 1.6% with 90 days notice).
  • Their banking facilities are somewhat limited; they operate by transferring money to their shared bank account with your account as a reference number; they have some online banking, but notice for notice accounts can only be given by post.

Charity Bank

Charity Bank only lend money to charities and social enterprises from savings deposits. They treat their employees and customers well, foregoing bonuses and hard sales targets in favour of a living wage. Their board are also unpaid volunteers. Their shareholders are all charities, trusts, or foundations, hopefully giving a good direction.
  • They operate only savings accounts, paying around 0.6% for 33 day notice accounts. This number is noticeably lower than others available, as they favour cheaper rates to borrowing charities over higher interest to their savers.
  • Their accounts are entirely operated by post.

Shared Interest

Shared Interest are a co-operative who provide unsecured loans to the fair trade industry; both to producers and buyers, in an attempt to cover the gaps that exist in the production/sales lifecycle. They are run very strongly as a co-operative, including a shadow-board of randomly selected members to oversee the board.
  • They operate share accounts, rather than deposit accounts, so there is no protection scheme for your money. This is a theoretically risky account.
  • Their interest rate is rather low, currently at 0.5%, and their roughly aim to target base rate minus 4%. They also offer an option to automatically donate this interest.
  • Their board is currently 37.5% female, which is almost unheard of in the industry.
  • They have a somewhat loose links to Christianity, which may be an issue for some.

So why my decisions?

For my current account choice (Metro Bank), I sided mostly with convenience. A bank which is open seven days a week, who charge no foreign transaction fees, and who seem to actually care about the experience of their customers is exciting. The alternatives here were: Reliance (except Salvation Army), Al Rayan (whose offering seems fraught with inconvenience, e.g. problems using cards in America or at fuel pumps), building societies (either not local to me, or not necessarily as convenient), or credit unions (for whom operating current accounts is actually quite expensive, and with whom a current account seems somewhat inconvenient). When Triodos offer a current account, this may change.

For savings, I found Shared Interest too risky for my profile. I found Charity Bank quite interesting, but a) I’d like to know more about their lending criteria, and b) I’m not entirely convinced that pushing loans on charities is the best way to support them. I find credit unions more compelling than building societies. And I find Triodos and Ecology both quite compelling, the former more so than the latter.

Please help me!

If there are any interesting offerings I’ve missed, please let me know! If you’d like to talk about these issues more, disagree with anything I’ve written, or generally have an interest, I’d love to hear from you too! Get in touch - dawagner AT gmail DOT com :)

Tuesday, 4 September 2012

One of the most nefarious language features I've ever seen

Take a look at the following code:

file1.py:
from package.lib import do_something_which_raises
from otherpackage.myexception import MyException
try:
  do_something_which_raises()
except MyException:
  print 'Oops, raised'

file2.py:
from otherpackage.myexception import MyException
def do_something_which_raises():
  raise MyException()

What would you say if I told you that this code didn't print 'Oops, raised!', but instead bubbled MyException all the way up to terminate execution?

"Sure, you have two classes named MyException".

You can see that both files import exactly the same exception class from exactly the same package  I promise you, there is only one file named myexception.py, which contains exactly one class named MyException.  If you walk all the entries of sys.path, you will only reach one definition of MyException.

"That's not possible!" I hear you say.

That's certainly what I thought.  But I was faced with the fact that I'd moved a couple of files around in my source tree, fixed up the package references, and three quarters of my tests were failing. I had no bloody clue why.

It turns out, class equality in python depends not on whether two types were defined the same, but rather, whether they were loaded exactly the same.

You see, file1.py and file2.py are in different directories; the structure is actually:

./file1.py
./package/file2.py
./package/otherpackage/myexception.py

but both top_level and top_level/package are in my sys.path.  . is always implicitly in sys.path, and I added /path/to/package to my $PYTHONPATH.

So when, in file1, I import from otherpackage.myexception, Python actually goes "otherpackage... Well, I don't have a file named that in the current directory... Or a folder named that... Let's start going through $PYTHONPATH".

For file2, it goes "Aha! The current directory has a folder named otherpackage! And inside it is a file named myexception! And that defines a class named MyException! You're sorted!"

But as far as Python knows, these files were reached differently.  They have different paths to themselves (one is "/path/to/package" / "otherpackage/myexception.py" and the other is "./" "otherpackage/myexception.py" - the fact that, "." happens to be the same as "/path/to/package" is neither here nor there.  Python's package cache, apparently, doesn't do that level of resolution1 (I guess it assumes no one would be silly enough as to have overlapping sys.path entries).

So when I said "you will only reach one definition of MyException", I was only telling half a truth.  You will only reach one definition, but you will reach it twice.  And that confuses python.

So here's a warning for you.  Don't have overlapping sys.path entries.  And if you do, always reference every definition therein from a consistent top-level folder.  Because if you get this wrong, equality isn't equality, and everything goes to hell.



1: This is conjecture; I've looked through the source of the equality functions, and exception definitions, in both 2.7 and 3.3 to see whether anything funky was going on there, I've not yet delved in to the package cache, but this seems like reasonable conjecture, I'll probably get around to reading through the package cache source at some point, and may follow up then with a blog post and/or patch to python.

Monday, 27 August 2012

Python, driving you to do the right thing, sometimes

This week I've been writing my first real Python.  I mean, I've hacked together 100-line scripts before, but I've been writing real code from scratch, structured, with tests and everything.

I've noticed some really nice things, and some really horrible things.

Nice thing: Python doesn't let you hash mutable collections:

>>> hash(set())  # mutable
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unhashable type: 'set'
>>> hash(frozenset())  # immutable
133156838395276

>>> hash([1,2,3])  # mutable
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unhashable type: 'list'
>>> hash((1,2,3))  # immutable
2528502973977326415

This is great, because if you hash something, it tends to be in order to store it in a set or dict or something, and if you change the thing after hashing it, you've broken the contract of the set/dict.

Awesome.  Guiding you toward doing the right thing.

Horrible thing: unittest1 has a method: assertRaises.  Great - a simple, concise way of asserting that a single call raises an exception.

Except I wanted to assert that something is raised, without specifying what, so I tried skipping the exception type from the call to assertRaises.


>>> import unittest
>>>
>>> def does_raise():
...   raise Exception()
...
>>> def does_not_raise():
...   return 1
...
>>> class TC(unittest.TestCase):
...   def test_raises(self):
...     self.assertRaises(Exception, does_raise)
...   def test_does_not_raise(self):
...     self.assertRaises(does_not_raise)
...
>>> unittest.main()
..
----------------------------------------------------------------------
Ran 2 tests in 0.002s

OK

WAIT! I called self.assertRaises with a method which *does not raise*! It's in the name and everything! It feels like if the first arg to assertRaises is a callable, and not a type, unittest could perhaps at least warn, if not throw.

If I hadn't run this test before implementing the backing code to see it fail, and seen it passed, I would have blindly been assuming that my code correctly raised an exception (as my test showed!) when it didn't!


1: unittest from Python 2.7, backported as unittest2 before

Saturday, 25 August 2012

Your app isn't all UI, don't test it like it is!

There are two common pieces of advice I give about testing software:
  1. Get a walking skeleton up quickly, and drive your development from user-acceptance tests (testing from the UI down), inspired by the fantastic Growing Object-Oriented Software Guided By Tests
  2. You should have very, very few UI tests1.
These two pieces of advice seem to directly contradict each other.  They do and they don't.

The idea behind 1 is that you start with a single UI test, and you write many smaller tests as you implement the sub-features required to make the UI test pass, but that having the UI tests gives you the really useful confidence that you get from actually being able to prove your app does all the things you say it does, and discourages you from writing code not required to make the UI test pass.

The idea behind 2 is that UI tests are slow, give you poorly localised feedback, are often flaky, and require writing a lot more test infrastructure and harnesses than smaller tests.  So you want to avoid them.

My experience with trying to follow both pieces of advice at the same time has been pretty poor.  Driving from UI tests encourages me to write UI tests for every single thing.  Which is useful, but: a) slow, b) means I'm distracted from getting things done at a lower level, because I'm busy writing UI tests and infrastructure, and c) means I end up with a lot of UI tests.

There's a side-project I've been meaning to write for a while now, and a few times I've started it, gotten bogged down with UI writing and UI testing, gotten bored, and stopped.  Yesterday, I started it again, but with a different approach, and it feels much better:

Don't have a UI.

UIs are great; they let people interact with your system.  But that's not the thing I want to be testing when I'm implementing logic, or interacting with a data store, or doing any of the many other things which are nothing to do with the UI.  When developing core functionality, the UI is really just a way to trigger calls in to your core library.  UI tests are great for showing that your UI makes the right calls, and everything is wired up properly, but I definitely don't want them giving me confidence that my core library works! I want much, much smaller tests for that!

So what have I been doing? Well, I've been writing the library first.  I know roughly what actions my UI needs to perform in my backend, so I can write code to perform those actions.  My code is unit-tested and has some larger tests showing that it interacts with a (fake, in-memory) data store correctly and such, but there are absolutely no UI tests, because there is absolutely no UI! A whole host of things not to distract me!

Now, at some point I'm going to have a problem.  One of the drivers for the Walking Skeleton is that integrations are hard, and should be done early so you can find integration problems, and at some point, I'm going to need to integrate my UI (when I write it) with my library, but I'm ok with leaving that to a later date.  I like that my UI is largely independent of the implementation.  It means my UI tests will really only be testing the UI, and how it integrates with the backend.  It means when I'm trying to write the UI using whatever web framework I happen to choose, I can really focus on the UI and the web framework, and not be thinking about a database, or whether my loop can really always terminate.  And hopefully, it means that I can write just a few UI tests, which exercise a few useful paths through my system, and show it works, without needing to test every little thing.  Because I know every little thing works, I just don't know whether the UI pokes it correctly.  So my UI tests can test exactly that.


There are other benefits too. My UI should be more independent of my backend, so I should be able to more easily switch out one or the other with a whole new implementation. Because my library has been designed to be an API for a UI, rather than being the implementation of the UI, opening up that API in other ways should be easier too, as I have a well defined interface. But mostly, it lets me focus on the thing I'm trying to do right now, without being distracted by a largely separate part of the system.

This is something I've been finding works great for me - How do you manage the trade-off between the confidence UI tests give you, and the pain they entail? How do you keep your test-pyramid bottom-heavy?



1: People are often surprised that, given most of my day job involves making UI testing easier, I don't think people should be doing it. I think of it as a necessary evil, which should be minimised. I really liked what someone said at CITCON in London this year, roughly: "Every time someone on my team wants to add a UI test, they have to justify to the team why the thing they're testing can't possibly be tested at a lower level. As a result, we don't have many UI tests, and that's just fine." - sorry I can't attribute the line!

Monday, 5 December 2011

Scripted/Command Line Android Emulator Installation

If you're anything like me, you tend to script things like installation, because inevitably, you're going to do it again, and you'll never quite remember the right sequence of steps needed. In particular, I was trying to install the Android emulator, such that I can run tests in it without any manual intervention; able to spin up machines and have them configure themselves to Just Work.

It turns out, the Android emulator goes out of its way to make this tricky. You need a handful of components to get the Android emulator running:
* The Platform Tools
* The version-specific SDK
* The ABI/system images

Two of these things are easily installed from the command line! One of them seems only accessible through the UI! (That last one, there)

But fortunately, if you happen to know the URL where the zip file containing the ABI is, you can download it, and unzip it to the right place!

Then, on first launch, you get a modal dialog asking whether you want to send usage statistics to Google. Useful. But annoying, when you're not a user using the tool!

So, here is my script to do all the magic you need to actually get your android emulator working (on Linux):

wget http://dl.google.com/android/android-sdk_r15-linux.tgz
tar xf android-sdk_r15-linux.tgz
cd android-sdk-linux
tools/android update sdk --no-ui -t android-14,platform-tools
mkdir -p system-images/android-14
cd system-images/android-14
wget https://dl-ssl.google.com/android/repository/sysimg_armv7a-14_r01.zip
unzip sysimg_armv7a-14_r01.zip
mkdir -p ~/.android
echo "pingId=0" > ~/.android/ddms.cfg

[Edit to fix typo in script]